Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Solving Redis Cluster Cross-Slot
Errors with Hash Tags.

Your payments service is migrating from standalone Redis to a 6-node Redis Cluster (3 primaries, 3 replicas, 16,384 hash slots). Two core workflows immediately fail with CROSSSLOT Keys in request don't hash to the same slot: (1) an atomic Lua transfer/ledger script touching user:1042:balance, user:1042:ledger, and user:1042:nonce, and (2) a batch dashboard read calling MGET across 50 different users (user:101:profile ... user:150:profile). Redesign key naming, Lua execution, and client batching so both workflows succeed with < 2ms p99 latency and zero hot-shard skew.
Start the journey5 stages
Loading diagram…
Solving Redis Cluster Cross-Slot Errors with Hash Tags architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your payments service is migrating from standalone Redis to a 6-node Redis Cluster (3 primaries, 3 replicas, 16,384 hash slots). Two core workflows immediately fail with `CROSSSLOT Keys in request don't hash to the same slot`: (1) an atomic Lua transfer/ledger script touching `user:1042:balance`, `user:1042:ledger`, and `user:1042:nonce`, and (2) a batch dashboard read calling `MGET` across 50 different users (`user:101:profile ... user:150:profile`). Redesign key naming, Lua execution, and client batching so both workflows succeed with < 2ms p99 latency and zero hot-shard skew.

From patterns to decisions.

Your payments service is migrating from standalone Redis to a 6-node Redis Cluster (3 primaries, 3 replicas, 16,384 hash slots). Two core workflows immediately fail with `CROSSSLOT Keys in request don't hash to the same slot`: (1) an atomic Lua transfer/ledger script touching `user:1042:balance`, `user:1042:ledger`, and `user:1042:nonce`, and (2) a batch dashboard read calling `MGET` across 50 different users (`user:101:profile ... user:150:profile`). Redesign key naming, Lua execution, and client batching so both workflows succeed with < 2ms p99 latency and zero hot-shard skew.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

Redis Cluster partitions keys across.

Redis Cluster partitions keys across 16,384 slots using CRC16(key) & 16383; keys without matching hash tags land on different slots and different primaries.

ChoiceWhat it buysWhat it costs
Trade-off 1No cross-shard 2PC keeps Redis Cluster throughput linear and sub-millisecond.

Compute how CRC16 hashes `user

1042:balance vs user:1042:nonce` to different slots.

Step 2

Note that Redis Cluster rejects any multi-key command spanning >1 slot because it has no cross-shard 2PC.

Alternative 1

Using a Redis proxy (Twemproxy/Envoy) or single-shard Redis if cross-key transactions are mandatory everywhere.

Failure Mode 1

Silent runtime CROSSSLOT exceptions in production if not tested against a multi-shard cluster.

import binascii

def redis_slot(key: str) -> int:
    s = key.find("{")
    if s != -1:
        e = key.find("}", s + 1)
        if e != -1 and e != s + 1:
            key = key[s + 1:e]
    return binascii.crc_hqx(key.encode("utf-8"), 0) & 16383
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Use {user.

Use {user:<id>} hash tags for per-user Lua atomicity and group cross-user MGET keys by primary node into concurrent per-node pipelines.

ChoiceWhat it buysWhat it costs
Trade-off 13 parallel pipelines complete in ~0.9ms vs 25ms sequential.
Trade-off 2Non-transactional per-shard pipeline returns values even if keys are on different slots of the same primary.

Format per-user keys as {user

<id>}:balance, {user:<id>}:ledger, {user:<id>}:nonce.

Step 2

Compute slot for each key in batch MGET and map to its primary node.

Step 3

Execute one pipeline per primary concurrently and reassemble results in input order.

Alternative 1

Using redis-py RedisCluster.mget_nonatomic() which internally groups by node.

Failure Mode 1

Calling MGET within a single node's pipeline still throws CROSSSLOT if keys are in different slots on that same primary—use pipe.get(k) per key.

import binascii
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, List, Optional

def user_keys(user_id: int) -> tuple[str, str, str]:
    tag = f"{{user:{user_id}}}"
    return f"{tag}:balance", f"{tag}:ledger", f"{tag}:nonce"

def compute_slot(key: str) -> int:
    s = key.find("{")
    if s != -1:
        e = key.find("}", s + 1)
        if e != -1 and e > s + 1:
            key = key[s + 1:e]
    return binascii.crc_hqx(key.encode("utf-8"), 0) & 16383

def scatter_gather_mget(cluster_nodes: Dict[str, object], slot_to_node: Dict[int, str], keys: List[str]) -> List[Optional[str]]:
    grouped: Dict[str, List[tuple[int, str]]] = defaultdict(list)
    for idx, k in enumerate(keys):
        node_id = slot_to_node[compute_slot(k)]
        grouped[node_id].append((idx, k))

    results: List[Optional[str]] = [None] * len(keys)

    def fetch_node(node_id: str, items: List[tuple[int, str]]):
        pipe = cluster_nodes[node_id].pipeline(transaction=False)
        for _, k in items:
            pipe.get(k)
        vals = pipe.execute()
        return [(idx, val) for (idx, _), val in zip(items, vals)]

    with ThreadPoolExecutor(max_workers=len(grouped)) as pool:
        futures = [pool.submit(fetch_node, nid, items) for nid, items in grouped.items()]
        for fut in futures:
            for idx, val in fut.result():
                results[idx] = val
    return results
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

A single primary owns 5,461.

A single primary owns 5,461 slots; MGET on two keys in different slots on the same primary still fails with CROSSSLOT.

ChoiceWhat it buysWhat it costs
Trade-off 1Pipelined GETs use slightly more framing bytes than a single MGET but are 100% slot-safe.

Step 1

Verify that CROSSSLOT is enforced at the slot level, not the physical node level.

Step 2

Confirm that pipelining GET k1, GET k2, ..., GET k17 in one TCP write gives 1-RTT batching with zero CROSSSLOT risk.

Alternative 1

Grouping strictly by exact slot (5,461 groups) — impractical for 50 keys.

Failure Mode 1

Assuming same-node equals same-slot breaks immediately during cluster resharding.

# Primary 1 owns slots 0..5460. Slot 10 and Slot 20 are on the SAME node,
# yet MGET slot10_key slot20_key STILL returns CROSSSLOT!
# Solution: pipeline individual GETs.
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Primary 1 returns -MOVED 4120.

Primary 1 returns -MOVED 4120 <Primary2_IP> for that key in the pipeline; the client refreshes its slot map and retries only the redirected keys.

ChoiceWhat it buysWhat it costs
Trade-off 1Per-key retry avoids re-querying the 49 keys that already succeeded.

Step 1

Inspect per-command errors inside the pipeline response array.

Step 2

Trigger a single-flight CLUSTER SHARDS refresh on -MOVED.

Step 3

Retry only the keys that received -MOVED or -ASK.

Alternative 1

Using redis-py RedisCluster built-in retry + pipeline routing.

Failure Mode 1

Unbounded retry loops if a cluster is in CLUSTERDOWN state.

# Retry only keys whose pipeline slot returned MOVED/ASK rather than re-fetching all 50 keys
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Execute the authoritative atomic debit/credit in a transactional database (Postgres/Spanner) and project balances to Redis via CDC/outbox, or use a two-phase idempotent Saga.

ChoiceWhat it buysWhat it costs
Trade-off 1Guarantees financial correctness while keeping Redis read scaling intact.

Step 1

Perform UPDATE accounts ... atomically in the primary RDBMS.

Emit an outbox/CDC event to update `{user

1042}:balance and {user:2099}:balance` in Redis.

Step 3

Use versioned SET (Lua if new_ver > cur_ver) so out-of-order CDC updates never regress balances.

Alternative 1

Two-phase reservation keys in Redis with TTL rollback (complex edge cases on crash).

Failure Mode 1

Out-of-order CDC delivery overwriting a newer balance if version checks are omitted.

-- Monotonic cache projection Lua per user slot
-- if tonumber(ARGV[2]) > tonumber(redis.call('GET', KEYS[2]) or 0) then ...

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Colocate strictly atomic entity keys on one slot using {hash_tags} (e.g., {user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.
  • 02Trade-off tip: Pipelined GETs use slightly more framing bytes than a single MGET but are 100% slot-safe.
  • 03Watch out for: Unbounded retry loops if a cluster is in CLUSTERDOWN state.
Behavioral Progression · Redis Cluster Hash Tags Cross Slot

Redis Cluster Hash Tags Cross Slot — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Colocate strictly atomic entity keys on one slot using {hashtags} (e.g., {user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.

Synthesis / One connected design

Putting Solving Redis Cluster Cross-Slot Errors with Hash Tags Together

Colocate strictly atomic entity keys on one slot using `{hash_tags}` (e.g., `{user:1042}:balance`), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.

A candidate architecture, not a universal answer

Putting Solving Redis Cluster Cross-Slot Errors with Hash Tags Together

Step 1
Step 2
Step 3
  1. Format per-user keys as {user:<id>}:balance, {user:<id>}:ledger, {user:<id>}:nonce.
  2. Compute slot for each key in batch MGET and map to its primary node.
  3. Execute one pipeline per primary concurrently and reassemble results in input order.
Primary 1 returns `-MOVED 4120 <Primary2_IP>` for that key in the pipeline; the client refreshes its slot map and retries only the redirected keys.
Interview Readiness Check

Primary 1 returns -MOVED 4120 <Primary2_IP> for that key in the pipeline; the client refreshes its slot map and retries only the redirected keys.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Solving Redis Cluster Cross-Slot Errors with Hash Tags

Colocate strictly atomic entity keys on one slot using {hash_tags} (e.g., {user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.