Your payments service is migrating from standalone Redis to a 6-node Redis Cluster (3 primaries, 3 replicas, 16,384 hash slots). Two core workflows immediately fail with `CROSSSLOT Keys in request don't hash to the same slot`: (1) an atomic Lua transfer/ledger script touching `user:1042:balance`, `user:1042:ledger`, and `user:1042:nonce`, and (2) a batch dashboard read calling `MGET` across 50 different users (`user:101:profile ... user:150:profile`). Redesign key naming, Lua execution, and client batching so both workflows succeed with < 2ms p99 latency and zero hot-shard skew.
From patterns to decisions.
Your payments service is migrating from standalone Redis to a 6-node Redis Cluster (3 primaries, 3 replicas, 16,384 hash slots). Two core workflows immediately fail with `CROSSSLOT Keys in request don't hash to the same slot`: (1) an atomic Lua transfer/ledger script touching `user:1042:balance`, `user:1042:ledger`, and `user:1042:nonce`, and (2) a batch dashboard read calling `MGET` across 50 different users (`user:101:profile ... user:150:profile`). Redesign key naming, Lua execution, and client batching so both workflows succeed with < 2ms p99 latency and zero hot-shard skew.
Redis Cluster partitions keys across.
Redis Cluster partitions keys across 16,384 slots using CRC16(key) & 16383; keys without matching hash tags land on different slots and different primaries.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | No cross-shard 2PC keeps Redis Cluster throughput linear and sub-millisecond. |
Compute how CRC16 hashes `user
1042:balance vs user:1042:nonce` to different slots.
Step 2
Note that Redis Cluster rejects any multi-key command spanning >1 slot because it has no cross-shard 2PC.
Alternative 1
Using a Redis proxy (Twemproxy/Envoy) or single-shard Redis if cross-key transactions are mandatory everywhere.
Failure Mode 1
Silent runtime CROSSSLOT exceptions in production if not tested against a multi-shard cluster.
import binascii
def redis_slot(key: str) -> int:
s = key.find("{")
if s != -1:
e = key.find("}", s + 1)
if e != -1 and e != s + 1:
key = key[s + 1:e]
return binascii.crc_hqx(key.encode("utf-8"), 0) & 16383
Use {user.
Use {user:<id>} hash tags for per-user Lua atomicity and group cross-user MGET keys by primary node into concurrent per-node pipelines.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | 3 parallel pipelines complete in ~0.9ms vs 25ms sequential. | |
| Trade-off 2 | Non-transactional per-shard pipeline returns values even if keys are on different slots of the same primary. |
Format per-user keys as {user
<id>}:balance, {user:<id>}:ledger, {user:<id>}:nonce.
Step 2
Compute slot for each key in batch MGET and map to its primary node.
Step 3
Execute one pipeline per primary concurrently and reassemble results in input order.
Alternative 1
Using redis-py RedisCluster.mget_nonatomic() which internally groups by node.
Failure Mode 1
Calling MGET within a single node's pipeline still throws CROSSSLOT if keys are in different slots on that same primary—use pipe.get(k) per key.
import binascii
from collections import defaultdict
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, List, Optional
def user_keys(user_id: int) -> tuple[str, str, str]:
tag = f"{{user:{user_id}}}"
return f"{tag}:balance", f"{tag}:ledger", f"{tag}:nonce"
def compute_slot(key: str) -> int:
s = key.find("{")
if s != -1:
e = key.find("}", s + 1)
if e != -1 and e > s + 1:
key = key[s + 1:e]
return binascii.crc_hqx(key.encode("utf-8"), 0) & 16383
def scatter_gather_mget(cluster_nodes: Dict[str, object], slot_to_node: Dict[int, str], keys: List[str]) -> List[Optional[str]]:
grouped: Dict[str, List[tuple[int, str]]] = defaultdict(list)
for idx, k in enumerate(keys):
node_id = slot_to_node[compute_slot(k)]
grouped[node_id].append((idx, k))
results: List[Optional[str]] = [None] * len(keys)
def fetch_node(node_id: str, items: List[tuple[int, str]]):
pipe = cluster_nodes[node_id].pipeline(transaction=False)
for _, k in items:
pipe.get(k)
vals = pipe.execute()
return [(idx, val) for (idx, _), val in zip(items, vals)]
with ThreadPoolExecutor(max_workers=len(grouped)) as pool:
futures = [pool.submit(fetch_node, nid, items) for nid, items in grouped.items()]
for fut in futures:
for idx, val in fut.result():
results[idx] = val
return results
A single primary owns 5,461.
A single primary owns 5,461 slots; MGET on two keys in different slots on the same primary still fails with CROSSSLOT.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Pipelined GETs use slightly more framing bytes than a single MGET but are 100% slot-safe. |
Step 1
Verify that CROSSSLOT is enforced at the slot level, not the physical node level.
Step 2
Confirm that pipelining GET k1, GET k2, ..., GET k17 in one TCP write gives 1-RTT batching with zero CROSSSLOT risk.
Alternative 1
Grouping strictly by exact slot (5,461 groups) — impractical for 50 keys.
Failure Mode 1
Assuming same-node equals same-slot breaks immediately during cluster resharding.
# Primary 1 owns slots 0..5460. Slot 10 and Slot 20 are on the SAME node,
# yet MGET slot10_key slot20_key STILL returns CROSSSLOT!
# Solution: pipeline individual GETs.
Primary 1 returns -MOVED 4120.
Primary 1 returns -MOVED 4120 <Primary2_IP> for that key in the pipeline; the client refreshes its slot map and retries only the redirected keys.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Per-key retry avoids re-querying the 49 keys that already succeeded. |
Step 1
Inspect per-command errors inside the pipeline response array.
Step 2
Trigger a single-flight CLUSTER SHARDS refresh on -MOVED.
Step 3
Retry only the keys that received -MOVED or -ASK.
Alternative 1
Using redis-py RedisCluster built-in retry + pipeline routing.
Failure Mode 1
Unbounded retry loops if a cluster is in CLUSTERDOWN state.
# Retry only keys whose pipeline slot returned MOVED/ASK rather than re-fetching all 50 keys
Implementation & Guardrails (5).
Execute the authoritative atomic debit/credit in a transactional database (Postgres/Spanner) and project balances to Redis via CDC/outbox, or use a two-phase idempotent Saga.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Guarantees financial correctness while keeping Redis read scaling intact. |
Step 1
Perform UPDATE accounts ... atomically in the primary RDBMS.
Emit an outbox/CDC event to update `{user
1042}:balance and {user:2099}:balance` in Redis.
Step 3
Use versioned SET (Lua if new_ver > cur_ver) so out-of-order CDC updates never regress balances.
Alternative 1
Two-phase reservation keys in Redis with TTL rollback (complex edge cases on crash).
Failure Mode 1
Out-of-order CDC delivery overwriting a newer balance if version checks are omitted.
-- Monotonic cache projection Lua per user slot
-- if tonumber(ARGV[2]) > tonumber(redis.call('GET', KEYS[2]) or 0) then ...
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Colocate strictly atomic entity keys on one slot using
{hash_tags}(e.g.,{user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines. - 02Trade-off tip: Pipelined GETs use slightly more framing bytes than a single MGET but are 100% slot-safe.
- 03Watch out for: Unbounded retry loops if a cluster is in
CLUSTERDOWNstate.
Redis Cluster Hash Tags Cross Slot — Normal → Load / Fault → Recovery
Colocate strictly atomic entity keys on one slot using {hashtags} (e.g., {user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.
Putting Solving Redis Cluster Cross-Slot Errors with Hash Tags Together
Colocate strictly atomic entity keys on one slot using `{hash_tags}` (e.g., `{user:1042}:balance`), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.
Putting Solving Redis Cluster Cross-Slot Errors with Hash Tags Together
- Format per-user keys as {user:<id>}:balance, {user:<id>}:ledger, {user:<id>}:nonce.
- Compute slot for each key in batch MGET and map to its primary node.
- Execute one pipeline per primary concurrently and reassemble results in input order.
Interview Readiness Check
Primary 1 returns -MOVED 4120 <Primary2_IP> for that key in the pipeline; the client refreshes its slot map and retries only the redirected keys.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Solving Redis Cluster Cross-Slot Errors with Hash Tags
Colocate strictly atomic entity keys on one slot using {hash_tags} (e.g., {user:1042}:balance), and replace cross-entity MGET/MSET with slot-grouped parallel scatter-gather pipelines.
