System Design HLD Example: Distributed Rate Limiter

Scenario context

A distributed rate limiter protects APIs from abuse and "noisy neighbors" by enforcing request quotas across a cluster of servers. The core technical challenge is Atomic State Management—solved by using Redis Lua scripts to perform a "check-and-increment" operation in a single atomic step, preventing race conditions that would allow users to bypass their limits.

Observable outcome: If the counter exceeds the configured policy limit, the gateway immediately returns a 429 Too Many Requests response without forwarding the request to the application tier — effectively shedding load before it reaches d…

Stage 6 — Deep Dive: How Each Algorithm Works and When the Atomic Redis Pattern Is Required

• Internals: How Redis Lua Scripts Provide Atomic Enforcement Across a Distributed Cluster Redis executes Lua scripts on its single-threaded event loop — the same loop that handles all GET, SET, INCR, and EXPIRE commands.
• When a Rate Limiter Service instance sends a EVALSHA command with a Lua script, Redis processes it from start to finish without executing any other command in between.
• This is not a transaction (which requires MULTI/EXEC and does not prevent interleaving of other clients' commands) — it is true atomicity at the command level.
• The Sliding Window Counter Lua script reads two keys — the current window counter and the previous window counter — calculates the weighted estimate (prev count × remaining fraction) + curr count , and either increments…

graph TD
    A[Request Arrives] --> B{Check Tokens in Bucket}
    B -->|Tokens >= 1| C[Consume 1 Token]
    C --> D[Allow Request]
    B -->|Tokens = 0| E[Reject with 429]
    F[Refill Timer] -->|Add tokens at fixed rate| B
    G[Burst Scenario] -->|Multiple requests arrive simultaneously| B

Observable outcome: The Sliding Window Counter Lua script reads two keys — the current window counter and the previous window counter — calculates the weighted estimate (prev count × remaining fraction) + curr count , and either increments…

End-to-end outcome

After working through the stages, you should be able to explain:

• The Policy Service Is as Critical as the Enforcement Engine.
• Teams that hardcode rate limits in application configuration files spend hours debugging why limit changes require a full deployment.
• • A distributed rate limiter must use shared state (Redis) — not local per-server counters — to enforce limits correctly across a cluster of application servers.
• • Redis Lua scripts provide the atomic check-and-increment operation required to prevent race conditions where two concurrent requests both slip past a limit boundary.

Alternatives considered

Decision Guide: Choosing the Right Algorithm and Enforcement Strategy

Decision Guide: Choosing the Right Algorithm and Enforcement Strategy

Failure and degraded modes

Fail-Open vs. Fail-Closed: The Most Important Operational Decision

• Detection: If the Redis cluster goes down, what should the rate limiter do?
• User or operator impact: If the Redis cluster goes down, what should the rate limiter do?

The Race Condition Window and Its Real-World Impact

• Detection: Even with Redis Lua atomic scripts, there is a small timing window for race conditions when using cluster-mode Redis.
• User or operator impact: Even with Redis Lua atomic scripts, there is a small timing window for race conditions when using cluster-mode Redis.

Local Cache for Policy Lookups

• Detection: The Policy Service stores rate limit configurations in Postgres and pushes them to Redis.
• User or operator impact: The Policy Service stores rate limit configurations in Postgres and pushes them to Redis.