System Design HLD Example: API Gateway for Microservices

Scenario context

An API Gateway centralizes "cross-cutting concerns" like authentication, rate limiting, and routing at the edge of your infrastructure. The architectural crux is the separation of the Control Plane (managing configurations) from the Data Plane (high-performance request proxying). By using an in-memory Radix Tree for routing and Redis for distributed rate limiting, the gateway ensures that backend ser…

This walkthrough focuses on The "Messy Front Door" Problem.

Before making a design choice, clarify:

• Imagine you’ve successfully migrated your monolith to 200 microservices.
• Each service has its own URL, its own way of checking user permissions, and its own rate limits.
• Now, your mobile app needs to render a single "Home Screen" that requires data from five different services: User Profile, Notifications, Order History, Recommendations, and Promotional Banners.

Concepts in play: system-design-hld-architectures, caching, rate-limiting, api-gateway

Stage 1 — API Gateway: Use Cases & Requirements

• Actors & Journeys - External Client: A mobile app, browser, or third-party partner calling public APIs.
• • Service Developer: Owns a backend microservice; wants to expose an endpoint without writing auth/rate-limiting boilerplate.
• • Platform Engineer: Manages the gateway's global policies, such as "Block all traffic from IP range X" or "Enable Canary for Service Y." - Security Auditor: Needs a centralized log of every request that entered the sys…
• In/Out Scope - In-Scope: Request routing, protocol translation (HTTP to gRPC), authentication/authorization, rate limiting, request/response transformation, and canary traffic splitting.

Observable outcome: In/Out Scope - In-Scope: Request routing, protocol translation (HTTP to gRPC), authentication/authorization, rate limiting, request/response transformation, and canary traffic splitting.

Stage 2 — The Gateway Mechanics: The Plugin Pipeline

• A request doesn't just "jump" to the backend.
• It passes through a Linear Filter Chain .
• • Pre-Auth Filters: IP Whitelisting, DDoS protection.
• • Auth Filters: JWT signature verification, API key lookup.

Observable outcome: - Auth Filters: JWT signature verification, API key lookup.

Stage 3 — Estimations & Design Goals

• Capacity Math (The Enterprise Scale) - Requests per Second (RPS): 100,000 (Steady state).
• • Active Consumers: 50,000 API Keys.
• • Log Volume: $100K \text{ requests/sec} \times 1KB \text{ per log} = \mathbf{100 \text{ MB/sec}}$ of log data.
• Scaling Targets - Max Latency Overhead: 5ms.

Observable outcome: Scaling Targets - Max Latency Overhead: 5ms.

Stage 4 — High-Level Design: The Distributed Gateway Architecture

• mermaid Internals: Radix Tree Path Lookup and Token Authentication Caching A naive routing implementation stores routes in a list and searches linearly — O(N) per request.
• With 2,000 routes and 100,000 RPS, this burns significant CPU.
• A Radix Tree (Patricia Trie) compresses route paths into a branching prefix structure.
• Looking up /api/v1/users/42 traverses the tree character-group by character-group until a leaf node bound to an upstream URL is reached.

Observable outcome: Looking up /api/v1/users/42 traverses the tree character-group by character-group until a leaf node bound to an upstream URL is reached.

End-to-end outcome

Imagine you’ve successfully migrated your monolith to 200 microservices.

Alternatives considered

Foundations: Data Plane vs. Control Plane

The most critical architectural distinction in a modern gateway is the split between the Data Plane and the Control Plane.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.