How do websites survive unexpected traffic spikes? | Load Balancing Explained

Scenario context

Load balancing is the practice of distributing incoming network traffic across a cluster of backend servers to optimize capacity utilization, minimize latency, and ensure high availability. By acting as a reverse proxy, a load balancer prevents server saturation, handles SSL termination, and dynamically manages server pools via health checks.

Imagine it is 09:00 AM on a Friday, and a highly anticipated ticket sale…

This walkthrough focuses on The Architectural Problem: Handling Unexpected Traffic Spikes.

Before making a design choice, clarify:

• Imagine it is 09:00 AM on a Friday, and a highly anticipated ticket sale goes live.
• Within seconds, millions of users hit the web application simultaneously.
• If all these requests are routed to a single application server, the server's CPU will spike, physical memory (RAM) will be exhausted, request queues will fill up, and the server will ultimately crash under the load.

Concepts in play: load-balancing, api-security-oauth2, system-design-hld-architectures, api-gateway

Stage 1 — Load Balancing Algorithms

• Load balancers use specific algorithms to determine which backend server should handle an incoming request.
• The choice of algorithm depends on the nature of the workload and the server pool capacity.
• Round Robin and Weighted Round Robin - Round Robin: The load balancer loops through the list of servers sequentially, sending each new request to the next server in line.
• It assumes all servers have equal capacity and all requests require equal processing time.

Observable outcome: It assumes all servers have equal capacity and all requests require equal processing time.

Stage 2 — High Availability & Health Checks

• A load balancer is a single point of failure (SPOF) if not configured correctly.
• To ensure high availability and prevent downtime, production systems use redundant clustering and passive health monitoring.
• Active-Passive and Active-Active Clusters - Active-Passive Setup: Two load balancers share a virtual IP (VIP) using protocols like Keepalived or VRRP (Virtual Router Redundancy Protocol).
• The active node handles all traffic.

Observable outcome: The active node handles all traffic.

Stage 3 — NGINX Load Balancing Configuration

• The following NGINX configuration demonstrates a Layer 7 Weighted Round Robin load balancer with health check failover.

http {
    # Define a logical upstream pool of servers
    upstream backend_pool {
        server 10.0.0.4:8080 weight=3 max_fails=3 fail_timeout=10s;
        server 10.0.0.5:8080 weight=1 max_fails=3 fail_timeout=10s;
        server 10.0.0.6:8080 backup; # Handles traffic only when others are down
    }

    server {
        listen 80;
        server_name api.abstractalgorithms.dev;

        location / {
            # Route all incoming requests to the upstream pool
            proxy_pass http://backend_pool;

            # Forward client metadata to backend instances
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }
}

Observable outcome: The following NGINX configuration demonstrates a Layer 7 Weighted Round Robin load balancer with health check failover.

Stage 4 — Deep Dive: Connection Queuing and TCP Backlog Tuning

• When a massive traffic spike hits the load balancer, the backend application servers may not be able to keep up, even with load balancing.
• In this scenario, how the load balancer and the OS manage the connection queue is critical.
• When a client initiates a connection, the TCP handshake passes through two queues in the operating system: 1.
• SYN Queue (Incomplete Connections): Stores connections that have received a SYN packet but have not yet completed the 3-way handshake (waiting for the client's ACK).

Client ---- SYN ----> [ SYN Queue ]
Client <--- SYN-ACK -
Client ---- ACK ----> [ Accept Queue ] ---- accept() ----> Application

Observable outcome: SYN Queue (Incomplete Connections): Stores connections that have received a SYN packet but have not yet completed the 3-way handshake (waiting for the client's ACK).

Stage 5 — A Load Balancer Is Not an Autoscaler

• A load balancer distributes work across capacity that already exists.
• It cannot rescue a fleet whose instances are all saturated.
• Surviving a spike requires layers: CDN caching absorbs cacheable traffic; admission control rejects excess work; the load balancer routes accepted requests; bounded queues smooth short bursts; and autoscaling adds capac…
• ~~~mermaid flowchart LR U[Users] -- E[CDN and edge cache] E -- G[Rate limit and admission control] G -- LB[Load balancer] LB -- A[Healthy app instances] A -- Q[Bounded async queue] A -- D[(Database and cache)] M[Metrics…

Observable outcome: ~~~mermaid flowchart LR U[Users] -- E[CDN and edge cache] E -- G[Rate limit and admission control] G -- LB[Load balancer] LB -- A[Healthy app instances] A -- Q[Bounded async queue] A -- D[(Database and cache)] M[Metrics…

Stage 6 — Overload Protection and Backpressure

• • Rate limits protect fairness per user, token, tenant, or IP.
• • Concurrency limits cap expensive in-flight work even when request rate looks modest.
• • Bounded queues absorb short bursts; unbounded queues convert overload into memory exhaustion and extreme latency.
• • Load shedding returns a fast 429 or 503 for optional work rather than timing out every request.

Observable outcome: - Load shedding returns a fast 429 or 503 for optional work rather than timing out every request.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Load balancing distributes capacity; overload protection preserves it.
• • Autoscaling is delayed, so headroom and load shedding handle sudden spikes.

Alternatives considered

Load balancers operate at different layers of the OSI model, primarily Layer 4 (Transport Layer) and Layer 7 (Application Layer). The choice between L4 and L7 significantly impacts performance, scalability, and routing capabilities.

Load balancers operate at different layers of the OSI model, primarily Layer 4 (Transport Layer) and Layer 7 (Application Layer).

Layer 4 (Transport Layer) Load Balancing

Layer 4 load balancers make routing decisions based solely on transport-level protocols (TCP/UDP).

Layer 7 (Application Layer) Load Balancing

Layer 7 load balancers operate at the application level, parsing the full HTTP/HTTPS payload.

Decision Guide: Selecting the Right Load Balancer

Use this guide to choose the appropriate load balancing strategy for your system: - Choose a Layer 4 Load Balancer if: - You need maximum throughput and minimum latency (e.g., millions of packets per second).

Failure and degraded modes

Trade-offs and Architectural Challenges

• Detection: Implementing load balancing introduces several design trade-offs: 1.
• User or operator impact: Implementing load balancing introduces several design trade-offs: 1.

NGINX Example with Failure Bounds

• Detection: ~~~nginx upstream api pool { least conn; zone api pool 64k; server api-1:8080 max fails=3 fail timeout=10s slow start=30s; server api-2:8080 max fails=3 fail timeout=10s slow start=30s; keepalive 64;…
• User or operator impact: ~~~nginx upstream api pool { least conn; zone api pool 64k; server api-1:8080 max fails=3 fail timeout=10s slow start=30s; server api-2:8080 max fails=3 fail timeout=10s slow start=30s; keepalive 64;…

When one zone slows down, health checks may remain green while latency rises. Clients retry, the proxy retries again, and the remaining healthy instances receive several times the original traffic. First cap retries, shed optional endpoints, and remove demonstrably slow targets. Scale only after preventing request multiplication; otherwise new capacity can be consumed immediately by retries.

• Detection: When one zone slows down, health checks may remain green while latency rises.
• User or operator impact: When one zone slows down, health checks may remain green while latency rises.

15-Minute Spike Drill

• Detection: Increase staging traffic tenfold in under one minute.
• User or operator impact: Increase staging traffic tenfold in under one minute.