Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Tuning Reverse Proxy Buffers
to Prevent 502s.

Your Nginx 1.24 reverse proxy fronts an OAuth2/OIDC SSO gateway and a GraphQL API at 6,000 RPS. After a product release added 45 fine-grained RBAC claims to the session JWT cookie, 6.5% of requests fail with 502 Bad Gateway. Nginx error logs show upstream sent too big header while reading response header from upstream and 400 Request Header Or Cookie Too Large. Each Nginx node has 16 GB RAM and handles up to 10,000 concurrent connections. Fix both the immediate 502 outage and the underlying token bloat without risking Nginx OOM.
Start the journey5 stages
Loading diagram…
Tuning Reverse Proxy Buffers to Prevent 502s architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Your Nginx 1.24 reverse proxy fronts an OAuth2/OIDC SSO gateway and a GraphQL API at 6,000 RPS. After a product release added 45 fine-grained RBAC claims to the session JWT cookie, 6.5% of requests fail with `502 Bad Gateway`. Nginx error logs show `upstream sent too big header while reading response header from upstream` and `400 Request Header Or Cookie Too Large`. Each Nginx node has 16 GB RAM and handles up to 10,000 concurrent connections. Fix both the immediate 502 outage and the underlying token bloat without risking Nginx OOM.

From patterns to decisions.

Your Nginx 1.24 reverse proxy fronts an OAuth2/OIDC SSO gateway and a GraphQL API at 6,000 RPS. After a product release added 45 fine-grained RBAC claims to the session JWT cookie, 6.5% of requests fail with `502 Bad Gateway`. Nginx error logs show `upstream sent too big header while reading response header from upstream` and `400 Request Header Or Cookie Too Large`. Each Nginx node has 16 GB RAM and handles up to 10,000 concurrent connections. Fix both the immediate 502 outage and the underlying token bloat without risking Nginx OOM.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Nginx's proxy_buffer_size (4 KB default.

Nginx's proxy_buffer_size (4 KB default on x86_64) must hold the entire upstream response header; a 9.4 KB Set-Cookie overflows it, causing Nginx to abort and emit 502.

ChoiceWhat it buysWhat it costs
Trade-off 1Log inspection immediately distinguishes header overflow from upstream crashes.

Step 1

Inspect /var/log/nginx/error.log for upstream sent too big header.

Step 2

Measure exact Set-Cookie byte length with curl -sI or wc -c.

Alternative 1

Check upstream_status vs status in Nginx access logs.

Failure Mode 1

Wasting time restarting healthy upstream pods when the proxy buffer is the bottleneck.

# /var/log/nginx/error.log:
# [error] 142#142: *99812 upstream sent too big header while reading response header from upstream
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Implementation & Guardrails (2).

Set proxy_buffer_size 16k and large_client_header_buffers 4 16k in Nginx, and store full RBAC claims in Redis behind a 64-byte opaque session ID.

ChoiceWhat it buysWhat it costs
Trade-off 1Max 144 KB/connection keeps 10,000 connections under 1.44 GB RAM.

Step 1

Update nginx.conf with 16k/32k buffer directives and validate with nginx -t.

Step 2

Modify SSO login handler to write claims to Redis and issue an opaque sid cookie.

Alternative 1

Use proxy_buffering off only for SSE/streaming endpoints, never for general HTTP APIs.

Failure Mode 1

Setting proxy_busy_buffers_size < proxy_buffer_size fails nginx -t.

server {
    large_client_header_buffers 4 16k;
    location / {
        proxy_pass http://sso_backend;
        proxy_buffering on;
        proxy_buffer_size 16k;
        proxy_buffers 8 16k;
        proxy_busy_buffers_size 32k;
    }
}
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Browsers enforce a 4,096-byte per-cookie.

Browsers enforce a 4,096-byte per-cookie limit and CDNs enforce 16 KB total header limits; only opaque server-side tokens prevent future RBAC growth from breaking clients.

ChoiceWhat it buysWhat it costs
Trade-off 1Adds 1 Redis lookup (< 1 ms) in exchange for < 128B cookies and instant session revocation.

Replace `Set-Cookie

jwt=<9.4KB> with Set-Cookie: sid=<64B>`.

Step 2

Expire the old jwt cookie (Max-Age=0) on next user visit.

Alternative 1

Keep a tiny signed JWT with sub + auth_ver and fetch RBAC permissions from Redis.

Failure Mode 1

Forgetting response.delete_cookie("legacy_jwt") leaves users sending 9.4 KB headers until expiry.

sid = secrets.token_urlsafe(32)
await redis.set(f"sess:{sid}", json.dumps(claims), ex=28800)
response.set_cookie("sid", sid, httponly=True, secure=True, samesite="lax")
response.delete_cookie("legacy_jwt")
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Once the 128 KB proxy_buffers.

Once the 128 KB proxy_buffers fill, Nginx spills the excess body to disk (proxy_temp_path), increasing disk I/O unless proxy_max_temp_file_size is tuned.

ChoiceWhat it buysWhat it costs
Trade-off 1Disk spill protects RAM from OOM on occasional large responses while freeing upstream app workers quickly.

Step 1

Monitor nginx_http_upstream_temp_files or disk I/O.

Step 2

Cap proxy_max_temp_file_size 64m; (or 0 to stream directly to fast clients).

Alternative 1

Enforce GraphQL pagination (first: 50) so responses stay < 128 KB.

Failure Mode 1

Setting proxy_max_temp_file_size 0 forces upstream app workers to stay blocked until slow mobile clients finish downloading.

proxy_max_temp_file_size 64m;
proxy_temp_file_write_size 32k;
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Add an integration test asserting.

Add an integration test asserting total Set-Cookie and response header size stays < 2,048 bytes on login and token refresh endpoints.

ChoiceWhat it buysWhat it costs
Trade-off 1Catches regressions in seconds during PR checks.

Step 1

Generate a worst-case user with 100+ RBAC roles in CI.

Step 2

Assert len(response.headers['Set-Cookie']) < 2048.

Alternative 1

Alert in production if http_response_header_bytes_p99 > 4096.

Failure Mode 1

Testing only with a default admin user that has 1 role instead of a 100-role user.

def test_login_cookie_size_budget(client, max_roles_user):
    resp = client.post("/oauth/token", json=max_roles_user)
    cookie_bytes = sum(len(v) for k, v in resp.headers.items() if k.lower() == "set-cookie")
    assert cookie_bytes < 2048, f"Cookie bloat detected: {cookie_bytes} bytes"

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Align proxy_buffer_size / proxy_buffers to 16 KB–32 KB for immediate relief, then eliminate cookie/JWT bloat using opaque Redis-backed reference tokens (< 128 bytes).
  • 02Trade-off tip: Adds 1 Redis lookup (< 1 ms) in exchange for < 128B cookies and instant session revocation.
  • 03Watch out for: Setting proxy_max_temp_file_size 0 forces upstream app workers to stay blocked until slow mobile clients finish downloading.
Behavioral Progression · Tuning Nginx Proxy Buffers 502

Tuning Nginx Proxy Buffers 502 — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Align proxybuffersize / proxybuffers to 16 KB–32 KB for immediate relief, then eliminate cookie/JWT bloat using opaque Redis-backed reference tokens (< 128 bytes).

Synthesis / One connected design

Putting Tuning Reverse Proxy Buffers to Prevent 502s Together

Align `proxy_buffer_size` / `proxy_buffers` to 16 KB–32 KB for immediate relief, then eliminate cookie/JWT bloat using opaque Redis-backed reference tokens (< 128 bytes).

A candidate architecture, not a universal answer

Putting Tuning Reverse Proxy Buffers to Prevent 502s Together

Step 1
Step 2
  1. Update nginx.conf with 16k/32k buffer directives and validate with nginx -t.
  2. Modify SSO login handler to write claims to Redis and issue an opaque sid cookie.
Once the 128 KB `proxy_buffers` fill, Nginx spills the excess body to disk (`proxy_temp_path`), increasing disk I/O unless `proxy_max_temp_file_size` is tuned.
Interview Readiness Check

Once the 128 KB proxy_buffers fill, Nginx spills the excess body to disk (proxy_temp_path), increasing disk I/O unless proxy_max_temp_file_size is tuned.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Tuning Reverse Proxy Buffers to Prevent 502s

Align proxy_buffer_size / proxy_buffers to 16 KB–32 KB for immediate relief, then eliminate cookie/JWT bloat using opaque Redis-backed reference tokens (< 128 bytes).