Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Mitigating Serverless Cold-Start
Latency Spikes.

A Java 21 Spring Boot checkout API on AWS Lambda (and Cloud Run for EU) normally handles 150 RPS at 45 ms p95. When marketing push notifications trigger a 3,000 RPS spike within 10 seconds, 400+ concurrent cold starts fire simultaneously. Each cold start takes 3.4 seconds (JVM + Spring context + RDS Postgres TLS handshake), breaching the 2 s API Gateway timeout and exhausting Postgres max_connections (500).
Start the journey5 stages
Loading diagram…
Mitigating Serverless Cold-Start Latency Spikes architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A Java 21 Spring Boot checkout API on AWS Lambda (and Cloud Run for EU) normally handles 150 RPS at 45 ms p95. When marketing push notifications trigger a 3,000 RPS spike within 10 seconds, 400+ concurrent cold starts fire simultaneously. Each cold start takes 3.4 seconds (JVM + Spring context + RDS Postgres TLS handshake), breaching the 2 s API Gateway timeout and exhausting Postgres max_connections (500).

TL;DR30-second simplified summary
▾

A Java 21 Spring Boot checkout API on AWS Lambda (and Cloud Run for EU) normally handles 150 RPS at 45 ms p95. When marketing push notifications trigger a 3,000 RPS spike within 10 seconds, 400+ concurrent cold starts fire simultaneously. Each cold start takes 3.4 seconds (JVM + Spring context + RDS Postgres TLS handshake), breaching the 2 s API Gateway timeout and exhausting Postgres max_connections (500).

Combine SnapStart / container image optimization with predictive Provisioned Concurrency (min-instances), lazy initialization, and an external connection pooler (RDS Proxy / PgBouncer).

From patterns to decisions.

A Java 21 Spring Boot checkout API on AWS Lambda (and Cloud Run for EU) normally handles 150 RPS at 45 ms p95. When marketing push notifications trigger a 3,000 RPS spike within 10 seconds, 400+ concurrent cold starts fire simultaneously. Each cold start takes 3.4 seconds (JVM + Spring context + RDS Postgres TLS handshake), breaching the 2 s API Gateway timeout and exhausting Postgres max_connections (500).

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Cold starts stack microVM boot.

Cold starts stack microVM boot, JVM/Spring DI scanning, and TLS DB handshakes, while per-sandbox connection pools multiply linearly with concurrency.

ChoiceWhat it buysWhat it costs
Trade-off 1Direct JDBC avoids proxy latency (~1 ms) but cannot survive serverless horizontal scaling.

Decompose the 3.4s

sandbox init + JVM startup + Spring classpath scanning + DB TLS handshake.

Step 2

Multiply 400 sandboxes * 5 default Hikari connections = 2,000 connection attempts > 500 Postgres limit.

Step 3

Show how failed requests trigger client retries, spawning even more cold starts.

Alternative 1

HTTP-based serverless data APIs (higher per-query latency).

Failure Mode 1

Connection exhaustion cascades into 500 errors even on already-warm instances.

# Broken serverless DB config
spring:
  datasource:
    url: jdbc:postgresql://prod-rds.internal:5432/checkout
    hikari:
      maximum-pool-size: 10 # 100 Lambdas = 1,000 connections -> Postgres OOM/reject
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Combines Lambda SnapStart + CRaC.

Combines Lambda SnapStart + CRaC lifecycle hooks, RDS Proxy with Hikari max=1, and Cloud Run concurrency=80 with startup CPU boost.

ChoiceWhat it buysWhat it costs
Trade-off 1SnapStart adds ~15s to CI/CD publish time to capture the microVM snapshot.

Step 1

Enable SnapStart on PublishedVersions in SAM/Terraform.

Step 2

Implement CRaC Resource beforeCheckpoint/afterRestore to manage sockets safely.

Step 3

Point Spring Boot datasource to RDS Proxy with maximum-pool-size: 1.

Alternative 1

GraalVM Native Image (~80 ms cold start, but breaks some Spring reflection libraries).

Failure Mode 1

Forgetting to register CRaC hooks leaves closed/stale socket handles in the restored heap.

@Component
public class DbSnapshotHook implements org.crac.Resource {
    private final HikariDataSource dataSource;
    public DbSnapshotHook(HikariDataSource ds) {
        this.dataSource = ds;
        Core.getGlobalContext().register(this);
    }
    @Override public void beforeCheckpoint(Context<? extends Resource> ctx) {
        dataSource.evictConnection(null); // Flush idle sockets before snapshot
    }
    @Override public void afterRestore(Context<? extends Resource> ctx) {
        // Connections lazily re-established to RDS Proxy on first query
    }
}
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Lambda isolates 1 request per.

Lambda isolates 1 request per microVM, whereas Cloud Run routes up to N concurrent HTTP requests into a single multi-threaded container.

ChoiceWhat it buysWhat it costs
Trade-off 1Higher concurrency slashes cold starts and cost, but a single OOM crash affects up to 80 in-flight requests.

Step 1

Show that 3,000 RPS at 50 ms requires 150 concurrency: 150 Lambda sandboxes vs only 2–3 Cloud Run containers.

Step 2

Explain --cpu-boost during Cloud Run startup to halve JVM classloading time.

Step 3

Note thread-safety and heap sizing requirements for 80 concurrent Spring MVC/WebFlux threads.

Alternative 1

Lower concurrency (e.g., 20) for CPU-bound workloads to avoid thread contention.

Failure Mode 1

ThreadLocal leaks in multi-concurrent containers causing cross-request data corruption.

# Cloud Run service spec
spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/minScale: "3"
        run.googleapis.com/startup-cpu-boost: "true"
    spec:
      containerConcurrency: 80
      containers:
        - resources:
            limits: { cpu: "2", memory: "2Gi" }
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

RDS Proxy and PgBouncer in.

RDS Proxy and PgBouncer in transaction mode pin a client to a dedicated backend Postgres connection if session-scoped state is mutated.

ChoiceWhat it buysWhat it costs
Trade-off 1Disabling server-side prepared statements adds minimal SQL parse CPU (~0.2 ms) but enables true 10:1 connection multiplexing.

Identify common pinning triggers

prepared statements, SET commands, temporary tables, or advisory locks.

Step 2

Add prepareThreshold=0 to the PostgreSQL JDBC URL to disable server-side prepared statements.

Step 3

Move any session-level settings into transaction-local SET LOCAL.

Alternative 1

Using PostgreSQL 16+ / PgBouncer 1.21+ with protocol-level prepared statement tracking.

Pinning turns RDS Proxy into a 1

1 pass-through, re-exposing Postgres to connection exhaustion.

spring:
  datasource:
    url: jdbc:postgresql://checkout-proxy.proxy-xyz.us-east-1.rds.amazonaws.com:5432/checkout?sslmode=require&prepareThreshold=0
    hikari:
      maximum-pool-size: 1
      connection-timeout: 1000
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Track AWS Lambda InitDuration /.

Track AWS Lambda InitDuration / Cloud Run container_startup_latency alongside RDS Proxy DatabaseConnectionsPinned.

ChoiceWhat it buysWhat it costs
Trade-off 1Synthetic burst tests incur brief staging compute cost but catch accidental SnapStart or DI regressions before production.

Step 1

Alert if Lambda InitDuration p99 > 500 ms on published SnapStart aliases.

Step 2

Run a step-burst k6 test (150 -> 3,000 RPS in 10 s) in staging before major campaigns.

Step 3

Verify RDS Proxy ClientConnections vs DatabaseConnections ratio stays >= 4:1.

Alternative 1

Canary alias traffic shifting (5% for 15 min) with InitDuration rollback alarms.

Failure Mode 1

Deploying to $LATEST instead of a published version silently bypasses SnapStart.

-- CloudWatch Logs Insights query for cold start InitDuration
filter @type = "REPORT" and ispresent(@initDuration)
| stats pct(@initDuration, 50) as p50, pct(@initDuration, 99) as p99, count(*) as cold_starts by bin(5m)

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Combine SnapStart / container image optimization with predictive Provisioned Concurrency (min-instances), lazy initialization, and an external connection pooler (RDS Proxy / PgBouncer).
  • 02Trade-off tip: Higher concurrency slashes cold starts and cost, but a single OOM crash affects up to 80 in-flight requests.
  • 03Watch out for: Pinning turns RDS Proxy into a 1:1 pass-through, re-exposing Postgres to connection exhaustion.
Put it together · Synthesis

Putting Mitigating Serverless Cold-Start Latency Spikes Together

Combine SnapStart / container image optimization with predictive Provisioned Concurrency (min-instances), lazy initialization, and an external connection pooler (RDS Proxy / PgBouncer).

A candidate architecture, not a universal answer

Putting Mitigating Serverless Cold-Start Latency Spikes Together

Step 1
Step 2
Step 3
  1. Enable SnapStart on PublishedVersions in SAM/Terraform.
  2. Implement CRaC Resource beforeCheckpoint/afterRestore to manage sockets safely.
  3. Point Spring Boot datasource to RDS Proxy with maximum-pool-size: 1.
RDS Proxy and PgBouncer in transaction mode pin a client to a dedicated backend Postgres connection if session-scoped state is mutated.
Interview Readiness Check

RDS Proxy and PgBouncer in transaction mode pin a client to a dedicated backend Postgres connection if session-scoped state is mutated.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Mitigating Serverless Cold-Start Latency Spikes

Combine SnapStart / container image optimization with predictive Provisioned Concurrency (min-instances), lazy initialization, and an external connection pooler (RDS Proxy / PgBouncer).