Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

End-to-End Request Tracing
with Correlation IDs.

A checkout request traverses 6 hops: API Gateway -> order-service -> inventory-service (gRPC) -> Kafka payment.requested -> payment-worker (Python) -> notification-service. Currently, each service logs unstructured plain text with only local thread IDs, and async CompletableFuture/Kafka hops lose all request context. During a peak-hour incident where 2% of orders are charged twice or stuck in PENDING, on-call engineers spend 4+ hours grepping 120 GB/hour of logs across 6 services by approximate timestamps. Design an end-to-end correlation and distributed tracing architecture that links every log line, metric exemplar, gRPC call, async thread, and Kafka message under a single immutable ID with < 1.5% CPU/network overhead.
Start the journey5 stages
Loading diagram…
End-to-End Request Tracing with Correlation IDs architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A checkout request traverses 6 hops: API Gateway -> `order-service` -> `inventory-service` (gRPC) -> Kafka `payment.requested` -> `payment-worker` (Python) -> `notification-service`. Currently, each service logs unstructured plain text with only local thread IDs, and async `CompletableFuture`/Kafka hops lose all request context. During a peak-hour incident where 2% of orders are charged twice or stuck in `PENDING`, on-call engineers spend 4+ hours grepping 120 GB/hour of logs across 6 services by approximate timestamps. Design an end-to-end correlation and distributed tracing architecture that links every log line, metric exemplar, gRPC call, async thread, and Kafka message under a single immutable ID with < 1.5% CPU/network overhead.

From patterns to decisions.

A checkout request traverses 6 hops: API Gateway -> `order-service` -> `inventory-service` (gRPC) -> Kafka `payment.requested` -> `payment-worker` (Python) -> `notification-service`. Currently, each service logs unstructured plain text with only local thread IDs, and async `CompletableFuture`/Kafka hops lose all request context. During a peak-hour incident where 2% of orders are charged twice or stuck in `PENDING`, on-call engineers spend 4+ hours grepping 120 GB/hour of logs across 6 services by approximate timestamps. Design an end-to-end correlation and distributed tracing architecture that links every log line, metric exemplar, gRPC call, async thread, and Kafka message under a single immutable ID with < 1.5% CPU/network overhead.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Context breaks at ingress (no.

Context breaks at ingress (no traceparent), Java async executors (ThreadLocal drop), gRPC/Kafka boundaries (no header injection), and unstructured text logs.

ChoiceWhat it buysWhat it costs
factorZero-Code OTel Java Agent vs. Manual SDK; pro: OTel Java Agent auto-instruments Spring, gRPC, CompletableFuture, and Kafka.; con: Adds ~100ms startup time and ~30 MB RAM per JVM.

Step 1

Check Envoy ingress for traceparent generation.

Step 2

Trace ThreadLocal MDC loss inside CompletableFuture.supplyAsync().

Step 3

Inspect Kafka ProducerRecord headers and Python worker extraction.

Alternative 1

Micrometer Tracing + OpenTelemetry bridge in Spring Boot 3.

name

Stale MDC on Reused Threads; description: Without MDC.clear(), threads log previous requests' trace IDs.; mitigation: Always clear MDC in finally.

# Envoy HttpConnectionManager tracing config
generate_request_id: true
tracing:
  provider:
    name: envoy.tracers.opentelemetry
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Propagate W3C traceparent across Envoy.

Propagate W3C traceparent across Envoy, Java Context.taskWrapping, Kafka RecordHeaders, and Python structlog.

ChoiceWhat it buysWhat it costs
factorKafka Headers vs. Payload Field; pro: Keeps business schemas clean and works with Schema Registry.; con: Requires consumers to read `msg.headers()`.

Step 1

Generate traceparent at Envoy and return X-Trace-Id.

Step 2

Wrap Java executors with Context.taskWrapping / MdcTaskDecorator.

Step 3

Inject traceparent into Kafka RecordHeaders and extract in Python.

Alternative 1

CloudEvents traceparent extension attribute.

name

Header Stripping by Legacy Proxy; description: Older Kafka/HTTP bridges may drop non-ASCII or custom headers.; mitigation: Use standard lowercase ASCII traceparent.

from opentelemetry.propagate import inject, extract
headers = []
inject(carrier := {})
for k, v in carrier.items():
    headers.append((k, v.encode()))
producer.produce('payment.requested', value=payload, headers=headers)
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

External IDs can be spoofed.

External IDs can be spoofed, non-unique, or injection vectors; mint a fresh 128-bit trace_id at Envoy and index trace_id as a keyword while sampling spans.

ChoiceWhat it buysWhat it costs
factorTail-Based Span Sampling (5% + 100% Errors); pro: Cuts APM trace storage cost by ~90% while keeping 100% of failure waterfalls.; con: Requires stateful OTel Collector tail-sampling tier routing by `trace_id`.

Step 1

Store client-supplied X-Request-ID in a separate client_request_id attribute.

Step 2

Index trace_id as an exact keyword field in OpenSearch/Loki and drop verbose DEBUG logs in prod.

Alternative 1

Head-based 5% sampling + 100% structured JSON logs.

name

Tail Sampler Split-Brain; description: If spans for the same trace_id hit different OTel Collectors, tail sampling drops half the trace.; mitigation: Use OTel loadbalancing exporter hashing on traceID.

{
  "ts": "2026-11-12T19:30:01.123Z",
  "level": "ERROR",
  "service": "payment-worker",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "client_request_id": "mobile-ios-9912",
  "msg": "idempotency_conflict"
}
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

Create a new batch span.

Create a new batch span with Span Links pointing to each message's traceparent rather than making 500 parents.

ChoiceWhat it buysWhat it costs
factorSpan Links vs. Direct Parent; pro: Preserves exact trace-to-batch causality without breaking UI waterfalls.; con: Not all legacy APM UIs render Span Links prominently.

Step 1

Extract SpanContext from each of the 500 Kafka messages.

Step 2

Start a batch span with links=[Link(ctx) for ctx in contexts] and process each item with its own child span.

Alternative 1

Process messages individually inside the poll loop with per-message child spans.

name

Giant 10,000-Span Trace; description: Linking a cron batch to 10k traces crashes Jaeger/Tempo UI.; mitigation: Cap links per batch span to 128.

links = [trace.Link(trace.get_current_span(extract(h)).get_span_context()) for h in batch_headers]
with tracer.start_as_current_span("consume_batch", links=links):
    for msg in batch:
        handle_message(msg)
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Use SQLCommenter (OTel DB instrumentation) to prepend /*traceparent='00-...'*/ to every SQL query.

ChoiceWhat it buysWhat it costs
factorSQLCommenter; pro: Links DB locks/slow queries directly to upstream HTTP traces.; con: Can fragment `pg_stat_statements` if `traceparent` isn't stripped by `pg_stat_statements` normalization (PostgreSQL 14+ normalizes comments safely).

Step 1

Enable otel.instrumentation.jdbc.statement-sanitizer.enabled=true and sqlcommenter.

Step 2

PostgreSQL slow-query logs (log_min_duration_statement = 200ms) automatically capture the trace_id.

Alternative 1

Set SET LOCAL application_name = '<trace_id>' inside transactions.

name

Prepared Statement Cache Miss; description: Dynamic comments in query text can bust app-side prepared statement caches.; mitigation: Only enable SQLCommenter on slow-path or use SET LOCAL.

SELECT * FROM orders WHERE id = $1 /*traceparent='00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01'*/;

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Generate and propagate W3C traceparent + X-Correlation-ID at the edge gateway, bridge them across Java ExecutorService/Reactor threads and Kafka headers, and emit JSON logs indexed by trace_id.
  • 02Trade-off tip: factor: Tail-Based Span Sampling (5% + 100% Errors); pro: Cuts APM trace storage cost by ~90% while keeping 100% of failure waterfalls.; con: Requires stateful OTel Collector tail-sampling tier routing by trace_id.
  • 03Watch out for: name: Giant 10,000-Span Trace; description: Linking a cron batch to 10k traces crashes Jaeger/Tempo UI.; mitigation: Cap links per batch span to 128.
Behavioral Progression · Distributed Tracing Correlation Ids

Distributed Tracing Correlation Ids — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Generate and propagate W3C traceparent + X-Correlation-ID at the edge gateway, bridge them across Java ExecutorService/Reactor threads and Kafka headers, and emit JSON logs indexed by traceid.

Synthesis / One connected design

Putting End-to-End Request Tracing with Correlation IDs Together

Generate and propagate W3C `traceparent` + `X-Correlation-ID` at the edge gateway, bridge them across Java `ExecutorService`/Reactor threads and Kafka headers, and emit JSON logs indexed by `trace_id`.

A candidate architecture, not a universal answer

Putting End-to-End Request Tracing with Correlation IDs Together

Step 1
Step 2
Step 3
  1. Generate traceparent at Envoy and return X-Trace-Id.
  2. Wrap Java executors with Context.taskWrapping / MdcTaskDecorator.
  3. Inject traceparent into Kafka RecordHeaders and extract in Python.
Create a new batch span with **Span Links** pointing to each message's `traceparent` rather than making 500 parents.
Interview Readiness Check

Create a new batch span with Span Links pointing to each message's traceparent rather than making 500 parents.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

End-to-End Request Tracing with Correlation IDs

Generate and propagate W3C traceparent + X-Correlation-ID at the edge gateway, bridge them across Java ExecutorService/Reactor threads and Kafka headers, and emit JSON logs indexed by trace_id.