A checkout platform processing 18,000 RPS uses 1% uniform head-based trace sampling to control APM ingestion costs ($14k/mo). Customers report intermittent 6-second freezes on 0.5% of orders, but APM shows almost zero slow traces because 99% of those rare outliers are dropped at the ingress gateway before the downstream lock contention even happens.
From patterns to decisions.
A checkout platform processing 18,000 RPS uses 1% uniform head-based trace sampling to control APM ingestion costs ($14k/mo). Customers report intermittent 6-second freezes on 0.5% of orders, but APM shows almost zero slow traces because 99% of those rare outliers are dropped at the ingress gateway before the downstream lock contention even happens.
Head-based sampling decides at the.
Head-based sampling decides at the ingress hop before execution begins, discarding 99% of slow requests blindly and keeping only ~0.9 slow traces/sec without bias toward anomalies.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Head-based sampling requires zero collector memory but provides zero anomaly prioritization. |
Step 1
Explain that trace_id sampling happens at hop 1 before DB locks or downstream timeouts occur.
Step 2
Show that 99% of diagnostic traces for the 0.5% tail are dropped immediately.
Step 3
Note that if a downstream service also samples independently, traces become fragmented.
Alternative 1
Debug-header triggered tracing (only works when caller knows a request will be slow).
Failure Mode 1
Missing root cause for p99.9 outages because no slow trace survived the 1% coin flip.
# Broken head-based sampling config in app SDK
OTEL_TRACES_SAMPLER: "traceidratio"
OTEL_TRACES_SAMPLER_ARG: "0.01" # Blindly drops 99% of 6-second freezes
Two-tier OTel Collector config.
Two-tier OTel Collector config: Tier 1 exports SpanMetrics and routes by traceID; Tier 2 buffers for 30s and applies latency, error, and 0.2% probabilistic policies.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Buffering 216k spans/sec for 30s requires ~26 GB fleet RAM across 4 stateful pods. |
Step 1
Configure Tier 1 loadbalancing exporter with routing_key: traceID.
Step 2
Configure Tier 2 memory_limiter (12 GiB) to prevent OOMKills.
Step 3
Define tail_sampling policies for status_code ERROR, latency >= 1500ms, and 0.2% baseline.
Alternative 1
Dynamic head-based adaptive sampling per endpoint (still misses per-request lock contention).
Failure Mode 1
Routing Tier 1 -> Tier 2 through a round-robin Kubernetes Service splits traces across pods.
# Tier 2 Tail-Sampling Collector (otel-tail-collector.yaml)
processors:
memory_limiter:
check_interval: 1s
limit_mib: 12288
spike_limit_mib: 2048
tail_sampling:
decision_wait: 30s
num_traces: 250000
expected_new_traces_per_sec: 5000
policies:
- name: errors-policy
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow-traces-policy
type: latency
latency: { threshold_ms: 1500 }
- name: baseline-sample
type: probabilistic
probabilistic: { sampling_percentage: 0.2 }
Implementation & Guardrails (3).
Tail sampling intentionally over-represents slow and errored traces (100% of >1.5s vs 0.2% of fast), which severely skews any latency percentiles computed post-sampling.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Computing spanmetrics on 216k spans/sec adds ~15% CPU to Tier 1 gateways. |
Show the math
keeping 100% of 0.5% slow traces and 0.2% of 99.5% fast traces makes slow traces ~71% of the exported dataset.
Step 2
Place spanmetrics connector in Tier 1 before loadbalancing/tail_sampling.
Step 3
Export exact Prometheus histograms independently of trace sampling.
Alternative 1
Instrumenting Prometheus histograms directly in every microservice SDK.
Failure Mode 1
Dashboards alerting on fake 90% error rates because metrics were derived from tail-sampled traces.
# Tier 1 SpanMetrics + TraceID LoadBalancer
connectors:
spanmetrics:
histogram:
explicit:
buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1500ms, 5000ms]
exporters:
loadbalancing:
routing_key: traceID
protocol:
otlp:
tls: { insecure: true }
resolver:
dns:
hostname: otel-tail-collector-headless.monitoring.svc.cluster.local
A 3x spike pushes buffered.
A 3x spike pushes buffered payload to ~35 GB raw (~75 GB with Go overhead), exceeding 64 GB fleet RAM unless memory_limiter and attribute stripping are configured.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Dropping bulky attributes saves ~40% RAM per span at the cost of not seeing full SQL strings in traces. |
Step 1
Size Tier 2 pods with 2.4x headroom (4 x 16 GB or 6 x 16 GB).
Step 2
Configure memory_limiter to refuse/drop new spans cleanly instead of OOMKilling the pod and losing all 30s of buffered traces.
Step 3
Strip large debug attributes (e.g., full SQL bodies > 512 bytes) at Tier 1 before buffering.
Alternative 1
Reducing decision_wait dynamically or spilling spans to local NVMe.
Failure Mode 1
OOMKill wipes out the entire 30s in-memory buffer of all 135,000 active traces simultaneously.
processors:
attributes/truncate:
actions:
- key: db.statement
action: delete # Strip bulky SQL payloads before 30s Tier 2 buffering
Monitor collector decision latency, orphan.
Monitor collector decision latency, orphan span rate, and exporter drop counters in Prometheus.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Synthetic 2,000 ms canary adds negligible cost (1 trace/min) while guaranteeing end-to-end pipeline verification. |
Step 1
Check otelcol_processor_tail_sampling_sampling_trace_dropped_too_early for traces exceeding 30s.
Step 2
Alert on otelcol_processor_refused_spans (memory_limiter triggers).
Step 3
Run a synthetic canary trace every 60s with a forced 2,000ms sleep and assert all 12 spans arrive in APM.
Alternative 1
Periodic manual trace inspection.
Failure Mode 1
Clock skew or >30s async spans causing partial orphan traces.
-- PromQL alerts for Tail-Sampling health
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m]) > 0
rate(otelcol_processor_refused_spans[1m]) > 0
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.
- 02Trade-off tip: Computing spanmetrics on 216k spans/sec adds ~15% CPU to Tier 1 gateways.
- 03Watch out for: OOMKill wipes out the entire 30s in-memory buffer of all 135,000 active traces simultaneously.
Tail Based Sampling Apm Tracing — Normal → Load / Fault → Recovery
Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.
Putting Capturing Tail Outliers with Tail-Based Trace Sampling Together
Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.
Putting Capturing Tail Outliers with Tail-Based Trace Sampling Together
- Configure Tier 1 loadbalancing exporter with routing_key: traceID.
- Configure Tier 2 memory_limiter (12 GiB) to prevent OOMKills.
- Define tail_sampling policies for status_code ERROR, latency >= 1500ms, and 0.2% baseline.
Interview Readiness Check
A 3x spike pushes buffered payload to ~35 GB raw (~75 GB with Go overhead), exceeding 64 GB fleet RAM unless memory_limiter and attribute stripping are configured.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is Distributed Tracing the right default for a production team?
Capturing Tail Outliers with Tail-Based Trace Sampling
Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.
