Skip to main content
← PROBLEMS / ENGINEERING
An end-to-end domain learning journey

Capturing Tail Outliers with
Tail-Based Trace Sampling.

A checkout platform processing 18,000 RPS uses 1% uniform head-based trace sampling to control APM ingestion costs ($14k/mo). Customers report intermittent 6-second freezes on 0.5% of orders, but APM shows almost zero slow traces because 99% of those rare outliers are dropped at the ingress gateway before the downstream lock contention even happens.
Start the journey5 stages
Loading diagram…
Capturing Tail Outliers with Tail-Based Trace Sampling architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

A checkout platform processing 18,000 RPS uses 1% uniform head-based trace sampling to control APM ingestion costs ($14k/mo). Customers report intermittent 6-second freezes on 0.5% of orders, but APM shows almost zero slow traces because 99% of those rare outliers are dropped at the ingress gateway before the downstream lock contention even happens.

From patterns to decisions.

A checkout platform processing 18,000 RPS uses 1% uniform head-based trace sampling to control APM ingestion costs ($14k/mo). Customers report intermittent 6-second freezes on 0.5% of orders, but APM shows almost zero slow traces because 99% of those rare outliers are dropped at the ingress gateway before the downstream lock contention even happens.

01
ENGINEERING · STAGE 01L4 · FUNDAMENTALS

Head-based sampling decides at the.

Head-based sampling decides at the ingress hop before execution begins, discarding 99% of slow requests blindly and keeping only ~0.9 slow traces/sec without bias toward anomalies.

ChoiceWhat it buysWhat it costs
Trade-off 1Head-based sampling requires zero collector memory but provides zero anomaly prioritization.

Step 1

Explain that trace_id sampling happens at hop 1 before DB locks or downstream timeouts occur.

Step 2

Show that 99% of diagnostic traces for the 0.5% tail are dropped immediately.

Step 3

Note that if a downstream service also samples independently, traces become fragmented.

Alternative 1

Debug-header triggered tracing (only works when caller knows a request will be slow).

Failure Mode 1

Missing root cause for p99.9 outages because no slow trace survived the 1% coin flip.

# Broken head-based sampling config in app SDK
OTEL_TRACES_SAMPLER: "traceidratio"
OTEL_TRACES_SAMPLER_ARG: "0.01" # Blindly drops 99% of 6-second freezes
02
ENGINEERING · STAGE 02L5 · APPLIED DESIGN

Two-tier OTel Collector config.

Two-tier OTel Collector config: Tier 1 exports SpanMetrics and routes by traceID; Tier 2 buffers for 30s and applies latency, error, and 0.2% probabilistic policies.

ChoiceWhat it buysWhat it costs
Trade-off 1Buffering 216k spans/sec for 30s requires ~26 GB fleet RAM across 4 stateful pods.

Step 1

Configure Tier 1 loadbalancing exporter with routing_key: traceID.

Step 2

Configure Tier 2 memory_limiter (12 GiB) to prevent OOMKills.

Step 3

Define tail_sampling policies for status_code ERROR, latency >= 1500ms, and 0.2% baseline.

Alternative 1

Dynamic head-based adaptive sampling per endpoint (still misses per-request lock contention).

Failure Mode 1

Routing Tier 1 -> Tier 2 through a round-robin Kubernetes Service splits traces across pods.

# Tier 2 Tail-Sampling Collector (otel-tail-collector.yaml)
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 12288
    spike_limit_mib: 2048
  tail_sampling:
    decision_wait: 30s
    num_traces: 250000
    expected_new_traces_per_sec: 5000
    policies:
      - name: errors-policy
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow-traces-policy
        type: latency
        latency: { threshold_ms: 1500 }
      - name: baseline-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 0.2 }
03
ENGINEERING · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Tail sampling intentionally over-represents slow and errored traces (100% of >1.5s vs 0.2% of fast), which severely skews any latency percentiles computed post-sampling.

ChoiceWhat it buysWhat it costs
Trade-off 1Computing spanmetrics on 216k spans/sec adds ~15% CPU to Tier 1 gateways.

Show the math

keeping 100% of 0.5% slow traces and 0.2% of 99.5% fast traces makes slow traces ~71% of the exported dataset.

Step 2

Place spanmetrics connector in Tier 1 before loadbalancing/tail_sampling.

Step 3

Export exact Prometheus histograms independently of trace sampling.

Alternative 1

Instrumenting Prometheus histograms directly in every microservice SDK.

Failure Mode 1

Dashboards alerting on fake 90% error rates because metrics were derived from tail-sampled traces.

# Tier 1 SpanMetrics + TraceID LoadBalancer
connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [10ms, 50ms, 100ms, 250ms, 500ms, 1500ms, 5000ms]
exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls: { insecure: true }
    resolver:
      dns:
        hostname: otel-tail-collector-headless.monitoring.svc.cluster.local
04
ENGINEERING · STAGE 04L6 · SCALE & TRADE-OFFS

A 3x spike pushes buffered.

A 3x spike pushes buffered payload to ~35 GB raw (~75 GB with Go overhead), exceeding 64 GB fleet RAM unless memory_limiter and attribute stripping are configured.

ChoiceWhat it buysWhat it costs
Trade-off 1Dropping bulky attributes saves ~40% RAM per span at the cost of not seeing full SQL strings in traces.

Step 1

Size Tier 2 pods with 2.4x headroom (4 x 16 GB or 6 x 16 GB).

Step 2

Configure memory_limiter to refuse/drop new spans cleanly instead of OOMKilling the pod and losing all 30s of buffered traces.

Step 3

Strip large debug attributes (e.g., full SQL bodies > 512 bytes) at Tier 1 before buffering.

Alternative 1

Reducing decision_wait dynamically or spilling spans to local NVMe.

Failure Mode 1

OOMKill wipes out the entire 30s in-memory buffer of all 135,000 active traces simultaneously.

processors:
  attributes/truncate:
    actions:
      - key: db.statement
        action: delete # Strip bulky SQL payloads before 30s Tier 2 buffering
05
ENGINEERING · STAGE 05L7 · STAFF / PRINCIPAL

Monitor collector decision latency, orphan.

Monitor collector decision latency, orphan span rate, and exporter drop counters in Prometheus.

ChoiceWhat it buysWhat it costs
Trade-off 1Synthetic 2,000 ms canary adds negligible cost (1 trace/min) while guaranteeing end-to-end pipeline verification.

Step 1

Check otelcol_processor_tail_sampling_sampling_trace_dropped_too_early for traces exceeding 30s.

Step 2

Alert on otelcol_processor_refused_spans (memory_limiter triggers).

Step 3

Run a synthetic canary trace every 60s with a forced 2,000ms sleep and assert all 12 spans arrive in APM.

Alternative 1

Periodic manual trace inspection.

Failure Mode 1

Clock skew or >30s async spans causing partial orphan traces.

-- PromQL alerts for Tail-Sampling health
rate(otelcol_processor_tail_sampling_sampling_trace_dropped_too_early[5m]) > 0
rate(otelcol_processor_refused_spans[1m]) > 0

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.
  • 02Trade-off tip: Computing spanmetrics on 216k spans/sec adds ~15% CPU to Tier 1 gateways.
  • 03Watch out for: OOMKill wipes out the entire 30s in-memory buffer of all 135,000 active traces simultaneously.
Behavioral Progression · Tail Based Sampling Apm Tracing

Tail Based Sampling Apm Tracing — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.

Synthesis / One connected design

Putting Capturing Tail Outliers with Tail-Based Trace Sampling Together

Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.

A candidate architecture, not a universal answer

Putting Capturing Tail Outliers with Tail-Based Trace Sampling Together

Step 1
Step 2
Step 3
  1. Configure Tier 1 loadbalancing exporter with routing_key: traceID.
  2. Configure Tier 2 memory_limiter (12 GiB) to prevent OOMKills.
  3. Define tail_sampling policies for status_code ERROR, latency >= 1500ms, and 0.2% baseline.
A 3x spike pushes buffered payload to ~35 GB raw (~75 GB with Go overhead), exceeding 64 GB fleet RAM unless memory_limiter and attribute stripping are configured.
Interview Readiness Check

A 3x spike pushes buffered payload to ~35 GB raw (~75 GB with Go overhead), exceeding 64 GB fleet RAM unless memory_limiter and attribute stripping are configured.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is Distributed Tracing the right default for a production team?

Key takeaway

Capturing Tail Outliers with Tail-Based Trace Sampling

Switch from pure head-based probabilistic sampling to OpenTelemetry Collector tail-based sampling that buffers spans in memory for 30 s and keeps 100% of errors and high-latency traces.