Deterministic Replay with eBPF‑backed Event Sourcing
Capture every syscall in production and replay it exactly—no guesswork.
🤖 Use eBPF tracepoints + a lightweight event store to build a deterministic replay engine that can run in production without affecting latency.
🎯 System Scenario & Requirements
System Goal
🤖 Use eBPF tracepoints + a lightweight event store to build a deterministic replay engine that can run in production without affecting latency.
Actors & Components
- Service App
- eBPF Loader
- Event Store
- Replay Engine
- Service
- eBPF Hook
- Perf Ring
- Kafka/File
Functional Requirements
- 🤖 Use eBPF tracepoints + a lightweight event store to build a deterministic replay engine that can run in production without affecting latency.
- In a microservice cluster, a race condition causes a one‑off “null pointer” crash.
- Traditional logs miss the exact ordering of concurrent syscalls that produced the fault, so reproducing the bug in a staging environment fails.
Constraints & Assumptions
- Operational latency, consistency levels, and network partition boundaries must be enforced.
⚡ Execution Stages & Decision Trace
Stage 1 — The Solution: eBPF + Event Sourcing
eBPF Tracepoints hook into sys enter / sys exit for every process that belongs to the target service.
Each event is serialized into a fixed‑size struct containing: timestamp (ns), pid , tid , syscall id , args[] , return code .
Deep Dive
Deep Dive: Stage 1 — The Solution: eBPF + Event Sourcing
- eBPF Tracepoints hook into
sys_enter/sys_exitfor every process that belongs to the target service. - Each event is serialized into a fixed‑size struct containing:
timestamp(ns),pid,tid,syscall_id,args[],return_code.
- The eBPF program writes the struct into a perf event ring buffer.
- A user‑space consumer reads the ring buffer, writes each event to an append‑only event log (e.g., Kafka or a local file).
- A replay engine consumes the log, re‑injects syscalls via
ptraceorseccomp‑based sandbox, preserving the original timestamp order.
Because the events are stored in a canonical, immutable log, the replay is deterministic even if the original system had clock drift or out‑of‑order packet arrivals.
Observable outcome: Because the events are stored in a canonical, immutable log , the replay is deterministic even if the original system had clock drift or out‑of‑order packet arrivals.
Deep Dive: Stage 2 — Architecture Kit
Observable outcome: You can explain how “Architecture Kit” changes the architecture or implementation path.
Deep Dive: Stage 3 — Trade‑offs
- Overhead: The eBPF program adds ~1 µs per syscall; fine‑grained tracing on high‑throughput services may need sampling.
- Storage: Persisting every syscall can grow fast; consider compression or selective filtering.
- Re‑execution cost: Replaying syscalls in a sandbox can be slower than the original run, but it’s isolated and safe for debugging.
By combining eBPF’s zero‑overhead tracing with an immutable event store, you get a deterministic, reproducible debugging surface that works in production and lets you trace back to the exact sequence that caused a bug.
Observable outcome: By combining eBPF’s zero‑overhead tracing with an immutable event store, you get a deterministic, reproducible debugging surface that works in production and lets you trace back to the exact sequence that caused a bug.
📊 System Outcome & Trade-offs
Observed User & Operator Behavior
User: Clients observe deterministic state without lost updates or race conditions.
Operator: Operators can verify stage outcomes against the scenario requirements.
Key Architectural Trade-offs
- System guarantees are traded against replication overhead and failover convergence times.
Key learnings
- eBPF Tracepoints hook into sys enter / sys exit for every process that belongs to the target service.
- Apply cqrs-and-event-sourcing to Architecture Kit.
- Overhead: The eBPF program adds ~1 µs per syscall; fine‑grained tracing on high‑throughput services may need sampling.
Apply Concepts from this Scenario
Test your understanding of the architecture decisions and trade-offs presented in this case study.