Event Sourcing Snapshot Desynchronization After Domain Aggregate Logic Changes

System Design Interview•~30 min read•medium

Event-sourced banking platform calculating account balance by replaying historical events and cached snapshots.

1. Understand Requirements & Scope

Event-sourced banking platform calculating account balance by replaying historical events and cached snapshots. Failure cascade dynamic: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown.

Functional Requirements:

Operational Invariant: "In event-sourced architectures, snapshot schemas and aggregate logic must be strictly versioned; raw events remain the sole source of truth."
Production Hardening: Version aggregate snapshots explicitly (e.g. 'Snapshot_v2'). When business logic evolves, invalidate old snapshots and recompute fresh snapshots from the immutable raw event stream
Prevent cascading failure by isolating downstream degradation at the ingress boundary.

2. Back-of-the-Envelope Capacity Estimation

• Incident Scenario Baseline: Event-sourced banking platform calculating account balance by replaying historical events and cached snapshots.

• Failure Cascade Path: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown

• Sub-millisecond rejection / shed latency for overloaded requests

3. High-Level Architecture Flow

Version aggregate snapshots explicitly (e.g. 'Snapshot_v2'). When business logic evolves, invalidate old snapshots and recompute fresh snapshots from the immutable raw event stream (Event Versioning / Upcasting pattern). Invariant: "In event-sourced architectures, snapshot schemas and aggregate logic must be strictly versioned; raw events remain the sole source of truth."

Ingress → Bounded Admission Controller → Adaptive Concurrency / Queue Gate → Protected Worker Engine; Overload Shed Path → HTTP 429 / Circuit Fallback (<0.1ms).

Fig 1. Architecture data flow and component topology.

4. Production Implementation & Code

// HARDENED: Fast local Bloom filter / Redis revocation list verification
Claims claims = Jwts.parser().setSigningKey(secret).parseClaimsJws(token).getBody();
if (revocationBloomFilter.mightContain(claims.getId()) && redis.exists("revoked:" + claims.getId())) {
    throw new SecurityException("Revoked token access attempt rejected");
}

Hardened Implementation Sequence

Ingress Boundary Defense: Version aggregate snapshots explicitly (e.g. 'Snapshot_v2'). When business logic evolves, invalidate old snapshots and recompute fresh snapshots from
Failure Containment: Eliminate unconstrained buffering and propagate physical backpressure.
Invariant Enforcement: In event-sourced architectures, snapshot schemas and aggregate logic must be strictly versioned; raw events remain the sole source of truth.

5. Key Decisions & Operational Mitigations

Architectural Decisions & Trade-Offs

•Ingress Admission: Naive unconstrained buffering causes queue bloat and GC exhaustion. Mitigation: Bounded ingress gate with fast non-blocking shedding.
•Resource Strategy: Static capacity allocations collapse under brownouts. Mitigation: Dynamic adaptive limits scaling to real-time physical bounds.
•Outage Impact: Cascading lock convoys and memory saturation. Mitigation: Isolated failure domains keep P99 latency flat and protected.

Operational Hazards & Mitigations

•Cascading Failure: A developer updates the business logic for calculating fees on transactions. Old snapshots cached in storage reflect the old fee logic, while event replays compute the new fee logic, resulting in inconsistent balances for identical accounts.
•Breakdown Chain: 100k QPS → TTL Expiration → Cache Miss Herd → Origin Slam → 💥 DB Meltdown
•Shedding Failure: Without fast rejection at the boundary, queued dead work triggers multi-second GC pauses, CPU thrashing, and cascading 504 Gateway Timeouts.

Interactive Knowledge Check • Question 1 of 3↓

Interactive Practice Quiz•Question 1 of 3

intermediate

Canary vs Blue-Green Release Strategies

What is the key advantage of Canary Deployments over Blue-Green Deployments for large-scale microservice clusters?