Cassandra Write Path & Tombstone Gotchas
Why deletes in Cassandra create tombstones that can degrade read latency, and how LSM-tree flushes work.
Cassandra turns deletes into special markers called tombstones.
🎯 System Scenario & Requirements
System Goal
Cassandra turns deletes into special markers called tombstones.
Actors & Components
- Volatile Memory
- Persistent Storage
- Memtable (Sorted RAM)
- CommitLog (Append Only)
- SSTable (Immutable Disk)
Functional Requirements
- Cassandra turns deletes into special markers called tombstones.
- Excessive deletes lead to heavy tombstone scanning and GC stalls during reads.
- In Apache Cassandra, write operations avoid expensive random disk I/O by executing sequential appends.
Constraints & Assumptions
- When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries: The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers.
- This causes sudden CPU spikes, triggers TombstoneOverloadedException , and causes severe JVM stop-the-world garbage collection pauses.
⚡ Execution Stages & Decision Trace
Stage 1 — The Solution & Trade-offs
When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries: The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers.
This causes sudden CPU spikes, triggers TombstoneOverloadedException , and causes severe JVM stop-the-world garbage collection pauses.
Deep Dive
Deep Dive: Stage 1 — The Solution & Trade-offs
When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries:
- The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers. This causes sudden CPU spikes, triggers
TombstoneOverloadedException, and causes severe JVM stop-the-world garbage collection pauses. - Best Practice Solution: Avoid database anti-patterns that rely on frequent single-row updates or deletes. For time-series telemetry or logging workloads with automatic expiration, configure TimeWindowCompactionStrategy (TWCS) so entire SSTable files drop cleanly when their window expires without triggering tombstone scans.
Observable outcome: For time-series telemetry or logging workloads with automatic expiration, configure TimeWindowCompactionStrategy (TWCS) so entire SSTable files drop cleanly when their window expires without triggering tombstone scans.
📊 System Outcome & Trade-offs
Observed User & Operator Behavior
User: Clients observe deterministic state without lost updates or race conditions.
Operator: Operators can verify stage outcomes against the scenario requirements.
Key Architectural Trade-offs
- When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries: The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers.
- This causes sudden CPU spikes, triggers TombstoneOverloadedException , and causes severe JVM stop-the-world garbage collection pauses.
⚠️ Failure Modes & Degraded Operation
The Solution & Trade-offs
When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries: The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers. This causes sudden CPU spikes, triggers TombstoneOverloadedException , and causes severe JVM stop-the-world garbage collection pauses. Best Practice Solution: Avoid database anti-patterns that rely on frequent single-row updates or deletes. For time-series telemetry or logging workloads with automatic expiration, configure TimeWindowCompactionStrategy (TWCS) so entire SSTable files drop cleanly when their window expires without triggering tombstone scans.
Key learnings
- When a read query executes, the Cassandra coordinator must scan across multiple SSTables on disk and merge the latest values while filtering out tombstoned entries: The Tombstone Pitfall: If your workload generates frequent updates, row deletions, or rapid TTL expirations on high-volume tables, read queries must scan through thousands of tombstone markers.
Apply Concepts from this Scenario
Test your understanding of the architecture decisions and trade-offs presented in this case study.