📦 Snap & Roll: Raft Log Compaction in Serverless Functions

Scenario context

In serverless Raft, the log can balloon fast because functions are stateless. Persisting compacted snapshots to object storage lets you restart a node in seconds and keep state machine size manageable. Serverless functions (e.g., AWS Lambda, Azure Functions) terminate after a short run. A Raft node that stores its log in memory loses it on shutdown. If you restart the function, you must replay the entire log from th…

This walkthrough focuses on The Gotcha.

Before making a design choice, clarify:

• Serverless functions (e.g., AWS Lambda, Azure Functions) terminate after a short run.
• A Raft node that stores its log in memory loses it on shutdown.
• If you restart the function, you must replay the entire log from the last snapshot.

Concepts in play: consistency-replication, caching, database-indexing, git-internals-and-versioning

Stage 1 — The Solution & Trade‑offs

• Persist Snapshots – After every N log entries or at a regular interval, serialize the state machine’s current state and the last committed index.
• Store this blob in durable object storage (S3, Blob, etc.).
• Store Metadata – Keep a small metadata file that records the snapshot’s index, term, and a checksum.
• This allows a new function instance to validate the snapshot before loading.

Observable outcome: This allows a new function instance to validate the snapshot before loading.

Stage 2 — Architecture Kit Diagram

• { "version": 1, "title": "Serverless Raft Snapshot Flow", "description": "How a stateless function loads a Raft snapshot from object storage", "view": "data-flow", "pattern": "reactive-pull", "direction": "LR", "groups"…

Observable outcome: { "version": 1, "title": "Serverless Raft Snapshot Flow", "description": "How a stateless function loads a Raft snapshot from object storage", "view": "data-flow", "pattern": "reactive-pull", "direction": "LR", "groups"…

Stage 3 — When to Use

• • High‑frequency writes where the log can outgrow memory in a few minutes.
• • Cost‑sensitive environments where you want to keep the function runtime short.
• • Regulatory compliance requiring state to be recoverable after a crash or function termination.
• By treating snapshots as first‑class citizens in your serverless Raft design, you keep state machine size under control and ensure that cold starts remain a brief, predictable event rather than a cascading delay.

Observable outcome: By treating snapshots as first‑class citizens in your serverless Raft design, you keep state machine size under control and ensure that cold starts remain a brief, predictable event rather than a cascading delay.

End-to-end outcome

Serverless functions (e.g., AWS Lambda, Azure Functions) terminate after a short run.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.

• Persist Snapshots – After every N log entries or at a regular interval, serialize the state machine’s current state and the last committed index.
• { "version": 1, "title": "Serverless Raft Snapshot Flow", "description": "How a stateless function loads a Raft snapshot from object storage", "view": "data-flow", "pattern": "reactive-pull", "direction": "LR", "groups"…
• • High‑frequency writes where the log can outgrow memory in a few minutes.