Skip to main content
← PROBLEMS / ARCHITECTURE
An end-to-end domain learning journey

Calibrating SQS Visibility
Timeouts & Heartbeats.

An invoice-OCR and billing-settlement service reads jobs from an SQS Standard queue (invoice-jobs) configured with the default VisibilityTimeout = 30s. Normal jobs finish in 8–12 seconds, but multi-page PDFs take 48s at p95 and up to 110s at p99. Under peak load (400 jobs/sec), 14% of jobs are picked up by a second worker at t=30s while Worker A is still running OCR and calling the payment gateway. This causes duplicate gateway charges, redundant GPU OCR runs, and ReceiptHandleIsInvalid / stale-delete errors when Worker A finishes at t=75s. Design the visibility-timeout lifecycle, heartbeat extension mechanism, and idempotency layer to eliminate duplicate side effects without delaying crash recovery for dead workers.
Start the journey5 stages
Loading diagram…
Calibrating SQS Visibility Timeouts & Heartbeats architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

An invoice-OCR and billing-settlement service reads jobs from an SQS Standard queue (`invoice-jobs`) configured with the default `VisibilityTimeout = 30s`. Normal jobs finish in 8–12 seconds, but multi-page PDFs take 48s at p95 and up to 110s at p99. Under peak load (400 jobs/sec), 14% of jobs are picked up by a second worker at t=30s while Worker A is still running OCR and calling the payment gateway. This causes duplicate gateway charges, redundant GPU OCR runs, and `ReceiptHandleIsInvalid` / stale-delete errors when Worker A finishes at t=75s. Design the visibility-timeout lifecycle, heartbeat extension mechanism, and idempotency layer to eliminate duplicate side effects without delaying crash recovery for dead workers.

From patterns to decisions.

An invoice-OCR and billing-settlement service reads jobs from an SQS Standard queue (`invoice-jobs`) configured with the default `VisibilityTimeout = 30s`. Normal jobs finish in 8–12 seconds, but multi-page PDFs take 48s at p95 and up to 110s at p99. Under peak load (400 jobs/sec), 14% of jobs are picked up by a second worker at t=30s while Worker A is still running OCR and calling the payment gateway. This causes duplicate gateway charges, redundant GPU OCR runs, and `ReceiptHandleIsInvalid` / stale-delete errors when Worker A finishes at t=75s. Design the visibility-timeout lifecycle, heartbeat extension mechanism, and idempotency layer to eliminate duplicate side effects without delaying crash recovery for dead workers.

01
ARCHITECTURE · STAGE 01L4 · FUNDAMENTALS

A static 900s visibility timeout.

A static 900s visibility timeout delays crash recovery to 15 minutes whenever a Spot worker is terminated mid-job, and still does not protect against SQS Standard duplicate deliveries.

ChoiceWhat it buysWhat it costs
Trade-off 1Heartbeats require a background thread/task per in-flight message, whereas a static timeout is zero-code.

Step 1

Note that 85% of jobs finish in 10s, so a 900s timeout penalizes normal Spot interruption recovery by 90x.

Step 2

Explain that Spot instances receive SIGKILL/termination frequently; a 45s timeout + heartbeat bounds recovery to 45s.

Step 3

Emphasize that SQS Standard can still deliver duplicates on network retries, requiring idempotency regardless of timeout.

Alternative 1

Route known large PDFs (>10 pages) to a separate invoice-jobs-large queue with VisibilityTimeout=180s.

Failure Mode 1

If HEARTBEAT_INTERVAL_SEC is too close to INITIAL_VISIBILITY_SEC (e.g., 40s vs 45s), a single 6s network blip causes duplicate delivery.

# Calculating initial visibility vs heartbeat interval
INITIAL_VISIBILITY_SEC = 45   # fast dead-worker recovery
HEARTBEAT_INTERVAL_SEC = 20   # < INITIAL_VISIBILITY_SEC / 2
MAX_JOB_DURATION_SEC = 900    # hard cap to prevent infinite extension
02
ARCHITECTURE · STAGE 02L5 · APPLIED DESIGN

Use a threading.

Use a threading.Event to signal heartbeat shutdown in a finally block, capped at max_duration_sec, paired with a DynamoDB conditional PutItem.

ChoiceWhat it buysWhat it costs
Trade-off 1Using a daemon thread per job is simple for I/O-bound workers; for 50+ concurrent jobs per process, a single shared `ScheduledExecutorService` / timer wheel is more memory-efficient.

Step 1

Acquire an idempotency lease in DynamoDB before starting expensive OCR.

Step 2

Start a daemon threading.Thread that waits on stop_event.wait(20) and calls change_message_visibility.

Step 3

On completion, mark DynamoDB COMPLETED, set stop_event.set(), and call delete_message.

Alternative 1

Asyncio background task (asyncio.create_task) if using aioboto3.

Failure Mode 1

Swallowing non-ReceiptHandleIsInvalid errors silently without logging can mask IAM permission errors (sqs:ChangeMessageVisibility).

import threading
import time
import boto3

sqs = boto3.client("sqs")

class VisibilityHeartbeat:
    def __init__(self, queue_url: str, receipt_handle: str, extend_sec=45, interval_sec=20, max_sec=900):
        self.queue_url = queue_url
        self.receipt_handle = receipt_handle
        self.extend_sec = extend_sec
        self.interval_sec = interval_sec
        self.max_sec = max_sec
        self._stop = threading.Event()
        self._thread = threading.Thread(target=self._run, daemon=True)

    def _run(self):
        start = time.monotonic()
        while not self._stop.wait(self.interval_sec):
            if time.monotonic() - start >= self.max_sec:
                break
            try:
                sqs.change_message_visibility(
                    QueueUrl=self.queue_url,
                    ReceiptHandle=self.receipt_handle,
                    VisibilityTimeout=self.extend_sec,
                )
            except sqs.exceptions.ReceiptHandleIsInvalid:
                break

    def __enter__(self):
        self._thread.start()
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self._stop.set()
        self._thread.join(timeout=2)
03
ARCHITECTURE · STAGE 03L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (3).

Treat ConditionalCheckFailedException as a signal that another worker owns an active lease or already completed the invoice; never force-overwrite it.

ChoiceWhat it buysWhat it costs
Trade-off 1Using `lease_expires_at` requires bounded clock skew between workers (use NTP/Amazon Time Sync Service).

Step 1

Read the existing DynamoDB item to check status.

Step 2

If status == COMPLETED, delete the SQS message immediately and return.

Step 3

If status == IN_PROGRESS and lease_expires_at > now, abort processing and let SQS visibility expire naturally (or extend slightly) without charging.

Alternative 1

PostgreSQL INSERT ... ON CONFLICT DO NOTHING for teams already running RDS.

Failure Mode 1

If lease_sec (60s) is shorter than actual OCR time without renewing the DynamoDB lease alongside SQS visibility, a second worker could steal the DynamoDB lock.

from botocore.exceptions import ClientError

def claim_invoice_lease(ddb_table, invoice_id: str, now_epoch: int, lease_sec: int = 60) -> str:
    try:
        ddb_table.put_item(
            Item={"invoice_id": invoice_id, "status": "IN_PROGRESS", "lease_expires_at": now_epoch + lease_sec},
            ConditionExpression="attribute_not_exists(invoice_id) OR (#s = :prog AND lease_expires_at < :now)",
            ExpressionAttributeNames={"#s": "status"},
            ExpressionAttributeValues={":prog": "IN_PROGRESS", ":now": now_epoch},
        )
        return "CLAIMED"
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            item = ddb_table.get_item(Key={"invoice_id": invoice_id}).get("Item", {})
            return "ALREADY_COMPLETED" if item.get("status") == "COMPLETED" else "LOCKED_BY_OTHER"
        raise
04
ARCHITECTURE · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

Defense-in-depth: conditional transition to COMPLETED in DynamoDB + Idempotency-Key at the payment gateway prevents double charges even if two workers run OCR concurrently.

ChoiceWhat it buysWhat it costs
Trade-off 1Wasted GPU OCR compute on the rare 0.01% GC-pause overlap, but 0% risk of duplicate financial charges.

Both workers pass `Idempotency-Key

invoice_{invoice_id}` to Stripe/Adyen, so the gateway executes at most one charge.

Step 2

Before returning success, Worker A attempts a conditional UpdateItem requiring status = IN_PROGRESS and its own worker_token.

Step 3

If Worker B already claimed and completed it, Worker A's conditional write fails and Worker A skips DeleteMessage.

Alternative 1

Fencing token passed to an internal settlement service that rejects lower tokens.

Failure Mode 1

Generating a random UUID per attempt for Idempotency-Key instead of deterministic invoiceId breaks deduplication.

// Passing deterministic Idempotency-Key to payment gateway
HttpRequest req = HttpRequest.newBuilder()
    .uri(URI.create("https://api.payments.internal/v1/charges"))
    .header("Idempotency-Key", "invoice-" + invoiceId)
    .header("Content-Type", "application/json")
    .POST(HttpRequest.BodyPublishers.ofString(payload))
    .build();
05
ARCHITECTURE · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

No. For jobs > 15 minutes, decouple the SQS message from execution duration using AWS Step Functions (waitForTaskToken) or an async job table + ECS/Batch.

ChoiceWhat it buysWhat it costs
Trade-off 1Step Functions adds per-state-transition cost, but cleanly separates queueing from multi-hour compute.

Step 1

Receive the SQS message, persist a PENDING job record in DynamoDB/RDS, submit an AWS Batch/ECS task, and delete the SQS message immediately.

Step 2

Track long-running task liveness via ECS task state events or Step Functions heartbeats instead of holding SQS visibility locks for hours.

Alternative 1

Split 500-page PDFs into 10-page chunks so each SQS job stays under 30 seconds.

Failure Mode 1

Deleting the SQS message before durably persisting the job record risks silent job loss on crash.

# Step Functions Task Token pattern for >15m jobs
RunLongOcrJob:
  Type: Task
  Resource: arn:aws:states:::ecs:runTask.waitForTaskToken
  TimeoutSeconds: 5400
  HeartbeatSeconds: 120

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (ChangeMessageVisibility every T/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.
  • 02Trade-off tip: Using lease_expires_at requires bounded clock skew between workers (use NTP/Amazon Time Sync Service).
  • 03Watch out for: Generating a random UUID per attempt for Idempotency-Key instead of deterministic invoiceId breaks deduplication.
Behavioral Progression · Sqs Visibility Timeout And Idempotency

Sqs Visibility Timeout And Idempotency — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (ChangeMessageVisibility every T/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.

Synthesis / One connected design

Putting Calibrating SQS Visibility Timeouts & Heartbeats Together

Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (`ChangeMessageVisibility` every `T/2`) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.

A candidate architecture, not a universal answer

Putting Calibrating SQS Visibility Timeouts & Heartbeats Together

Step 1
Step 2
Step 3
  1. Acquire an idempotency lease in DynamoDB before starting expensive OCR.
  2. Start a daemon threading.Thread that waits on stop_event.wait(20) and calls change_message_visibility.
  3. On completion, mark DynamoDB COMPLETED, set stop_event.set(), and call delete_message.
Defense-in-depth: conditional transition to `COMPLETED` in DynamoDB + `Idempotency-Key` at the payment gateway prevents double charges even if two workers run OCR concurrently.
Interview Readiness Check

Defense-in-depth: conditional transition to COMPLETED in DynamoDB + Idempotency-Key at the payment gateway prevents double charges even if two workers run OCR concurrently.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Calibrating SQS Visibility Timeouts & Heartbeats

Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (ChangeMessageVisibility every T/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.