An invoice-OCR and billing-settlement service reads jobs from an SQS Standard queue (`invoice-jobs`) configured with the default `VisibilityTimeout = 30s`. Normal jobs finish in 8–12 seconds, but multi-page PDFs take 48s at p95 and up to 110s at p99. Under peak load (400 jobs/sec), 14% of jobs are picked up by a second worker at t=30s while Worker A is still running OCR and calling the payment gateway. This causes duplicate gateway charges, redundant GPU OCR runs, and `ReceiptHandleIsInvalid` / stale-delete errors when Worker A finishes at t=75s. Design the visibility-timeout lifecycle, heartbeat extension mechanism, and idempotency layer to eliminate duplicate side effects without delaying crash recovery for dead workers.
From patterns to decisions.
An invoice-OCR and billing-settlement service reads jobs from an SQS Standard queue (`invoice-jobs`) configured with the default `VisibilityTimeout = 30s`. Normal jobs finish in 8–12 seconds, but multi-page PDFs take 48s at p95 and up to 110s at p99. Under peak load (400 jobs/sec), 14% of jobs are picked up by a second worker at t=30s while Worker A is still running OCR and calling the payment gateway. This causes duplicate gateway charges, redundant GPU OCR runs, and `ReceiptHandleIsInvalid` / stale-delete errors when Worker A finishes at t=75s. Design the visibility-timeout lifecycle, heartbeat extension mechanism, and idempotency layer to eliminate duplicate side effects without delaying crash recovery for dead workers.
A static 900s visibility timeout.
A static 900s visibility timeout delays crash recovery to 15 minutes whenever a Spot worker is terminated mid-job, and still does not protect against SQS Standard duplicate deliveries.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Heartbeats require a background thread/task per in-flight message, whereas a static timeout is zero-code. |
Step 1
Note that 85% of jobs finish in 10s, so a 900s timeout penalizes normal Spot interruption recovery by 90x.
Step 2
Explain that Spot instances receive SIGKILL/termination frequently; a 45s timeout + heartbeat bounds recovery to 45s.
Step 3
Emphasize that SQS Standard can still deliver duplicates on network retries, requiring idempotency regardless of timeout.
Alternative 1
Route known large PDFs (>10 pages) to a separate invoice-jobs-large queue with VisibilityTimeout=180s.
Failure Mode 1
If HEARTBEAT_INTERVAL_SEC is too close to INITIAL_VISIBILITY_SEC (e.g., 40s vs 45s), a single 6s network blip causes duplicate delivery.
# Calculating initial visibility vs heartbeat interval
INITIAL_VISIBILITY_SEC = 45 # fast dead-worker recovery
HEARTBEAT_INTERVAL_SEC = 20 # < INITIAL_VISIBILITY_SEC / 2
MAX_JOB_DURATION_SEC = 900 # hard cap to prevent infinite extension
Use a threading.
Use a threading.Event to signal heartbeat shutdown in a finally block, capped at max_duration_sec, paired with a DynamoDB conditional PutItem.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using a daemon thread per job is simple for I/O-bound workers; for 50+ concurrent jobs per process, a single shared `ScheduledExecutorService` / timer wheel is more memory-efficient. |
Step 1
Acquire an idempotency lease in DynamoDB before starting expensive OCR.
Step 2
Start a daemon threading.Thread that waits on stop_event.wait(20) and calls change_message_visibility.
Step 3
On completion, mark DynamoDB COMPLETED, set stop_event.set(), and call delete_message.
Alternative 1
Asyncio background task (asyncio.create_task) if using aioboto3.
Failure Mode 1
Swallowing non-ReceiptHandleIsInvalid errors silently without logging can mask IAM permission errors (sqs:ChangeMessageVisibility).
import threading
import time
import boto3
sqs = boto3.client("sqs")
class VisibilityHeartbeat:
def __init__(self, queue_url: str, receipt_handle: str, extend_sec=45, interval_sec=20, max_sec=900):
self.queue_url = queue_url
self.receipt_handle = receipt_handle
self.extend_sec = extend_sec
self.interval_sec = interval_sec
self.max_sec = max_sec
self._stop = threading.Event()
self._thread = threading.Thread(target=self._run, daemon=True)
def _run(self):
start = time.monotonic()
while not self._stop.wait(self.interval_sec):
if time.monotonic() - start >= self.max_sec:
break
try:
sqs.change_message_visibility(
QueueUrl=self.queue_url,
ReceiptHandle=self.receipt_handle,
VisibilityTimeout=self.extend_sec,
)
except sqs.exceptions.ReceiptHandleIsInvalid:
break
def __enter__(self):
self._thread.start()
return self
def __exit__(self, exc_type, exc_val, exc_tb):
self._stop.set()
self._thread.join(timeout=2)
Implementation & Guardrails (3).
Treat ConditionalCheckFailedException as a signal that another worker owns an active lease or already completed the invoice; never force-overwrite it.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Using `lease_expires_at` requires bounded clock skew between workers (use NTP/Amazon Time Sync Service). |
Step 1
Read the existing DynamoDB item to check status.
Step 2
If status == COMPLETED, delete the SQS message immediately and return.
Step 3
If status == IN_PROGRESS and lease_expires_at > now, abort processing and let SQS visibility expire naturally (or extend slightly) without charging.
Alternative 1
PostgreSQL INSERT ... ON CONFLICT DO NOTHING for teams already running RDS.
Failure Mode 1
If lease_sec (60s) is shorter than actual OCR time without renewing the DynamoDB lease alongside SQS visibility, a second worker could steal the DynamoDB lock.
from botocore.exceptions import ClientError
def claim_invoice_lease(ddb_table, invoice_id: str, now_epoch: int, lease_sec: int = 60) -> str:
try:
ddb_table.put_item(
Item={"invoice_id": invoice_id, "status": "IN_PROGRESS", "lease_expires_at": now_epoch + lease_sec},
ConditionExpression="attribute_not_exists(invoice_id) OR (#s = :prog AND lease_expires_at < :now)",
ExpressionAttributeNames={"#s": "status"},
ExpressionAttributeValues={":prog": "IN_PROGRESS", ":now": now_epoch},
)
return "CLAIMED"
except ClientError as e:
if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
item = ddb_table.get_item(Key={"invoice_id": invoice_id}).get("Item", {})
return "ALREADY_COMPLETED" if item.get("status") == "COMPLETED" else "LOCKED_BY_OTHER"
raise
Implementation & Guardrails (4).
Defense-in-depth: conditional transition to COMPLETED in DynamoDB + Idempotency-Key at the payment gateway prevents double charges even if two workers run OCR concurrently.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Wasted GPU OCR compute on the rare 0.01% GC-pause overlap, but 0% risk of duplicate financial charges. |
Both workers pass `Idempotency-Key
invoice_{invoice_id}` to Stripe/Adyen, so the gateway executes at most one charge.
Step 2
Before returning success, Worker A attempts a conditional UpdateItem requiring status = IN_PROGRESS and its own worker_token.
Step 3
If Worker B already claimed and completed it, Worker A's conditional write fails and Worker A skips DeleteMessage.
Alternative 1
Fencing token passed to an internal settlement service that rejects lower tokens.
Failure Mode 1
Generating a random UUID per attempt for Idempotency-Key instead of deterministic invoiceId breaks deduplication.
// Passing deterministic Idempotency-Key to payment gateway
HttpRequest req = HttpRequest.newBuilder()
.uri(URI.create("https://api.payments.internal/v1/charges"))
.header("Idempotency-Key", "invoice-" + invoiceId)
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(payload))
.build();
Implementation & Guardrails (5).
No. For jobs > 15 minutes, decouple the SQS message from execution duration using AWS Step Functions (waitForTaskToken) or an async job table + ECS/Batch.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Step Functions adds per-state-transition cost, but cleanly separates queueing from multi-hour compute. |
Step 1
Receive the SQS message, persist a PENDING job record in DynamoDB/RDS, submit an AWS Batch/ECS task, and delete the SQS message immediately.
Step 2
Track long-running task liveness via ECS task state events or Step Functions heartbeats instead of holding SQS visibility locks for hours.
Alternative 1
Split 500-page PDFs into 10-page chunks so each SQS job stays under 30 seconds.
Failure Mode 1
Deleting the SQS message before durably persisting the job record risks silent job loss on crash.
# Step Functions Task Token pattern for >15m jobs
RunLongOcrJob:
Type: Task
Resource: arn:aws:states:::ecs:runTask.waitForTaskToken
TimeoutSeconds: 5400
HeartbeatSeconds: 120
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (
ChangeMessageVisibilityeveryT/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects. - 02Trade-off tip: Using
lease_expires_atrequires bounded clock skew between workers (use NTP/Amazon Time Sync Service). - 03Watch out for: Generating a random UUID per attempt for
Idempotency-Keyinstead of deterministicinvoiceIdbreaks deduplication.
Sqs Visibility Timeout And Idempotency — Normal → Load / Fault → Recovery
Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (ChangeMessageVisibility every T/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.
Putting Calibrating SQS Visibility Timeouts & Heartbeats Together
Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (`ChangeMessageVisibility` every `T/2`) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.
Putting Calibrating SQS Visibility Timeouts & Heartbeats Together
- Acquire an idempotency lease in DynamoDB before starting expensive OCR.
- Start a daemon
threading.Threadthat waits onstop_event.wait(20)and callschange_message_visibility. - On completion, mark DynamoDB
COMPLETED, setstop_event.set(), and calldelete_message.
Interview Readiness Check
Defense-in-depth: conditional transition to COMPLETED in DynamoDB + Idempotency-Key at the payment gateway prevents double charges even if two workers run OCR concurrently.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Calibrating SQS Visibility Timeouts & Heartbeats
Because SQS guarantees at-least-once delivery, pair a calibrated visibility heartbeat (ChangeMessageVisibility every T/2) with a strict idempotent consumer store so neither long jobs nor worker crashes cause duplicate side effects.
