Dead-Letter Queue (DLQ) Silent Poisoning & Explosive Disk Accumulation

Hardened Implementation Steps

1. 1Ingress Boundary Defense: Strict SLA Set strict Prometheus alerts on DLQ queue depth (>0 for >5 minutes), implement automated redrive tools, and enforce maximum retention TTLs
2. 2Failure Containment: Eliminate unconstrained buffering and propagate physical backpressure.
3. 3Invariant Enforcement: Every Dead-Letter Queue must have high-severity alerts on queue depth; DLQs are diagnostic holding pens, not black holes.

Production Code Contrast & Implementation

// HARDENED: Full Jitter exponential backoff with retry budget
public Response callService(Request req) {
    if (!retryBudget.acquireToken()) {
        throw new RetryBudgetExhaustedException("Shedding retries under cluster distress");
    }
    // Full Jitter: Uniform random backoff breaks synchronized wave patterns
    long sleepMs = (long) (Math.random() * Math.min(maxBackoffMs, baseMs * Math.pow(2, attempt)));
    scheduledExecutor.schedule(() -> executeCall(req), sleepMs, TimeUnit.MILLISECONDS);
}

Alternatives Evaluated

• →Naive Approach: A bug causes 10% of production traffic to fail validation and divert to the DLQ. Nobody is monitoring the DLQ. Millions of messages accumulate over 3
• →Production-Hardened Pattern: Strict SLA Set strict Prometheus alerts on DLQ queue depth (>0 for >5 minutes), implement automated redrive tools, and enforce maximum retention TTLs

Extension Implementation Steps

1. 1Track real-time latency gradients and inflight concurrency.
2. 2Reject excess traffic at the ingress boundary within <0.1ms.
3. 3Monitor P99 latency and verify autonomous recovery when downstream pressure subsides.

Follow-Up Trade-Offs

• ⚖Architectural Invariant: "Every Dead-Letter Queue must have high-severity alerts on queue depth; DLQs are diagnostic holding pens, not black holes."
• ⚖Physical Resource Boundary: Dynamic admission control ensures worker threads never buffer dead requests during downstream brownouts.