Integration Architecture Patterns: Orchestration, Choreography, Schema Contracts, and Idempotent Receivers

Scenario context

Integration failures usually come from weak contracts, unsafe retries, and missing ownership rather than from choosing the wrong transport. Orchestration, choreography, schema contracts, and idempotent receivers are patterns for making cross-boundary behavior explicit.

TLDR: Integration architecture succeeds when boundaries are explicit: contract first, idempotency by default, bounded retries, and clear workf…

This walkthrough focuses on Real-World Applications: Realistic Scenario: Payment Callback and ERP Sync.

Before making a design choice, clarify:

• Stripe: Idempotency Keys and Bounded Retries Stripe delivers 200M+ webhook events/day with at-least-once guarantees.
• After the 2016 amplification incident, they designed retry logic with true exponential backoff (1s → 2s → 4s → ...
• up to 3 days), a hard cap of 25 retries , and a mandatory Idempotency-Key header.

Concepts in play: event-driven-architecture, graph-traversal-algorithms, circuit-breaker, database-indexing

Stage 1 — When to Use Orchestration, Choreography, Contracts, and Idempotent Receivers

• Quick decision rule - External partner callbacks: idempotency + contract checks first.
• • Ordered business process: orchestration first.
• • Broad internal event mesh: choreography + strict contract governance.
• 📊 Orchestration: Central Coordinator This sequence diagram illustrates the orchestration pattern: a central Orchestrator drives the workflow by calling ServiceA, ServiceB, and ServiceC in sequence, passing results down…

sequenceDiagram
    participant O as Orchestrator
    participant A as ServiceA
    participant B as ServiceB
    participant C as ServiceC
    O->>A: Call ServiceA
    A-->>O: Result A
    O->>B: Call ServiceB (with A result)
    B-->>O: Result B
    O->>C: Call ServiceC (with B result)
    C-->>O: Result C
    Note over O: Aggregates all results

Observable outcome: 📊 Orchestration: Central Coordinator This sequence diagram illustrates the orchestration pattern: a central Orchestrator drives the workflow by calling ServiceA, ServiceB, and ServiceC in sequence, passing results down…

Stage 2 — How Reliable Integration Works End-to-End

• Ingress validates signature/auth and schema version.
• System checks idempotency key before side effects.
• Workflow is orchestrated or evented according to ownership model.
• Retries run with backoff and cap.

Observable outcome: Retries run with backoff and cap.

Stage 3 — How to Implement: Integration Hardening Checklist

• Define canonical envelope fields and contract versioning policy.
• Add schema validation at ingress (before business logic).
• Implement idempotency store with retention policy.
• Configure exponential backoff with max attempts.

Observable outcome: Configure exponential backoff with max attempts.

Stage 4 — Deep Dive: Contract Drift and Duplicate Safety Internals

• The Internals: Dedupe Keys, Ownership, and Replay Semantics Idempotency is stateful.
• The receiver must store processed IDs durably and define key scope clearly (global message ID vs partner+operation+business key).
• Replay semantics should be explicit: - replay for recovery, - replay for backfill, - replay for audit.
• Each mode may have different side-effect rules.

Observable outcome: Each mode may have different side-effect rules.

Stage 5 — Integration Flow: Safe External Callback Boundary

• This flowchart traces the safe ingestion path for an external webhook or API call: authentication and signature verification at the gateway, schema and version validation, then an idempotency lookup that short-circuits…
• New events are persisted as canonical records before being forwarded to the orchestrator or event bus, and failures trigger retry-with-backoff then dead-letter routing.
• The takeaway is that every external boundary crossing requires layered validation — no partner payload should reach business logic without passing all three checks.

flowchart TD
  A[Partner webhook or API call] --> B[Gateway auth and signature check]
  B --> C[Schema and version validator]
  C --> D[Idempotency lookup]
  D --> E{Already processed?}
  E -->|Yes| F[Return success idempotently]
  E -->|No| G[Persist canonical event]
  G --> H[Orchestrator or event bus]
  H --> I[Consumer side effects]
  I --> J{Failure?}
  J -->|Yes| K[Retry with backoff then DLQ]
  J -->|No| L[Emit completion with correlation ID]

Observable outcome: The takeaway is that every external boundary crossing requires layered validation — no partner payload should reach business logic without passing all three checks.

Stage 6 — Practical Example: Minimal Idempotent Receiver Contract

• Python: Idempotent Webhook Receiver with Retry and Dead-Letter Handling Required envelope fields (Slack/Stripe production pattern): - event id — stable producer-assigned key; never reuse across distinct events - event t…
• Validate contract before side effects.
• Persist dedupe key transactionally with the business outcome.
• Return idempotent success for duplicates — same response, no work done.

import asyncio, logging
from typing import Optional

logger = logging.getLogger(__name__)

# ── Idempotency store (Postgres-backed, 72-hour retention) ───────────────────
async def is_already_processed(conn, source: str, event_id: str) -> bool:
    row = await conn.fetchrow(
        "SELECT 1 FROM processed_events "
        "WHERE source=$1 AND event_id=$2 AND processed_at > NOW() - INTERVAL '72 hours'",
        source, event_id
    )
    return row is not None

async def mark_processed(conn, source: str, event_id: str, result: dict):
    await conn.execute(
        "INSERT INTO processed_events (source, event_id, result, processed_at) "
        "VALUES ($1, $2, $3, NOW()) ON CONFLICT DO NOTHING",
        source, event_id, str(result)
    )

# ── Idempotent handler with exponential backoff + DLQ routing ────────────────
async def handle_webhook(event: dict, conn, dlq_publish) -> dict:
    source   = event.get("source", "unknown")
    event_id = event.get("event_id")
    corr_id  = event.get("correlation_id")

    if not event_id:
        raise ValueError("event_id required — cannot guarantee idempotency")

    if await is_already_processed(conn, source, event_id):
        logger.info(f"Duplicate skipped: {source}/{event_id} [{corr_id}]")
        return {"status": "duplicate_skipped", "event_id": event_id}

    max_attempts = 3
    for attempt in range(max_attempts):
        try:
            result = await apply_business_logic(event)
            await mark_processed(conn, source, event_id, result)
            return {"status": "processed", "event_id": event_id}
        except Exception as exc:
            backoff = 2 ** attempt   # 1s → 2s → 4s (Stripe retry pattern)
            if attempt < max_attempts - 1:
                logger.warning(f"Retry {attempt+1}/{max_attempts} in {backoff}s: {exc}")
                await asyncio.sleep(backoff)
            else:
                logger.error(f"Exhausted retries, routing to DLQ: {event_id}")
                await dlq_publish(event, error=str(exc), correlation_id=corr_id)
                return {"status": "dlq_routed", "event_id": event_id}

async def apply_business_logic(event: dict) -> dict:
    return {"processed": True, "type": event.get("event_type")}

Observable outcome: Return idempotent success for duplicates — same response, no work done.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Protocol choice matters less than contract and replay discipline.
• • Orchestration and choreography are complementary, not mutually exclusive.

Alternatives considered

Decision Guide: Choosing Integration Style Quickly

If replay behavior is undefined, pause architecture changes until that is clarified.

Failure and degraded modes

Why Integration Failures Are Usually Boundary Failures

• Detection: Most integration incidents are not protocol failures.
• User or operator impact: Most integration incidents are not protocol failures.

Trade-offs & Failure Modes: Pros, Cons, and Risks

• Detection: 📊 Choreography: Event-Driven Flow This sequence diagram contrasts with orchestration by showing a choreography model: ServiceA publishes EventA to an EventBus, ServiceB reacts autonomously and publi…
• User or operator impact: 📊 Choreography: Event-Driven Flow This sequence diagram contrasts with orchestration by showing a choreography model: ServiceA publishes EventA to an EventBus, ServiceB reacts autonomously and publi…