The 8 Fallacies of Distributed Systems

Scenario context

The first time I heard "the network is reliable", I nodded. The second time was when our payment service was retrying duplicate charges because the network wasn't reliable and we hadn't handled it. Orders were double-billed, customers called support, and the root cause traced back to a single missing idempotency check — a check that would have been obvious if we had taken this fallacy seriously from day one. These 8…

This walkthrough focuses on The Eight Assumptions That Will Break Your System.

Before making a design choice, clarify:

• The first time I heard "the network is reliable", I nodded.
• The second time was when our payment service was retrying duplicate charges because the network wasn't reliable and we hadn't handled it.
• Orders were double-billed, customers called support, and the root cause traced back to a single missing idempotency check — a check that would have been obvious if we had taken this fallacy seriously from day one.

Concepts in play: consistency-replication, circuit-breaker, system-design-hld-architectures, caching

Stage 1 — What Makes a System "Distributed" in the First Place

• Before the fallacies can bite you, you need to know what kind of system exposes you to them.
• A distributed system is any application where components run on separate machines and communicate over a network.
• That sounds obvious — until you realise it covers microservices, mobile apps talking to a backend, two data centre regions sharing a database, and even a monolith that makes third-party API calls.
• The key insight: a network call is fundamentally different from a function call.

Observable outcome: The key insight: a network call is fundamentally different from a function call.

Stage 2 — Network Fallacies 1–4: Reliability, Latency, Bandwidth, and Security

• Fallacy 1: The network is reliable.
• A single function call that succeeds 99.9% of the time fails once every 1,000 requests, and distributed systems call each other thousands of times per second.
• Design response: Retry with exponential backoff.
• Use circuit breakers (Hystrix, Resilience4j).

Observable outcome: Use circuit breakers (Hystrix, Resilience4j).

Stage 3 — Infrastructure Fallacies 5–8: Topology, Administration, Cost, and Compatibility

• Fallacy 5: Topology doesn't change.
• Auto-scaling adds and removes instances.
• Hardcoding IP addresses breaks as soon as a node is replaced.
• Design response: Use service discovery (Consul, Kubernetes DNS, AWS Cloud Map).

Observable outcome: Design response: Use service discovery (Consul, Kubernetes DNS, AWS Cloud Map).

Stage 4 — From False Assumption to Production Outage

• Every fallacy follows the same lifecycle: the assumption goes unchallenged in local development, test environments mask the problem, and then production traffic amplifies the gap until something fails.
• The painful part: teams almost always reach R — but only after the outage forces their hand.
• The goal of knowing these fallacies is to reach R before the first incident, not after.

flowchart TD
    F[False Assumption baked into code] --> D[No defensive code written]
    D --> T[Tests pass on localhost / staging]
    T --> P[Hits real production traffic]
    P --> O[Outage or silent data corruption]
    O --> M[Team adds retry / circuit breaker / mTLS]
    M --> R[Resilient system]

Observable outcome: The goal of knowing these fallacies is to reach R before the first incident, not after.

Stage 5 — Real-World Applications: Where Each Fallacy Bites in Production

• Abstract warnings become concrete when mapped to documented production failures.
• A new service failing in all eight ways simultaneously is rare.
• A new service failing in two or three of them in the first quarter after launch is extremely common.

Observable outcome: A new service failing in two or three of them in the first quarter after launch is extremely common.

Stage 6 — A Pre-Flight Checklist for Every New Service Call

• Use this at design-review time to catch fallacy violations before they reach production.
• Reliability and latency (Fallacies 1 & 2) - [ ] Timeout configured explicitly (never rely on OS defaults) - [ ] Retry with exponential backoff and random jitter - [ ] Circuit breaker so a failing downstream cannot exhau…

Observable outcome: Reliability and latency (Fallacies 1 & 2) - [ ] Timeout configured explicitly (never rely on OS defaults) - [ ] Retry with exponential backoff and random jitter - [ ] Circuit breaker so a failing downstream cannot exhau…

End-to-end outcome

The first time I heard "the network is reliable", I nodded.

Alternatives considered

Decision Guide: Choosing Your Defenses Against the 8 Fallacies

Map each fallacy to a specific architectural defense.

Failure and degraded modes

Deep Dive: The Practical Antidote for Distributed Failures

• Detection: The minimal production checklist for every service call: - Timeout set (never rely on OS default) - Retry with exponential backoff + jitter - Circuit breaker to stop cascade - Bulkhead (limit concurr…
• User or operator impact: The minimal production checklist for every service call: - Timeout set (never rely on OS default) - Retry with exponential backoff + jitter - Circuit breaker to stop cascade - Bulkhead (limit concurr…

Trade-offs & Failure Modes: Why These Fallacies Persist

• Detection: These fallacies are taught in university — and still violated in code every week because: - Local development masks network problems (everything runs on localhost) - Unit tests don't simulate network…
• User or operator impact: These fallacies are taught in university — and still violated in code every week because: - Local development masks network problems (everything runs on localhost) - Unit tests don't simulate network…