Designing for High Availability: The Road to 99.99% Reliability

Scenario context

High Availability (HA) is the art of eliminating Single Points of Failure (SPOFs). By using Active-Active redundancy, automated health checks, and global failover via GSLB, you can achieve "Four Nines" (99.99%) reliability—limiting downtime to just 52 minutes per year.

Imagine you are the Lead Architect for a global payment processor. It’s Black Friday, and traffic is peaking. Your system is distributed across 20 n…

This walkthrough focuses on The "Everything Fails" Problem.

Before making a design choice, clarify:

• Imagine you are the Lead Architect for a global payment processor.
• It’s Black Friday, and traffic is peaking.
• Your system is distributed across 20 nodes, and your database is a robust, sharded SQL cluster.

Concepts in play: api-gateway, load-balancing, consistency-replication, system-design-hld-architectures

Stage 1 — The Basics of Redundancy and Health

• The foundation of HA is simple: Never have just one of anything.
• If you have one load balancer, one database, or one region, you have a SPOF.
• To move beyond SPOFs, we use Redundancy .
• This isn't just about having "more servers"; it's about how those servers interact.

Observable outcome: This isn't just about having "more servers"; it's about how those servers interact.

Stage 2 — How Automated Failover Keeps the Lights On

• The core mechanic of HA is the Failover Loop .
• This is a continuous cycle of monitoring, detection, and reconfiguration.
• Heartbeating: Nodes constantly send small "I am alive" packets to a controller or to each other.
• Detection: If a heartbeat is missed $X$ times, the node is marked as "Unhealthy." 3.

Observable outcome: Detection: If a heartbeat is missed $X$ times, the node is marked as "Unhealthy." 3.

Stage 3 — Visualizing the Flow of Automated Failover

• Explanation of the Diagram: The diagram shows a standard horizontal failover flow.
• The Load Balancer acts as the central traffic director.
• It performs regular health checks on all active nodes.
• When Server 1 fails to respond (detected via a timeout or heartbeat miss), the Load Balancer immediately marks it as unhealthy and reroutes 100% of the traffic to Server 2.

graph TD
    User((User)) --> LB[Load Balancer]
    LB -->|Health Check: OK| S1[Server 1: Active]
    LB -->|Health Check: OK| S2[Server 2: Active]

    subgraph Failure_Scenario
        S1 -.->|Crash| S1_Down((X))
        LB -->|Timeout| S1_Down
        LB -->|Reroute| S2
    end

Observable outcome: When Server 1 fails to respond (detected via a timeout or heartbeat miss), the Load Balancer immediately marks it as unhealthy and reroutes 100% of the traffic to Server 2.

Stage 4 — Deep Dive: The Internals of High Availability

• Achieving 99.99% requires looking under the hood at how traffic actually moves across the wire during a failure.
• 🛡️ The Internals: Virtual IPs and VRRP How does a Load Balancer fail over if it is the one that crashes?
• Two physical load balancers share a single IP address using a protocol called VRRP (Virtual Router Redundancy Protocol) .
• Both nodes listen, but only the "Master" responds to ARP requests for that IP.

Observable outcome: Both nodes listen, but only the "Master" responds to ARP requests for that IP.

Stage 5 — Advanced Concepts: Global Traffic Management (GSLB)

• To survive an entire region going dark, you need Global Server Load Balancing (GSLB) .
• This happens at the DNS level.
• When a user in London requests api.example.com , a GSLB-enabled DNS server (like Cloudflare or Route53) looks at the health of your data centres.
• If the London region is at 90% capacity or reporting errors, the DNS server returns the IP address for the Dublin region instead.

Observable outcome: If the London region is at 90% capacity or reporting errors, the DNS server returns the IP address for the Dublin region instead.

Stage 6 — Real-World Applications: Designing for 99.99%

• Case Study 1: The Payment Gateway (Active-Passive DB) A payment gateway uses a primary SQL database in us-east-1 and a synchronous replica in us-west-2 .
• • Process: Write to Primary → Sync to Replica → Commit.
• • Failure Scenario: us-east-1 go down.
• • HA Logic: The monitoring system detects the outage, promotes the us-west-2 replica to Primary, and updates the application connection strings.

Observable outcome: - HA Logic: The monitoring system detects the outage, promotes the us-west-2 replica to Primary, and updates the application connection strings.

End-to-end outcome

After working through the stages, you should be able to explain:

• Hardware will fail, so build software that doesn't care.
• Assume every disk, cable, and power supply has a timer counting down to zero.
• • Eliminate SPOFs: Redundancy at every layer (DNS, LB, App, DB).
• • Automate Detection: Use health checks with aggressive but safe timeouts.

Alternatives considered

Decision Guide: Choosing your HA Strategy

Decision Guide: Choosing your HA Strategy

Failure and degraded modes

Trade-offs & Failure Modes

• Detection: Cost: Running an Active-Active multi-region setup doubles your cloud bill.
• User or operator impact: Cost: Running an Active-Active multi-region setup doubles your cloud bill.