System Design Service Discovery and Health Checks: Routing Traffic to Healthy Instances

Scenario context

Service discovery is how clients find the right service instance at runtime, and health checks are how systems decide whether an instance should receive traffic. Together, they turn dynamic infrastructure from guesswork into deterministic routing.

TLDR: If you scale beyond static IPs, discovery plus health-aware routing becomes a core reliability primitive.

In small systems, service communication can start w…

This walkthrough focuses on Why Service Discovery Is the Invisible Backbone of Modern Systems.

Before making a design choice, clarify:

• In small systems, service communication can start with fixed hostnames and static configuration.
• That model breaks quickly once autoscaling, rolling deploys, and multi-zone failover enter the picture.
• In production, service instances come and go all day: - New instances launch during traffic spikes.

Concepts in play: api-gateway, database-indexing, caching, canary-deployments

Stage 1 — The Two Discovery Models You Must Distinguish in Interviews

• Service discovery usually appears in one of two patterns.
• Client-side discovery: the client queries a service registry and chooses a backend instance directly.
• This is common in microservice SDKs where clients include load-balancing logic.
• Server-side discovery: the client calls a stable endpoint (for example, a load balancer or API gateway), and that component resolves healthy backends.

Observable outcome: Server-side discovery: the client calls a stable endpoint (for example, a load balancer or API gateway), and that component resolves healthy backends.

Stage 2 — How Discovery and Health Checks Work End-to-End

• A robust discovery path is usually a loop, not a one-time lookup.
• Service instance starts and registers itself with metadata.
• Registry stores endpoint, zone, version, and status.
• Clients or proxies query for candidate instances.

Observable outcome: Clients or proxies query for candidate instances.

Stage 3 — Discovery Flow: Registration to Health-Aware Routing

• This model captures the key principle: discovery and health checks are continuous control loops, not setup-time configuration.
• 📊 Service Registration and Client Discovery This sequence diagram traces the full lifecycle of service registration and client-driven discovery.
• OrderService registers with the registry and sends heartbeats every 5 seconds to maintain its healthy status; when a Client queries for available endpoints, the registry returns only healthy instances, and the Client ro…
• Discovery is not a one-time lookup — it is a continuous health-maintained contract that ensures clients never route to stale or unhealthy backends.

flowchart TD
    A[Instance boots] --> B[Register with service registry]
    B --> C[Heartbeat and metadata updates]
    C --> D{Healthy and ready?}
    D -->|Yes| E[Add to routing pool]
    D -->|No| F[Exclude from routing pool]
    E --> G[Client or proxy resolves target]
    G --> H[Request served]
    F --> I[Recovery or restart]
    I --> C

sequenceDiagram
    participant S as OrderService
    participant R as Service Registry
    participant C as Client
    S->>R: Register: host port tags
    R-->>S: ACK registration
    S->>R: Heartbeat every 5s
    C->>R: Discover order-service
    R-->>C: Return healthy endpoints
    C->>S: Route request
    S-->>C: Response

Observable outcome: Discovery is not a one-time lookup — it is a continuous health-maintained contract that ensures clients never route to stale or unhealthy backends.

Stage 4 — Real-World Applications: API Gateways, Payments, and Internal Platforms

• HashiCorp Consul at scale: Consul's gossip protocol propagates health changes across a cluster in ~200ms on typical LAN deployments.
• The deregister critical service after field automatically removes services that remain unhealthy beyond a configurable window — preventing stale endpoints from silently accumulating in the registry.
• When a service registers with Consul, it provides its name, port, and metadata tags alongside a health check definition.
• The check specifies the HTTP path to poll, the polling interval, the per-check timeout, and — critically — a window after which Consul automatically deregisters a service that remains in a critical state.

Observable outcome: The check specifies the HTTP path to poll, the polling interval, the per-check timeout, and — critically — a window after which Consul automatically deregisters a service that remains in a critical state.

Stage 5 — Practical Example: Evolving a Checkout Service Beyond Static Backends

• Imagine a checkout service initially routed via hardcoded backend IPs.
• Problems appear during traffic spikes: - New app instances launch but receive no traffic.
• • One bad instance still receives requests for minutes.
• • Rolling deploys create intermittent errors from stale endpoint lists.

Observable outcome: - Rolling deploys create intermittent errors from stale endpoint lists.

Stage 6 — Dynamic Registration and Health-Aware Discovery in Production

• Spring Cloud Netflix Eureka is a client-side service registry built into the Spring Cloud ecosystem; Spring Cloud Consul provides the same programming model backed by HashiCorp Consul's gossip-based registry.
• Both integrate with a discovery client abstraction and a health indicator framework to expose liveness and readiness semantics to the control plane automatically.
• How it works in practice: A microservice opts into dynamic discovery by declaring itself as a discovery client at startup.
• From that point, registration and heartbeat renewal happen automatically — the developer does not manage registry calls manually.

Observable outcome: From that point, registration and heartbeat renewal happen automatically — the developer does not manage registry calls manually.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Service discovery is a control-plane capability, not just a DNS trick.
• • Health checks must distinguish process liveness from real request readiness.

Alternatives considered

Decision Guide: Choosing a Discovery Strategy

When unsure in interviews, start with server-side discovery for simpler client behavior, then discuss where client-side control may be worth the complexity.

Failure and degraded modes

The Internals: Registries, Heartbeats, TTLs, and Routing Metadata

• Detection: Most systems maintain a control plane with these pieces: - Registry store for service instances and metadata.
• User or operator impact: Most systems maintain a control plane with these pieces: - Registry store for service instances and metadata.

Performance Analysis: Lookup Latency, Convergence Time, and Flapping

• Detection: Discovery systems are often judged by three metrics.
• User or operator impact: Discovery systems are often judged by three metrics.

Trade-offs & Failure Modes: Where Discovery Can Go Wrong

• Detection: The interview-quality answer always includes one sentence like: "I would define clear health semantics and failure thresholds before tuning load-balancer algorithms."
• User or operator impact: The interview-quality answer always includes one sentence like: "I would define clear health semantics and failure thresholds before tuning load-balancer algorithms."