System Design Service Discovery and Health Checks: Routing Traffic to Healthy Instances

TLDR: Service discovery is how clients find the right service instance at runtime, and health checks are how systems decide whether an instance should receive traffic. Together, they turn dynamic infrastructure from guesswork into deterministic routing.

TLDR: If you scale beyond static IPs, discovery plus health-aware routing becomes a core reliability primitive.

📖 Why Service Discovery Is the Invisible Backbone of Modern Systems

In small systems, service communication can start with fixed hostnames and static configuration. That model breaks quickly once autoscaling, rolling deploys, and multi-zone failover enter the picture.

In production, service instances come and go all day:

• New instances launch during traffic spikes.
• Old instances terminate during scale-down.
• Deployments replace instances in waves.
• Network partitions make some endpoints temporarily unreachable.

If clients keep a stale list of backends, requests fail even when healthy capacity exists elsewhere. Service discovery solves this by making endpoint lookup dynamic and health-aware.

For interviews, this is a key signal: strong candidates explain that scaling services is not only about compute. It is also about continuously correct routing decisions.

🔍 The Two Discovery Models You Must Distinguish in Interviews

Service discovery usually appears in one of two patterns.

Client-side discovery: the client queries a service registry and chooses a backend instance directly. This is common in microservice SDKs where clients include load-balancing logic.

Server-side discovery: the client calls a stable endpoint (for example, a load balancer or API gateway), and that component resolves healthy backends.

Interview-friendly takeaway: neither model is universally better. The right choice depends on organizational maturity, traffic behavior, and operational ownership.

⚙️ How Discovery and Health Checks Work End-to-End

A robust discovery path is usually a loop, not a one-time lookup.

1. Service instance starts and registers itself with metadata.
2. Registry stores endpoint, zone, version, and status.
3. Clients or proxies query for candidate instances.
4. Health checks evaluate liveness/readiness continuously.
5. Unhealthy nodes are removed from traffic until recovery.

Health checks are often split into two types:

• Liveness check: is the process alive enough to restart decision logic?
• Readiness check: can this instance safely serve real traffic now?

A frequent production pitfall is using only liveness checks. That can keep a process alive but still route traffic to an instance that cannot serve real requests because dependencies are down.

🧠 Deep Dive: What Actually Makes Discovery Reliable Under Failure

The Internals: Registries, Heartbeats, TTLs, and Routing Metadata

Most systems maintain a control plane with these pieces:

• Registry store for service instances and metadata.
• Heartbeat protocol to refresh instance presence.
• TTL eviction logic to remove stale endpoints.
• Watch/stream mechanism to push updates to clients or proxies.

When an instance registers, it usually publishes metadata like zone, version, and tags (canary, stable, gpu). Routing layers can then enforce traffic policies, such as zone-affinity or canary rollout splits.

A practical sequence looks like this:

1. Instance sends heartbeat every N seconds.
2. Registry updates last_seen timestamp.
3. If heartbeat expires beyond TTL, endpoint is marked unhealthy.
4. Load balancer excludes endpoint from selection set.

This flow is simple but safety-critical. Aggressive TTLs reduce stale routing risk but can amplify flapping during transient network spikes. Conservative TTLs lower churn but keep bad endpoints in circulation longer.

Performance Analysis: Lookup Latency, Convergence Time, and Flapping

Discovery systems are often judged by three metrics.

Lookup latency: if discovery calls are synchronous and slow, p95 request latency rises. Many systems cache discovery results briefly to reduce lookup overhead.

Convergence time: this is the delay between a backend failure and traffic stop. Faster convergence improves reliability but requires aggressive health-check cadence and low-control-plane lag.

Flapping: if health checks are too strict, instances bounce between healthy/unhealthy states, creating churn and cascading retries. Hysteresis and multi-sample thresholds help avoid this.

In interviews, saying "I would optimize for stable convergence, not just fastest possible eviction" shows operational maturity.

📊 Discovery Flow: Registration to Health-Aware Routing

flowchart TD
    A[Instance boots] --> B[Register with service registry]
    B --> C[Heartbeat and metadata updates]
    C --> D{Healthy and ready?}
    D -->|Yes| E[Add to routing pool]
    D -->|No| F[Exclude from routing pool]
    E --> G[Client or proxy resolves target]
    G --> H[Request served]
    F --> I[Recovery or restart]
    I --> C

This model captures the key principle: discovery and health checks are continuous control loops, not setup-time configuration.

📊 Service Registration and Client Discovery

sequenceDiagram
    participant S as OrderService
    participant R as Service Registry
    participant C as Client
    S->>R: Register: host port tags
    R-->>S: ACK registration
    S->>R: Heartbeat every 5s
    C->>R: Discover order-service
    R-->>C: Return healthy endpoints
    C->>S: Route request
    S-->>C: Response

This sequence diagram traces the full lifecycle of service registration and client-driven discovery. OrderService registers with the registry and sends heartbeats every 5 seconds to maintain its healthy status; when a Client queries for available endpoints, the registry returns only healthy instances, and the Client routes its request directly. Discovery is not a one-time lookup — it is a continuous health-maintained contract that ensures clients never route to stale or unhealthy backends.

📊 Health Check Lifecycle: Unhealthy to Deregister

sequenceDiagram
    participant R as Registry
    participant S as Service Instance
    R->>S: GET /ready every 5s
    S-->>R: 200 OK
    Note over S: DB connection lost
    R->>S: GET /ready
    S-->>R: 503 Unhealthy
    R->>R: Mark instance DOWN
    R->>R: Remove from routing pool
    Note over S: DB connection restored
    S->>R: Re-register
    R->>S: GET /ready
    S-->>R: 200 OK
    R->>R: Add to routing pool

This sequence diagram shows what happens when a service instance loses a critical dependency. The registry polls the instance every 5 seconds; when a database connection is lost and the instance returns 503, the registry marks it DOWN and removes it from the routing pool — no manual intervention required. Once the database connection is restored and the instance re-registers and returns 200 OK, the registry automatically adds it back to the pool, completing the self-healing loop.

🌍 Real-World Applications: API Gateways, Payments, and Internal Platforms

HashiCorp Consul at scale: Consul's gossip protocol propagates health changes across a cluster in ~200ms on typical LAN deployments. The deregister_critical_service_after field automatically removes services that remain unhealthy beyond a configurable window — preventing stale endpoints from silently accumulating in the registry.

When a service registers with Consul, it provides its name, port, and metadata tags alongside a health check definition. The check specifies the HTTP path to poll, the polling interval, the per-check timeout, and — critically — a window after which Consul automatically deregisters a service that remains in a critical state. This final setting prevents stale endpoints from silently accumulating in the registry when a service crashes without sending a graceful shutdown signal, which is one of the most common sources of phantom traffic in production environments. Version and environment tags such as v2 and stable are also registered at this point, enabling routing layers to enforce canary splits or environment isolation without modifying service code.

Kubernetes endpoint controller: Kubernetes removes a failing pod from its EndpointSlice within the kube-proxy sync interval — typically < 1 second on a healthy cluster — when the pod's readiness probe fails. This is faster than any DNS TTL-based failover mechanism.

Kubernetes readiness and liveness probes are configured declaratively in the pod specification with separate polling intervals, initial delays, and consecutive-failure thresholds for each check type. A readiness probe can start polling sooner and use a shorter polling period because temporarily withholding traffic from a pod that is not yet ready is a safe, reversible action; a liveness probe uses a longer initial delay because triggering a pod restart is destructive and should be reserved for genuinely unrecoverable states.

The /ready endpoint returns 200 only when the service has established its database connection and warmed its local caches. The /healthz endpoint returns 200 as long as the process is responsive — a liveness failure triggers a pod restart, which is destructive and should be reserved for genuinely deadlocked or unrecoverable processes.

Envoy xDS health propagation: Envoy's Endpoint Discovery Service (EDS) receives health status from a control plane such as Istio, Consul Connect, or a custom xDS server. Status changes propagate to all connected Envoy proxies in < 50ms in a well-tuned mesh — orders of magnitude faster than DNS TTL expiry. This speed is what enables effective circuit-breaking and near-instant unhealthy-instance removal in a service mesh.

Failure scenario: a payments team used only liveness probes — no readiness probes — on their transaction-processing service. During a scheduled database maintenance window, pods stayed alive but could not process transactions. The load balancer continued routing requests to these pods for 6 minutes until engineers manually drained them. Adding a readiness probe that checks database connectivity eliminated this failure class entirely over the following 12 months.

⚖️ Trade-offs & Failure Modes: Where Discovery Can Go Wrong

The interview-quality answer always includes one sentence like: "I would define clear health semantics and failure thresholds before tuning load-balancer algorithms."

🧭 Decision Guide: Choosing a Discovery Strategy

When unsure in interviews, start with server-side discovery for simpler client behavior, then discuss where client-side control may be worth the complexity.

🧪 Practical Example: Evolving a Checkout Service Beyond Static Backends

Imagine a checkout service initially routed via hardcoded backend IPs.

Problems appear during traffic spikes:

• New app instances launch but receive no traffic.
• One bad instance still receives requests for minutes.
• Rolling deploys create intermittent errors from stale endpoint lists.

A safer evolution path:

1. Introduce a service registry with instance metadata.
2. Route through a load balancer that consumes registry updates.
3. Add readiness checks that include payment-db connectivity.
4. Add zone-aware balancing to reduce cross-zone latency.

Expected outcome:

This is a strong interview answer because it keeps architecture evolution incremental and justified by failures.

🛠️ Dynamic Registration and Health-Aware Discovery in Production

Spring Cloud Netflix Eureka is a client-side service registry built into the Spring Cloud ecosystem; Spring Cloud Consul provides the same programming model backed by HashiCorp Consul's gossip-based registry. Both integrate with a discovery client abstraction and a health indicator framework to expose liveness and readiness semantics to the control plane automatically.

How it works in practice: A microservice opts into dynamic discovery by declaring itself as a discovery client at startup. From that point, registration and heartbeat renewal happen automatically — the developer does not manage registry calls manually. The same framework allows custom health indicators to extend the built-in health check, so a payment service can include its database connectivity as part of its reported health status. When the database becomes unreachable, the health indicator reports the service as down, and the registry removes it from the routing pool without any manual intervention. On the client side, load-balanced HTTP clients resolve service names through the registry at call time rather than using hardcoded URLs, so each outgoing request automatically targets a healthy instance chosen from the current live pool.

Discovery client registration behavior is controlled through application configuration: the client is told which registry agent to contact, which HTTP path to use as the health check endpoint, how frequently to poll it, and whether to automatically deregister when the application shuts down. Graceful deregistration on shutdown is particularly important for rolling deployments — when enabled, the framework deregisters the instance before the process exits, eliminating the stale-endpoint window that commonly causes 502 errors in blue-green and canary rollouts. Tagging instances with version and environment labels such as v2 and stable allows routing layers to enforce traffic policies — canary splits, environment isolation, or weighted routing — without modifying service code or redeploying.

For a full deep-dive on Spring Cloud service discovery with Eureka and Consul, a dedicated follow-up post is planned.

📚 Lessons Learned

• Service discovery is a control-plane capability, not just a DNS trick.
• Health checks must distinguish process liveness from real request readiness.
• Faster failover is useful only when flapping is controlled.
• Registry availability and correctness directly affect data-plane reliability.
• Discovery design should align with team ownership and platform maturity.

📌 TLDR: Summary & Key Takeaways

• Dynamic systems need dynamic endpoint resolution.
• Discovery and health checks are tightly coupled reliability primitives.
• Readiness semantics matter more than raw check frequency.
• Control-plane failures can become data-plane outages if unmanaged.
• Start simple, then add richer routing metadata and policies as scale grows.