System problems.
Real requirements. Concrete designs. Master system design, concurrency, and architecture trade-offs.
- 01Architecture
Active-Active Multi-Master Split-Brain Recovery
Your global SaaS runs an active-active multi-master relational topology (PostgreSQL BDR / MySQL Group Replication style async bi-directional replication) between `us-east-1` and `eu-west-1` serving 6,000 writes/sec. A 9-minute transatlantic fiber cut partitions the regions while both continue accepting writes. Both regions update the same `account_balances` and `subscription_entitlements` rows. On recovery, the replication engine applies Last-Write-Wins (LWW) using `updated_at` wall-clock timestamps. Due to a 180ms NTP drift in `eu-west-1`, 1,400 valid `us-east-1` payments and plan upgrades are silently overwritten without error logs. Redesign the replication conflict-detection, ordering, and recovery pipeline to eliminate silent data loss.
hard · ~30m
- 02Architecture
Handling Asymmetric Network Partitions & Split-Brain
Your 3-node metadata control plane (Nodes A, B, C across us-east-1a, 1b, 1c) runs a Raft-backed coordination service handling 4,500 lock acquisitions/sec at p99 < 12ms. A top-of-rack BGP/ACL misconfiguration drops all TCP/UDP packets between AZ-1a (Node A) and AZ-1c (Node C), while A<->B and B<->C links remain healthy at 0.8ms RTT. Within 500ms, Node C's election timeout fires, it increments its Raft term, and Node B votes for C—disrupting Leader A, which then times out and bumps its term again, causing continuous term inflation and 100% lock-service unavailability. Redesign the failure detection, consensus election protocol, and downstream fencing to survive non-transitive partitions without livelock or split-brain writes.
hard · ~30m
- 03Engineering
Non-Blocking Async & Structured Logging
A 20-pod Spring Boot order-ingestion service (200 Tomcat worker threads/pod, gp3 EBS 3,000 IOPS limit) enables verbose DEBUG/JSON request logging during an incident. At 1,800 RPS/pod, each request writes 6 log lines (~4.5 KB). With Logback's synchronous `FileAppender` (`immediateFlush=true`), disk queue depth hits 128 and write latency spikes to 45 ms per line—blocking all 200 worker threads in `FileOutputStream.writeBytes` and dropping throughput to near zero.
medium · ~30m
- 04Engineering
Automating TLS Certificate Rotation & Expiry Alerts
At 00:00 UTC, 100% of global web and mobile traffic to `api.globalshop.com` fails immediately with `SSL_ERROR_EXPIRED_CERT_ALERT` / `ERR_CERT_DATE_INVALID`. Yet your internal Prometheus/Kubernetes dashboards are all green—origin ALBs, pods, and health checks return `200 OK` because the expired certificate sits on the **CDN Edge** (and internal monitors bypassed the CDN or used `insecureSkipVerify: true`). Furthermore, the automated ACME renewal cron failed silently 21 days ago when a DNS TXT record permission was revoked. Design a zero-touch TLS lifecycle and edge-aware observability architecture.
medium · ~30m
- 05Architecture
Avoiding Partition Hotspots on Monotonic Keys
An IoT telemetry platform ingests 250,000 events/sec into a 16-node distributed wide-column/KV cluster (ScyllaDB/Cassandra or DynamoDB) using `event_date_hour` or `created_at_ms` as the partition key and `device_id` as the clustering key. During peak ingestion, a single storage node hits 98% CPU, 100% NVMe write saturation, and p99 write latency of 1,400 ms, while the other 15 nodes sit at 6% CPU. Redesign the partition key schema and query path to distribute writes evenly across all 16 nodes while still supporting time-range queries.
medium · ~30m
- 06Architecture
Why Implicit Type Coercion Breaks B-Tree Indexes
Your user-auth service runs on PostgreSQL 15 (and has a legacy MySQL 8 shard) with a 50-million-row `users` table (`phone_number VARCHAR(20)` with a B-Tree index `idx_users_phone`). After a Java/Python API refactor, `SELECT id, password_hash FROM users WHERE phone_number = ?` jumps from **2 ms** to **18 seconds**, spiking DB CPU to 100% and disk read IOPS to 40,000. Diagnose why the B-Tree index is ignored and deploy a zero-downtime fix.
medium · ~30m
- 07Engineering
Zero-Downtime Blue-Green Database Deployments
A payments service deploys via Blue-Green behind an AWS ALB sharing a single PostgreSQL 15 database. Release v42 (Green) runs a Flyway migration that renames `billing_zip` to `postal_code` and adds a non-nullable `tax_jurisdiction VARCHAR(16) NOT NULL` column without a default. During Green validation, 12% of requests fail with a serialization bug, so the on-call engineer flips the ALB target group back to Blue (v41). Immediately, Blue fails with 100% HTTP 500s (`column billing_zip does not exist` and `null value in column tax_jurisdiction violates not-null constraint`). Design a zero-downtime, instant-rollback Blue-Green schema evolution and deployment pipeline.
hard · ~30m
- 08Low-Level Design
Bounding Task Queues to Prevent Heap OOM
An event-ingestion service reads 15,000 events/sec from Kafka/HTTP and dispatches each event (average payload ~8 KB) into `Executors.newFixedThreadPool(16)` to write to an Elasticsearch cluster. When Elasticsearch experiences a 45-second GC pause and drops to 200 writes/sec, the ingestion service does not slow down or return backpressure. Instead, `newFixedThreadPool(16)` buffers `675,000+` `Runnable` closures in its internal `LinkedBlockingQueue`, consuming 5.4 GB of heap and crashing all pods with `java.lang.OutOfMemoryError: Java heap space`. Redesign the executor and queue-bounding strategy to prevent heap collapse and surface backpressure cleanly.
medium · ~30m
- 09Architecture
Bulkhead Pattern: Isolating Faults & Capacity
A checkout BFF (`checkout-bff`, 200 Tomcat worker threads, single shared Apache `PoolingHttpClientConnectionManager` with `maxTotal=200`) handles 1,200 req/s. Each `/checkout/summary` request fans out in parallel to three downstream services: `InventoryService` (p99=25 ms, critical), `PricingService` (p99=40 ms, critical), and `RecommendationService` (p99=80 ms, non-critical). During a DB lock storm in `RecommendationService`, its response time spikes to 10 seconds (matching the shared HttpClient socket timeout). Within 4 seconds, all 200 shared HTTP connections and Tomcat threads block waiting on `RecommendationService`. Requests to `InventoryService` and `PricingService` fail with connection-pool timeouts (`503/504`), causing a 100% checkout outage. Design a bulkhead isolation, capacity-partitioning, and fallback architecture.
medium · ~30m
- 10Low-Level Design
Bypassing Python GIL Contention in CPU Pipelines
A Python 3.12 data-processing service computes SHA-256/custom pure-Python checksums, JSON canonicalization, and regex transforms on 50,000 payloads/min. To utilize a 16-vCPU machine, the team wrapped the worker in `ThreadPoolExecutor(max_workers=16)`. Instead of a 16x speedup, throughput dropped from 1,200 items/sec (1 thread) to 740 items/sec (16 threads), while `sys` CPU time jumped to 35%. Design a scalable Python execution architecture that achieves near-linear 16-core scaling without excessive pickle serialization overhead.
medium · ~30m
- 11Architecture
CAP & PACELC Consistency-Latency Trade-Offs
A global e-commerce order and catalog service runs on a geo-distributed consensus database (3-region Raft cluster across `us-east-1`, `eu-west-1`, and `ap-south-1`; RTTs: US-EU 75ms, EU-IN 130ms, US-IN 195ms). Every read and write uses default linearizable quorum consistency (`PC/EC`). During normal operation with zero network partitions, `ap-south-1` users experience 390ms+ p99 checkout latency and 150ms+ catalog read latency, breaching the 120ms p99 SLA. Redesign the data consistency and placement architecture using PACELC principles.
medium · ~30m
- 12Architecture
Preventing Cassandra Tombstone Accumulation
A background job-scheduler backed by Cassandra/ScyllaDB ingests 200M short-lived tasks/day (`TTL = 86400s` or explicit `DELETE` upon completion) into `PRIMARY KEY (queue_name, task_id)`. Workers poll `SELECT * FROM tasks WHERE queue_name = 'default' LIMIT 50`. Within 12 hours, queries fail with `TombstoneOverwhelmingException` (exceeding `tombstone_failure_threshold = 100000`) and p99 read latency hits 4.2 seconds. Redesign the data model, compaction strategy, and access pattern to eliminate tombstone scans.
hard · ~30m
Showing 1–12 of 109 problems