System Design HLD Example: File Storage and Sync (Dropbox and Google Drive)

Scenario context

Cloud sync systems separate immutable blob storage (S3) from atomic metadata operations (PostgreSQL), using chunk-level deduplication to optimize storage costs and delta-sync events to minimize bandwidth.

Dropbox serves 700 million registered users who edit files simultaneously across desktop, mobile, and web clients. The hard problem is not storing bytes — object stores like S3 handle petabyte durability reliably.…

This walkthrough focuses on File Storage & Sync: Use Cases & Requirements.

Before making a design choice, clarify:

• A Dropbox-style system must bridge the gap between high-latency file transfers and low-latency metadata operations.
• Functional Requirements - File Upload/Download : Support files up to 50GB via chunked, resumable transfers.
• • Versioning : Maintain a history of file versions (e.g., last 30 days) to allow recovery from accidental edits or ransomware.

Concepts in play: caching, vector-databases-rag, consistency-replication, system-design-hld-architectures

Stage 1 — Basics: Baseline Architecture

• The system is split into two distinct planes: the Data Plane (heavy bytes) and the Control Plane (metadata).
• Block Service : Handles file chunking, hashing, and storage in an object store (like AWS S3).
• Metadata Service : Manages the file hierarchy, version history, and sharing permissions in a relational database (PostgreSQL).
• Sync Service : Propagates metadata change events to clients using WebSockets or Long Polling.

Observable outcome: Sync Service : Propagates metadata change events to clients using WebSockets or Long Polling.

Stage 2 — Mechanics: Key Logic

• The 4 MB Chunking Strategy Instead of treating a file as a single blob, we split it into fixed-size chunks (e.g., 4 MB).
• • Benefit : If an upload fails at chunk 10 of 100, we only retry chunk 10.
• A client can upload 4 chunks simultaneously to saturate bandwidth.
• Content-Addressed Storage (Deduplication) Every chunk is hashed (SHA-256).

sequenceDiagram
    participant C as Client
    participant AG as API Gateway
    participant BS as Block Service
    participant RC as Redis Cache
    participant S3 as Object Storage
    participant MS as Metadata Service
    C->>AG: Upload request with chunk hashes
    AG->>BS: Check chunk existence
    BS->>RC: Query known chunk hashes
    RC-->>BS: Return missing chunks list
    BS-->>C: Upload only missing chunks
    C->>BS: Stream missing chunk data
    BS->>S3: Store new chunks durably
    S3-->>BS: Storage confirmed
    C->>MS: Commit file metadata
    MS-->>C: Return new file version ID

Observable outcome: Content-Addressed Storage (Deduplication) Every chunk is hashed (SHA-256).

Stage 3 — Estimations & Design Goals

• • Scale : 100M Daily Active Users.
• • Storage : 100M users 10GB avg = 1 Exabyte total storage.
• • Throughput : 1M writes per second (metadata), 10GB/s aggregate upload.
• • Dedup Ratio : Typically 30-50% in consumer clouds due to common OS files and shared media.

Observable outcome: - Dedup Ratio : Typically 30-50% in consumer clouds due to common OS files and shared media.

Stage 4 — High-Level Design

• architecture { "version": 1, "title": "High-Level Design - Subsystem Overview", "description": "Decoupled system architecture isolating ingress, stateless logic, and persistent datastores.", "view": "high-level", "patte…
• The Block Service handles the heavy byte operations — checking Redis for a known chunk hash, writing new chunks to S3, and acknowledging receipt to the client.
• The Metadata Service handles the logical file tree — atomically updating the folder structure in PostgreSQL and emitting a change event to Kafka.
• The Sync Service consumes those Kafka events and fans them out to all other devices via WebSocket, completing the cross-device propagation path within seconds of the original save.

graph TD
    Client[Client App] -->|1. Init Upload| Meta[Metadata API]
    Client -->|2. Upload Chunks| Block[Block Service]
    Block -->|3. Check Dedup| Redis[(Redis Hash Cache)]
    Block -->|4. Store Bytes| S3[Object Storage - S3]
    Block -->|5. ACK Chunks| Client
    Client -->|6. Commit| Meta
    Meta -->|7. Update Tree| DB[(PostgreSQL)]
    Meta -->|8. Emit Event| Kafka{Kafka}
    Kafka -->|9. Fan Out| Sync[Sync Service]
    Sync -->|10. Notify| OtherClients[Other Devices]

Observable outcome: The Sync Service consumes those Kafka events and fans them out to all other devices via WebSocket, completing the cross-device propagation path within seconds of the original save.

Stage 5 — Deep Dive: Content-Addressed Storage, Delta Sync Cursor, and Conflict Resolution Mechanics

• The three hardest problems in a cloud file sync system are storing bytes efficiently without duplication, propagating changes to devices without polling, and resolving conflicts when two devices edit the same file concu…
• Internals: Content-Addressed Chunk Storage and the SHA-256 Deduplication Gate Every file uploaded to the system is split into fixed-size chunks — typically 4MB — before any byte reaches the network.
• Each chunk is hashed with SHA-256 to produce a 64-character hex digest that serves as both the chunk's unique identifier and its object key in S3.
• Before uploading any chunk, the client sends the server a list of proposed chunk hashes (a "check-before-upload" request).

flowchart TD
    A["Local Change Detected"] --> B["Compute SHA-256 Chunk Hashes"]
    B --> C["Check-Before-Upload Gate - Redis"]
    C --> D{"All Chunks Already Known?"}
    D -->|Yes| E["Skip Upload — Link Metadata Only"]
    D -->|No| F["Upload Missing Chunks to S3"]
    F --> G["Commit Metadata to PostgreSQL"]
    E --> G
    G --> H{"Concurrent Edit Detected?"}
    H -->|No| I["Emit Change Event to Kafka"]
    H -->|Yes| J["Fork Both Versions"]
    J --> K["Surface Conflict Marker to User"]

Observable outcome: Before uploading any chunk, the client sends the server a list of proposed chunk hashes (a "check-before-upload" request).

Stage 6 — Real-World File Storage Architectures: Dropbox, Google Drive, and iCloud

• Dropbox Magic Pocket is Dropbox's proprietary object storage system, built to replace their dependence on AWS S3 for cost reasons.
• Magic Pocket is a custom erasure-coded storage cluster that stores file chunks across spinning-disk JBOD servers in multiple data centers.
• Dropbox's key architectural decision was to separate the chunk storage layer (Magic Pocket) from the metadata layer (MySQL with Vitess for sharding), exactly mirroring the pattern described in this guide.
• Moving from S3 to Magic Pocket reduced Dropbox's storage costs by approximately 40% while improving their control over durability guarantees.

Observable outcome: Moving from S3 to Magic Pocket reduced Dropbox's storage costs by approximately 40% while improving their control over durability guarantees.

End-to-end outcome

A Dropbox-style system must bridge the gap between high-latency file transfers and low-latency metadata operations.

Alternatives considered

Decision Guide: Chunk Size, Storage Backend, and Sync Strategy

Decision Guide: Chunk Size, Storage Backend, and Sync Strategy

Failure and degraded modes

Trade-offs and Failure Modes in File Storage and Sync Design

• Detection: Conflict on Concurrent Offline Edits: When Device A renames a file while Device B deletes it offline, both sync on reconnect.
• User or operator impact: Conflict on Concurrent Offline Edits: When Device A renames a file while Device B deletes it offline, both sync on reconnect.

Lessons Learned from File Sync Failures in Production

• Detection: A Missing Chunk Reference Caused Silent Data Loss for 0.01% of Files.
• User or operator impact: A Missing Chunk Reference Caused Silent Data Loss for 0.01% of Files.