Skip to main content
← PROBLEMS / AI-SYSTEMS
An end-to-end domain learning journey

Design an Enterprise RAG System at
Scale (10M+ Documents with ACLs).

Ingest documents via incremental CDC connectors into sharded Qdrant/Milvus + Elasticsearch BM25 with synced Roaring Bitmap ACLs, fuse via RRF, rerank Top-50 -> Top-5, and verify LLM citations.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC permission sync, < 500 ms retrieval + rerank latency, zero cross-tenant/ACL leaks, and inline citation verification.

From patterns to decisions.

Ingest documents via incremental CDC connectors into sharded Qdrant/Milvus + Elasticsearch BM25 with synced Roaring Bitmap ACLs, fuse via RRF, rerank Top-50 -> Top-5, and verify LLM citations.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Scale Envelope & Latency Budget.

Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC permission sync, < 500 ms retrieval + rerank latency, zero cross-tenant/ACL leaks, and inline citation verification.

STAGE 01 · CORE INTUITION

Scale Envelope & Latency Budget

Scale EnvelopeIngress / BaselineLatency BudgetScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Size throughput, storage, and p99 latency targets.

2.Level (Foundation): Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC p

3.Operational Invariant: Size throughput, storage, and p99 latency targets.

Caption: Size throughput, storage, and p99 latency targets.

02
STAGE 02 · CORE MECHANICSfoundation

Core Architecture & Data Flow.

On query, the gateway resolves the user's transitive Okta/AD group memberships (cached 60s), executes ACL-prefiltered HNSW + BM25 search across sharded Qdrant/Elasticsearch nodes, merges Top-50 via RRF, and reranks to Top-5 with a GPU Cross-Encoder.

STAGE 02 · CORE MECHANICS

Core Architecture & Data Flow

Core ArchitectureInput StateData FlowCore MechanismValidation GateInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Assemble the primary data path and storage layout.

2.Level (Foundation): On query, the gateway resolves the user's transitive Okta/AD group memberships (cached 60s), executes ACL-prefiltered HNSW + BM25 search acr

3.Operational Invariant: Assemble the primary data path and storage layout.

Caption: Assemble the primary data path and storage layout.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

ACL-Scoped Hybrid Retrieval Guard.

def search_enterprise_rag(user_principals: set[str], query: str, dense_idx, bm25_idx, reranker) -> list[dict]:
    if not user_principals:
        raise PermissionError('Fail closed: user_principals cannot be empty')
    dense_hits = dense_idx.search_prefiltered(query, principals=user_principals, limit=50)
    sparse_hits = bm25_idx.search_prefiltered(query, principals=user_principals, limit=50)
    fused = reciprocal_rank_fusion(dense_hits, sparse_hits, top_n=50)
    return reranker.rerank(query, [doc_id for doc_id, _ in fused], top_k=5)
STAGE 03 · SAFE IMPLEMENTATION

ACL-Scoped Hybrid Retrieval Guard

ACL-Scoped HybridRetrieValidated InputCore EngineAtomic StepGuard & VerifyError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.

2.Level (Intermediate): def search_enterprise_rag(user_principals: set[str], query: str, dense_idx, bm25_idx, reranker) -> list[dict]:

3.Operational Invariant: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.

Caption: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Sharding & Hot-Spot Mitigation.

Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

STAGE 04 · SCALING & TRADE-OFFS

Sharding & Hot-Spot Mitigation

ShardingHot PathHot-Spot MitigationPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Prevent single-partition bottlenecks at peak load.

2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.

3.Operational Invariant: Prevent single-partition bottlenecks at peak load.

Caption: Prevent single-partition bottlenecks at peak load.

05
STAGE 05 · FAULT ISOLATIONadvanced

Failure Modes & Graceful Degradation.

Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

STAGE 05 · FAULT ISOLATION

Failure Modes & Graceful Degradation

Failure ModesFault TriggerGraceful DegradationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Keep critical paths healthy during partial outages.

2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.

3.Operational Invariant: Keep critical paths healthy during partial outages.

Caption: Keep critical paths healthy during partial outages.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Consistency, Idempotency & Recovery.

Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

STAGE 06 · RECOVERY & CONSISTENCY

Consistency, Idempotency & Recovery

ConsistencyIdempotencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Guarantee correctness across retries and node crashes.

2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.

3.Operational Invariant: Guarantee correctness across retries and node crashes.

Caption: Guarantee correctness across retries and node crashes.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why decouple ACL metadata updates from chunk vector re-embedding?

Recommended Learning Flow