Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC permission sync, < 500 ms retrieval + rerank latency, zero cross-tenant/ACL leaks, and inline citation verification.
From patterns to decisions.
Ingest documents via incremental CDC connectors into sharded Qdrant/Milvus + Elasticsearch BM25 with synced Roaring Bitmap ACLs, fuse via RRF, rerank Top-50 -> Top-5, and verify LLM citations.
Scale Envelope & Latency Budget.
Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC permission sync, < 500 ms retrieval + rerank latency, zero cross-tenant/ACL leaks, and inline citation verification.
Scale Envelope & Latency Budget
1.Stage Objective: Size throughput, storage, and p99 latency targets.
2.Level (Foundation): Design an enterprise RAG platform indexing 10M+ multi-format documents (~150M chunks) from Drive, Confluence, and Git with incremental CDC p
3.Operational Invariant: Size throughput, storage, and p99 latency targets.
Caption: Size throughput, storage, and p99 latency targets.
Core Architecture & Data Flow.
On query, the gateway resolves the user's transitive Okta/AD group memberships (cached 60s), executes ACL-prefiltered HNSW + BM25 search across sharded Qdrant/Elasticsearch nodes, merges Top-50 via RRF, and reranks to Top-5 with a GPU Cross-Encoder.
Core Architecture & Data Flow
1.Stage Objective: Assemble the primary data path and storage layout.
2.Level (Foundation): On query, the gateway resolves the user's transitive Okta/AD group memberships (cached 60s), executes ACL-prefiltered HNSW + BM25 search acr
3.Operational Invariant: Assemble the primary data path and storage layout.
Caption: Assemble the primary data path and storage layout.
ACL-Scoped Hybrid Retrieval Guard.
def search_enterprise_rag(user_principals: set[str], query: str, dense_idx, bm25_idx, reranker) -> list[dict]:
if not user_principals:
raise PermissionError('Fail closed: user_principals cannot be empty')
dense_hits = dense_idx.search_prefiltered(query, principals=user_principals, limit=50)
sparse_hits = bm25_idx.search_prefiltered(query, principals=user_principals, limit=50)
fused = reciprocal_rank_fusion(dense_hits, sparse_hits, top_n=50)
return reranker.rerank(query, [doc_id for doc_id, _ in fused], top_k=5)ACL-Scoped Hybrid Retrieval Guard
1.Stage Objective: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.
2.Level (Intermediate): def search_enterprise_rag(user_principals: set[str], query: str, dense_idx, bm25_idx, reranker) -> list[dict]:
3.Operational Invariant: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.
Caption: Fail closed if user principal groups are missing and enforce pre-filtered hybrid search.
Sharding & Hot-Spot Mitigation.
Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
Sharding & Hot-Spot Mitigation
1.Stage Objective: Prevent single-partition bottlenecks at peak load.
2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
3.Operational Invariant: Prevent single-partition bottlenecks at peak load.
Caption: Prevent single-partition bottlenecks at peak load.
Failure Modes & Graceful Degradation.
Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
Failure Modes & Graceful Degradation
1.Stage Objective: Keep critical paths healthy during partial outages.
2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
3.Operational Invariant: Keep critical paths healthy during partial outages.
Caption: Keep critical paths healthy during partial outages.
Consistency, Idempotency & Recovery.
Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
Consistency, Idempotency & Recovery
1.Stage Objective: Guarantee correctness across retries and node crashes.
2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
3.Operational Invariant: Guarantee correctness across retries and node crashes.
Caption: Guarantee correctness across retries and node crashes.
Telemetry, Circuit Breakers & Failure Modes.
Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.
Production Guardrails
1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.
2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de
3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Practice: Make the trade-off yourself.
Why decouple ACL metadata updates from chunk vector re-embedding?