Guide to Using RAG with LangChain and ChromaDB/FAISS

Scenario context

RAG (Retrieval-Augmented Generation) gives an LLM access to your private documents at query time. You chunk and embed documents into a vector store (ChromaDB or FAISS), retrieve the relevant chunks at query time, and inject them into the LLM's prompt. The model answers from real data instead of hallucinating.

TLDR: RAG combines a vector database (ChromaDB) with an LLM — retrieve relevant document chunks a…

This walkthrough focuses on Why LLMs Hallucinate and RAG Fixes It.

Before making a design choice, clarify:

• An LLM's knowledge is frozen at its training cutoff.
• Ask it about your internal documentation, your product catalog, or a document uploaded today — it will generate a plausible-sounding answer from its weights, not from your data.
• RAG bridges this gap: - No fine-tuning required (fine-tuning is expensive and won't help for dynamic data) - Works with any LLM - Updates in real time — add new documents to the vector store and they're immediately sear…

Concepts in play: vector-databases-rag, vector-embeddings-and-similarity, llm-agents-and-langgraph, sorting-and-searching-algorithms

Stage 1 — Key Concepts: Embeddings and Vector Similarity

• When you search for "refund policy" in a document, a keyword search returns results containing those exact words.
• But what if your policy document says "money-back guarantee"?
• An embedding is a list of numbers — a high-dimensional vector (e.g., 1,536 numbers for OpenAI's text-embedding-3-small ) that encodes the semantic meaning of a piece of text.
• Similar meanings produce vectors that point in similar directions in that space.

sequenceDiagram
    participant D as Documents
    participant S as TextSplitter
    participant E as EmbeddingModel
    participant V as ChromaDB
    participant U as User
    participant R as Retriever
    participant L as LLM

    D->>S: Split into chunks
    S->>E: Embed each chunk
    E->>V: Store vectors + metadata
    U->>R: Submit query
    R->>E: Embed query
    E->>V: Cosine similarity search
    V-->>R: Top-K chunks
    R->>L: Inject chunks into prompt
    L-->>U: Grounded answer + citations

flowchart LR
    DL["DocumentLoader (PDF, TXT, HTML)"]
    TS[TextSplitter chunk_size=500]
    EM["Embeddings (OpenAI / HuggingFace)"]
    VS["VectorStore (ChromaDB / FAISS)"]
    RT[Retriever top_k=4]
    LLM["LLM (gpt-4o-mini)"]
    ANS[Answer]

    DL --> TS --> EM --> VS --> RT --> LLM --> ANS

Observable outcome: Similar meanings produce vectors that point in similar directions in that space.

Stage 2 — Step 1: Chunking and Embedding Your Documents

• Before storing, split your documents into chunks and convert each to an embedding.
• Overlap (typically 10–20% of chunk size) preserves context across chunk boundaries.

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,    # ~500 tokens: self-contained enough for focused retrieval without losing sentence coherence
    chunk_overlap=50,  # 10% overlap ensures sentences split at boundaries appear in both adjacent chunks
)
chunks = splitter.split_documents(docs)

Observable outcome: Overlap (typically 10–20% of chunk size) preserves context across chunk boundaries.

Stage 3 — Step 2: Storing and Searching with ChromaDB

• ChromaDB stores embeddings locally (persisted to disk).
• FAISS is an alternative — in-memory only, faster for pure search, no built-in persistence.

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",
)

# Retrieve top-4 most relevant chunks for a query
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

Observable outcome: FAISS is an alternative — in-memory only, faster for pure search, no built-in persistence.

Stage 4 — Deep Dive: The RetrievalQA Chain in LangChain

• The chain type="stuff" injects all retrieved chunks into one prompt.
• For larger context windows, use "map reduce" (summarize each chunk separately, then combine) or "refine" .
• 🔬 Internals RAG pipelines split documents into overlapping chunks (typically 512–1024 tokens with 10–20% overlap), embed each chunk with a sentence encoder, and store embeddings in a vector index.
• At query time, the query is embedded and the top-k chunks are retrieved by approximate nearest-neighbor search (HNSW or IVF).

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",   # "stuff" = inject all context at once
    retriever=retriever,
    return_source_documents=True,
)

result = qa_chain.invoke({"query": "What is our refund policy?"})
print(result["result"])
print(result["source_documents"])

flowchart LR
    Q[User Question] --> Embed[Embed question]
    Embed --> Retrieve[Retrieve top-k chunks from ChromaDB]
    Retrieve --> Prompt[Inject chunks into LLM prompt]
    Prompt --> LLM[LLM call]
    LLM --> A[Answer with citations]

Observable outcome: At query time, the query is embedded and the top-k chunks are retrieved by approximate nearest-neighbor search (HNSW or IVF).

Stage 5 — The RAG Pipeline: End-to-End Flow

• The complete RAG workflow has two phases: an offline indexing phase that builds the vector store, and an online retrieval phase that grounds each LLM response in retrieved evidence.

flowchart TD
    A[Documents] --> B[Chunking]
    B --> C[Embedding Model]
    C --> D[ChromaDB Vector Index]
    E[User Query] --> F[Query Embedding]
    F --> G[ANN Search]
    G --> D
    G --> H[Top-k Chunks]
    H --> I[LLM + Context Window]
    I --> J[Grounded Answer]

Observable outcome: The complete RAG workflow has two phases: an offline indexing phase that builds the vector store, and an online retrieval phase that grounds each LLM response in retrieved evidence.

Stage 6 — Real-World Applications of RAG

• RAG powers document Q&A systems across every industry.
• The core pattern is identical — the differences are in which documents get indexed and what guardrails are required.
• Hybrid search in production: Pure semantic search sometimes misses exact terms — product codes, error numbers, names.
• Production systems often combine keyword search (BM25) with vector search, then rerank with a cross-encoder model.

Observable outcome: Production systems often combine keyword search (BM25) with vector search, then rerank with a cross-encoder model.

End-to-end outcome

After working through the stages, you should be able to explain:

• Chunking strategy is the single most important RAG parameter.
• Test chunk size values of 200, 500, and 1000 on your own documents.

Alternatives considered

Decision Guide: RAG Architecture End-to-End

RAG splits neatly into two phases that run at different times: - Indexing phase (offline, runs once per document set): Load documents → split into chunks → embed each chunk → persist vectors in ChromaDB.

Failure and degraded modes

Trade-offs & Failure Modes: Retrieval Quality Traps

• Detection: Trade-offs & Failure Modes: Retrieval Quality Traps
• User or operator impact: Trade-offs & Failure Modes: Retrieval Quality Traps