LangChain RAG: Retrieval-Augmented Generation in Practice

Scenario context

Imagine you are the lead developer at a legal firm. Your team spends months building an AI assistant to help associates query hundreds of proprietary case files, internal memos, and client contracts. You wire up GPT-4 and deploy it. On day one, a senior partner asks: "What were the liability exclusions in the Hartwell vs. Meridian settlement?" The assistant replies with a confident, well-structured answer — and ev…

This walkthrough focuses on The Stale Knowledge Problem: Why LLMs Confidently Get It Wrong.

Before making a design choice, clarify:

• Imagine you are the lead developer at a legal firm.
• Your team spends months building an AI assistant to help associates query hundreds of proprietary case files, internal memos, and client contracts.
• You wire up GPT-4 and deploy it.

Concepts in play: vector-databases-rag, llm-agents-and-langgraph, vector-embeddings-and-similarity, database-indexing

Stage 1 — Grounding the Model: The Core Idea Behind RAG

• A useful mental model: think of a closed-book exam versus an open-book exam.
• A vanilla LLM is a closed-book test — it can only recall what it memorized during training.
• RAG turns it into an open-book exam.
• Before the LLM answers, the system hunts through a library for the most relevant passages and places them in front of the model.

Observable outcome: Before the LLM answers, the system hunts through a library for the most relevant passages and places them in front of the model.

Stage 2 — The RAG Pipeline Step by Step: From Documents to Answers

• Building RAG involves two distinct pipelines: ingestion (run once or on schedule) and retrieval (run per query).
• Ingestion: Getting Documents Into the Vector Store Step 1 — Document Loading.
• LangChain's DocumentLoader abstractions handle PDFs, HTML pages, plain text, Notion exports, and more.
• Each loader returns a list of Document objects (content + metadata).

pip install langchain langchain-community chromadb faiss-cpu sentence-transformers openai

from langchain_community.document_loaders import TextLoader, PyPDFLoader, WebBaseLoader

# Load a local text file
loader = TextLoader("company_policy.txt")
docs = loader.load()

# Load a PDF
pdf_loader = PyPDFLoader("org_chart.pdf")
pdf_docs = pdf_loader.load()

# Load a web page
web_loader = WebBaseLoader("https://example.com/faq")
web_docs = web_loader.load()

all_docs = docs + pdf_docs + web_docs

Observable outcome: Each loader returns a list of Document objects (content + metadata).

Stage 3 — Deep Dive: How Vector Retrieval Actually Works

• The Internals: Embeddings, Cosine Similarity, and FAISS Indexes When you embed a sentence like "remote work requires manager approval" , the embedding model produces a vector with hundreds of dimensions — typically 384…
• Each dimension captures a latent semantic feature learned during model training.
• When a query arrives, you embed it into the same vector space and measure cosine similarity — the cosine of the angle between the query vector and every stored document vector.
• A score of 1.0 means the vectors point in exactly the same direction (semantically identical); 0.0 means orthogonal (unrelated).

Observable outcome: A score of 1.0 means the vectors point in exactly the same direction (semantically identical); 0.0 means orthogonal (unrelated).

Stage 4 — Retrieval Strategies: MMR, Metadata Filters, and Contextual Compression

• Plain similarity search can return redundant chunks — five chunks saying the same thing in slightly different words.
• Maximal Marginal Relevance (MMR) balances relevance against diversity: each subsequent result must be both similar to the query and dissimilar to already-selected results.
• Metadata filtering narrows search to a document subset before running similarity search — useful when your index holds multi-tenant or multi-topic documents.
• Contextual Compression is a post-retrieval refinement step.

retriever = vectorstore.as_retriever(
    search_type="mmr",
    search_kwargs={"k": 5, "fetch_k": 20, "lambda_mult": 0.6}
)
# lambda_mult: 1.0 = pure similarity, 0.0 = pure diversity

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 4, "filter": {"source": "company_policy.txt"}}
)

Observable outcome: Contextual Compression is a post-retrieval refinement step.

Stage 5 — Visualizing the Full RAG Pipeline

• The pipeline splits cleanly into two halves: everything above the Vector Store node is the ingestion path (run offline).
• Everything from User Query down is the retrieval + generation path (run per request).

graph TD
    A[Raw Documents PDFs  TXT  HTML] --> B[Document Loaders]
    B --> C[Text Splitter RecursiveCharacterTextSplitter]
    C --> D[Embedding Model OpenAI  HuggingFace]
    D --> E[(Vector Store FAISS  Chroma)]

    F[User Query] --> G[Query Embedder]
    G --> H{Similarity Search Top-k Chunks}
    E --> H
    H --> I[Context Formatter format_docs]
    I --> J[Prompt Template System + Context + Question]
    J --> K[LLM GPT-4o  Claude  Local]
    K --> L[Final Answer]

    style A fill:#e8f5e9,stroke:#388e3c
    style E fill:#e3f2fd,stroke:#1976d2
    style K fill:#fce4ec,stroke:#c62828
    style L fill:#fff9c4,stroke:#f57f17

Observable outcome: Everything from User Query down is the retrieval + generation path (run per request).

Stage 6 — Real-World Application: A Legal Firm's Document Intelligence System

• Let us return to the legal firm.
• The team ingests three document types: - Case summaries (PDF): 2,000+ files covering past litigation outcomes.
• • Internal policy memos (TXT): HR policy, billing rates, escalation procedures.
• • Client contracts (PDF): active engagement letters with custom terms.

Observable outcome: - Client contracts (PDF): active engagement letters with custom terms.

End-to-end outcome

After working through the stages, you should be able to explain:

• Evaluate retrieval before the LLM.
• A common trap is treating the whole pipeline as a black box and tuning prompts when the real culprit is poor retrieval.
• • RAG solves the LLM knowledge-cutoff problem by retrieving relevant document chunks at query time and injecting them as context before generation.
• • The ingestion pipeline (load → split → embed → index) runs offline; the retrieval pipeline (embed query → search → augment prompt → generate) runs per request.

Alternatives considered

Decision Guide: When to Build a RAG Pipeline

Decision Guide: When to Build a RAG Pipeline

Failure and degraded modes

Trade-offs & Failure Modes in Production RAG

• Detection: RAG dramatically reduces hallucinations but introduces its own failure taxonomy.
• User or operator impact: RAG dramatically reduces hallucinations but introduces its own failure taxonomy.