RAG Explained: How to Give Your LLM a Brain Upgrade

Scenario context

LLMs have a training cut-off and no access to private data. RAG (Retrieval-Augmented Generation) solves both problems by retrieving relevant documents from an external store and injecting them into the prompt before generation. No retraining required.

A standard LLM is like a student who has memorized everything in a textbook — but cannot consult notes during the exam. Helpful for general questions; unreliable…

This walkthrough focuses on The Open-Book Exam Analogy.

Before making a design choice, clarify:

• A standard LLM is like a student who has memorized everything in a textbook — but cannot consult notes during the exam.
• Helpful for general questions; unreliable when the answer changed after the book was printed.
• A RAG-enhanced LLM is like the same student with an open-book policy.

Concepts in play: vector-databases-rag, vector-embeddings-and-similarity, database-indexing, sorting-and-searching-algorithms

Stage 1 — The Three-Step RAG Pipeline

• Every RAG system, regardless of framework, follows the same three steps: 1.
• Retrieve — Convert the query to an embedding vector.
• Search a vector database for the nearest stored document embeddings.
• Augment — Inject the retrieved chunks into the prompt as context.

graph TD
    A[User Query] --> B[Embed Query
vec = embed_model]
    B --> C[Vector DB Similarity Search
top-k cosine nearest neighbors]
    C --> D[Retrieved Document Chunks]
    D --> E[Augmented Prompt
System + Context + Query]
    E --> F[LLM]
    F --> G[Grounded Response]

Observable outcome: Augment — Inject the retrieved chunks into the prompt as context.

Stage 2 — How Retrieval Actually Works: Embeddings and Cosine Similarity

• Every piece of text — document or query — is transformed into a dense vector by an embedding model (e.g., text-embedding-3-small , nomic-embed-text ).
• Semantic similarity between query vector $q$ and document vector $d i$ is measured by cosine similarity : $$ ext{sim}(q, d i) = rac{q \cdot d i}{\ q\ \,\ d i\ }$$ The vector store returns the top-k document chunks with…
• Minimal Python RAG skeleton (LangChain): 📊 RAG Query Pipeline

from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import FAISS
from langchain.chains import RetrievalQA

# 1. Build index from documents
texts = ["Redis is an in-memory key-value store.", "PostgreSQL supports ACID transactions."]
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_texts(texts, embeddings)

# 2. Build retrieval chain
llm = ChatOpenAI(model="gpt-4o-mini")
qa_chain = RetrievalQA.from_chain_type(llm=llm, retriever=vectorstore.as_retriever())

# 3. Query
result = qa_chain.invoke("What is Redis used for?")
print(result["result"])
# → "Redis is an in-memory key-value store used for caching..."

sequenceDiagram
    participant U as User
    participant E as Embed Model
    participant V as Vector DB
    participant L as LLM
    U->>E: Query text
    E->>V: Query embedding
    V-->>E: Top-K chunks
    E-->>L: Augmented prompt
    L-->>U: Grounded response

Observable outcome: Minimal Python RAG skeleton (LangChain): 📊 RAG Query Pipeline

Stage 3 — End-to-End RAG Data Flow

• This diagram shows the complete data journey — from document ingestion through query answering — in a single view.
• The offline indexing pipeline and the online query pipeline share exactly one thing: the embedding model.
• Using different models for indexing and querying is a common mistake that causes retrieval to silently fail, because the vector spaces will not align.
• Critical constraint: Always use the same embedding model version for both indexing and querying.

graph TD
    subgraph Offline Indexing
        A[Raw Documents] --> B[Chunker: split into 300-500 token pieces]
        B --> C[Embedding Model: chunk  float32 vector]
        C --> D[Vector Store: FAISS / Pinecone / pgvector]
    end
    subgraph Online Query
        E[User Query] --> F[Embed query with same model]
        F --> G[Cosine similarity search in Vector Store]
        G --> H[Top-k relevant chunks retrieved]
        H --> I[Augmented Prompt: system + chunks + query]
        I --> J[LLM generates grounded response]
    end
    D --> G

Observable outcome: Critical constraint: Always use the same embedding model version for both indexing and querying.

Stage 4 — Deep Dive: Vector Search and Embedding Space

• Every chunk and query is projected into the same high-dimensional vector space by the embedding model.
• Cosine similarity measures the angle between two vectors — not their length — so short and long chunks are compared fairly.
• Vector databases like FAISS and Pinecone use approximate nearest-neighbor (ANN) algorithms (e.g., HNSW) to search millions of vectors in milliseconds, trading a tiny recall loss for a 100×+ speed gain over exact exhaust…

Observable outcome: Vector databases like FAISS and Pinecone use approximate nearest-neighbor (ANN) algorithms (e.g., HNSW) to search millions of vectors in milliseconds, trading a tiny recall loss for a 100×+ speed gain over exact exhaust…

Stage 5 — Real-World Applications: RAG in Production: Indexing Pipeline

• Before queries can be answered, documents must be indexed.
• The indexing pipeline runs offline (and on updates): Chunking strategy matters.
• Too large: retrieval returns diluted context.
• Too small: chunks lose semantic coherence.

graph LR
    A[Raw Documents
.pdf, .md, .html] --> B[Chunker
split into 300-500 token chunks]
    B --> C[Embedding Model
chunk  float32 vector]
    C --> D[Vector Store
FAISS, Pinecone, Weaviate, pgvector]
    D --> E[Index ready for retrieval]

flowchart TD
    D[Raw Documents] --> C[Chunking]
    C --> EM[Embedding Model]
    EM --> VS[Vector Store]
    VS --> IDX[Indexed & Searchable]

Observable outcome: Too small: chunks lose semantic coherence.

Stage 6 — What to Learn Next

• • Tokenization Explained: How LLMs Understand Text - LLM Terms: A Helpful Glossary - Advanced AI Agents: RAG and the Future of Intelligence

Observable outcome: - Tokenization Explained: How LLMs Understand Text - LLM Terms: A Helpful Glossary - Advanced AI Agents: RAG and the Future of Intelligence

End-to-end outcome

A standard LLM is like a student who has memorized everything in a textbook — but cannot consult notes during the exam.

Alternatives considered

Decision Guide: Choosing Between RAG, Fine-tuning, and Prompt Engineering

Decision Guide: Choosing Between RAG, Fine-tuning, and Prompt Engineering

Failure and degraded modes

Trade-offs & Failure Modes: When RAG Works Well — and When It Doesn't

• Detection: RAG excels when: - Data changes frequently — update the index without retraining.
• User or operator impact: RAG excels when: - Data changes frequently — update the index without retraining.