LLM Evaluation Frameworks: How to Measure Model Quality (RAGAS, DeepEval, TruLens)

Scenario context

📏 Traditional ML metrics (accuracy, F1) fail for LLMs because there's no single "correct" answer. RAGAS measures RAG pipeline quality with faithfulness, answer relevance, and context precision. DeepEval provides unit-test-style LLM evaluation. TruLens enables continuous monitoring in production. The secret: reference-free evaluation using LLMs to judge LLMs.

<!--

You launched a RAG chatbot for customer support. U…

This walkthrough focuses on The LLM Evaluation Crisis: When Cosine Similarity Lies.

Before making a design choice, clarify:

• You launched a RAG chatbot for customer support.
• Users are complaining it gives wrong answers like "Your warranty expired in 1847" or "Please contact our Mars office." But your monitoring dashboard shows everything is green: 0.95 cosine similarity scores, 200ms retrie…
• The fundamental problem: traditional ML metrics don't work for LLMs.

Concepts in play: vector-databases-rag, deep-learning-foundations, vector-embeddings-and-similarity, consistency-replication

Stage 1 — The Three Dimensions of LLM Quality

• LLM evaluation has evolved beyond accuracy into three semantic dimensions: 1.
• Faithfulness (Grounding) Does the answer stay true to the provided context?
• A faithful answer only uses information explicitly stated in the retrieved documents.
• Bad (Unfaithful): - Context: "Our return policy is 30 days" - Answer: "You have 60 days to return items" ❌ Good (Faithful): - Answer: "You have 30 days to return items" ✅ 2.

Observable outcome: Bad (Unfaithful): - Context: "Our return policy is 30 days" - Answer: "You have 60 days to return items" ❌ Good (Faithful): - Answer: "You have 30 days to return items" ✅ 2.

Stage 2 — How LLM Evaluation Frameworks Actually Work

• Modern LLM evaluation frameworks solve the "infinite valid answers" problem with LLM-as-Judge : using another LLM to evaluate the quality of the first LLM's output.
• The core insight: If humans can judge whether an answer is faithful, relevant, and accurate, then a sufficiently capable LLM can too.
• Here's the general pattern all frameworks follow: This reference-free evaluation approach means you don't need perfect ground truth answers—you just need the evaluator LLM to understand what makes a good response.

def evaluate_llm_response(question, context, answer, ground_truth=None):
    # Step 1: Design evaluation prompt
    eval_prompt = f"""
    Question: {question}
    Context: {context}  
    Answer: {answer}

    Rate the faithfulness of this answer on a scale 1-5:
    1 = Completely contradicts context
    5 = Perfectly grounded in context only

    Provide your rating and reasoning.
    """

    # Step 2: Get judgment from evaluator LLM
    evaluation = evaluator_llm.generate(eval_prompt)

    # Step 3: Extract score and reasoning
    return parse_score_and_reasoning(evaluation)

Observable outcome: Here's the general pattern all frameworks follow: This reference-free evaluation approach means you don't need perfect ground truth answers—you just need the evaluator LLM to understand what makes a good response.

Stage 3 — The Internals: Reference-Free Evaluation with LLM-as-Judge

• Let's dive deeper into how this actually works.
• The magic happens in the evaluation prompt design: Faithfulness Evaluation Prompt (RAGAS Style) The Chain-of-Thought Pattern Notice the prompt asks the evaluator to: 1.
• Break down the answer into claims 2.
• Check each claim against context 3.

FAITHFULNESS_PROMPT = """
Given the following QUESTION, CONTEXT and ANSWER, analyze whether the ANSWER is faithful to the CONTEXT.

The ANSWER is faithful if:
1. Every claim in the ANSWER can be inferred from the CONTEXT  
2. The ANSWER doesn't contradict any information in the CONTEXT
3. The ANSWER doesn't add information not present in the CONTEXT

QUESTION: {question}

CONTEXT: {context}

ANSWER: {answer}

First, identify all the claims made in the ANSWER.
Then, for each claim, check if it's supported by the CONTEXT.
Finally, provide a faithfulness score from 0 to 1.

Claims Analysis:
[Your analysis here]

Faithfulness Score: [0.0 to 1.0]
"""

def evaluate_answer_relevance(question, answer):
    # Generate questions this answer could address
    gen_questions = llm.generate(f"""
    Generate 3 questions that could be answered by: {answer}
    """)

    # Calculate similarity between original and generated questions  
    similarities = []
    for gen_q in gen_questions:
        sim = cosine_similarity(
            embed(question), 
            embed(gen_q)
        )
        similarities.append(sim)

    # High relevance = generated questions similar to original
    return mean(similarities)

Observable outcome: Check each claim against context 3.

Stage 4 — Visualizing the Evaluation Pipeline

• The evaluation happens in parallel with answer generation, providing real-time quality gates and continuous monitoring data.

graph TD
    A[User Query] --> B[RAG Retrieval]
    B --> C[Retrieved Documents]
    C --> D[LLM Generation]
    D --> E[Generated Answer]

    E --> F[Evaluation Framework]
    C --> F
    A --> F

    F --> G[Faithfulness Check]
    F --> H[Relevance Check]  
    F --> I[Context Quality Check]

    G --> J[Evaluator LLM]
    H --> J
    I --> J

    J --> K[Evaluation Scores]
    K --> L{Quality Gate}

    L -->|Pass| M[Return Answer to User]
    L -->|Fail| N[Fallback Response]

    K --> O[Monitoring Dashboard]
    O --> P[Alert if Quality Drops]

Observable outcome: The evaluation happens in parallel with answer generation, providing real-time quality gates and continuous monitoring data.

Stage 5 — Real-World Applications: RAGAS, DeepEval, TruLens in Production

• Case Study 1: E-commerce RAG with RAGAS An online retailer uses RAG to answer product questions.
• Their evaluation pipeline: Input: "Is this laptop good for gaming?" Retrieved Context: Product specs, reviews, gaming performance data Generated Answer: "Yes, with an RTX 4070 and 32GB RAM, this laptop handles modern ga…
• Case Study 2: Legal Document Analysis with DeepEval A law firm uses LLMs to analyze contracts.
• Their unit-test-style evaluation: Production Impact: Prevented 3 cases where the LLM missed critical liability clauses by maintaining 0.9 faithfulness scores across their evaluation suite.

# DeepEval test case
@pytest.mark.llm_eval
def test_contract_analysis_quality():
    contract_text = load_contract("sample.pdf")
    analysis = llm.analyze_contract(contract_text)

    # Assert semantic quality thresholds
    assert_faithfulness(analysis, contract_text, threshold=0.9)
    assert_no_hallucination(analysis, contract_text)  
    assert_contains_key_terms(analysis, ["liability", "termination", "payment"])

Observable outcome: Their unit-test-style evaluation: Production Impact: Prevented 3 cases where the LLM missed critical liability clauses by maintaining 0.9 faithfulness scores across their evaluation suite.

Stage 6 — RAGAS in Practice: Evaluating a RAG Pipeline End-to-End

• Let's build a complete evaluation pipeline using RAGAS to assess a customer support RAG system: Sample Output: Integrating with CI/CD: This creates an automated quality gate that prevents deploying RAG systems with degr…

# Installation
# pip install ragas langchain chromadb openai

import os
from langchain.document_loaders import TextLoader
from langchain.text_splitter import CharacterTextSplitter  
from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_recall, context_precision
from datasets import Dataset

# Set up OpenAI API key
os.environ["OPENAI_API_KEY"] = "your-api-key-here"

# Step 1: Create a simple RAG pipeline
def create_rag_pipeline():
    # Load and chunk documents
    loader = TextLoader("customer_support_docs.txt")
    documents = loader.load()

    text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
    texts = text_splitter.split_documents(documents)

    # Create vector store
    embeddings = OpenAIEmbeddings()
    vectorstore = Chroma.from_documents(texts, embeddings)

    # Create RAG chain
    llm = OpenAI(temperature=0)
    rag_chain = RetrievalQA.from_chain_type(
        llm=llm,
        chain_type="stuff", 
        retriever=vectorstore.as_retriever(search_kwargs={"k": 3})
    )

    return rag_chain, vectorstore

# Step 2: Create evaluation dataset
def create_evaluation_dataset():
    """Create a dataset with questions, ground truth, and retrieved contexts"""

    evaluation_data = {
        "question": [
            "What is your return policy?",
            "How do I track my order?", 
            "What payment methods do you accept?",
            "How do I cancel my subscription?",
            "What's your customer service phone number?"
        ],
        "ground_truths": [
            ["We accept returns within 30 days of purchase with original receipt."],
            ["You can track orders using the tracking number sent via email."],
            ["We accept credit cards, PayPal, and bank transfers."],  
            ["Subscriptions can be canceled in your account settings or by contacting support."],
            ["Our customer service number is 1-800-SUPPORT (1-800-786-7678)."]
        ]
    }

    return evaluation_data

# Step 3: Generate answers and retrieve contexts
def generate_rag_responses(rag_chain, vectorstore, questions):
    """Generate answers and get retrieved contexts for evaluation"""

    answers = []
    contexts = []

    for question in questions:
        # Generate answer
        result = rag_chain({"query": question})
        answers.append(result["result"])

        # Get retrieved contexts
        retrieved_docs = vectorstore.similarity_search(question, k=3)
        context_list = [doc.page_content for doc in retrieved_docs]
        contexts.append(context_list)

        print(f"Q: {question}")
        print(f"A: {result['result']}")
        print(f"Retrieved {len(context_list)} context chunks")
        print("-" * 50)

    return answers, contexts

# Step 4: Run RAGAS evaluation
def run_ragas_evaluation(questions, answers, contexts, ground_truths):
    """Evaluate the RAG pipeline using RAGAS metrics"""

    # Create evaluation dataset
    eval_dataset = Dataset.from_dict({
        "question": questions,
        "answer": answers, 
        "contexts": contexts,
        "ground_truths": ground_truths
    })

    # Define metrics to evaluate
    metrics = [
        faithfulness,        # Does answer stay true to retrieved context?
        answer_relevancy,    # Does answer address the question?  
        context_recall,      # Did retrieval find all relevant info?
        context_precision,   # How much retrieved context was useful?
    ]

    # Run evaluation
    print("Running RAGAS evaluation...")
    result = evaluate(eval_dataset, metrics=metrics)

    return result

# Step 5: Analyze results and set quality gates
def analyze_evaluation_results(result):
    """Analyze RAGAS scores and determine if quality gates pass"""

    print("\n🔍 RAGAS Evaluation Results:")
    print("=" * 40)

    # Quality thresholds
    quality_gates = {
        "faithfulness": 0.8,      # 80% faithfulness minimum
        "answer_relevancy": 0.75,  # 75% relevance minimum  
        "context_recall": 0.7,     # 70% recall minimum
        "context_precision": 0.6   # 60% precision minimum
    }

    all_passed = True

    for metric, score in result.items():
        threshold = quality_gates.get(metric, 0.0)
        status = "✅ PASS" if score >= threshold else "❌ FAIL" 

        if score < threshold:
            all_passed = False

        print(f"{metric:20}: {score:.3f} (threshold: {threshold:.1f}) {status}")

    print("\n" + "=" * 40)
    overall_status = "✅ All quality gates passed!" if all_passed else "❌ Quality gates failed - investigate!"
    print(f"Overall Status: {overall_status}")

    return all_passed

# Main execution
def main():
    # Create RAG pipeline
    print("Setting up RAG pipeline...")
    rag_chain, vectorstore = create_rag_pipeline()

    # Create evaluation dataset  
    eval_data = create_evaluation_dataset()
    questions = eval_data["question"]
    ground_truths = eval_data["ground_truths"]

    # Generate responses
    print("\nGenerating RAG responses...")
    answers, contexts = generate_rag_responses(rag_chain, vectorstore, questions)

    # Run RAGAS evaluation
    print("\nRunning RAGAS evaluation...")
    result = run_ragas_evaluation(questions, answers, contexts, ground_truths)

    # Analyze results
    quality_passed = analyze_evaluation_results(result)

    # Production decision
    if quality_passed:
        print("\n🚀 RAG pipeline ready for production!")
    else:
        print("\n🔧 RAG pipeline needs improvement before deployment.")
        print("   - Check document retrieval quality")  
        print("   - Tune chunk size and overlap")
        print("   - Improve prompt engineering")
        print("   - Consider different embedding models")

if __name__ == "__main__":
    main()

🔍 RAGAS Evaluation Results:
========================================
faithfulness        : 0.857 (threshold: 0.8) ✅ PASS
answer_relevancy     : 0.923 (threshold: 0.8) ✅ PASS  
context_recall       : 0.743 (threshold: 0.7) ✅ PASS
context_precision    : 0.681 (threshold: 0.6) ✅ PASS

========================================
Overall Status: ✅ All quality gates passed!

🚀 RAG pipeline ready for production!

Observable outcome: Let's build a complete evaluation pipeline using RAGAS to assess a customer support RAG system: Sample Output: Integrating with CI/CD: This creates an automated quality gate that prevents deploying RAG systems with degr…

End-to-end outcome

After working through the stages, you should be able to explain:

• Key Insights from LLM Evaluation in Production 1.
• Evaluation is Not Optional Anymore Unlike traditional software where bugs are obvious, LLM quality degradation is subtle.
• • Traditional ML metrics fail for LLMs because there's no single correct answer—cosine similarity measures retrieval but not generation quality • The three pillars of LLM evaluation are faithfulness (grounding in contex…
• A consistent 0.8 faithfulness is better than an inconsistent 0.9.

Alternatives considered

Decision Guide: Choosing the Right Framework

Decision Guide: Choosing the Right Framework

Failure and degraded modes

Performance vs Cost Trade-offs**

• Detection: High-Frequency Evaluation (Real-time): - ✅ Catch issues immediately - ❌ 10x higher API costs - ❌ Adds 200-500ms latency Batch Evaluation (Daily/Weekly): - ✅ Lower cost, no user latency - ❌ Issues go…
• User or operator impact: High-Frequency Evaluation (Real-time): - ✅ Catch issues immediately - ❌ 10x higher API costs - ❌ Adds 200-500ms latency Batch Evaluation (Daily/Weekly): - ✅ Lower cost, no user latency - ❌ Issues go…

Common Failure Modes**

• Detection: Evaluator Bias: The judge LLM has systematic biases 2.
• User or operator impact: Evaluator Bias: The judge LLM has systematic biases 2.

Mitigation Strategies**

• Detection: Multi-Evaluator Consensus: Use 2-3 different LLMs and take the median score: Human Validation Loop: Randomly sample 1-5% of evaluations for human review to catch systematic evaluator errors.
• User or operator impact: Multi-Evaluator Consensus: Use 2-3 different LLMs and take the median score: Human Validation Loop: Randomly sample 1-5% of evaluations for human review to catch systematic evaluator errors.