LLM Observability: Tracing, Logging, and Debugging Production AI Systems

Scenario context

🔍 LLM observability is radically different from traditional APM—non-deterministic outputs, variable token costs, and multi-step reasoning chains require specialized tracing. LangSmith provides native LangChain integration, OpenTelemetry offers standardization, and LangFuse delivers open-source flexibility. The key: instrument every prompt, capture every token, and alert on cost spikes before they hit your budget. -…

This walkthrough focuses on The LLM Observability Challenge: Why Traditional APM Falls Short.

Before making a design choice, clarify:

• Traditional monitoring assumes deterministic systems.
• Give the same input, get the same output, measure latency and error rates.
• LLMs shatter these assumptions: Non-deterministic outputs mean the same prompt can produce different responses based on temperature settings, model state, or even cosmic rays.

Concepts in play: llm-agents-and-langgraph, api-gateway, llm-prompt-engineering, sorting-and-searching-algorithms

Stage 1 — The Three Pillars of LLM Observability

• LLM observability rests on three pillars that extend traditional monitoring: Traces: Full Prompt-to-Response Journeys Unlike HTTP request traces, LLM traces capture: - Prompt templates with variable substitution - Conte…
• Metrics: Token-Aware Performance Indicators Key metrics specific to LLMs: Logs: Structured Events with LLM Context Traditional logs capture HTTP status codes.
• LLM logs need: - Prompt construction events - Model selection and routing decisions - Context retrieval results with relevance scores - Tool execution outcomes - Output validation failures Each log entry includes the fu…

Observable outcome: LLM logs need: - Prompt construction events - Model selection and routing decisions - Context retrieval results with relevance scores - Tool execution outcomes - Output validation failures Each log entry includes the fu…

Stage 2 — How LangSmith Traces Every Token and Decision in Your AI Pipeline

• LangSmith is LangChain's native observability platform, designed specifically for LLM applications.
• It captures the full execution graph of chains, agents, and tools with zero configuration.
• Automatic Instrumentation LangSmith automatically traces all LangChain components: LangSmith captures: - Prompt template with variables - Token counts (input and output) - Model parameters (temperature, max tokens) - Ex…

import os
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import ChatPromptTemplate
from langsmith import traceable

# Set LangSmith API key - no other config needed
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your_langsmith_key"

# This chain is automatically traced
llm = ChatOpenAI(temperature=0.7)
prompt = ChatPromptTemplate.from_template(
    "You are a helpful assistant. Answer this question: {question}"
)
chain = LLMChain(llm=llm, prompt=prompt)

# Every invoke() call creates a trace
response = chain.invoke({"question": "What is quantum computing?"})

from langsmith import traceable
import requests

@traceable
def fetch_context_documents(query: str, top_k: int = 5) -> list:
    """Retrieve relevant documents for RAG context."""
    # Custom retrieval logic
    response = requests.post("http://vector-db:8000/search", {
        "query": query,
        "limit": top_k
    })

    docs = response.json()["documents"]

    # LangSmith automatically captures inputs, outputs, and metadata
    return docs

@traceable  
def synthesize_response(question: str, context_docs: list) -> str:
    """Generate response using retrieved context."""
    context = "\n".join([doc["content"] for doc in context_docs])

    prompt = f"""
    Context: {context}

    Question: {question}

    Answer based only on the context provided.
    """

    # This LLM call is also traced automatically
    llm = ChatOpenAI(temperature=0.1)
    response = llm.invoke(prompt)

    return response.content

# Usage creates nested traces
def answer_question(user_question: str):
    docs = fetch_context_documents(user_question)
    answer = synthesize_response(user_question, docs)
    return answer

Observable outcome: Automatic Instrumentation LangSmith automatically traces all LangChain components: LangSmith captures: - Prompt template with variables - Token counts (input and output) - Model parameters (temperature, max tokens) - Ex…

Stage 3 — Deep Dive: OpenTelemetry for Standardized LLM Instrumentation

• The Internals OpenTelemetry (OTel) provides vendor-neutral observability for LLM applications through standardized spans and metrics.
• The LLM semantic conventions define specific attributes for AI workloads: Span Structure: Memory Layout: OTel stores traces in memory as linked span objects, with each span containing attributes, events, and links to pa…
• When memory limits are reached, spans are exported to backends like Jaeger or DataDog.
• State Management: The tracer maintains active span context using thread-local storage or async context variables, enabling automatic parent-child relationships across async operations.

operation.name: "llm.completion"
llm.vendor: "openai" 
llm.model.name: "gpt-4"
llm.model.version: "2024-02-15-preview"
llm.temperature: 0.7
llm.max_tokens: 1000
llm.token_count.prompt: 156
llm.token_count.completion: 89  
llm.latency.time_to_first_token: 1.2s

Observable outcome: State Management: The tracer maintains active span context using thread-local storage or async context variables, enabling automatic parent-child relationships across async operations.

Stage 4 — Visualizing LLM Request Flows with Distributed Tracing

• LLM applications create complex execution graphs that traditional request tracing can't capture.
• Here's how a typical RAG agent execution looks with proper instrumentation: Each node in this graph becomes a span in your distributed trace, with the following key attributes:

graph TD
    A[User Question] --> B[Intent Classification]
    B --> C[Vector Search]
    B --> D[Knowledge Graph Query]  
    C --> E[Document Ranking]
    D --> F[Entity Resolution]
    E --> G[Context Assembly]
    F --> G
    G --> H[Prompt Construction]
    H --> I[LLM Generation]
    I --> J[Output Validation]
    J --> K{Valid Response?}
    K -->|Yes| L[Response Formatting]
    K -->|No| M[Fallback Generation]
    M --> N[Secondary LLM Call]
    N --> L
    L --> O[User Response]

    style B fill:#e1f5fe
    style C fill:#e8f5e8  
    style D fill:#e8f5e8
    style I fill:#fff3e0
    style N fill:#fff3e0
    style J fill:#fce4ec

Observable outcome: Here's how a typical RAG agent execution looks with proper instrumentation: Each node in this graph becomes a span in your distributed trace, with the following key attributes:

Stage 5 — Real-World Debugging: From Prompt Drift to Cost Explosions

• Case Study 1: The Hallucinating Support Bot Situation: A customer support chatbot started giving incorrect refund policies, causing compliance issues.
• Investigation with LangSmith: 1.
• Trace Analysis: Filtered traces by "refund" keyword, found 23% contained hallucinated policy details 2.
• Prompt Inspection: Discovered the knowledge base retrieval was returning outdated documents 3.

@traceable
def validate_context_quality(retrieved_docs: list, threshold: float = 0.6):
    avg_score = sum(doc["relevance_score"] for doc in retrieved_docs) / len(retrieved_docs)

    if avg_score < threshold:
        # Log warning and trigger alert
        logger.warning(f"Context quality below threshold: {avg_score:.2f}")
        send_slack_alert(f"RAG context quality degraded to {avg_score:.2f}")

    return avg_score

# Cost tracking with OTel metrics
from opentelemetry import metrics

meter = metrics.get_meter(__name__)
cost_counter = meter.create_counter(
    "llm_cost_total_usd",
    description="Total LLM API costs"
)
token_histogram = meter.create_histogram(
    "llm_tokens_per_request", 
    description="Token usage distribution"
)

def track_llm_costs(model_name: str, input_tokens: int, output_tokens: int):
    # OpenAI GPT-4 pricing: $0.03/1K input, $0.06/1K output  
    input_cost = (input_tokens / 1000) * 0.03
    output_cost = (output_tokens / 1000) * 0.06
    total_cost = input_cost + output_cost

    # Record metrics with attributes
    cost_counter.add(total_cost, {"model": model_name, "cost_type": "api_usage"})
    token_histogram.record(input_tokens + output_tokens, {"model": model_name})

Observable outcome: Prompt Inspection: Discovered the knowledge base retrieval was returning outdated documents 3.

Stage 6 — Practical Examples: Instrumenting a LangChain Agent with LangSmith

• Example 1: Customer Support Agent with Tool Usage LangSmith Dashboard Output: - Agent Execution Trace: Shows the reasoning steps, tool calls, and final response - Tool Usage Metrics: Number of API calls, response times,…

import os
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.chat_models import ChatOpenAI
from langchain.prompts import ChatPromptTemplate  
from langchain.tools import Tool
from langsmith import traceable
import requests

# Enable LangSmith tracing
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "your_langsmith_key"
os.environ["LANGCHAIN_PROJECT"] = "customer-support-agent"

@traceable
def search_knowledge_base(query: str) -> str:
    """Search internal knowledge base for support articles."""
    response = requests.post("http://kb-api:8000/search", {
        "query": query,
        "max_results": 3
    })
    articles = response.json()["articles"]

    # LangSmith captures this function's inputs/outputs automatically
    return "\n".join([f"Article: {a['title']}\n{a['content']}" for a in articles])

@traceable  
def get_order_status(order_id: str) -> str:
    """Fetch current order status from order management system."""
    response = requests.get(f"http://orders-api:8000/orders/{order_id}")
    order = response.json()

    return f"Order {order_id}: Status = {order['status']}, Expected delivery = {order['delivery_date']}"

# Define tools for the agent
tools = [
    Tool(
        name="search_knowledge_base",
        description="Search support articles and documentation",
        func=search_knowledge_base
    ),
    Tool(
        name="get_order_status", 
        description="Get current status of customer orders by order ID",
        func=get_order_status
    )
]

# Create the agent
llm = ChatOpenAI(temperature=0.1, model="gpt-4")
prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a helpful customer support agent. Use the available tools to help customers.

    Available tools:
    - search_knowledge_base: Find relevant support articles
    - get_order_status: Check order status by order ID

    Always be polite and provide specific, actionable information."""),
    ("human", "{input}"),
    ("assistant", "{agent_scratchpad}")
])

agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

# Usage - all steps automatically traced in LangSmith
def handle_support_request(customer_message: str, customer_id: str):
    with traceable(
        name="support_request",
        metadata={"customer_id": customer_id}
    ):
        response = agent_executor.invoke({
            "input": customer_message
        })
        return response["output"]

# Example usage
result = handle_support_request(
    "Hi, I need help with my order #12345. It was supposed to arrive yesterday.",
    customer_id="cust_789"
)
print(result)

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langsmith import traceable
import numpy as np

class ObservableRAG:
    def __init__(self, vector_store_path: str):
        self.embeddings = OpenAIEmbeddings()
        self.vectorstore = Chroma(
            persist_directory=vector_store_path,
            embedding_function=self.embeddings
        )
        self.llm = ChatOpenAI(temperature=0.1, model="gpt-4")

    @traceable
    def retrieve_with_quality_check(self, query: str, k: int = 5) -> dict:
        """Retrieve documents and assess context quality."""

        # Get similarity search results with scores
        docs_with_scores = self.vectorstore.similarity_search_with_score(query, k=k)

        # Calculate context quality metrics
        scores = [score for _, score in docs_with_scores]
        avg_similarity = np.mean(scores)  
        min_similarity = min(scores)
        score_variance = np.var(scores)

        docs = [doc for doc, _ in docs_with_scores]

        # Log quality metrics for monitoring
        context_quality = {
            "avg_similarity": avg_similarity,
            "min_similarity": min_similarity, 
            "score_variance": score_variance,
            "num_docs_retrieved": len(docs),
            "query_length": len(query.split())
        }

        # Alert on poor context quality
        if avg_similarity < 0.6:
            self._log_quality_alert("Low average similarity", context_quality)

        if min_similarity < 0.3:
            self._log_quality_alert("Very low minimum similarity", context_quality)

        return {
            "documents": docs,
            "quality_metrics": context_quality,
            "similarity_scores": scores
        }

    @traceable
    def _log_quality_alert(self, alert_type: str, metrics: dict):
        """Log context quality alerts for monitoring."""
        print(f"ALERT: {alert_type} - Metrics: {metrics}")
        # In production, send to monitoring system

    @traceable
    def generate_answer(self, query: str) -> dict:
        """Generate answer with full observability."""

        # Step 1: Retrieve and assess context
        retrieval_result = self.retrieve_with_quality_check(query)
        docs = retrieval_result["documents"]
        quality = retrieval_result["quality_metrics"]

        # Step 2: Build context-aware prompt
        context = "\n\n".join([doc.page_content for doc in docs])

        prompt = f"""
        Use the following context to answer the question. If the context doesn't contain 
        relevant information, say so explicitly.

        Context:
        {context}

        Question: {query}

        Answer:
        """

        # Step 3: Generate response
        response = self.llm.invoke(prompt)

        # Step 4: Return with full metadata
        return {
            "answer": response.content,
            "context_quality": quality,
            "num_context_chars": len(context),
            "source_documents": len(docs)
        }

# Usage with automatic LangSmith tracing
rag = ObservableRAG("./chroma_db")

result = rag.generate_answer("What are the key principles of microservices architecture?")
print(f"Answer: {result['answer']}")
print(f"Context Quality: {result['context_quality']}")

Observable outcome: Example 1: Customer Support Agent with Tool Usage LangSmith Dashboard Output: - Agent Execution Trace: Shows the reasoning steps, tool calls, and final response - Tool Usage Metrics: Number of API calls, response times,…

End-to-end outcome

After working through the stages, you should be able to explain:

• Key Insights from Real Deployments Token Budget Monitoring is Critical: One startup burned through their entire monthly OpenAI budget in 3 days because a recursive agent got stuck in a reasoning loop.
• Always implement token limits per session.
• • LLM observability is fundamentally different: Non-deterministic outputs, variable costs, and multi-step reasoning require specialized tracing beyond traditional APM - Three pillars matter most: Traces (full prompt-res…

Alternatives considered

Decision Guide: Choosing Your LLM Observability Stack

Decision Guide: Choosing Your LLM Observability Stack

LangFuse provides comprehensive LLM observability without vendor lock-in, offering features comparable to LangSmith with full self-hosting capabilities.

LangFuse provides comprehensive LLM observability without vendor lock-in, offering features comparable to LangSmith with full self-hosting capabilities.

Key Features:

• Distributed tracing for multi-step LLM workflows - Cost tracking with granular token-level attribution - A/B testing and evaluation workflows - Dataset management for prompt engineering - Team collaboration with shared dashboards

Self-Hosted Setup:

Self-Hosted Setup:

Cost Analysis Dashboard:

LangFuse vs. LangSmith Comparison:

Failure and degraded modes

Performance vs. Vendor Lock-in

• Detection: LangSmith provides the richest LLM-specific features but locks you into the LangChain ecosystem: Pros: - Zero-config automatic instrumentation for LangChain - LLM-specific UI for prompt debugging and…
• User or operator impact: LangSmith provides the richest LLM-specific features but locks you into the LangChain ecosystem: Pros: - Zero-config automatic instrumentation for LangChain - LLM-specific UI for prompt debugging and…

Correctness vs. Cost Trade-offs

• Detection: Comprehensive tracing captures every prompt and response, but storage costs scale with volume: Failure Modes: - Sampling bias: Critical edge cases missed in sampled data - Storage explosion: Full pro…
• User or operator impact: Comprehensive tracing captures every prompt and response, but storage costs scale with volume: Failure Modes: - Sampling bias: Critical edge cases missed in sampled data - Storage explosion: Full pro…