Context Window Management: Strategies for Long Documents and Extended Conversations

Scenario context

🧠 Context windows are LLM memory limits. When conversations grow past 4K-128K tokens, you need strategies: sliding windows (cheap, lossy), summarization (balanced), RAG (selective), map-reduce (scalable), or selective memory (precise). LangChain provides ConversationSummaryMemory and ConversationTokenBufferMemory for production use.

Your customer service bot works great for the first 3-4 conversational tur…

This walkthrough focuses on The $0.08 Problem: When Context Windows Become Budget Killers.

Before making a design choice, clarify:

• Your customer service bot works great for the first 3-4 conversational turns.
• The customer explains their billing issue, the bot asks clarifying questions, and everything flows smoothly.
• By turn 10, however, the bot has completely forgotten the customer's original problem and starts asking the same questions again.

Concepts in play: llm-prompt-engineering, core-algorithm-patterns, vector-databases-rag, llm-agents-and-langgraph

Stage 1 — Token Economics: The Hidden Cost of Long Conversations

• Before exploring strategies, you need to understand the economics.
• Context window management isn't just a technical problem—it's a cost optimization problem.
• A single conversation turn might seem cheap, but tokens add up: - User message: "My billing statement shows a charge I don't recognize" = 12 tokens - Assistant response: "I'd be happy to help..." (typical 150-word respo…
• Add system prompts, retrieved documents, and examples, and you're approaching the 4K limit of smaller models.

import tiktoken

def count_tokens(text, model="gpt-4"):
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

conversation = [
    {"role": "user", "content": "My billing statement shows a charge I don't recognize"},
    {"role": "assistant", "content": "I'd be happy to help you identify that charge..."}
]

total_tokens = sum(count_tokens(msg["content"]) for msg in conversation)
print(f"Conversation tokens: {total_tokens}")
# Output: Conversation tokens: 212

Observable outcome: Add system prompts, retrieved documents, and examples, and you're approaching the 4K limit of smaller models.

Stage 2 — Strategy 1: Sliding Window - The Brutally Simple Approach

• The sliding window strategy is context management's equivalent of a FIFO queue: keep the most recent N messages and discard the rest.
• When you hit your token limit, drop the oldest messages until you're under budget.
• Maintain a conversation buffer 2.
• Before each API call, count total tokens 3.

class SlidingWindowMemory:
    def __init__(self, max_tokens=3000, model="gpt-4"):
        self.max_tokens = max_tokens
        self.messages = []
        self.encoding = tiktoken.encoding_for_model(model)

    def add_message(self, role, content):
        self.messages.append({"role": role, "content": content})
        self._truncate_if_needed()

    def _count_tokens(self, messages):
        return sum(len(self.encoding.encode(msg["content"])) for msg in messages)

    def _truncate_if_needed(self):
        while len(self.messages) > 1 and self._count_tokens(self.messages) > self.max_tokens:
            # Keep system message, remove oldest user/assistant pair
            if self.messages[1]["role"] == "user":
                self.messages.pop(1)  # Remove user message
                if len(self.messages) > 1 and self.messages[1]["role"] == "assistant":
                    self.messages.pop(1)  # Remove corresponding assistant message
            else:
                self.messages.pop(1)

    def get_messages(self):
        return self.messages

Observable outcome: Before each API call, count total tokens 3.

Stage 3 — Strategy 2: Summarization-Based Compression

• Instead of dropping old messages, summarization compresses them.
• When approaching token limits, use the LLM to create a condensed summary of early conversation turns, then replace the original messages with this summary.
• Detect when approaching token limit (e.g., 80% of max) 2.
• Extract older messages for summarization 3.

class SummarizationMemory:
    def __init__(self, max_tokens=3000, summarize_threshold=0.8):
        self.max_tokens = max_tokens
        self.threshold = int(max_tokens * summarize_threshold)
        self.messages = []
        self.summary = ""

    def add_message(self, role, content):
        self.messages.append({"role": role, "content": content})
        if self._count_tokens() > self.threshold:
            self._summarize_and_compress()

    def _summarize_and_compress(self):
        # Take first half of messages for summarization
        split_point = len(self.messages) // 2
        messages_to_summarize = self.messages[:split_point]

        # Generate summary (this would call your LLM)
        new_summary = self._generate_summary(messages_to_summarize)

        # Update state
        self.summary += f"\nPrevious conversation: {new_summary}"
        self.messages = self.messages[split_point:]

Observable outcome: Extract older messages for summarization 3.

Stage 4 — Strategy 3: RAG Instead of Stuffing

• Rather than cramming entire documents or conversation histories into context, use Retrieval-Augmented Generation (RAG) to pull in only the relevant pieces for each query.
• The RAG approach for conversations: 1.
• Store all conversation turns in a vector database 2.
• For each new user input, retrieve semantically similar past exchanges 3.

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter

class RAGContextManager:
    def __init__(self):
        self.embeddings = OpenAIEmbeddings()
        self.vectorstore = Chroma(embedding_function=self.embeddings)
        self.conversation_history = []

    def add_conversation_turn(self, user_msg, assistant_msg):
        # Store conversation turn for future retrieval
        turn = f"User: {user_msg}\nAssistant: {assistant_msg}"
        self.vectorstore.add_texts([turn])
        self.conversation_history.append({"user": user_msg, "assistant": assistant_msg})

    def get_relevant_context(self, query, k=3):
        # Retrieve most relevant past exchanges
        relevant_docs = self.vectorstore.similarity_search(query, k=k)
        return [doc.page_content for doc in relevant_docs]

    def build_context_aware_prompt(self, current_query):
        relevant_context = self.get_relevant_context(current_query)
        context_section = "\n".join([f"Previous relevant exchange:\n{ctx}" 
                                    for ctx in relevant_context])

        return f"""
        {context_section}

        Current user query: {current_query}

        Please respond based on the relevant context above.
        """

Observable outcome: For each new user input, retrieve semantically similar past exchanges 3.

Stage 5 — Strategy 4: Map-Reduce for Long Document Processing

• When dealing with documents too large for any context window, map-reduce breaks them into manageable chunks, processes each chunk independently, then combines results.
• Split : Divide large document into overlapping chunks 2.
• Map : Process each chunk with a focused prompt 3.
• Reduce : Combine chunk results into final output For a 50,000-word research paper, you might: 1.

from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import ReduceDocumentsChain, MapReduceDocumentsChain
from langchain.chains.llm import LLMChain
from langchain.prompts import PromptTemplate

def create_map_reduce_chain(llm):
    # Map phase: summarize each chunk
    map_template = """
    Summarize the following text in 2-3 sentences, focusing on key insights:
    {text}

    Summary:"""

    map_prompt = PromptTemplate(template=map_template, input_variables=["text"])
    map_chain = LLMChain(llm=llm, prompt=map_prompt)

    # Reduce phase: combine summaries
    reduce_template = """
    Combine the following summaries into a coherent overview:
    {text}

    Combined summary:"""

    reduce_prompt = PromptTemplate(template=reduce_template, input_variables=["text"])
    reduce_chain = LLMChain(llm=llm, prompt=reduce_prompt)

    # Chain them together
    reduce_documents_chain = ReduceDocumentsChain(
        llm_chain=reduce_chain,
        document_variable_name="text"
    )

    return MapReduceDocumentsChain(
        llm_chain=map_chain,
        reduce_documents_chain=reduce_documents_chain,
        document_variable_name="text"
    )

Observable outcome: Reduce : Combine chunk results into final output For a 50,000-word research paper, you might: 1.

Stage 6 — Strategy 5: Selective Memory - Entity Extraction and Structured State

• The most sophisticated approach maintains structured state about important entities, relationships, and facts, updating this state as conversations progress.
• Instead of keeping raw conversation history, extract and maintain semantic knowledge.
• Selective memory components: 1.
• Entity extraction : Identify people, places, products, issues 2.

class SelectiveMemory:
    def __init__(self):
        self.entities = {}  # {entity_id: {type, attributes, mentions}}
        self.relationships = {}  # {entity1_id: {entity2_id: relationship_type}}
        self.facts = {}  # {fact_id: {statement, confidence, last_updated}}

    def extract_and_update(self, message):
        # This would use NER and relation extraction
        entities = self._extract_entities(message)
        relationships = self._extract_relationships(message)
        facts = self._extract_facts(message)

        for entity in entities:
            self._update_entity(entity)

        for rel in relationships:
            self._update_relationship(rel)

        for fact in facts:
            self._update_fact(fact)

    def get_relevant_context(self, query):
        # Reconstruct context from structured knowledge
        relevant_entities = self._find_relevant_entities(query)
        context = self._build_context_from_entities(relevant_entities)
        return context

    def _build_context_from_entities(self, entities):
        context_items = []
        for entity_id in entities:
            entity = self.entities[entity_id]
            context_items.append(f"{entity['type']} {entity['name']}: {entity['summary']}")
        return "\n".join(context_items)

Observable outcome: Entity extraction : Identify people, places, products, issues 2.

End-to-end outcome

After working through the stages, you should be able to explain:

• Key insights from deploying context window strategies at scale: 1.
• Token counting is non-negotiable : Estimate costs before implementation.

Alternatives considered

Decision Guide: Choosing Your Context Strategy

Decision Guide: Choosing Your Context Strategy

Failure and degraded modes

Trade-offs & Failure Modes: When Strategies Break

• Detection: Common Failure Modes & Mitigations : Cost vs.
• User or operator impact: Common Failure Modes & Mitigations : Cost vs.