How AI Coding Agents Work: Models, Context, Sessions, and Memory

A refactor forgets your rules when the context window overflows

You are 45 minutes into a payment refactor with an AI coding agent. You established the ground rules: use the PaymentProcessor interface, never call LegacyBillingClient, and make every new service implement Retryable. The agent acknowledged this, generated StripePaymentProcessor, and started PayPalPaymentProcessor.

Then the context window filled up.

The next response imports LegacyBillingClient — the class you forbade — and ignores Retryable entirely. The agent did not get worse. It literally never saw your earlier instructions. Those tokens were evicted when the window overflowed.

That failure is architectural, not accidental. Every coding agent is a stateless model plus an orchestration layer that rebuilds context on every API call. The sections below follow one realistic task from user message through tool calls, context assembly, and session limits so you can predict — and mitigate — behaviors like the one above.

Stage 1 — Separate the language model from the coding agent

A plain LLM such as GPT-4o or Claude 3.5 Sonnet is a stateless text transformer: tokens in, tokens out. It cannot read your repository, run tests, or remember what you said three turns ago unless the orchestrator puts that information back into the prompt.

An AI coding agent adds three layers on top of the model:

1. Tool registry — read_file, edit_file, run_command, search_codebase, and similar actions the model can request but never execute directly.
2. Orchestration loop — logic that parses model output, runs tools, injects results, and repeats until the task is complete.
3. Context manager — code that decides which files, history, and instructions fit inside the finite token window for each call.

llm-agents-and-langgraph covers the orchestration pattern: the model plans, the runtime acts, observations return to context, and the loop continues.

Copilot, Cursor, Claude Code, Aider, and Devin differ mainly in how aggressively they use RAG, how large a context window they load, and how deep their tool registries go — not in whether the underlying model is stateless.

`` Dev["Developer / User Prompt"] --> Orch["Orchestration Loop (ReAct / Event Loop)"] subgraph AgentRuntime ["Agent Runtime Architecture"] Orch <--> Tools["Tool Registry (read_file, edit_file, terminal, search)"] Orch <--> Ctx["Context Manager (Window Budgeting & RAG Retrieval)"] end Orch <--> Model["Stateless Foundation LLM (Claude 3.5 Sonnet / GPT-4o)"]

**Observable outcome:** You can name the three agent layers and explain why swapping models does not change statelessness.

## Stage 2 — Budget every turn in tokens, not words

Three facts explain most confusing agent behavior:

- **Tokens** are the billing and limit unit. Roughly 1,000 tokens ≈ 750 English words ≈ 50–80 lines of code.
- **Context window** is everything the model can attend to in one call. Content outside it does not exist to the model.
- **Statelessness** means each API call is independent. Follow-up messages are new requests that include serialized prior turns.

When you send turn 20 of a refactor, the orchestrator is not "continuing a chat." It is constructing a fresh prompt that happens to contain the text of earlier turns — until truncation removes them.

**llm-prompt-engineering** applies here because prompt assembly is deliberate engineering: what you include, omit, and prioritize directly changes output quality.

``
    UserMsg["User sends message"] --> Orchestrator["Orchestrator receives message"]
    Orchestrator --> LoadCache["Load prior turns from session cache"]
    LoadCache --> RAGQuery["Query vector store for relevant source files"]
    RAGQuery --> Assemble["Assemble context window from prompt, history, and files"]
    Assemble --> WindowCheck{"Context fits in token window?"}
    WindowCheck -->|Yes| CallLLM["Send complete context to LLM API as new stateless request"]
    WindowCheck -->|No| Evict["Evict or summarize oldest conversation turns"]
    Evict --> CallLLM
    CallLLM --> Parse["Parse LLM response"]
    Parse -->|Tool call detected| RunTool["Execute tool and capture result"]
    RunTool --> Inject["Inject tool_result message into context"]
    Inject --> CallLLM
    Parse -->|Final answer| Respond["Return answer to user"]
    Respond --> PersistTurn["Persist turn to session cache for next request"]

Observable outcome: You can explain why a forgotten constraint is almost always a context eviction event, not model degradation.

Stage 3 — Run the ReAct loop until the task is done

Non-trivial coding tasks use a Reason → Act → Observe loop. A single user request often triggers several LLM calls before a final answer.

`` Start([Task received]) --> Reason[Reason about current state] Reason --> Decide{Tool call needed?} Decide -->|Yes| Pick[Select tool and arguments] Pick --> Exec[Execute tool] Exec --> Observe[Observe tool result] Observe --> Inject[Inject result into context] Inject --> Reason Decide -->|No| Generate[Generate final response] Generate --> Done([Return to user])

Example: "Add retry logic to `uploadFile`."

1. **Reason:** Need the current `uploadFile` implementation → **Act:** `read_file("src/storage/FileUploader.java")`.
2. **Observe:** File content enters context.
3. **Reason:** Check whether `RetryPolicy` already exists → **Act:** `search_codebase("RetryPolicy")`.
4. **Observe:** `RetryPolicy.java` found in `src/core/`.
5. **Generate:** Emit the diff.

A realistic pagination task — "Add cursor-based pagination to `GET /users`" — often needs four LLM calls: assemble context, read repository file, apply edit, run tests, observe failure, fix DTO annotation, rerun tests. The agent is not one generation; it is a chain of stateless calls with growing context.

**Observable outcome:** You can trace one user request through multiple tool round-trips and name what each observation adds to context.

## Stage 4 — Shape what the model sees on each API call

Every agent-to-LLM request shares the same skeleton:

```plaintext
system:    [repository context, style guide, tool schemas, agent persona]
messages:  [conversation history — user and assistant turns]
tools:     [JSON schema of available tools]
user:      [current user message]

The system prompt is fixed cost paid on every call. In Cursor it typically includes .cursorrules, open files, tool schemas, and style constraints — often 2,000–4,000 tokens before the user speaks.

Modern APIs expose tools as JSON schemas. The model emits structured tool_use blocks; the orchestrator executes them and returns tool_result messages. The model never touches the filesystem directly.

{
  "name": "read_file",
  "description": "Read the contents of a file at the given path",
  "input_schema": {
    "type": "object",
    "properties": {
      "path": {
        "type": "string",
        "description": "Relative path from the repository root"
      }
    },
    "required": ["path"]
  }
}

Agents stream tokens so the UI shows progress and the orchestrator can detect a tool-call block mid-stream, stop generation, execute the tool, and re-prompt — instead of waiting for a complete response.

Observable outcome: You can list what sits in system, messages, and tools, and explain why tool schemas consume context budget.

Stage 5 — Fit a large codebase into a finite window

Context management is the central engineering constraint. On a long refactor, history alone can consume 60,000 tokens; add a few large files and a 128K window is nearly full.

Dumping the whole repository fails for any real codebase. Agents use RAG: embed files, store vectors locally, retrieve top-k chunks for each query, inject only relevant code. vector-embeddings-and-similarity explains why nearest-neighbor retrieval over embeddings beats keyword grep for "find code related to this task."

sequenceDiagram
    title Context Assembly & Tool Invocation Sequence
    participant U as User
    participant Orch as Orchestrator
    participant CB as Context Builder
    participant VS as Vector Store
    participant LLM as LLM API
    U->>Orch: Fix retry logic in uploadFile
    Orch->>CB: Assemble context
    CB->>VS: Embed query, retrieve top chunks
    VS-->>CB: FileUploader, RetryPolicy, HttpClient
    CB->>CB: Prepend system prompt and tool schemas
    CB->>CB: Append history up to token budget
    CB->>LLM: Send assembled context
    LLM-->>Orch: Tool call read_file
    Orch->>CB: Inject tool result
    CB->>LLM: Re-send with result
    LLM-->>Orch: Final edit diff

Two more constraints shape assembly:

• Lost in the middle: models recall content at the start and end of context better than the middle. Put durable rules in the system prompt, not buried mid-history.
• Overflow strategies: hard truncation (drops old turns), summarization (extra LLM call, preserves semantics), priority eviction (drop tool output before user rules), or RAG refresh (re-select files each turn).

Observable outcome: You can describe what RAG retrieves, what it cannot guarantee, and which overflow strategy likely caused a forgotten rule.

Stage 6 — Persist session state the model will never hold

The LLM has no memory. The session is an agent-side log serialized between turns — local JSON in Copilot, LanceDB-backed history in Cursor, markdown history in Aider.

`` Start["START
(Empty history)"] --> Active["ACTIVE
(State in memory)"] Active --> Idle["IDLE
(Cached to disk)"] Idle --> Expired["EXPIRED
(History cleared)"]

Every turn appends to the log, and the entire log is resent on the next call. By turn 20 the log may dominate the budget — which is why agents expose flags such as Aider's `--max-chat-history-tokens` to summarize before silent truncation.

Autonomous systems like Devin add **sub-agents** with separate windows. They share state through external caches (summaries, scratchpad files), not a shared context window — multiplying orchestration surface area.

**Observable outcome:** You can explain why "the agent remembered yesterday" means history replay, not model memory.

## Stage 7 — Navigate four memory tiers and tool output limits

Coding agents juggle four memory types with different speed, capacity, and persistence:

| Tier | What it is | Lifetime | Example |
| --- | --- | --- | --- |
| 1 — In-context | Tokens in the current API call | One request | Open file + last tool result |
| 2 — Session cache | Serialized conversation and scratch state | Until expiry | Copilot local JSON, Redis for sub-agents |
| 3 — Vector store | Embedded codebase chunks | Project lifetime | Cursor LanceDB index |
| 4 — Parametric | Model weights from pretraining | Static until retrain | Patterns from other codebases |

When RAG misses a file, the model fills gaps from tier 4 — which is why hallucinated package paths appear even with retrieval enabled.

Tool results bloat context fast. A 2,000-line test log can cost 4,000+ tokens in one turn. Production agents truncate or summarize tool output (Claude Code caps tool results around 4K tokens by default).

**Observable outcome:** You can map a symptom — stale file edit, hallucinated import, forgotten rule — to the memory tier or budget decision behind it.

## End-to-end outcome

Return to the opening refactor. The agent violated `PaymentProcessor` and `Retryable` because those constraints lived only in early conversation turns that truncation removed. Constraints that must survive a long session belong in `.cursorrules` or the system prompt — the only context guaranteed to reload every call.

After working through the stages, you should be able to:

- Trace a user request through context assembly, multiple ReAct iterations, and final output.
- Estimate what consumes the token budget and when eviction will occur.
- Place durable rules where overflow cannot erase them.
- Distinguish retrieval failure (missing file in RAG) from parametric hallucination (invented API).

A focused pagination change in Cursor might cost ~14,800 tokens and ~45 seconds across four LLM calls — not because generation is slow, but because read → edit → test → fix is four separate stateless requests.

## Alternatives considered

### Load the full repository (long context only)
Works for small repos under ~100K tokens. No retrieval overhead; the model sees everything. Fails for large monorepos.

### RAG-first retrieval
Required above ~500K tokens of code. Adds ~200–400ms per turn but saves tens of thousands of input tokens. Risk: wrong chunks retrieved → hallucination fills the gap.

### Summarization plus durable rules file
Best for 30+ turn refactors. Accepts 3–8s summarization latency to preserve session thread. Must pair with `.cursorrules` so summarized history does not drop hard constraints.

### Single-shot completion (no agent loop)
Appropriate for "explain this function." Agent overhead — multiple tool calls — is wasted when no observation cycle is needed.

| Situation | Recommended approach |
| --- | --- |
| Small repo (< 100K tokens) | Load all relevant files |
| Large monorepo | RAG + short focused sessions |
| Long refactor session | Summarization + `.cursorrules` |
| Multi-step bug fix with tests | Full ReAct loop |

## Failure and degraded modes

### Context overflow (silent truncation)
Oldest turns drop without warning. The model does not signal loss. **Mitigation:** short sessions; put rules in system prompt.

### Hallucinated paths and APIs
Model invents classes not in retrieved context. **Mitigation:** linter or compiler feedback injected as tool results; re-read before edit.

### Infinite tool loops
Repeated `search_codebase` with rephrased queries. **Mitigation:** max-iteration guards (Claude Code ~10 calls per turn; Aider ~5 retries per edit).

### Stale file snapshot
Agent edits from a read taken before the user manually changed the file. **Mitigation:** hash check or automatic re-read immediately before `edit_file`.

### Cascading hallucination
Bad path in `edit_file` → "file not found" → model creates duplicate file → drift compounds. **Mitigation:** stop on tool error; require user confirmation for new files.

## Configuring agents for durable constraints

`.cursorrules` prepends project rules to every system prompt — the most direct fix for the opening refactor failure:

```plaintext
# .cursorrules
You are working in a Spring Boot 3.x codebase.

Rules:
- All payment services must implement PaymentProcessor
- Never use LegacyBillingClient — deprecated
- All new services must use @Retryable(maxAttempts = 3)

Aider exposes explicit history control:

aider --model claude-3-5-sonnet-20241022 \
      --context-window 150000 \
      --max-chat-history-tokens 30000 \
      src/payment/FileUploader.java src/core/RetryPolicy.java

These are integration-level controls. They do not change model weights; they change what the orchestrator reliably includes when the window is full.

• Separate the language model from the orchestrator: tools, memory, and retries are product architecture, not model magic.
• Context windows are finite; durable rules belong in always-included files and retrieval, not only in early chat turns that can be dropped.
• Treat tool results as untrusted inputs that can be stale, partial, or wrong — verify before merging.
• Bound loops, file scopes, and attempt counts; unbounded agent autonomy is an availability and cost failure mode.
• Evaluate agents on task completion with constraints held, not on fluent explanation of what they intended to do.