Skip to main content
← CONCEPTS / AI & MACHINE LEARNING SYSTEMS
An end-to-end domain learning journey

Fill-in-the-Middle (FIM) & Speculative
IDE Code Completion.

Fill-in-the-Middle (FIM) rearranges code around the cursor into <PRE> prefix <SUF> suffix <MID> so autoregressive models condition on both preceding and following code.
Start the journey7 sections
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Fill-in-the-Middle (FIM) rearranges code around the cursor into `<PRE> prefix <SUF> suffix <MID>` so autoregressive models condition on both preceding and following code.

TL;DR30-second simplified summary
▾

Standard left-to-right LLM generation only sees code before the cursor, ignoring closing braces, return statements, and types below the cursor. Fill-in-the-Middle (FIM) solves this by splitting the editor buffer at the cursor and packing tokens in Prefix-Suffix-Middle (<PRE> prefix <SUF> suffix <MID>) order during training and inference.

Because IDE autocomplete fires on nearly every pause in typing, latency budgets are brutal (< 150 ms end-to-end). Production engines pair a fast 7B/13B FIM model with RadixAttention prefix caching (since the top of the file rarely changes across keystrokes), Tree-sitter AST symbol retrieval across open tabs, and Speculative Decoding (or prompt-lookup drafting) to stream ghost text before the user types the next character.

From patterns to decisions.

Fill-in-the-Middle (FIM) rearranges code around the cursor into `<PRE> prefix <SUF> suffix <MID>` so autoregressive models condition on both preceding and following code.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Core Intuition & Failure Mode.

Fill-in-the-Middle (FIM) rearranges code around the cursor into <PRE> prefix <SUF> suffix <MID> so autoregressive models condition on both preceding and following code.

STAGE 01 · CORE INTUITION

Core Intuition & Failure Mode

Core IntuitionIngress / BaselineFailure ModeScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Understand the baseline failure mode.

2.Level (Foundation): Fill-in-the-Middle (FIM) rearranges code around the cursor into `<PRE> prefix <SUF> suffix <MID>` so autoregressive models condition on both

3.Operational Invariant: Understand the baseline failure mode.

Caption: Understand the baseline failure mode.

02
STAGE 02 · CORE MECHANICSfoundation

Architectural Mechanics.

Because IDE autocomplete fires on nearly every pause in typing, latency budgets are brutal (< 150 ms end-to-end). Production engines pair a fast 7B/13B FIM model with RadixAttention prefix caching (since the top of the file rarely changes across keystrokes), Tree-sitter AST symbol retrieval across open tabs, and Speculative Decoding (or prompt-lookup drafting) to stream ghost text before the user types the next character.

STAGE 02 · CORE MECHANICS

Architectural Mechanics

ArchitecturalMechanicsInput StateCore EngineCore MechanismGuard & VerifyInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Trace the core mechanism end-to-end.

2.Level (Foundation): Because IDE autocomplete fires on nearly every pause in typing, latency budgets are brutal (< 150 ms end-to-end). Production engines pair a

3.Operational Invariant: Trace the core mechanism end-to-end.

Caption: Trace the core mechanism end-to-end.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

FIM Prompt Packer with Token Budget.

def build_fim_prompt(prefix: str, suffix: str, ast_context: str, max_chars: int = 12000) -> str:
    if not prefix and not suffix:
        raise ValueError('Editor buffer cannot be completely empty')
    ctx_budget = min(len(ast_context), max_chars // 4)
    pre_budget = (max_chars - ctx_budget) * 2 // 3
    suf_budget = max_chars - ctx_budget - pre_budget
    trimmed_ctx = ast_context[:ctx_budget]
    trimmed_pre = prefix[-pre_budget:] if pre_budget > 0 else ''
    trimmed_suf = suffix[:suf_budget] if suf_budget > 0 else ''
    return f'<PRE>{trimmed_ctx}\n{trimmed_pre}<SUF>{trimmed_suf}<MID>'
STAGE 03 · SAFE IMPLEMENTATION

FIM Prompt Packer with Token Budget

FIM Prompt Packerwith TValidated InputCore EngineAtomic StepGuard & VerifyError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Pack prefix, suffix, and AST neighbor snippets within a strict latency-safe token budget.

2.Level (Intermediate): def build_fim_prompt(prefix: str, suffix: str, ast_context: str, max_chars: int = 12000) -> str:

3.Operational Invariant: Pack prefix, suffix, and AST neighbor snippets within a strict latency-safe token budget.

Caption: Pack prefix, suffix, and AST neighbor snippets within a strict latency-safe token budget.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Throughput & Latency Trade-Offs.

Evaluate asymptotic bounds, cache locality, and tail-latency impact under peak load.

STAGE 04 · SCALING & TRADE-OFFS

Throughput & Latency Trade-Offs

ThroughputHot PathLatency Trade-OffsPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Quantify operational complexity and memory bounds.

2.Level (Intermediate): Evaluate asymptotic bounds, cache locality, and tail-latency impact under peak load.

3.Operational Invariant: Quantify operational complexity and memory bounds.

Caption: Quantify operational complexity and memory bounds.

05
STAGE 05 · FAULT ISOLATIONadvanced

Blast-Radius & Failure Isolation.

Isolate shared resources, apply bounded queues, and prevent retry amplification.

STAGE 05 · FAULT ISOLATION

Blast-Radius & Failure Isolation

Blast-RadiusFault TriggerFailure IsolationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Contain failures without cascading across tenants or nodes.

2.Level (Advanced): Isolate shared resources, apply bounded queues, and prevent retry amplification.

3.Operational Invariant: Contain failures without cascading across tenants or nodes.

Caption: Contain failures without cascading across tenants or nodes.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Durability, Consistency & Recovery.

Use idempotent state transitions, atomic commits, and deterministic replay.

STAGE 06 · RECOVERY & CONSISTENCY

Durability, Consistency & Recovery

DurabilityConsistencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Preserve invariants across crashes and restarts.

2.Level (Advanced): Use idempotent state transitions, atomic commits, and deterministic replay.

3.Operational Invariant: Preserve invariants across crashes and restarts.

Caption: Preserve invariants across crashes and restarts.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why does FIM place `<SUF>` before `<MID>` in `<PRE> prefix <SUF> suffix <MID>`?

Recommended Learning Flow