System Design: Designing an Autonomous AI Coding Agent (Devin at Scale)

Scenario context

Designing an autonomous AI coding agent at scale is not a prompt engineering task; it is a complex systems problem. The system requires secure multitenancy via Firecracker microVMs, a low-latency workspace syncing event loop, and a dual planning-execution engine backed by durable state to survive compile-and-retry loops.

When building an autonomous AI developer like Devin or Swe-agent, developers often start w…

This walkthrough focuses on Part 1: Approach — Requirements and Scale Challenges.

Before making a design choice, clarify:

• When building an autonomous AI developer like Devin or Swe-agent, developers often start with a simple script that calls the OpenAI API, reads local files, runs local compilers, and writes code.
• In production, this naive setup fails immediately: The Infinite Loop Resource Burn: An LLM planner gets stuck in an compile-fail-retry cycle, compiling a broken package 500 times in 2 minutes, consuming all system CPU a…
• Sandbox Security Escape: An agent writes a python script that runs malicious code locally, deletes files, or attempts to port-scan other nodes in the network.

Concepts in play: event-driven-architecture, llm-agents-and-langgraph, caching, api-gateway

Stage 1 — High-Level Architecture and System Mechanics

• The diagram below details the components of our autonomous agent system, splitting execution into a control plane and a sandbox execution plane.
• This high-level architecture separates orchestration from isolated compute execution.
• The API Gateway routes user configurations.
• The Agent Orchestrator manages the task planning and state database updates, while delegating command execution to the Sandbox Supervisor.

graph TD
    Client[Web Browser Client] -->|HTTP/Websocket| Gateway[API Gateway & Router]
    Gateway -->|gRPC| Orchestrator[Agent Orchestrator Service]
    Gateway -->|Stream| LogCollector[Log & Stream Collector]
    Orchestrator -->|Queue| TaskQueue[Task Message Queue]
    Orchestrator -->|State Updates| StateDB[(State Database)]
    Orchestrator -->|Locking| Cache[(Redis Session Cache)]

    TaskQueue --> SandboxSupervisor[Sandbox Supervisor Service]
    SandboxSupervisor -->|Control API| VMHost[MicroVM Compute Host]

    VMHost --> VM1[MicroVM Sandbox 1]
    VMHost --> VM2[MicroVM Sandbox 2]

    VM1 -->|gRPC Stream| LogCollector
    VM2 -->|gRPC Stream| LogCollector

Observable outcome: The Agent Orchestrator manages the task planning and state database updates, while delegating command execution to the Sandbox Supervisor.

Stage 2 — Deep Dive: Execution Sandboxing and Workspace Syncing

• Building a production-ready system requires looking closely at how virtualization runs commands securely and syncs workspace directories between host and VM.
• The Internals To run untrusted agent code, we avoid standard Docker containers because they share the host OS kernel.
• Instead, we use Firecracker microVMs.
• Firecracker leverages KVM to create lightweight virtual machines.

Observable outcome: Firecracker leverages KVM to create lightweight virtual machines.

Stage 3 — Advanced Concepts: Multi-Tenant Sandbox Security and Event Streaming

• Secure Network Sandboxing Each microVM is isolated using a dedicated virtual network interface (TUN/TAP).
• Traffic routing is restricted using host-level iptables rules: VMs cannot access the metadata API endpoints of the host cloud provider (e.g., AWS IMDSv2).
• Outbound internet access is restricted to whitelisted package registries (e.g., npmjs.org, pypi.org) using a proxy gateway to prevent data exfiltration.
• Event-Driven Orchestration Loop The orchestrator planning loop runs asynchronously.

Observable outcome: Event-Driven Orchestration Loop The orchestrator planning loop runs asynchronously.

Stage 4 — Visualizing the System Flows

• Write Path: Job Submission and Sandbox Boot Flow The diagram below details the sequence of steps that occur when a user triggers a new coding task.
• As shown in this sequence, the database record is persisted first.
• This ensures that even if the compute host hosting the supervisor crashes while cloning the repository, the state engine can recover the task and retry on a separate host.
• Read Path: Live Terminal and Diffs Streaming Flow This diagram describes how command execution output inside the VM reaches the user's browser dashboard.

flowchart TD
    User([User]) -->|POST /jobs| Gateway[API Gateway]
    Gateway -->|Schedule| Orchestrator[Orchestrator]
    Orchestrator -->|1. Create State Record| StateDB[(State Database)]
    Orchestrator -->|2. Send Start Event| TaskQueue[Task Queue]
    TaskQueue -->|3. Consume Event| Supervisor[Sandbox Supervisor]
    Supervisor -->|4. Clone Git Repo| HostStorage[Host Storage NVMe]
    Supervisor -->|5. Boot microVM with Virtio-FS| VM[MicroVM Sandbox]
    VM -->|6. Start execution agent daemon| Exec([Active Agent Loop])

flowchart TD
    VM[MicroVM Sandbox] -->|1. Emit console stdout/stderr| AgentDaemon[Agent Daemon]
    AgentDaemon -->|2. gRPC stream chunk| LogCollector[Log Collector Service]
    LogCollector -->|3. Push updates| EventStream[Event Streaming Service]
    EventStream -->|4. Publish SSE| Gateway[API Gateway]
    Gateway -->|5. Render terminal view| Client([Web UI Dashboard])

Observable outcome: Read Path: Live Terminal and Diffs Streaming Flow This diagram describes how command execution output inside the VM reaches the user's browser dashboard.

Stage 5 — Real-World Applications: Devin at Scale

• Two major case studies showcase the application of this design: Massive Library Migrations: Upgrading an API client library across 1,000 distinct service repositories.
• The system spawns 1,000 parallel sandboxes, allowing the agents to compile, run tests, fix import errors, and submit pull requests simultaneously.
• Auto-Patching Security Alerts: Scanning code for CVE vulnerability patterns.
• The system clones repositories, executes static analysis tools inside the sandbox, edits files to patch dependencies, runs unit test suites, and flags failures for human review.

Observable outcome: The system clones repositories, executes static analysis tools inside the sandbox, edits files to patch dependencies, runs unit test suites, and flags failures for human review.

Stage 6 — Interview Delivery Example

• In a system design interview, structured presentation demonstrates seniority.
• Below is a 45-minute breakdown showing how to communicate this coding agent design.
• Requirements & Scope Definition (Minutes 0 - 5) Clarify boundaries: Ask if the interviewer wants to cover sandbox orchestration, model fine-tuning, or IDE integration.
• Focus on sandbox lifecycle and reliability loops.

Observable outcome: Focus on sandbox lifecycle and reliability loops.

End-to-end outcome

After working through the stages, you should be able to explain:

• Git Loop Lockups: When compiler checks fail, agents often attempt to revert commits, creating circular git loops.
• Implement git-ref guardrails to prevent agents from pushing changes without a clean history stream.
• Compute Isolation: Never run untrusted code on shared hosts; decouple orchestrators from compute execution using virtualization-level microVMs.
• State Persistency: Save execution metrics and history traces at each step transition boundary to enable crash recovery.

Alternatives considered

Decision Guide: Orchestrator vs. Sandbox Architecture

Use the matrix below to choose how to coordinate planning and execution layers based on your deployment constraints.

Failure and degraded modes

Designing an autonomous coder involves trading execution isolation against latency and host density.

• Detection: Designing an autonomous coder involves trading execution isolation against latency and host density.
• User or operator impact: Designing an autonomous coder involves trading execution isolation against latency and host density.

System-Specific Bottlenecks & Failure Modes

• Detection: Infinite Retry Storm: The agent tries to install an incompatible package, fails, and repeats the command.
• User or operator impact: Infinite Retry Storm: The agent tries to install an incompatible package, fails, and repeats the command.