Build vs Buy: Deploying Your Own LLM vs Using ChatGPT, Gemini, and Claude APIs

Scenario context

Use the API until you hit $10K/month or a hard data privacy requirement. Then add a semantic cache. Then evaluate hybrid routing. Self-hosting full model serving is only cost-effective at > 50M tokens/day with a dedicated MLOps team. The build vs buy decision is a spreadsheet problem, not an engineering identity problem.

Month one, a seed-stage startup routes every user query to GPT-4 Turbo. The product is a docume…

This walkthrough focuses on The $47,000 Monthly Bill That Killed the Roadmap.

Before making a design choice, clarify:

• Month one, a seed-stage startup routes every user query to GPT-4 Turbo.
• The product is a document analysis tool — users paste contracts, the model summarises obligations, highlights risk clauses, and answers follow-up questions.
• The team ships fast, investors are happy, and the demo is genuinely impressive.

Concepts in play: llm-agents-and-langgraph, caching, design-patterns-gof, llm-prompt-engineering

Stage 1 — The 7-Factor Framework for Choosing Your LLM Deployment Model

• Rather than making this decision by intuition, score your use case against seven factors.
• Each factor scores 0, 1, or 2 points.
• Sum the scores for a decision tier.
• Scoring rubric: - 0–4 points → API-first.

Observable outcome: Scoring rubric: - 0–4 points → API-first.

Stage 2 — Running the Numbers: A Cost Model That Actually Holds Up

• Before any architecture decision, open a spreadsheet.
• The numbers are not intuitive — and many teams that made the wrong call did so because they only looked at the first layer of the cost model.
• This section breaks the economics into three sub-questions: what does the pricing layer actually include, and where do throughput benchmarks really shift the break-even point?
• Internals: What the Pricing Layer Hides From You When you look at API token prices in isolation, you are seeing one dimension of a three-layer cost structure.

def monthly_cost_api(
    daily_input_tokens: int,
    daily_output_tokens: int,
    input_price_per_million: float,
    output_price_per_million: float,
) -> float:
    """Compute monthly API cost in USD."""
    monthly_input = daily_input_tokens * 30
    monthly_output = daily_output_tokens * 30
    return (
        monthly_input / 1_000_000 * input_price_per_million
        + monthly_output / 1_000_000 * output_price_per_million
    )

def monthly_cost_self_host(
    gpu_hourly_rate: float = 2.10,   # A40 reserved 1-year rate
    num_gpus: int = 2,
    utilization: float = 0.70,       # 70% avg utilization
) -> float:
    """Compute monthly self-hosting cost (GPU + 20% overhead for storage/networking)."""
    gpu_monthly = gpu_hourly_rate * num_gpus * 24 * 30
    overhead = gpu_monthly * 0.20
    return gpu_monthly + overhead

def break_even_daily_tokens(
    self_host_monthly: float,
    input_price_per_million: float,
    output_price_per_million: float,
    output_ratio: float = 0.20,  # output tokens as fraction of total
) -> int:
    """
    Estimate the daily token volume at which self-hosting becomes cheaper.
    Assumes output_ratio fraction of tokens are output tokens.
    """
    blended_price = (
        (1 - output_ratio) * input_price_per_million
        + output_ratio * output_price_per_million
    ) / 1_000_000  # cost per token
    monthly_tokens_needed = self_host_monthly / blended_price
    return int(monthly_tokens_needed / 30)

# --- Example: 10M input + 2M output tokens/day vs Claude 3.5 Sonnet ---
api_cost = monthly_cost_api(
    daily_input_tokens=10_000_000,
    daily_output_tokens=2_000_000,
    input_price_per_million=3.0,
    output_price_per_million=15.0,
)
self_host_cost = monthly_cost_self_host(gpu_hourly_rate=2.10, num_gpus=2)
be = break_even_daily_tokens(self_host_cost, 3.0, 15.0)

print(f"API (Claude 3.5 Sonnet):     ${api_cost:>10,.0f}/month")
print(f"Self-hosted Llama 3 70B:     ${self_host_cost:>10,.0f}/month")
print(f"Break-even daily volume:     {be / 1_000_000:.1f}M tokens/day")
print()

# --- Scenario: GPT-4o-mini (the hidden gem tier) ---
mini_cost = monthly_cost_api(
    daily_input_tokens=10_000_000,
    daily_output_tokens=2_000_000,
    input_price_per_million=0.15,
    output_price_per_million=0.60,
)
be_mini = break_even_daily_tokens(self_host_cost, 0.15, 0.60)

print(f"API (GPT-4o-mini):           ${mini_cost:>10,.0f}/month")
print(f"Self-hosted Llama 3 70B:     ${self_host_cost:>10,.0f}/month")
print(f"Break-even vs mini:          {be_mini / 1_000_000:.0f}M tokens/day")

API (Claude 3.5 Sonnet):     $    10,800/month
Self-hosted Llama 3 70B:     $     3,629/month
Break-even daily volume:     8.5M tokens/day

API (GPT-4o-mini):           $       504/month
Self-hosted Llama 3 70B:     $     3,629/month
Break-even vs mini:          242M tokens/day

Observable outcome: Internals: What the Pricing Layer Hides From You When you look at API token prices in isolation, you are seeing one dimension of a three-layer cost structure.

Stage 3 — Three Deployment Tiers and When Each One Fits

• Most production systems at scale use a tiered model.
• The following flowchart maps the key decision points to the three tiers.
• Read it top-to-bottom: each diamond is a yes/no question; the answers route you toward API-only, hybrid, or self-host-first.
• The flowchart shows that volume is the first filter but not the only one.

graph TD
    A[Start: Evaluate Your Use Case] --> B{Daily volume above 50M tokens?}
    B -->|No| C{PII or PHI in prompts?}
    B -->|Yes| D{Dedicated MLOps team?}
    C -->|No| E{Latency SLA under 200ms?}
    C -->|Yes| F{Data can leave premises?}
    E -->|No| G[Tier 1 - API Only]
    E -->|Yes| H[Tier 2 - Hybrid with vLLM for latency]
    F -->|No| I[Tier 3 - Self-Host or Private Endpoint]
    F -->|Yes| G
    D -->|No| J[Tier 2 - Hybrid]
    D -->|Yes| K{Fine-tuning on proprietary corpus needed?}
    K -->|No| J
    K -->|Yes| L[Tier 3 - Self-Host First]

graph TD
    App[Application Layer] --> Cache[Semantic Cache - Redis plus Embeddings]
    Cache -->|Cache hit| Response[Return Cached Response]
    Cache -->|Cache miss| Router[Model Router]
    Router --> Primary[Primary Model - GPT-4o or Claude 3.5 Sonnet]
    Router --> Fallback[Fallback Model - GPT-4o-mini or Gemini Flash]
    Primary --> Store[Store in Cache]
    Fallback --> Store
    Store --> Response

Observable outcome: The flowchart shows that volume is the first filter but not the only one.

Stage 4 — Real Deployments, Real Numbers: Who Chose What and Why

• Harvey AI — self-hosted, compliance-driven.
• Harvey builds legal AI for law firms and corporate legal departments.
• Their regulatory environment is non-negotiable: client privileged communications cannot traverse a third-party API.
• Harvey self-hosts fine-tuned models on their own infrastructure and uses their own legal corpus to improve model performance on contracts, case law, and regulatory filings.

Observable outcome: Harvey self-hosts fine-tuned models on their own infrastructure and uses their own legal corpus to improve model performance on contracts, case law, and regulatory filings.

Stage 5 — The One-Page Decision Checklist

• Use this table in your architecture review.
• Each question maps to a recommendation.
• If your answers put you in two different tiers, the stronger constraint wins (data privacy and volume tend to dominate).
• The single most important insight from this checklist: build your evaluation infrastructure before you build your serving infrastructure.

Observable outcome: The single most important insight from this checklist: build your evaluation infrastructure before you build your serving infrastructure.

Stage 6 — Building a Production LLM Router in Python

• The following router implements the core Tier 2 pattern: classify task complexity from prompt features, dispatch simple tasks to a self-hosted vLLM endpoint (served with an OpenAI-compatible API surface), route complex…
• The complexity classifier here is a heuristic — fast and transparent, but limited.
• In a production Tier 2 deployment, replace it with a trained classifier (a fine-tuned BERT or a small LLM that predicts routing category) for higher accuracy.
• The rest of the router structure remains the same.

import os
import time
from dataclasses import dataclass, field
from openai import OpenAI

@dataclass
class RoutingDecision:
    model: str
    endpoint: str
    estimated_cost_usd: float
    reason: str
    complexity: str = "unknown"

@dataclass
class RouterStats:
    total_requests: int = 0
    self_hosted_requests: int = 0
    api_requests: int = 0
    sla_fallbacks: int = 0
    total_estimated_cost_usd: float = 0.0

    def summary(self) -> str:
        pct_self = (
            100 * self.self_hosted_requests / self.total_requests
            if self.total_requests
            else 0
        )
        return (
            f"Requests: {self.total_requests} total | "
            f"{self.self_hosted_requests} self-hosted ({pct_self:.0f}%) | "
            f"{self.api_requests} API | "
            f"{self.sla_fallbacks} SLA fallbacks | "
            f"Est. cost: ${self.total_estimated_cost_usd:.4f}"
        )

class LLMRouter:
    """
    Routes LLM requests between a self-hosted vLLM endpoint and a managed API
    based on heuristic task complexity and a latency SLA guard.

    Replace _estimate_complexity() with a trained classifier in production.
    Replace the Anthropic call with LiteLLM proxy for multi-provider support.
    """

    SELF_HOST_ENDPOINT = os.getenv("VLLM_ENDPOINT", "http://localhost:8000/v1")
    SELF_HOST_MODEL = "meta-llama/Meta-Llama-3-70B-Instruct"

    # For the API path, use LiteLLM proxy (see the LiteLLM section below)
    # or the provider's native SDK. This example uses an OpenAI-compat layer.
    API_MODEL = "claude-3-5-sonnet-20241022"
    API_BASE_URL = os.getenv("LLM_GATEWAY_URL", "https://api.anthropic.com/v1")
    LATENCY_SLA_MS = 3_000  # 3-second SLA for self-hosted path

    # Approximate amortised costs (USD per 1K tokens)
    SELF_HOST_COST_PER_1K = 0.0005   # GPU cost amortised at 70% utilisation
    API_COST_INPUT_PER_1K = 0.003    # Claude Sonnet input
    API_COST_OUTPUT_PER_1K = 0.015   # Claude Sonnet output

    def __init__(self) -> None:
        self.self_host_client = OpenAI(
            base_url=self.SELF_HOST_ENDPOINT,
            api_key="not-needed",  # vLLM does not require an API key
        )
        self.api_client = OpenAI(
            api_key=os.environ["LLM_API_KEY"],
            base_url=self.API_BASE_URL,
        )
        self.stats = RouterStats()

    def _estimate_complexity(self, prompt: str) -> str:
        """
        Heuristic complexity classifier.
        Categories: simple | medium | complex
        Replace with a trained classifier for production accuracy.
        """
        words = prompt.split()
        word_count = len(words)

        has_code_request = any(
            kw in prompt.lower()
            for kw in ["implement", "write code", "debug", "refactor", "write a function"]
        )
        has_reasoning_request = any(
            kw in prompt.lower()
            for kw in ["explain why", "analyse", "compare", "design", "evaluate", "trade-off"]
        )
        has_multi_step = any(
            kw in prompt.lower()
            for kw in ["step by step", "plan", "first", "then", "finally", "outline"]
        )

        if word_count < 40 and not has_code_request and not has_reasoning_request:
            return "simple"
        if has_code_request or has_reasoning_request or has_multi_step or word_count > 250:
            return "complex"
        return "medium"

    def _build_routing_decision(self, prompt: str) -> RoutingDecision:
        complexity = self._estimate_complexity(prompt)
        # Rough token estimate: 1 word ≈ 1.3 tokens
        estimated_tokens = len(prompt.split()) * 1.3

        if complexity == "simple":
            return RoutingDecision(
                model=self.SELF_HOST_MODEL,
                endpoint=self.SELF_HOST_ENDPOINT,
                estimated_cost_usd=estimated_tokens / 1_000 * self.SELF_HOST_COST_PER_1K,
                reason=f"simple task ({len(prompt.split())} words) → self-hosted Llama 3",
                complexity=complexity,
            )
        # medium and complex → API
        cost = (
            estimated_tokens / 1_000 * self.API_COST_INPUT_PER_1K
            + (estimated_tokens * 0.5) / 1_000 * self.API_COST_OUTPUT_PER_1K
        )
        return RoutingDecision(
            model=self.API_MODEL,
            endpoint="api",
            estimated_cost_usd=cost,
            reason=f"{complexity} task → managed API ({self.API_MODEL})",
            complexity=complexity,
        )

    def _call_self_hosted(self, prompt: str, decision: RoutingDecision) -> str | None:
        """
        Call the self-hosted vLLM endpoint.
        Returns the response text, or None if the call fails or exceeds the SLA.
        """
        start = time.monotonic()
        try:
            resp = self.self_host_client.chat.completions.create(
                model=self.SELF_HOST_MODEL,
                messages=[{"role": "user", "content": prompt}],
                max_tokens=512,
                timeout=self.LATENCY_SLA_MS / 1_000,
            )
            elapsed_ms = (time.monotonic() - start) * 1_000
            if elapsed_ms > self.LATENCY_SLA_MS:
                return None  # SLA breach — fall back to API
            return resp.choices[0].message.content
        except Exception:
            return None  # Any error → fall back to API

    def _call_api(self, prompt: str) -> str:
        resp = self.api_client.chat.completions.create(
            model=self.API_MODEL,
            messages=[{"role": "user", "content": prompt}],
            max_tokens=512,
        )
        return resp.choices[0].message.content

    def complete(self, prompt: str) -> tuple[str, RoutingDecision]:
        """
        Route a prompt to the appropriate model and return (response, decision).
        Updates internal stats for cost and routing telemetry.
        """
        decision = self._build_routing_decision(prompt)
        self.stats.total_requests += 1

        if decision.endpoint == self.SELF_HOST_ENDPOINT:
            response = self._call_self_hosted(prompt, decision)
            if response is not None:
                self.stats.self_hosted_requests += 1
                self.stats.total_estimated_cost_usd += decision.estimated_cost_usd
                return response, decision
            # SLA breach or error: fall back to API
            decision.reason += " [SLA breach — fell back to API]"
            self.stats.sla_fallbacks += 1

        # API path
        response = self._call_api(prompt)
        self.stats.api_requests += 1
        self.stats.total_estimated_cost_usd += decision.estimated_cost_usd
        return response, decision

# --- Demo: route a mixed workload ---
if __name__ == "__main__":
    router = LLMRouter()

    test_prompts = [
        "Summarise this PR title: Fix null pointer in UserService",
        "What is the capital of France?",
        "Implement a thread-safe LRU cache in Python with O(1) get and put operations.",
        "Compare the trade-offs between eventual consistency and strong consistency in distributed databases.",
        "Translate: Hello, how are you?",
    ]

    for prompt in test_prompts:
        # In production, await router.complete() if using async
        response, dec = router.complete(prompt)
        print(f"[{dec.complexity.upper()}] {dec.reason}")
        print(f"  Est. cost: ${dec.estimated_cost_usd:.5f}")
        print(f"  Response:  {response[:100]}...")
        print()

    print("--- Session Summary ---")
    print(router.stats.summary())

flowchart TD
    Task["Task Type"]
    Changing{Requires frequent
knowledge updates?}
    RAG["Use RAG
(Dynamic retrieval)"]
    Finetune["Use Fine-tuning
(Static weights)"]

    Task --> Changing
    Changing -->|Yes| RAG
    Changing -->|No| Finetune

Observable outcome: The rest of the router structure remains the same.

End-to-end outcome

After working through the stages, you should be able to explain:

• Six non-obvious lessons from teams that have navigated this decision in both directions.
• Start API-first, self-host at a specific measurable threshold — never speculatively.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

Why This Decision Is Harder Than Most Engineering Trade-offs

• Detection: Teams that have navigated cloud database vendor lock-in or build-vs-buy for analytics pipelines think they understand this decision.
• User or operator impact: Teams that have navigated cloud database vendor lock-in or build-vs-buy for analytics pipelines think they understand this decision.

Hidden Risks and Failure Modes in Both Directions

• Detection: Knowing which tier is right does not guarantee a good outcome.
• User or operator impact: Knowing which tier is right does not guarantee a good outcome.