LLM Skill Registries, Routing Policies, and Evaluation for Production Agents

Scenario context

If tools are primitives and skills are reusable routines, then the skill registry + router + evaluator is your production control plane. This layer decides which skill runs, under what constraints, and how you detect regressions before users do.

---

In small demos, the model picks a tool and returns a decent answer. In production, that is not enough.

You need answers to operational questions:

• Which skil…

This walkthrough focuses on Why a Skill Registry Becomes the Agent Control Plane.

Before making a design choice, clarify:

• In small demos, the model picks a tool and returns a decent answer.
• In production, that is not enough.
• You need answers to operational questions: - Which skills are currently active?

Concepts in play: llm-agents-and-langgraph, vector-embeddings-and-similarity, database-indexing, consistency-replication

Stage 1 — Designing a Registry That Humans and Routers Can Trust

• A useful registry entry is both machine-readable and operator-readable.
• Minimum fields per skill: Dense systems should also include: - deprecation status, - fallback skill id, - required approvals (for sensitive actions), - evaluation baseline hash.
• A registry is a product artifact.
• Treat it like API surface area, not internal trivia.

Observable outcome: Treat it like API surface area, not internal trivia.

Stage 2 — Routing Pipeline: From User Intent to Skill Selection

• A production router should be explicit about stages.
• This pipeline prevents a common failure mode: the model picks a "kind of related" skill because a keyword looked similar.
• Router quality is usually more important than incremental prompt tweaks once you scale beyond a few skills.

flowchart TD
    A[User request] --> B[Intent and entity extraction]
    B --> C[Candidate skill retrieval from registry]
    C --> D[Policy gate: data/risk/compliance]
    D --> E[Score candidates: fit, cost, risk, freshness]
    E --> F{Confidence above threshold?}
    F -- Yes --> G[Select top skill]
    F -- No --> H[Fallback: safe default or human review]
    G --> I[Execute skill with trace]
    H --> I
    I --> J[Return response + route metadata]

Observable outcome: Router quality is usually more important than incremental prompt tweaks once you scale beyond a few skills.

Stage 3 — Deep Dive: Scoring, Constraints, and Runtime Guarantees

• Internals: hybrid routing usually beats single-strategy routing Most robust systems combine three routing signals: 1.
• Rule-based filters for non-negotiable constraints (risk, permissions, domain).
• Semantic match for intent-to-skill relevance.
• Operational priors from latency, error rate, and freshness.

Observable outcome: Operational priors from latency, error rate, and freshness.

Stage 4 — Evaluation Loop: Offline Replay, Shadow Routing, and Live Gates

• Recommended evaluation layers: For intermediate maturity, start with one robust offline suite and one shadow dashboard before touching canary automation.

sequenceDiagram
    participant D as Dataset Store
    participant R as Router
    participant E as Evaluator
    participant P as Prod Traffic

    D->>R: historical requests replay
    R-->>E: selected skill + confidence + trace
    E->>E: compute quality/safety/latency metrics
    E-->>R: threshold updates and alerts
    P->>R: live requests (shadow mode)
    R-->>E: shadow route decisions
    E-->>R: promote or rollback recommendation

Observable outcome: Recommended evaluation layers: For intermediate maturity, start with one robust offline suite and one shadow dashboard before touching canary automation.

Stage 5 — Real-World Applications: Rollout Patterns That Work in Real Teams

• Pattern 1: New skill onboarding checklist - Add skill metadata and policy fields to registry.
• • Add at least 20 representative replay prompts.
• • Verify schema-valid output rate and safety checks.
• • Enable shadow routing before any user-facing traffic.

Observable outcome: - Enable shadow routing before any user-facing traffic.

Stage 6 — Practical Examples: Registry and Router Skeleton

• Example 1: Registry document shape (JSON) These examples provide a concrete skill registry document shape and a hybrid route selection function that together form the skeleton of a production routing system.
• The JSON registry entry was chosen because it is the minimum viable artifact that makes routing decisions both machine-readable and operator-auditable — humans and automated routers share the same source of truth.
• When reading the Python routing function, focus on how eligibility filtering happens before score computation: this separation is what keeps policy enforcement from leaking into the scoring math and makes each layer ind…
• This is enough to power both routing decisions and operator dashboards.

{
  "skill_id": "incident_triage_v3",
  "version": "3.2.1",
  "description": "Analyze outage alerts, summarize impact, and create incident tickets.",
  "risk_level": "medium",
  "owner_team": "sre-platform",
  "allowed_data_domains": ["logs", "incidents"],
  "input_schema": {
    "type": "object",
    "required": ["service", "time_range", "alert_id"]
  },
  "output_schema": {
    "type": "object",
    "required": ["summary", "severity", "ticket_id"]
  },
  "slo": {
    "p95_latency_ms": 4000,
    "schema_valid_rate": 0.99
  },
  "fallback_skill_id": "incident_triage_safe_v1"
}

from typing import Dict, List

def allowed(skill: Dict, request: Dict) -> bool:
    if request.get("risk_tier") == "high" and skill.get("risk_level") == "high":
        return False
    required_domain = request.get("domain")
    return required_domain in skill.get("allowed_data_domains", [])

def route_score(skill: Dict, fit: float, latency_ms: float, reliability: float) -> float:
    risk_penalty = {"low": 0.05, "medium": 0.15, "high": 0.40}[skill.get("risk_level", "medium")]
    return 0.55 * fit - 0.20 * (latency_ms / 5000.0) - 0.15 * risk_penalty + 0.10 * reliability

def choose_skill(request: Dict, candidates: List[Dict]) -> Dict:
    scored = []
    for skill in candidates:
        if not allowed(skill, request):
            continue

        # Placeholder signals. In production these come from embedding match and telemetry.
        fit = skill.get("fit", 0.0)
        latency_ms = skill.get("p95_latency_ms", 3000)
        reliability = skill.get("schema_valid_rate", 0.95)

        scored.append((route_score(skill, fit, latency_ms, reliability), skill))

    if not scored:
        return {"action": "fallback", "reason": "no_allowed_candidate"}

    scored.sort(key=lambda pair: pair[0], reverse=True)
    best_score, best_skill = scored[0]

    if best_score < 0.35:
        return {"action": "fallback", "reason": "low_confidence", "score": best_score}

    return {"action": "execute", "skill_id": best_skill["skill_id"], "score": best_score}

Observable outcome: This is enough to power both routing decisions and operator dashboards.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Registry design quality determines routing quality more than people expect.
• • If metadata ownership is unclear, incident resolution slows down dramatically.

Alternatives considered

Decision Guide: What to Build First at Your Current Maturity

Decision Guide: What to Build First at Your Current Maturity

Failure and degraded modes

Trade-offs & Failure Modes: Failure Modes and Mitigations in Skill Routing Systems

• Detection: Also watch for this anti-pattern: using one global confidence threshold for every intent type.
• User or operator impact: Also watch for this anti-pattern: using one global confidence threshold for every intent type.