Semantic Tool Retrieval: Scaling Agentic Workflows with Large Toolsets

Scenario context

Semantic Tool Retrieval is a design pattern for agentic systems to manage and intelligently select tools from a large, expanding library. It works by embedding tool descriptions and user queries into a vector space, then using similarity search to find the most relevant tools, overcoming the limitations of static tool lists and prompt stuffing.

Modern AI agents, powered by Large Language Models (LLMs), derive much…

This walkthrough focuses on The Problem/Scenario.

Before making a design choice, clarify:

• Modern AI agents, powered by Large Language Models (LLMs), derive much of their utility from their ability to use external tools.
• These tools can range from simple calculators and search engines to complex APIs for interacting with databases, CRM systems, or even controlling robots.
• Initially, agents are often provided with a small, fixed set of tools, whose descriptions are directly injected into the LLM's prompt.

Concepts in play: vector-embeddings-and-similarity, llm-agents-and-langgraph, vector-databases-rag, sorting-and-searching-algorithms

Stage 1 — How the Pattern Works

• The Semantic Tool Retrieval pattern addresses the "tool explosion" problem by treating tools as searchable documents.
• It leverages techniques from Information Retrieval (IR) and Natural Language Processing (NLP) to enable agents to find tools based on their semantic relevance to a given query or task.
• The pattern typically involves the following components: 1.
• Tool Registry: A centralized repository that stores metadata for all available tools.

+-------------------+     +-------------------+     +--------------------+
|   Tool Registry   | --> | Embedding Model   | --> | Vector Database    |
| (Tool Definitions)|     | (Offline Process) |     | (Tool Embeddings)  |
+-------------------+     +-------------------+     +--------------------+
          ^                                                   ^
          |                                                   |
          |                                                   |
+-------------------+                               +-------------------+
|     AI Agent      |                               | Retrieval Logic   |
| (LLM + Orchestrator)|<----------------------------| (Similarity Search)|
+-------------------+                               +-------------------+
          ^                                                   |
          | (Tool Query)                                      | (Top-K Tools)
          +---------------------------------------------------+

Observable outcome: Tool Registry: A centralized repository that stores metadata for all available tools.

Stage 2 — Code Implementation

• Let's illustrate a simplified Semantic Tool Retrieval system using Python.
• We'll use a mock embedding model and a simple in-memory vector store for demonstration purposes.
• In a real-world scenario, you'd integrate with actual embedding APIs (e.g., OpenAI, Hugging Face sentence-transformers ) and a production-grade vector database (e.g., Pinecone, Weaviate, ChromaDB, FAISS).
• MockEmbeddingModel : A placeholder for a real embedding model.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
from typing import List, Dict, Any

# --- 1. Mock Embedding Model ---
# In a real scenario, this would be an actual LLM embedding API or a local model.
class MockEmbeddingModel:
    def embed(self, text: str) -> np.ndarray:
        """Generates a mock embedding vector for a given text."""
        # Simple hash-based embedding for demonstration
        # In reality, this would be a sophisticated NLP model.
        seed = sum(ord(c) for c in text) % 1000
        np.random.seed(seed)
        return np.random.rand(128) # 128-dimensional embedding vector

# --- 2. Tool Registry and Definition ---
class Tool:
    def __init__(self, name: str, description: str, parameters: Dict[str, Any]):
        self.name = name
        self.description = description
        self.parameters = parameters

    def to_dict(self):
        return {
            "name": self.name,
            "description": self.description,
            "parameters": self.parameters
        }

    def __str__(self):
        return f"Tool Name: {self.name}\nDescription: {self.description}\nParameters: {self.parameters}"

# --- 3. Vector Database (In-memory for simplicity) ---
class InMemoryVectorStore:
    def __init__(self):
        self.vectors: List[np.ndarray] = []
        self.metadata: List[Dict[str, Any]] = []

    def add_vector(self, vector: np.ndarray, tool_metadata: Dict[str, Any]):
        self.vectors.append(vector)
        self.metadata.append(tool_metadata)

    def search(self, query_vector: np.ndarray, k: int = 3) -> List[Dict[str, Any]]:
        if not self.vectors:
            return []

        similarities = cosine_similarity(query_vector.reshape(1, -1), np.array(self.vectors))
        # Get indices of top-k most similar items
        top_k_indices = np.argsort(similarities[0])[-k:][::-1]

        results = []
        for idx in top_k_indices:
            results.append(self.metadata[idx])
        return results

# --- Semantic Tool Retriever Orchestrator ---
class SemanticToolRetriever:
    def __init__(self, embedding_model: MockEmbeddingModel, vector_store: InMemoryVectorStore):
        self.embedding_model = embedding_model
        self.vector_store = vector_store
        self.tools_by_name: Dict[str, Tool] = {}

    def register_tool(self, tool: Tool):
        """Adds a tool to the registry and embeds its description."""
        self.tools_by_name[tool.name] = tool
        tool_text = f"Tool name: {tool.name}. Description: {tool.description}"
        embedding = self.embedding_model.embed(tool_text)
        self.vector_store.add_vector(embedding, tool.to_dict())
        print(f"Registered tool: {tool.name}")

    def retrieve_tools(self, query: str, k: int = 3) -> List[Tool]:
        """Retrieves top-k most relevant tools for a given query."""
        query_embedding = self.embedding_model.embed(query)
        retrieved_metadata = self.vector_store.search(query_embedding, k=k)

        # Reconstruct Tool objects from metadata
        retrieved_tools = []
        for meta in retrieved_metadata:
            tool_name = meta['name']
            if tool_name in self.tools_by_name:
                retrieved_tools.append(self.tools_by_name[tool_name])
            else:
                # This case should ideally not happen if registration is consistent
                print(f"Warning: Tool '{tool_name}' found in vector store but not in active registry.")
        return retrieved_tools

# --- Demonstration ---
if __name__ == "__main__":
    # Initialize components
    embedding_model = MockEmbeddingModel()
    vector_store = InMemoryVectorStore()
    tool_retriever = SemanticToolRetriever(embedding_model, vector_store)

    # Define a large set of tools
    tools = [
        Tool("getCurrentWeather", "Retrieves the current weather for a specified city.", {"city": "string"}),
        Tool("getDailyForecast", "Provides a 5-day weather forecast for a given location.", {"city": "string", "days": "integer"}),
        Tool("searchWikipedia", "Searches Wikipedia for information on a given topic.", {"query": "string"}),
        Tool("sendEmail", "Sends an email to a recipient with a specified subject and body.", {"to": "string", "subject": "string", "body": "string"}),
        Tool("createCalendarEvent", "Schedules a new event in the user's calendar.", {"title": "string", "start_time": "datetime", "end_time": "datetime"}),
        Tool("listFiles", "Lists files in a specified directory.", {"path": "string"}),
        Tool("translateText", "Translates text from one language to another.", {"text": "string", "source_lang": "string", "target_lang": "string"}),
        Tool("calculateExpression", "Evaluates a mathematical expression.", {"expression": "string"}),
        Tool("bookFlight", "Searches and books flights between two cities on specific dates.", {"origin": "string", "destination": "string", "date": "date"}),
        Tool("checkStockPrice", "Retrieves the current stock price for a given ticker symbol.", {"symbol": "string"}),
        Tool("getNewsHeadlines", "Fetches top news headlines for a specific category or region.", {"category": "string", "region": "string"}),
        Tool("summarizeDocument", "Summarizes a given text document or URL.", {"content": "string"}),
        Tool("setAlarm", "Sets an alarm for a specific time.", {"time": "string", "message": "string"}),
        Tool("findRestaurant", "Finds nearby restaurants based on cuisine and location.", {"cuisine": "string", "location": "string"}),
        Tool("convertCurrency", "Converts an amount from one currency to another.", {"amount": "float", "from_currency": "string", "to_currency": "string"}),
    ]

    # Register all tools
    print("--- Registering Tools ---")
    for tool in tools:
        tool_retriever.register_tool(tool)
    print("\n")

    # Simulate agent queries and retrieve relevant tools
    print("--- Simulating Agent Queries ---")

    query1 = "I need to know what the weather will be like next week in London."
    print(f"Query: '{query1}'")
    relevant_tools = tool_retriever.retrieve_tools(query1, k=2)
    for tool in relevant_tools:
        print(f"  - {tool.name}: {tool.description}")
    print("-" * 30)

    query2 = "Can you help me send an email to John about the meeting?"
    print(f"Query: '{query2}'")
    relevant_tools = tool_retriever.retrieve_tools(query2, k=1)
    for tool in relevant_tools:
        print(f"  - {tool.name}: {tool.description}")
    print("-" * 30)

    query3 = "What's the latest news on technology?"
    print(f"Query: '{query3}'")
    relevant_tools = tool_retriever.retrieve_tools(query3, k=2)
    for tool in relevant_tools:
        print(f"  - {tool.name}: {tool.description}")
    print("-" * 30)

    query4 = "I need to perform a calculation: 2 + 2 * 5"
    print(f"Query: '{query4}'")
    relevant_tools = tool_retriever.retrieve_tools(query4, k=1)
    for tool in relevant_tools:
        print(f"  - {tool.name}: {tool.description}")
    print("-" * 30)

    query5 = "I want to find a good Italian restaurant in New York."
    print(f"Query: '{query5}'")
    relevant_tools = tool_retriever.retrieve_tools(query5, k=1)
    for tool in relevant_tools:
        print(f"  - {tool.name}: {tool.description}")
    print("-" * 30)

Observable outcome: MockEmbeddingModel : A placeholder for a real embedding model.

Stage 3 — Complexity Analysis

• Understanding the computational complexity of Semantic Tool Retrieval is crucial for designing scalable agentic systems.
• Offline Phase (Tool Embedding and Indexing) Embedding Generation: For N tools, if each tool description takes L tokens on average, and the embedding model has a complexity of O(L embed) per token, the total time complex…
• L embed can be significant for large transformer models.
• This is a one-time or incremental cost.

Observable outcome: This is a one-time or incremental cost.

Stage 4 — Advanced Considerations

• While the basic pattern is powerful, several advanced techniques can further enhance Semantic Tool Retrieval: 1.
• Hybrid Retrieval: Combine semantic search with keyword-based search (e.g., BM25) for improved recall, especially for tools with very specific names or parameters that might not be captured well by embeddings alone.
• A common approach is to retrieve candidates using both methods and then re-rank them.
• Re-ranking: After retrieving an initial set of k tools, a secondary, more computationally intensive model (e.g., a cross-encoder re-ranker, or even the LLM itself) can be used to re-score and re-order the tools based on…

Observable outcome: Re-ranking: After retrieving an initial set of k tools, a secondary, more computationally intensive model (e.g., a cross-encoder re-ranker, or even the LLM itself) can be used to re-score and re-order the tools based on…

End-to-end outcome

Modern AI agents, powered by Large Language Models (LLMs), derive much of their utility from their ability to use external tools.

Alternatives considered

The source focuses on one primary design. A second architecture is not compared, so no alternative is invented here.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.