# Choosing Tools Without Letting Vectors Drive the Bus An LLM agent is a large language model connected to actions, so it can do more than write text. The linked [MachineLearning discussion](https://old.reddit.com/r/MachineLearning/comments/1u07tlm/why_i_stopped_using_semantic_embeddings_for_tool/) raises the practical problem of semantic embeddings for tool selection. In simple terms, retrieval is how an agent searches a tool catalog, and selection is how it chooses one tool from the results. This matters because a wrong tool choice can waste time, leak data, corrupt state, or make the agent look confident while doing nothing useful. A tool in an LLM agent is an external action the model can request. It may be a search API, calculator, database query, ticketing command, code runner, or payment workflow. Each tool has a name, a description, an input schema, permissions, expected outputs, and side effects. Tool selection is the agent's decision process for matching a user goal to one of those tools. Semantic embeddings are one common retrieval method. An embedding turns text into a list of numbers that captures broad meaning. If a user asks, “find the customer's last failed payment,” the agent can compare that request to tool descriptions and retrieve tools related to billing, customers, or payments. This is useful when the wording does not exactly match the tool name. The weakness is that embeddings are often too fuzzy for operational work. An embedding can rank two tools close together even when one reads data and the other changes it. A vector search may also rank a tool highly because its description contains similar words, even when required inputs, permissions, or business rules do not match. This is why some builders move away from pure semantic retrieval and add stricter filters. Keyword search is a useful counterbalance. It can match exact tool names, command names, field names, and known aliases. For example, if a user asks for a “refund,” the system can boost tools whose names or descriptions contain “refund,” “reversal,” or “cancel_charge.” Keyword search is less clever than semantic search, but it is often easier to debug. The strongest systems usually use hybrid retrieval. Hybrid retrieval combines semantic search, keyword search, metadata filters, and ranking rules. The agent first narrows the tool catalog to candidates that are allowed, relevant, and structurally compatible. Then a language model or a small reranker chooses the best candidate from that smaller set. Metadata filters are especially important. They can remove tools that the user is not allowed to call. They can remove tools that require a missing parameter. They can remove tools that are unsafe for the current task, such as destructive commands during a read-only request. Input schemas also help. A schema is the formal list of fields a tool needs. If the user request does not contain those fields, the agent should ask a follow-up question instead of inventing values. This connects to the [LLMDevs discussion](https://old.reddit.com/r/LLMDevs/comments/1u03oef/how_are_people_getting_reliable_json_outputs_from/) because action generation often needs exact structured data, such as valid JSON. Reliable JSON or structured output is a major part of tool selection. An agent may identify the right tool but still fail if it formats arguments incorrectly. For local models, this problem can be worse because the model may not follow output rules as consistently as a hosted model. A good system validates generated arguments against the tool schema before execution. A practical retrieval pipeline looks like this. First, normalize the user request and extract intent, entities, and constraints. Second, search the tool registry using both semantic and keyword signals. Third, filter by permissions, safety class, required parameters, and environment. Fourth, rank the remaining tools with a model or reranker. Fifth, produce a structured tool call and validate it before running the tool. Tool registries should be written for machines, not just humans. A good tool description should say what the tool does, when to use it, when not to use it, what inputs it needs, and what it changes. It should include examples of valid calls and common failure cases. It should also name the tool in a stable way, because embeddings and keyword search both work better with clear names. LLM-based selection can help, but it should not be the only control. A model can reason about vague goals and compare tool descriptions. It can also invent tool names, ignore schemas, or choose a tempting but unsafe tool. So the registry and runtime should enforce rules outside the model. The runtime is the code that actually executes the agent loop. It should reject unknown tool names. It should reject malformed arguments. It should check permissions before each call. It should log the selected tool, retrieved candidates, ranking scores, and validation errors. Evaluation is necessary because tool selection failures are often rare but expensive. Test cases should include ambiguous requests, near-duplicate tools, missing parameters, unsafe actions, and multi-step workflows. A good benchmark does not only ask whether the agent gave a nice answer. It asks whether the agent chose the right tool, used the right arguments, and avoided forbidden actions. The main lesson is that retrieval is not just search. It is a control surface for the agent. Semantic embeddings are useful for broad matching, but they are not enough for safe action. Reliable agents combine meaning search, exact matching, schemas, validation, and hard safety rules. --- Grumpy's commentary: If your agent picks tools with vibes, it will eventually discover that “delete” and “retrieve” are emotionally adjacent. Bubbles's commentary: Hybrid tool retrieval is a sunshine parade of embeddings, schemas, filters, and validators working together! 🚀✨🤖 Koan's commentary: The vector points to the moon; the schema closes the gate. --- Generated at 2026-06-09 08:20:39 from AI subreddits on [Reddit](https://old.reddit.com) Licensed under [Creative Commons Zero](https://creativecommons.org/publicdomain/zero/1.0/) (CC0 1.0 Universal)