Back to Blog
A wall of organized tools representing a large catalog an AI agent selects from

AI Agent Tool Selection: Dynamic Discovery for Large Catalogs

8 min read

Giving an agent 120 tools does not give it 120 useful choices for every request. Most schemas will be irrelevant, and similar names make the wrong action easier to choose. AI agent tool selection gets harder as the catalog grows, while context use and latency rise before the agent has done any work. Removing tools by hand is not a durable fix because permissions, workflow stages, and available integrations change.

A two-stage boundary fixes this. First, application code filters the catalog by trusted identity and policy. Second, a retrieval step selects a small task-relevant set and loads only those complete schemas into model context. The execution gate still validates any resulting call. Discovery decides what the model can consider. Validation decides whether a proposed action may run.

Why an all-tools prompt fails

Every tool definition occupies context with a name, description, parameter schema, examples, and sometimes output details. Similar tools compete for attention. A model might see find_customer, search_contacts, lookup_account, and get_profile without enough context to know which system owns the requested record.

Current framework guidance treats this as an operating problem. LangChain's dynamic tool selection documentation warns that too many tools can overload context and increase errors, then shows how to filter or register tools at runtime from state, permissions, feature flags, and conversation stage. A Kaiten MCP issue about excessive tool exposure gives the practitioner version of the same complaint: a large set of unused MCP tools consumes agent context.

A static shortlist creates a different failure. It can hide a tool that becomes relevant later, expose an action after a user's permission changes, or keep an old schema after a server update. The catalog therefore needs selection, authorization, and freshness rules rather than one fixed array.

Keep discovery separate from execution

Treat tool discovery as an application service with a narrow contract. It accepts trusted runtime context and a task description. It returns tool definitions that the model may consider for the next step.

It must not accept the user or model's claims about identity as authority. Resolve these inputs before retrieval:

  • authenticated user and organization
  • workload or delegated identity
  • granted roles and scopes
  • current workflow stage
  • enabled integrations and feature flags
  • environment and data classification

Run these checks before semantic retrieval so the search covers only the eligible catalog. Searching every tool and removing forbidden results afterward can leak sensitive tool names and descriptions into logs, traces, or model context. It can also leave fewer candidates than expected, which changes retrieval behavior in ways that are hard to test.

The discovery boundary does not replace call validation. A selected tool can still receive malformed arguments, an unauthorized resource identifier, or a duplicate write. Keep the typed execution gate described by the selected tool's schema and business policy after the model proposes a call.

Give each catalog entry operational metadata

A catalog entry needs more than a function schema. Store the metadata required for filtering, retrieval, refresh, and audit without sending all of it to the model. 

{
  "tool_id": "crm.orders.list_open",
  "version": "2026-08-13.1",
  "namespace": "crm.orders",
  "summary": "List open orders for an authorized customer account",
  "required_scopes": ["orders.read"],
  "allowed_stages": ["investigation", "reply_draft"],
  "risk": "read",
  "schema_hash": "catalog-managed-version",
  "source_server": "crm-mcp",
  "enabled": true
}

Use a stable internal identifier even if a provider exposes a different display name. Namespaces reduce collisions and give retrieval a useful first filter. OpenAI's Tool Search documentation recommends grouping deferred functions into clearly described namespaces and explains that the model can initially see a namespace or server description without receiving every contained parameter schema.

Keep authorization attributes outside the model-editable description. The sentence "administrators can delete orders" helps retrieval but cannot prove the current caller is an administrator. Policy data must come from the server-side catalog and identity system.

Filter first, retrieve second

Implement selection as a deterministic filter followed by ranked retrieval. The first pass should be explainable and fail closed. 

def select_tools(task, runtime, catalog, limit=8):
    eligible = [
        tool for tool in catalog
        if tool.enabled
        and tool.environment == runtime.environment
        and tool.required_scopes <= runtime.scopes
        and runtime.stage in tool.allowed_stages
        and tool.source_server in runtime.approved_servers
    ]

    ranked = semantic_rank(task, eligible)
    candidates = ranked[:limit]

    if retrieval_confidence(candidates) < runtime.minimum_confidence:
        return deterministic_fallback(task, eligible)

    return load_complete_schemas(candidates)

Build the retrieval text from the namespace, short summary, input concepts, and examples of suitable requests. Do not embed secrets, tenant names, or live permission data. Rebuild embeddings when the searchable description changes, and version the index alongside the catalog.

Choose the candidate limit with evaluation data rather than intuition. Eight tools may work for one catalog and hide necessary combinations in another. Measure recall before optimizing context cost. If the correct tool does not enter the candidate set, the model cannot recover through better prompting.

Do not send every tool through semantic ranking. A deterministic workflow step may require one known tool. An emergency stop or approval function may need to remain available whenever its state permits. Common read-only tools can stay preloaded if their schemas are small and their use is unambiguous. Dynamic discovery is for the part of the catalog whose relevance varies.

Load schemas only when needed

After retrieval, inject complete schemas for the selected candidates. Keep names, descriptions, and parameter requirements precise enough to separate neighboring actions. Examples should clarify distinctions, not inflate every definition.

Provider features can implement this pattern directly. OpenAI supports hosted and client-executed tool search with deferred loading. Client-executed search is the better fit when availability depends on tenant or project state because the application controls the eligible set. Anthropic's advanced tool use guidance also describes on-demand discovery for large libraries and reports lower context use in its own tool-search tests.

Keep a provider-neutral layer above these features. It lets the application keep one authorization and catalog model while translating selected entries into each model API's schema. It also makes the all-tools baseline and candidate-set experiments comparable across providers.

Refresh catalogs without stale authority

MCP provides the mechanics for catalog refresh. The MCP tools specification defines paginated tools/list discovery and a notifications/tools/list_changed event for servers that declare the capability.

Use those signals to update metadata and schemas, but do not let a notification silently widen access. A safe refresh sequence is:

  1. Fetch every catalog page into a temporary snapshot.
  2. Validate names and input schemas.
  3. Apply the server's trust policy and local allowlist.
  4. Compute additions, removals, and schema changes.
  5. Rebuild affected retrieval records.
  6. Publish the snapshot atomically with a new catalog version.

If refresh fails midway, keep serving the last validated snapshot. Mark removed or suspect tools unavailable in the policy layer immediately, even if the retrieval index still contains an old record. Authorization checks should read current policy, not a cached retrieval document.

Store the catalog version and selected tool IDs in each agent trace. Without those fields, a failed run cannot be reproduced after schemas change.

Define fallbacks before retrieval fails

Tool retrieval can fail because the index is unavailable, the query is ambiguous, the top score is weak, or the correct tool was filtered by policy. Handle each case separately.

When the index is down, use a small deterministic set of safe tools or pause the workflow. When confidence is low, ask for missing task information or load an approved namespace rather than guessing across the full catalog. When policy removed the likely tool, return a denied-capability result without revealing tools from another role or tenant. When no tool fits, allow the agent to answer without a tool if the task permits it.

Do not fall back to the entire catalog for consequential workflows. That restores the original context and exposure problem exactly when the selector is least reliable. If an all-tools fallback exists for low-risk development use, gate it by environment and record when it runs.

Test selection as its own system

A successful final answer cannot prove that discovery selected the right candidate set. Build a labeled fixture set where each task names the expected tool or acceptable tool set. Include ordinary tasks, ambiguous wording, similar tools, multi-tool tasks, permission changes, disabled integrations, renamed schemas, and requests for forbidden capabilities.

Compare the dynamic selector with an all-tools baseline on these measures:

  • candidate recall, meaning the expected tool entered the loaded set
  • false selection rate for irrelevant or confusing tools
  • denied-tool exposure in names, descriptions, and traces
  • input tokens used for tool definitions
  • discovery and end-to-end latency
  • tool-call validity and task completion

Run negative tests with two users who have different scopes. The same request should produce different eligible catalogs when policy requires it. Then change a schema, emit a catalog update, and prove that new runs use the new version while existing traces remain reproducible.

Roll out in shadow mode first. Generate the dynamic candidate set, but let the current production path make the actual decision. Compare both paths until recall and policy behavior meet the release threshold. Then enable dynamic selection for a low-risk namespace before expanding it.

Put AI agent tool selection into production

Start with one agent whose catalog is large enough to show the problem. Export its tools into versioned catalog records, add the deterministic eligibility filter, and label 30 to 50 real tasks with acceptable tools. Measure the all-tools baseline before changing the prompt.

Next, retrieve a bounded candidate set and shadow it against production. Do not ship on token savings alone. Require tool recall, denied-tool exposure, call validity, latency, and task completion to stay inside explicit thresholds. Once those checks pass, enable one namespace and keep the typed execution gate behind it.

This design fills the implementation gap in most tool-search guides. Discovery becomes a policy-aware, versioned subsystem with defined fallbacks and its own evaluation contract.

References

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the policy-aware discovery, deferred schema loading, and selection evaluation described above, at published fixed prices. Schedule a call to discuss your next project.