An AI agent connected to many tools needs to find the right one before it can use it. Loading the whole catalogue gives the agent details for tools it does not need, taking up space it needs for task context and consuming costly input tokens. We tested searching for the right tool and then loading its details in a catalogue of 234 Snowflake tools. This returned 98% less text than loading the whole catalogue, even after accounting for searches that missed.
The benchmark tool catalogue contains standard Snowflake API operations, including restoring a dropped table and generating a file download link. Each tool has a name, a one-line description and an input schema describing the parameters needed for a tool call.
Three ways to find a tool
Gatekeeper governs tool calls between AI agents (both unsupervised and human-interactive) and connected services. Before calling an unfamiliar tool, an agent needs its name and input schema. We compared three ways of getting that information:
- Full listing: return every tool’s schema in a single call.
- Names-then-expand: list all tool names, then load the schema for one selected tool. Requires two calls.
- Search-then-expand: run a keyword query, then load the schema for one selected tool. Requires two calls.
We measured the amount of text returned during discovery. The test selected the correct tool by script, so it measured the amount of text that each approach returned rather than whether a model could choose correctly.
| Discovery approach | Tokens | Result type |
|---|---|---|
| Full listing | 88,852 | Measured |
| Names then expand | 4,329 | Measured |
| Successful search then expand | 1,312 | Measured |
| Search with fallback after miss | 1,756 (est.) | Calculated estimate |
Token counts for the three measured paths are means across 36 tasks. The fallback row is calculated from the benchmark results. The gateway’s own instructions and tool definitions are counted separately.
How much text is saved with search
Figure 1. Tokens returned during discovery for full listing, names-then-expand and search-with-fallback. All approaches use the same build.
Listing every schema returned 88,852 tokens. Listing names and loading one schema reduced that to 4,329. When search found the intended tool, searching and loading its schema averaged 1,312 tokens.
Figure 2. A closer comparison of names-then-expand, successful search-then-expand and search-with-fallback.
Using search resulted in the intended tool on the first page for 71 of 80 queries. In the 9 cases where search failed to find the tool, we compute the full cost: the search plus a names-then-expand lookup. Counting those misses, search averaged 1,756 tokens per task; that’s 59.4% fewer than names-then-expand and 98.0% fewer than the full listing (default) technique.
Separate formatting from search
The tested build uses compact JSON rather than indented JSON. Formatting alone reduced tokens by 19.9% for the full listing and 26.2% for names-then-expand compared with earlier builds. All approaches in Figures 1 and 2 use the same compact format, so those comparisons isolate the discovery approach.
What happens when search misses
The missed queries showed how much keyword search depended on the catalogue’s wording. One Snowflake tool was described as “Undrop a table”. Requests to restore, recover or bring back a dropped table did not put it on the first results page. The word “table” matched 64 tools.
Figure 3. A missed query from the benchmark.
Other misses included “exchange tables” and “promote replica”. Our interpretation is that the requested verb was missing from the description while a common noun returns too many competing matches.
Finding the tool on the first page also differs from ranking it first. The intended tool ranked first in 32% of queries and within the top five in 60%. A model still needs to choose among the returned candidates; that choice was scripted in this benchmark.
What to test in a tool catalogue
- Use words that users use: The misses suggest synonyms to test in descriptions.
- Check retrieval and selection separately: A search can return the right tool without establishing that a model will choose it.
- Include the cost of missed searches: The extra query and fallback calls belong in the discovery-token calculation.
How we measured discovery
We used an automated test to send discovery requests to Gatekeeper over MCP (Model Context Protocol) and measured the text in each response.
The benchmark covered 36 labelled tasks, each repeated three times, and 80 plain-language queries, with two or three queries per task. Queries avoided the tool’s name and were written by someone who had seen the catalogue. Each search page contained up to 25 results.
Token counts came from Claude Code’s usage report for claude-opus-5-5. The counts excluded the small amount of text a client adds around tool results.
The updated run completed 1,425 discovery calls with zero failures.
Latency and fixed overhead
Gateway timings were similar for both approaches. Searching and then loading the selected tool’s schema took a median of 660 ms. Listing all tool names and then loading the selected tool’s schema took a median of 666 ms.
There were three samples per trial. Timings were measured from the client and excluded model processing and complete task duration.
Adding search support increased the gateway’s own instructions and tool definitions from approximately 8,193 to 8,720 tokens, an additional 527 tokens. This fixed overhead is separate from the returned discovery text. The benchmark did not model prompt caching or billed task cost.
Conclusions
Searching for a tool and then loading its details returned the least text of the three approaches in this Snowflake benchmark. After accounting for searches that missed, it returned an estimated 59.4% less text than listing all tool names and then loading one tool’s details, and 98% less than loading the whole catalogue.
The missed searches also pointed to a practical next step: test whether descriptions that use the words people actually search for make tools easier to find.
Source and related reading
Source: Station70 Research, Gatekeeper Discovery Benchmark. Measurements taken October 1–2, 2026.