Tool-Calling Design for Web Search in LLM Agents

The ability of an LLM agent to act on the web through function calling

Web search tool-calling looks simple from the outside: give an LLM agent a search function, and it returns grounded answers instead of hallucinated ones. That model is wrong, and it fails in a specific, traceable way. An agent has to decide when to search, keep track of what it already pulled, and figure out what to do with the results once they land. Most production failures trace back to one of those three steps breaking quietly, not loudly. That is why they survive a demo and die in production.

The underlying infrastructure was never built for this job. Web pages exist for humans clicking through browsers: navigation menus, ad slots, cookie banners, boilerplate footers nobody reads. None of that serves a model. A search wrapper that returns ten links forces an agent into ten more requests just to learn what's actually on those pages, and raw HTML burns tokens on markup before delivering one usable sentence. So the failures split across three layers, and each is invisible until you go looking for it: the model doesn't know when to call the tool, the model calls the tool but sends a bad query, or the tool returns results the model can't parse into anything useful. Fix one layer without the others and the failure doesn't disappear. It moves downstream to whichever layer you left alone.

Function calling is what lets a language model reach past its own weights and touch something real. The model gets a schema describing an external function, decides on its own whether that function is needed, emits a structured call with arguments, receives a result, and folds that result into the next step of reasoning. Run that loop enough times and a text generator starts behaving like an operator.

Inside the agent, the sequence is fairly mechanical. First it parses what the user actually wants. Then it checks that intent against whatever tool schemas are registered (a step often called tool routing) and decides whether the question can be answered from what the model already knows or whether a call is required. If a call is required, it generates a structured JSON invocation using the arguments the schema defines, and once the result comes back, that result becomes input to the next round of planning.

The dominant pattern for structuring this is ReAct-style prompting, where the model interleaves an explicit reasoning step, often literally labeled "Thought:", with an action step such as "Act: search(...)" before it fires the tool. The pattern holds that reasoning and acting reinforce each other rather than competing for the model's attention, and it has since become close to standard across agentic search systems and widely referenced in current agentic framework documentation.

A separate decision is whether to use function calling at all or adopt MCP (Model Context Protocol), an open standard introduced by Anthropic in late 2024 for connecting models to external tools,, and it deserves to be made deliberately rather than by default. Function calling fits when the model chooses among a handful of tools at runtime and the developer controls what's registered. MCP fits when the goal is dynamic tool discovery, plugging an agent into tools it wasn't explicitly wired for ahead of time. OAuth 2.1 support arrived in the protocol's March 2025 revision. Neither approach beats the other on raw capability; treating this as a capability question, rather than a design-intent question, is the first mistake most teams make here.

What carries forward from all of this is one dependency: the quality of any tool call rests entirely on what the model was told about the tool in the first place. That puts tool definition first in line.

Writing a tool definition the model will use correctly

A tool schema is the only instruction manual the model gets. There's no onboarding conversation, no chance to ask a follow-up question. If the description is vague, or the tool tries to do three things at once, the model will guess, and its guesses will be inconsistent across runs even when the underlying task hasn't changed.

A working web search tool definition needs several things present at once, not scattered across a rewrite two versions later. It needs a precise, narrow description of what the tool does and doesn't do, specific enough that the model can tell it apart from something like a vector store retriever sitting in the same toolbox. It needs a clear statement of when to call it: for information past the model's training cutoff, for anything time-sensitive, for claims that need a citation attached. It needs typed, documented parameters (a query string, maybe an optional domain filter, a recency constraint, a depth switch distinguishing a fast lookup from a thorough one). And it needs a documented return shape, so the model knows what fields to expect instead of pattern-matching against whatever text happens to come back.

Describing the tool as "search the web," full stop, is the single most common mistake teams make, and it's a costly one. It leaves the model to invent its own invocation logic, and it will invent something inconsistent from one run to the next. Bundling retrieval and extraction into a single schema is nearly as bad: the model can't reason separately about "find sources" and "pull content from a source" if both operations hide behind one opaque function call. Omitting the return shape is the quietest failure of the three, because nothing breaks visibly. The model just burns context guessing at structure, and that guessing introduces errors that become visible three steps later in the reasoning chain, far from where the actual mistake happened.

Tool-based abstraction frameworks like WALT point at the same underlying principle. WALT surfaces a website's own native operations, search, filter, sort, as separate, deterministic, callable tools instead of asking the model to reason through a UI step by step, offloading fragile, multi-step interface logic onto scripts that have already been validated. Atomic, single-purpose tools consistently beat one broad tool trying to cover everything. Any team still building the broad version is building the wrong thing, full stop.

There's a cost that occurs before any search even runs, too. Some tool-registry approaches inject every registered tool schema into the system prompt at once, which inflates the context window before the agent has done a single unit of useful work. Active tool discovery addresses this by pulling in only the schemas relevant to the current task rather than the full set upfront. That tradeoff matters more as the number of tools an agent carries grows, and it starts to matter a great deal once that number crosses into the dozens.

A useful gut check for anyone writing one of these schemas: read the description as if it were the only information available, with no other context, and ask honestly whether a correct call could be written from it alone. If the answer is no, the model has the same problem you do.

Forming queries the model should send

A tool wired up correctly still fails if the query going into it is wrong, and this failure mode is sneaky precisely because the tool call succeeds at the API level. Nothing errors out. The agent just gets back an answer to a question nobody actually asked.

The default failure is echo. LLMs tend to pass the user's own phrasing straight into the search call, and user phrasing is rarely a good search query. It's conversational, leaning on pronouns and implicit context from earlier in the conversation, on assumptions the search index has no way to resolve. "What about the second one?" means nothing to a retrieval system with no memory of what "the second one" refers to.

Query decomposition is the strongest lever available here, and it belongs in the default design, not bolted on later as an optimization. Complex tasks should break into sub-queries, each aimed at a single retrievable fact rather than the whole compound question at once. Some systems push this further with multi-agent pipelines, dedicating separate sub-agents to acronym resolution, keyphrase extraction, and sub-query generation, an approach that handles dense or domain-specific material more reliably, as recent work on multi-agent retrieval pipelines has shown. Even single-agent setups benefit from an explicit rewriting step sitting between the user's message and the actual search call.

Keyword matching and semantic search solve different problems, and conflating them is where a lot of query design goes wrong. Keyword matching works fine when the target is a known entity or an exact phrase: a product name, a case number, something with one right answer. It falls apart on intent-driven questions, where retrieval has to match what the user means rather than which words they used. Semantic search APIs process natural-language input based on meaning rather than term overlap, which lines up far better with how LLMs actually phrase things. Declarative search, describing what's needed and letting the system return ranked results, removes almost all of the query-crafting burden from the model. That's the right default for agentic pipelines, essential to how they should work from the start.

Source bias compounds the problem quietly. Left alone, LLM search policies gravitate toward highly connected general sources, encyclopedic sites being the obvious example, rather than sources that are actually authoritative for the domain in question. Query design has to build in domain or source-type constraints whenever credibility is the point, because the model won't reach for the right source on its own.

The single highest-leverage decision at this layer is whether the architecture loops at all. Research on agentic deep research has found that standard LLMs relying on basic keyword search perform poorly on complex, multi-hop research benchmarks. Systems built around iterative retrieval (search, reason over what came back, search again with a refined query) scored dramatically higher on the same benchmarks. The specific tool mattered far less than whether the system was built to loop. Teams arguing over which search API to buy are usually optimizing the wrong variable.

In practice, that means telling the model explicitly, in the system prompt, to rewrite queries before calling the tool rather than assuming it'll do so unprompted. It means feeding the model's own intermediate reasoning, the ReAct "Thought:" step, into that rewriting process, since the model's chain of thought is a better seed for a query than the user's raw message ever will be. And it means using recency constraints when time sensitivity actually matters, while leaving them off when breadth is worth more than freshness.

What the search API returns and how the format determines whether reasoning succeeds

Finding the right page is only half the problem. What the API hands back, and in what shape, decides whether the model can do anything useful with it.

Search APIs split roughly into two tiers, and picking the wrong one means building an extraction layer you didn't need to build. Traditional search-results APIs return the kind of thing a search engine shows a browser: titles, URLs, short snippets. That's high fidelity to the human search experience, but an extraction layer still has to run afterward before any of it reaches the model, since the API itself was never designed with an agent pipeline in mind. AI-native search APIs skip that gap. They return content already processed for a language model: extracted text, citations, structured fields, configurable depth settings, all shaped for a pipeline rather than a browser tab.

That processing does real work underneath. JavaScript-heavy pages, CAPTCHAs, PDFs, the sources a plain scraper trips over under real-world conditions, get handled before the content reaches the model. Passage-level extraction (chunking a page and surfacing the specific paragraphs relevant to the query rather than dumping the entire document) saves the model from wading through material it doesn't need. Relevance ranking tuned for meaning rather than click-through data serves an agent's purpose better than a ranking built to predict what a human eye would click next.

Latency deserves equal billing, because a production system has to behave predictably under real load, not just look good in a demo. Some APIs offer distinct modes: a faster, lighter-weight option for latency-sensitive work like voice agents or consumer chat, and a slower but higher-quality mode for the bulk of agent workloads where an extra second doesn't matter but accuracy does. Which mode is right depends entirely on where the search call sits in the larger pipeline.

Vendor dependency is a risk that has to be named directly, not hedged around. When an entire retrieval pipeline runs through one third-party service, an acquisition, a pricing change, or a policy shift on that vendor's end can destabilize a production system overnight. Keeping some control over the data pipeline, rather than treating a single vendor as bedrock, reduces that exposure.

The question to ask of any search API under evaluation is blunt: does it return something a model can reason over, or something built for a browser to render? Most vendor pitches dodge that question. Ask it directly rather than inferring the answer from a demo.

Shaping retrieved content into context the model can reason over

Retrieval isn't the finish line. What happens to content after it arrives, before it ever reaches the model's context window, is where a surprising number of production systems quietly fall apart, and it's the layer most teams underinvest in.

The core problem has a name: context rot. Research into model behavior with long contexts has consistently found that models degrade as input length grows. Models that perform reliably on shorter inputs show measurable accuracy drops once inputs grow long enough. The mechanism isn't mysterious: every token in the context window draws from a finite attention budget, and irrelevant material buries the useful material in zones of the context the model barely attends to.

This is why context engineering has emerged as its own discipline rather than an afterthought to prompting. Shopify's CEO, Tobi Lütke, coined the term in June 2025, framing it as the discipline of shaping what information a model receives before it acts. The framing matters because it shifts the emphasis: a complex agent system succeeds or fails based on the quality and completeness of what's fed into it, far more than on how cleverly the prompt itself is worded. Teams that keep tuning prompt wording while ignoring retrieval quality are polishing the wrong surface.

LangChain's 2025 systematization breaks the discipline into four strategies. Write means recording what was retrieved and when, so the system has a memory of its own actions. Select means pulling only the passages relevant to the current step of reasoning. Compress means condensing accumulated retrieval history without losing its meaning, through summarization, targeted extraction, or deduplication methods like diversity-based selection. Isolate means restricting what different sub-agents in a multi-agent pipeline can see, so each one works with only what it actually needs rather than the full shared history.

Google's ADK formalizes something similar with a three-tier context stack: storage sources such as sessions, memory, and artifacts feed into flows and processors, a compiler-style pipeline of named, ordered transformations, which then produce the compiled working context the model actually sees. Context, in this framing, is a compiled artifact, assembled deliberately, layer by layer. Treating it as an afterthought is how context rot happens when nobody manages the assembly.

How content gets split changes recall meaningfully. Semantic chunking, dividing text along natural argument or topic boundaries rather than at a fixed character count, improves recall over naive fixed-size chunking. A chunk boundary that falls in the middle of an argument loses the argument, no matter how good the retrieval was upstream. Position within the context window carries its own bias, too: content placed early gets disproportionate attention regardless of how important it actually is, so critical evidence needs to sit where the model will attend to it, matched to its importance rather than wherever it landed in the retrieval order.

In practice, that means filtering before injection: stripping navigation chrome, boilerplate, and off-topic passages before anything enters the window. It means ranking by relevance to the current sub-task rather than trusting the search API's default order. It means summarizing or extracting at the passage level for long documents instead of injecting full pages wholesale. And it means tracking what's already been retrieved in the agent's memory, so the same content doesn't get pulled and re-injected across multiple loop iterations, quietly eating the attention budget twice.

Connecting the layers into a search loop that holds up across turns

None of the decisions above operate in isolation. What an agent actually does emerges from how tool definition, query formation, and content shaping compose together across repeated turns. Most teams debug the wrong layer when something breaks, because the layer that fails is rarely the layer that shows the symptom.

A full ReAct-style search loop runs something like this. The model reasons first, in a Thought step, about what it knows and what's missing. The query gets rewritten, translating raw user intent into something a search index can work with. The tool call fires with typed, structured parameters. The API responds with content already shaped for machine consumption rather than a browser. That content passes through filtering, ranking, compression, and isolation before it touches the context window, and once it's there, as an observation, the model evaluates whether the goal has been met or whether another round of search is needed.

Most of the actual breakage happens at the seams between these steps, not inside any single one. It's so easy to miss in a demo. Between query formation and the API call, a rewritten query that's still keyword-shaped underperforms badly on a task that needs semantic retrieval, even though the tool call executes without error. Between the API response and content shaping, full unfiltered pages arriving with no extraction step means the model spends its attention budget on boilerplate instead of the fact it was sent to find. Neither failure throws an error message. Both just produce an agent that quietly gives worse answers than the architecture should allow. By the time that becomes visible in production, it's a lot more expensive to trace back to the seam that caused it.