Agentic Retrieval Loops and When Agents Should Re-Query the Web

A research agent answers a multi-hop legal question in production. It retrieves eight chunks, drafts a response, and ships it. Two days later, a customer flags a fabricated paragraph, a citation that doesn't exist, an inference nobody made. This is the failure mode agentic RAG exists to fix, and understanding exactly when an agent should re-query the web (not whether it can, but when it should) is the difference between a system that catches its own mistakes and one that ships them.

The trace from that case tells you everything. Six of the eight retrieved chunks were used correctly. The seventh fact was invented outright, stitched into the answer with the same confidence as the real material. There was no faithfulness score attached to the draft, no judge model checking the output against the source chunks, no re-query gate that could have caught the gap before the answer went out. The framework in use had every feature available: retrieval, generation, tool calls, memory. What it lacked was a policy, a self-check loop that would have looked at the draft, noticed the seventh claim had no support, and gone back for more evidence before shipping.

Classic RAG can't catch this because it's built for one-shot lookups, not for looking back. It's a linear function: query in, retrieve once, generate once, answer out. The model has no say over whether retrieval happens, how many times it happens, or whether the result is any good. According to research published as arXiv 2509.04820, 48% of traditional RAG failures trace back to the golden chunk simply not showing up in the top-k results. That's a problem outside model quality. That's a structural miss, the kind no amount of prompt tuning fixes, because the retriever pulled the wrong evidence and the pipeline had no way to notice or recover.

Single-pass retrieval is fast and cheap, and for most questions, that's fine. It falls apart on multi-hop questions, ambiguous questions, or anything time-sensitive where the corpus might be stale. The fix is a different pipeline architecture, not a smarter retriever bolted onto the same one-shot design. It's a policy layer that decides, on a per-query basis, when one retrieval pass is enough and when it isn't.

What makes agentic RAG a policy rather than a pipeline

Agentic RAG is a loop with a decision-maker sitting inside it, not a function with fixed steps. The agent decides, at runtime, whether to retrieve at all, what to retrieve, which source to pull from, and whether what came back is actually good enough to answer with.

Four primitives separate this from classic RAG, drawing on the taxonomy laid out in the FutureAGI survey on agentic RAG systems. First, the decision to retrieve: trivial questions get answered from the model's own knowledge, and the retriever only gets called when the agent judges it's needed. Second, query transformation: the agent rewrites or splits the user's question before retrieval runs, so a multi-hop question becomes several targeted sub-queries instead of one vague one. Third, iterative retrieval: retrieve, read, decide if there's enough evidence, and if not, retrieve again, looping until the agent judges it has what it needs or hits a step limit. Fourth, self-check: a faithfulness or groundedness judge scores the draft before anything ships, and if a claim gets flagged as unsupported, the loop kicks back to retrieval instead of letting the answer through.

These four primitives borrow directly from broader agentic AI patterns: reflection, planning, tool use, multi-agent handoffs. Applied to retrieval specifically, they turn what used to be a single database call into a decision function with inputs, thresholds, and failure modes of its own.

The practical contrast shows up in call counts. Classic RAG makes exactly one retrieval call per turn, always. Agentic RAG makes anywhere from one to six, and that range isn't sloppiness, it's the policy doing its job: easy questions get routed through fast, hard ones get worked harder. Every extra hop costs tokens and time, so the trade-off adds measurable overhead. A well-tuned system spends that cost only where the question actually demands it, which is why calling this "architecture" undersells it. It's an engineering discipline with its own failure modes, and those failure modes are exactly what showed up in the legal-question case.

The three architectural patterns production systems use to implement re-query decisions

Three patterns dominate how production teams actually implement the re-query decision, and each answers a slightly different version of the same question: when is the retrieved evidence not good enough?

Self-RAG, introduced by Asai et al. in 2023, has the model emit special reflection tokens as part of its own inference process. Should it retrieve at all for this query? Are the passages it pulled actually relevant? Does the generated answer hold up against them? Is the answer even useful to the user? That self-critique loop is precisely what catches unsupported claims before they ship, rather than surfacing only after a downstream complaint. Self-RAG fits best in regulated domains, legal, medical, financial, where the cost of a hallucination is measured in more than embarrassment.

Corrective RAG, or CRAG, from Yan et al. in 2024, takes a different approach. Instead of the model critiquing itself, a separate retrieval evaluator scores the quality of what came back from the retriever. If the evidence is weak, CRAG triggers a fallback path, often swapping a vector database lookup for a live web search. The re-query decision here isn't introspective, it's an external verdict on retrieval quality. In production, CRAG frequently gets paired with knowledge graph queries for questions that hinge on relationships between entities rather than raw semantic similarity.

Adaptive RAG works upstream of both. A query classifier looks at each incoming request before any retrieval runs and routes it by difficulty: simple factoid questions skip retrieval entirely, moderate questions get a single vector search, and only the genuinely complex multi-step questions get the full agentic loop with iteration and self-checking. Somewhere around 60 to 70% of production queries fall into the simple category, and routing those away from the expensive loop saves real cost without touching quality on the hard questions. Adaptive RAG makes explicit what the other two patterns leave implicit: not every question deserves a re-query, and the classifier making that call is itself a policy decision, with its own thresholds and its own failure modes.

None of these three are mutually exclusive. Production systems tend to stack them: CRAG's external evaluator can be the trigger that kicks off Self-RAG's re-retrieval branch, with Adaptive RAG's classifier deciding upfront whether either of them needs to run at all.

One-shot versus iterative retrieval: where the performance-latency tradeoff lives

Lin et al. lay out two strategies for improving on the standard fixed top-k, single-pass retrieval setup, and they land in very different places on the cost-versus-reliability curve.

The first, which the paper calls One-SHOT, drops the fixed top-k constraint. Instead of always pulling a set number of chunks, it selects as many chunks as fit inside a token budget, ranked by relevance-per-token, so evidence density goes up without retrieval happening more than once. Rule-based filtering sits on top to clean out weak matches. The second, iterative retrieval, has a reasoning-capable model issue its own intermediate queries, evaluate what comes back, and refine the context across multiple turns.

Iteration sounds like the obvious upgrade, but Lin et al. flag two specific ways it goes wrong. Query drift is when the agent's follow-up queries wander from the original intent, chasing a tangent a few hops in. Retrieval laziness is the opposite problem: the agent stops re-querying before it actually has enough evidence, satisfied too early. Both are failures in retrieval policy, which is exactly why they're hard to catch in a demo and easy to miss in production.

A complicating finding buried in this research deserves sitting with. SearchR1-32B, one of the models studied, issues only 1.1 to 1.2 searches per question on average, essentially behaving like a single-shot retriever even when it has the option to iterate. That suggests end-to-end performance in that case is driven mostly by how well the initial query was formed and by the base retriever's quality, not by how many times the loop runs. The design implication follows directly: for a model that's strong at initial query formulation, extra retrieval hops buy less marginal benefit, and the threshold for triggering a re-query should be calibrated to that model's own query quality rather than applied as a flat rule.

None of this is free. Agentic RAG runs 3 to 10 times the token cost of simple RAG, and p95 latency stretches from the 1 to 2 second range up to 4 to 15 seconds. The tradeoff: agentic RAG trades latency and token spend for faithfulness on hard questions. If the hardest questions in a given product are single-document lookups, classic RAG is the right call, not a lesser one.

The specific conditions that should trigger a re-query

A re-query should fire when there's a concrete signal that the evidence on hand isn't sufficient, not on a fixed schedule and not on a hunch. Five conditions do the job. A faithfulness judge flags a claim in the draft that the retrieved chunks don't actually support. Retrieved passages score below a relevance threshold, which is CRAG's evaluator pattern in action. Self-RAG's reflection tokens come back indicating the generated answer isn't grounded in what was retrieved. A multi-hop question has been partially answered but a required intermediate entity, a name, a date, a linking fact, is still missing. Or the evidence on hand is stale for a time-sensitive question, and recency weighting falls below an acceptable threshold.

Just as important is knowing when not to re-query, because over-triggering burns the same latency and token budget it's meant to protect. Simple factoid questions the model can answer from its own training don't need retrieval at all, which is Adaptive RAG's classifier doing its job upstream. Re-retrieving chunks the agent already pulled earlier in the same session is waste, and proper state carry-over prevents it. And when retrieved evidence is actually fine but the agent's confidence reads low because the question was phrased ambiguously, the fix is rewriting the query, not looping back for more of the same evidence.

Termination matters as much as triggering. Production systems typically cap retrieval at 4 to 6 steps per turn, because without a hard ceiling, loops burn tokens and time without ever converging on an answer. Beyond the step cap, an explicit "enough evidence" signal should end the loop on its own, whether that's a confidence score clearing a bar or a faithfulness check passing. Ragas-style automated metrics give concrete gates to check against: faithfulness at or above 0.9, answer relevancy at or above 0.85, context precision at or above 0.8.

This is where most teams underinvest: the opening hallucination case had a step limit tied to how many times retrieval could run, but nothing tied to whether the answer was actually faithful to what was retrieved. A step cap alone doesn't catch a fabricated fact; only a faithfulness gate does. One useful hedge against the latency cost of all this: when a question breaks into several sub-queries, running them in parallel rather than one after another cuts end-to-end latency without touching the re-query policy itself.

Tool routing: choosing the right retrieval source when re-querying

Production agentic RAG rarely leans on a single retriever, and treating retrieval as one pipeline regardless of query type is, according to AWS's Well-Architected Agentic AI Lens, a named failure mode in its own right. Different questions call for different sources.

Dense vector search handles semantic similarity and covers the majority of queries by default. BM25 or other sparse retrieval methods handle exact term matching, error codes, version numbers, IDs, the kind of precise string a vector embedding tends to smooth over and lose. Hybrid search, BM25 and dense fused together through something like Reciprocal Rank Fusion, covers the common case where both signals matter, and running vector search alone, without a sparse complement, is increasingly treated as a design mistake in 2026-era production stacks. Knowledge graph traversal, the approach behind Microsoft's open-sourced GraphRAG, handles relational questions that similarity search structurally can't answer, and shows a 4 to 10% F1 improvement over vector-only retrieval on multi-hop reasoning benchmarks. Web search comes in when the internal corpus is stale on something recent, the one re-query path that steps outside the vector database. And structured data, SQL against a real database, handles questions that are actually metrics, such as a count, a sum, or a comparison between two numbers.

CRAG's evaluator plays a direct role here too: when it judges internal evidence weak, it can trigger the switch from vector database to web search on its own, a corrective reflex rather than a branch planned in advance.

Web search as a re-query target brings its own infrastructure problem, though, one that's easy to miss until it bites. Standard SERP APIs typically return short snippets, often in the 150 to 300 character range, which is metadata about a page, not the content of it. That's not enough for an LLM to reason from when the whole point of the re-query was to get better evidence than the first pass produced. If the web retrieval leg comes back with shallow snippets, or breaks under anti-bot measures on the target site, the re-query has made things worse, not better. Routing to a source that can't deliver real content isn't a fix, it's just a different way of running out of evidence.

Context engineering: shaping what the agent sees after each retrieval hop

Context engineering means deliberately designing what an LLM sees on every single inference call: what gets selected, what gets filtered out, what gets compressed, how it's ordered, and how memory carries across turns. It's a different discipline from prompt engineering, and it matters more as loops get longer.

A single-turn chatbot only has to manage the context from one exchange. An agent at step 47 of a retrieval loop is carrying residue from steps 1 through 46, and both token budget and attention budget are finite. Most context failures in long-running agentic loops trace back to how that limited budget got spent, not to a weak initial prompt. Gartner flagged this shift directly for 2025, framing it as "Context Engineering Is In, Prompt Engineering Is Out," a signal that the field's center of gravity had moved from writing better instructions to managing better context.

Four building blocks matter most for shaping what an agent sees after a retrieval hop. Selection and filtering apply relevance scoring, recency weighting, and salience thresholds to decide which chunks even make it into context. Compression and distillation, whether extractive or abstractive, shrink context size without losing the meaning that matters, which becomes essential once iterative retrieval has piled up more chunks than the context window can hold. Temporal management draws the line between short-term and long-term memory, session versus cross-session memory, and decay policies that let old, no-longer-relevant evidence drop out. Context assembly and ordering, meanwhile, governs instruction hierarchy and where tool results and memory get placed relative to each other, since ordering shapes what the model actually attends to, not just what's technically present.

State carry-over is the specific move that stops retrieval laziness and wasted re-querying from compounding: passing prior chunks, or a summary of them, into the next retrieval prompt so the agent isn't looping back to fetch evidence it's already holding. The strategic point that this drives: as models converge in raw capability, the edge in enterprise AI shifts toward whichever system assembles the richest, most accurate, most current context, not whichever one has the marginally smarter model sitting on top of it.

What the web retrieval layer must do to support production re-query logic

The web search leg of a re-query loop has different demands than a single-pass RAG setup bolted onto a static internal corpus. It has to be fast enough that it doesn't blow the latency budget the rest of the loop is already spending down, reliable enough that it doesn't fail silently mid-loop, and deep enough that what comes back is actual content an LLM can reason from, not just a pointer to content.

Standard search APIs run into a structural limit here. They're built to return titles, URLs, and short snippets, useful for a human scanning a results page, not for a model that needs full page context to check a claim or synthesize evidence across sources. A snippet in the 150 to 300 character range simply doesn't carry enough material for faithfulness evaluation or for stitching together multi-hop evidence, no matter how well the rest of the loop is engineered.

Scraping tools carry a different weakness. They break when a target site changes its HTML structure, updates CSS class names, or tightens anti-bot defenses, and even well-regarded web scraping API services average somewhere in the 85 to 98% success rate range, according to Proxyway's Web Scraping API Report 2025. That failure rate is tolerable for a batch data collection job that can retry overnight. It's not tolerable for an agent that needs to re-query reliably, mid-inference, with a user waiting on the other end. Both categories of tool were built for a different job: surfacing links for people to click, or scraping pages in bulk, not feeding full, current, structured content into a live reasoning loop that has to decide, in seconds, whether it finally has enough to answer.