Query Decomposition and Sub-Query Planning for Multi-Step Web Research

Most retrieval-augmented systems still run on a pattern built for a simpler kind of question. A query goes in, a vector search fires, the top-k chunks land in the model's context window, and an answer comes out. For a factual lookup like "when was the Federal Reserve founded," that pipeline works fine. It falls apart the moment answering requires assembling evidence from multiple sources, or reasoning across two or more entities that don't share a document.

Complexity here has a specific shape, and it repeats across categories. Comparative questions that ask about two entities across several attributes at once produce it. Causal chains produce it too, since the answer to "why did X lead to Y" depends on connecting facts scattered across separate places. So do policy questions needing both a current data point and historical context, and synthesis questions pulling from domains that never cite each other to begin with.

A single query string is a compressed, lossy stand-in for what the user actually wants to know. Retrieval run against that string finds documents that sound like the query, not documents that collectively answer it. Ask a naive system "How do Tesla's battery innovations compare to traditional automakers, and what are the cost implications?" and it retrieves a pile of documents about Tesla and batteries, because that's what the surface phrasing points to. It has no mechanism for noticing that the question contains multiple distinct information needs: Tesla's battery technology, the comparable technology at legacy automakers, and the cost relationship between them. Nobody built that noticing into the retrieval step, so it doesn't happen.

What query decomposition does architecturally

Decomposition flips the system's posture from reactive to deliberate. Instead of answering the query as posed, the agent first works out what it would need to know to answer it, then goes and gets that, piece by piece. That shift turns retrieval into something closer to a research process than a lookup, and it's the difference that matters most in this whole discussion.

Mechanically, the agent breaks the user's query into a sequence of sub-questions, decides which ones matter most, and writes that plan down as an explicit structure in working memory before a single retrieval call goes out. Query expansion and paraphrasing don't do this. Both stay inside the boundary of a single information need, while decomposition is semantic parsing of intent, drawing a hard line between what needs to be retrieved and how that retrieved material gets used afterward.

One clean formalization of this comes from the Scope/Task pattern described in the AgenticScholar approach to analytical decomposition. Scope defines the entities or conceptual boundary under analysis, the "what." Task defines the analytical operation to run against that scope, the "how." An LLM, prompted with an instruction that separates these two components explicitly, produces a plan where each sub-query carries both labels. That separation dictates ordering later on: a system has to know what it's looking at before it can perform an operation on it.

Sequential versus parallel execution

Once a plan exists, the orchestrator has to decide how to run it. Two modes exist, and picking the wrong one is where a lot of production latency gets wasted for no good reason.

Sequential execution applies when one sub-query's answer feeds directly into the next: find the document, then pull the specific metric out of it. Each step gates the one after it, and there's no way around that ordering, since the second step literally cannot start without the first one's output.

Parallel execution applies when sub-queries are logically independent. Asking for two people's birth dates in order to compare them doesn't require finishing the first lookup before starting the second; both fire at once and get merged afterward. Production planners default to parallel whenever independence holds, and the logic is straightforward: sequential-by-default is the slower option when independence between sub-queries already holds. To make the decision machine-readable, sub-queries in practice carry some form of dependency marker, so the orchestrator knows at a glance which branches wait and which run immediately.

Treating sequential as the safe default is a mistake, and the numbers say so directly. The ParallelSearch paper found that on parallelizable questions, sequential approaches average 3.36 LLM calls to reach an answer, while a parallel decomposition strategy cuts that to 2.34 calls. That's a 12.7% improvement in outcome quality on parallelizable benchmarks, using only 69.6% of the LLM calls sequential execution would need. For a decision that comes down to a scheduling flag, that's a lot of performance left on the table by systems that default to sequential out of caution rather than necessity.

How agents order sub-questions and manage dependencies

Even within a single execution mode, order matters, and planners lean on a few consistent heuristics to decide it.

Evidence utility comes first. Whichever sub-query, once answered, narrows or constrains the others the most, goes first. The second heuristic aligns with the Scope/Task distinction: retrieve the entity boundary before running an analytical operation against it, since operating on an undefined scope produces garbage. The third is cost. Cheap, low-latency lookups, cached facts, structured lookups, anything short of a full document read, ought to happen before expensive operations that burn time and tokens.

That third heuristic has a name in production literature. The Progressive Evidence Acquisition with Cost-Aware Escalation pattern, described in a production paper out of Ontario Power Generation, formalizes it directly: start with low-cost, high-precision retrieval, and escalate to full-document reads only when the expected gain in evidence justifies the added latency and cost. Cheap before expensive, in other words, turned into an architectural rule instead of a vague intuition someone applies inconsistently.

A dependency structure underlies all of this. Sub-queries are ordered according to their data dependencies, and that ordering determines which steps must wait and which are safe to parallelize. Plans commonly conclude with a synthesis step that takes the outputs of prior sub-queries as its input rather than the user's original question. Conflicting evidence, if any appears during retrieval, needs to be reconciled before a final answer goes back to the user.

Guardrails that keep decomposition from becoming a liability

Sophistication cuts both ways here, and teams tend to underrate this until it costs them. The more capable a planner gets at generating sub-questions, the more capable it also gets at generating too many of them. Nothing in an unconstrained LLM call stops it from decomposing a moderately complex question into a sprawling, expensive tree of retrievals. Call it the autonomy paradox: better decomposition demands stricter operational limits, not looser ones. Any team that treats more sub-queries as automatically better evidence has the relationship backward, and that assumption is worth abandoning early rather than discovering the cost of it in a production bill.

Production systems handle this with hard caps, not gentle suggestions. PlannerAgent implementations commonly enforce a max_sub_queries parameter, with a default cited at 5, and enterprise deployments cap planning depth at 3 levels with circuit breakers built in to stop runaway expansion. These numbers are acknowledgments of a cost curve more than arbitrary limits. A single complex question can already expand into somewhere between 3 and 15 internal steps, each one carrying its own input context and generating its own output, and both sides of that ledger get billed. Estimates put agentic multi-hop token cost at 3 to 6 times a simple baseline query, and that multiplier climbs as traversal depth increases.

Cost isn't the only risk. LLMs generating query plans hallucinate structure just as easily as they hallucinate facts, returning malformed JSON, circular dependencies, or sub-query combinations that are logically impossible to satisfy. Production planners guard against this with schema validation, typically Pydantic schemas enforced strictly at the planning boundary. The systems that do this well fail fast rather than letting a broken plan propagate downstream into wasted retrieval calls.

Where decomposition fits in the broader agentic RAG stack

Decomposition occupies one specific point in the pipeline: the query understanding layer, positioned before retrieval, functioning as an optimization pass that keeps expensive vector searches from firing against poorly formed queries.

What happens after the plan is generated follows a fairly consistent shape across current systems. Each sub-query gets routed to a retriever, and by current production standards that retriever is hybrid: dense vector search fused with sparse BM25 lexical search, typically combined through Reciprocal Rank Fusion. Pure vector search alone is increasingly treated as a design mistake, a view reflected in retrieval benchmarking research and practitioner experience across production deployments. After retrieval, a cross-encoder reranker, tools like Cohere's Rerank 3.5 or Voyage AI's rerank-2.5 are commonly cited examples, re-scores the candidate passages before they reach the model's context. That reranking step alone is credited with a 5 to 15 point improvement in MRR on hard retrieval sets, not a small margin for a step that's still skipped in a lot of pipelines.

Some agentic RAG architectures add a retrieval evaluator on top of this that checks whether what came back is actually good evidence, and redirects to an alternate source when it isn't. That evaluation connects back to the sub-query level, since each individual sub-query's evidence quality gets checked, not the plan as a whole.

Adaptive RAG handles the cost side. A classifier sits in front of the whole pipeline and sorts incoming queries by difficulty: simple factual questions skip decomposition and go straight to a direct answer, moderately complex ones get a single-hop search, and only the genuinely multi-step questions get the full agentic treatment described above. An estimated 60 to 70% of production queries fall into that simple category, so classification alone saves a lot of compute without touching answer quality on the hard end of the distribution. For questions that hinge on relationships between entities rather than isolated facts, graph traversal can substitute for or sit alongside vector search at the individual sub-query level, and the planner's dependency graph maps onto that kind of traversal plan almost directly.

Requirements of the retrieval layer for a web search API

Every model has a training cutoff, and any sub-query touching something time-sensitive, current, or narrowly domain-specific has to be grounded against a live source instead of the model's internal knowledge. A stale index ranks among the most commonly cited production failure modes in RAG systems, which is the reason live web retrieval enters the picture.

Traditional search APIs weren't built for this, and the mismatch becomes visible fast under an agentic workload. Standard search APIs return raw HTML and heavy metadata, and the agent has to parse that output before using any of it, adding both token overhead and latency to every sub-query in a plan. That overhead doesn't stay small: a decomposed plan might fire a dozen sub-queries in parallel, and if each one carries its own parsing tax, the cost multiplies across every branch at once. Search snippets make it worse. They're written for a human skimming a results page, and they truncate at exactly the point where a reasoning agent needs more depth, not less.

Adapting a consumer search API to this job is the wrong move, not a workable shortcut. A retrieval layer built to serve a decomposed query plan needs machine-ready content, clean Markdown or structured text that goes straight into context without an intermediate cleanup pass. It needs latency that stays predictable under parallel load, because one slow call sitting on the critical path of an otherwise parallel plan reintroduces the exact bottleneck decomposition was supposed to remove. It needs content filtered against the specific sub-query's intent rather than the user's original phrasing, since relevance has to be scoped narrowly at each node of the plan. And it needs full content extraction rather than a truncated snippet, since an agent evaluating evidence quality needs the substance of a source, not its headline.

Infrastructure ownership decides all of this in practice. A search layer built as a wrapper around third-party services has no real control over the latency, depth, or structural consistency of what it hands back, and a decomposed plan running at production scale depends on all three holding steady. That's the argument for web knowledge infrastructure built ground-up for machine consumption, rather than retrofitted from tools designed for a search results page.

Building a decomposition strategy that holds up in production

A decomposition strategy that survives contact with real traffic tends to share a small set of design commitments, and skipping any one of them causes problems in production faster than most teams expect.

The query plan needs to exist as an explicit, validated data structure, and schema validation at the planning boundary isn't optional. Query complexity needs classification before any decomposition happens at all, since routing the majority of simple queries away from the full agentic path is where most of the cost savings actually live. Dependencies between sub-queries need explicit encoding, through flags an orchestrator can read, not implied by the order a prompt happened to list them in. Plan depth and sub-query count need enforced ceilings, tuned as configuration rather than hard-coded, following the same shape as the default-of-5 and 3-level patterns already in use elsewhere. The integration step, the synthesis query that reconciles everything gathered, deserves treatment as a first-class part of the plan rather than something bolted on at the end. And every decomposition decision is worth logging alongside its outcome, since decomposition quality is usually the upstream cause when retrieval quality goes wrong downstream.

The cost math backs up the investment. A complex question can expand into 3 to 15 internal steps at 3 to 6 times the token cost of a simple query, and a planner that routes easy questions away from that path while parallelizing the independent branches of hard ones earns back its own overhead fast.

Where this is heading, based on work like ParallelSearch's reinforcement-learning-trained planner, is toward systems that learn to recognize parallelizable structure on their own, rather than relying on someone hand-coding the heuristic. Builders working on planning layers now would do well to keep execution strategy configurable and swappable rather than wired in permanently, since the heuristics governing it are still moving and producing shifts that everyone building on top of them must contend with.

None of it matters without a retrieval layer capable of returning content actually worth reasoning over. The most carefully engineered decomposition strategy still hands its synthesis step nothing but shallow, stale, or noisy source material if the underlying retrieval can't do better. Context quality at the sub-query level is the ceiling on everything decomposition tries to accomplish, and no amount of planning sophistication raises that ceiling on its own.