Multi-Stage RAG Pipelines for Production AI Systems

Retrieval-Augmented Generation grounds a language model's output in external, verifiable data. That grounding is why RAG became the reference architecture for enterprise AI, a market valued in the multibillions in 2025 with a steep annual growth rate. Most teams still design RAG as if retrieval were one stage instead of five, and that mismatch is where production deployments actually break, more often than model quality itself.

The standard prototype looks like this: chunk a document, embed the chunks, run a vector similarity search, hand the top results to a generator. That's one stage with one failure mode, easy to reason about, but production doesn't allow that luxury. Data sources multiply and stop being tidy, query distributions turn unpredictable, mixing simple lookups with multi-hop reasoning and requests that need live web data no internal index has ever seen. Latency budgets, freshness requirements, and reliability targets stop being nice-to-haves and become contractual, and every one of those pressures exposes a stage that was never built to hold up under it. Chunking, retrieval, reranking, context assembly, and generation each deserve treatment as their own engineering discipline; that is the only frame that survives contact with production load.

How pipeline stages compound each other's errors

Errors in a RAG pipeline don't stay where they start. A bad chunk boundary degrades retrieval recall before retrieval even runs, poor retrieval poisons whatever the reranker sees, since a reranker can only reorder what's in front of it, and noisy reranked context then overwhelms the generator, which has no way of knowing that the real answer never made it into its input. By the time a user sees a wrong answer, the failure could have originated three stages upstream, and nothing in the final output says which one.

Here's the asymmetry teams keep missing: generation failures are loud, retrieval failures are quiet. If the wrong answer comes out of the model, someone notices immediately, the fix feels obvious, and that's where attention goes. But if the right document never got surfaced in the first place, nobody sees an error message, they see an incomplete or subtly off answer, and the missing document leaves no trace of its own absence. Analysis of RAG failures frequently points to retrieval as a primary point of breakdown, rather than generation. The generator is rarely the weakest link, yet it's usually the first thing teams tune, simply because it's the only stage they can see failing. That instinct burns engineering hours that should go toward chunking and retrieval instead.

Optimizing only the generator, through fine-tuning or a model upgrade, hits diminishing returns fast if retrieval underneath it was never fixed. Debugging a wrong answer means tracing back through every upstream stage, not just checking the prompt. Each stage needs its own evaluation, distinct from a single end-to-end answer-quality score that confirms something failed without saying what.

The rest of this piece follows a five-stage production model: chunking, embedding and indexing, dense retrieval, reranking, and generation. Each section below treats one or two of these stages as a design problem in its own right, with its own tradeoffs, rather than as a configuration knob on one big system.

Chunking as the silent upstream failure that no downstream fix can recover

Chunking gets treated as preprocessing, something to get through before the interesting work starts. That instinct is costly: chunking sets the ceiling on everything downstream, and no reranker or generator can recover a semantic unit that was severed at the wrong boundary. It is the single most underrated decision in the whole pipeline, and it deserves far more engineering time than most teams currently give it.

Hold chunking to one standard: semantic completeness. A chunk should answer a question on its own, without needing the paragraph before or after it. A chunk holding half an argument or half a procedure is worse than no chunk at all, because it ranks well enough to get retrieved and then delivers nothing useful once it's there.

The failure modes are mundane and everywhere. Fixed-size character splits cut sentences in the middle of a thought, because they don't know what a sentence is. Chunks sized too small carry no real meaning, while chunks sized too large can't be ranked with any precision, since they blend a relevant passage with three irrelevant ones. Splitting that ignores document structure, cutting across headers, tables, or code blocks, produces chunks that are structurally incoherent even when the underlying text is fine. Chunks stored with no metadata carry no provenance, so neither the reranker nor the generator has any way to know where a passage came from or how much to trust it.

None of this is cosmetic. Semantic chunking, structural chunking, and hierarchical parent-child chunking each trade granularity against context preservation differently, and picking one without understanding that tradeoff is a decision made by default rather than by design. Overlap strategies exist specifically to keep meaning from being lost at chunk boundaries. Prose, tables, code, and diagrams don't share an optimal chunk size, so a single splitting rule applied uniformly across a heterogeneous corpus is a design failure before a single query runs.

A chunk should carry a source, a position within its document, and a defined semantic scope. Pipelines that strip that context break observability for every stage that follows. Chunking strategy deserves version control and its own evaluation, measured against retrieval recall, not against some downstream proxy that arrives too late to matter.

Why hybrid retrieval has become the production baseline and what it demands to work correctly

Dense retrieval, vector similarity over embeddings, handles semantic paraphrase well. Ask a question with different words than the source document uses, and dense retrieval will often still find it. Yet it's weak on exact terminology: product names, part numbers, legal citations, anything where the precise string matters more than the concept behind it.

Sparse retrieval, the BM25 family, is the mirror image: precise on keyword matches, brittle on anything conceptual or paraphrased. Neither approach alone covers the query distribution a production system actually sees, and treating one as sufficient because it performed well in a demo is how teams end up rebuilding retrieval six months into a launch.

Hybrid search combines both signals, and it has become the default because it measurably improves recall across a wider range of queries than either method run alone. That gain isn't free, though. Dense and sparse scores live on entirely different scales, so before the two result sets can be fused, someone has to normalize them, and getting that normalization wrong quietly breaks the fusion even when both retrieval paths individually work fine. Fusion strategy itself, Reciprocal Rank Fusion being the common choice, is a tunable parameter with real consequences for ranking quality, not a default to set once and forget.

Latency compounds too. Running two retrieval paths, whether in parallel or in sequence, adds time that has to be accounted for in the stage's overall budget, and hybrid retrieval that isn't scoped against a latency target will eventually blow one. Both indexes need to stay current independently, and they fail differently: a stale dense index drifts semantically, while a stale BM25 index simply misses new terms outright. Freshness has to be managed on two tracks, not one.

For most production use cases, hybrid retrieval paired with reranking gives the best ratio of quality to cost. That claim only holds when both retrieval paths are tuned against real traffic and the fusion logic has been validated against the query distribution the system actually serves, rather than the query distribution used in a demo.

Web retrieval deserves its own mention here, because it's a distinct path with its own failure profile. When a query needs live or external knowledge no internal index holds, the snippets a search results page returns are structurally too short to feed a reranker or a generator anything useful. Full content extraction from the source page is a requirement for this retrieval path to function at all, not an optional enhancement.

Reranking as a distinct model layer with its own latency budget and failure modes

Reranking reads the full query alongside each candidate document and scores actual relevance, going well beyond what vector proximity or keyword overlap can approximate. That distinction is why reranking runs as a separate model, with its own latency budget, rather than as a post-processing trick bolted onto retrieval.

Cross-encoder rerankers are computationally expensive, because they process query and document together instead of comparing precomputed embeddings. That expense is exactly why rerankers operate over a small candidate set pulled from retrieval, never the full index. The dependency cuts both ways: the reranker can only promote the best document already in its candidate pool, with no mechanism to recover a document retrieval never surfaced in the first place, no matter how good the reranking model is.

Several failure modes live specifically at this layer. A candidate pool that's too small means the correct document was never retrieved, so the reranker has nothing worth promoting, while a candidate pool that's too large blows the stage's latency budget, since cross-encoder scoring cost scales with the number of candidates. A general-purpose reranker trained on broad web text can underperform badly on technical, legal, or scientific content, where domain vocabulary carries most of the signal. Teams that measure only end-to-end answer quality miss a specific, fixable problem hiding underneath it: a reranker that consistently demotes the correct document while promoting plausible-sounding but wrong ones.

The TREC 2025 RAG Track results give this weight beyond intuition. Pipelines that fed generation signals back into the ranking process scored an nDCG@30 of 0.6762, measurably ahead of multi-stage pointwise pipelines at 0.6371 and learned sparse-only approaches at 0.5838. The lesson runs deeper than the number: treating ranking as a closed stage, sealed off from what the generator actually needs, leaves quality on the table that a more integrated design would capture.

The design principle follows directly. Define the reranking stage's latency budget before choosing a reranker model, not after. That budget constrains how large a candidate set can be, which constrains how much retrieval recall the system can afford to work with, which in turn feeds back into how granular chunking needs to be in the first place. The stages aren't independent even when they're designed separately.

Context assembly as an engineering discipline, not a prompt formatting task

In June 2025, Shopify's Tobi Lütke proposed that context engineering — assembling everything the model needs so the task is plausibly solvable at all — was the right frame, and Andrej Karpathy endorsed the idea shortly after. Both framings treat this as a genuine engineering discipline, and no production pipeline should treat it as less.

A prompt is an instruction. Context is the full information environment the model has to reason inside: retrieved documents, conversation history, tool outputs, system constraints, all of it. Conflating the two is how teams end up debugging a "prompt problem" that's actually a context problem three layers deep.

Larger context windows don't make this discipline optional, whatever the marketing around frontier models implies. Models remain bottlenecked by the quality of input tokens well more than the quantity available to them. Irrelevant or noisy chunks distract the model's reasoning and add inference cost for no benefit. A model's effective attention degrades under context noise even when the window technically has room to spare; a bigger window poorly filled falls short of a smaller one well curated.

Context assembly, done properly, has to select which reranked chunks actually belong in the prompt for this specific query, which is not always the same as the top-k results by score, and it has to decide on ordering, since document position affects how a model weighs evidence, and leading with the highest-scored chunk isn't always the arrangement that produces the best answer. It has to attribute each chunk to its source so the generator can ground claims in citable evidence instead of blending sources into an unverifiable blur. And it often has to compress long chunks through extractive methods, preserving the relevant passage while fitting inside a fixed latency and token budget.

The data contract at this stage matters as much as it does at chunking. Assembled context is a structured artifact, with provenance, ordering logic, and a defined token budget, built with the same rigor as any other pipeline stage. Treating it as formatting loses exactly the precision Karpathy and Lütke describe.

How agentic RAG changes the pipeline's failure surface

Static RAG retrieves once per query and stops. Agentic RAG systems decide when to retrieve, what to retrieve, and whether to retrieve again, based on intermediate reasoning the model produces as it works through a task. That's a structural shift, and it changes what can go wrong more than it expands what's possible.

In agentic systems, reasoning and retrieval interleave: the agent produces intermediate reasoning, acts on it, receives new information, and updates its trajectory before deciding on the next step. Retrieval stops being a fixed preprocessing stage and becomes embedded inside a sequential decision process the model controls.

Retrieval errors now compound across reasoning steps, not just within a single pass through the pipeline. An agent can issue a retrieval call with an underspecified or drifted query, one that's technically related to the task but has lost precision somewhere in the reasoning chain, and quality degrades mid-trajectory with no clear signal marking the point of failure. Multi-agent orchestration patterns, an orchestrator directing retriever agents, an analyst, a critic, a writer, distribute retrieval across specialized components, and each of those components can fail on its own, independent of the others. Tool-use agents often treat web search, graph traversal, and internal knowledge base queries as interchangeable retrieval actions, when in fact each carries a different latency profile, a different freshness guarantee, an entirely different reliability curve.

Reflective retrieval designs have emerged as responses to this problem: systems that assess their own retrieval quality and conditionally skip or repeat a retrieval step. That reduces noise, but it opens a question with no clean answer yet: how does the agent decide it has enough context to stop retrieving?

GraphRAG, an approach using graph-based knowledge structures, addresses a related but distinct problem: multi-hop queries that no single document can answer on its own. It structures knowledge at a higher level of abstraction, which enables broader queries across a corpus. That capability comes at real cost: index construction and update latency that a simpler chunk-and-embed pipeline never has to carry.

The governance implication is direct. Once RAG becomes the retrieval backbone for an autonomous agent, quality control has to operate at the level of individual agent steps, validating each retrieval action as it happens, rather than waiting to evaluate only the final output. An agent trajectory that ends in a correct answer can still have taken three wrong turns getting there, and end-to-end evaluation will never show those turns.

What evaluation must look like when every stage can fail independently

End-to-end evaluation has a structural blind spot: a correct final answer can mask a failure two stages upstream, and an incorrect answer gives no information at all about which stage caused it. Both directions of that failure make answer-only evaluation close to useless for diagnosing a production pipeline. Teams that rely on it lack the visibility needed to diagnose where the pipeline is actually failing and calling it a good landing when they happen to hit the runway.

Stage-isolated evaluation is the baseline that fixes this, and it isn't optional if the goal is actually knowing where a pipeline breaks. Chunking gets measured against retrieval recall: did the relevant chunk actually make it into the candidate pool at all? Retrieval gets measured through its own stage-specific metrics, independent of whatever the reranker later does with those candidates, while reranking gets measured on the quality of its reordered candidate list: did the correct document actually move to the top, or did the reranker leave it buried? Context assembly gets measured on its own criteria, including budget adherence and attribution quality for what actually made it into the prompt. Generation gets measured on faithfulness to the supplied context, above all, alongside answer completeness and citation accuracy.

The TREC 2025 RAG Track scores cited earlier matter for a second reason beyond the numbers themselves: they demonstrate that retrieval and ranking quality can be evaluated as distinct concerns from generation quality. That's the methodology worth adopting, alongside the specific benchmark result.

Offline evaluation and runtime monitoring do different jobs, and neither substitutes for the other. Offline evals validate a pipeline change before it ships, while runtime monitoring catches distribution shift, data freshness failures, and retrieval degradation as they happen under live traffic, none of which an offline eval run against a static test set will ever surface. Both are necessary, and neither is optional.

For agentic systems, evaluation has to trace the full trajectory, not just the final output: which retrieval actions were taken, which were redundant, which introduced noise the agent never corrected for. Teams that build stage-by-stage evaluation infrastructure from the start are better positioned from demo-level accuracy to production reliability than teams that try to bolt evaluation on after launch, once the failures are already reaching users.

Web retrieval as a production infrastructure problem, not a search query

Web retrieval is a different engineering problem from internal corpus retrieval, because the content is external, heterogeneous, and changing continuously in ways no internal index has to contend with. Bot detection, JavaScript rendering requirements, and wildly inconsistent page structures make raw scraping brittle at any meaningful scale.

Platform dependency adds a supply-chain risk layer on top of that, and it's the risk most teams underweight until it costs them. Some major search API providers have moved to restrict or sunset developer access, and and other search API offerings have faced similar uncertainty. Teams that built web retrieval on a single external API as a permanent foundation are building on infrastructure that has already proven it can disappear on a vendor's timeline, not theirs.

What a search results API typically returns, titles, URLs, and snippets running a few hundred characters, is enough for a human being to glance at and decide whether to click through, yet it falls well short of what a reranker needs for genuine relevance signal, and it falls well short of what a generator needs to ground an answer in actual source content rather than a fragment stripped of context. Full-page content extraction, structured and attributed the same way an internal chunk would be, is the actual requirement here. Treating web search as a query fired off and a snippet pasted into a prompt carries the same risk as treating chunking as a preprocessing afterthought. It looks like a small detail right up until it's the reason the whole pipeline gives an answer nobody can trust.