Re-Ranking Web Retrieved Documents for LLM Context Windows
Positioning matters as much as relevance when filling an LLM's context window.
Retrieval-augmented generation has a math problem. Cast a wide enough net to catch every relevant document, and the model drowns in marginally relevant material; narrow the net to keep it focused, and it misses the answer outright. Re-ranking is the stage built to resolve that tension, sitting between retrieval and generation (taking a recall-heavy candidate set and turning it into a short, ordered list an LLM can actually work through). Most teams treat it as an optional quality knob, and that undersells its role: re-ranking is usually the difference between a system that works and one that quietly burns half its context window on documents the model was never going to read closely anyway.
How LLMs actually read a context window (and where attention degrades)
Position matters more than most retrieval pipelines account for, and most pipelines don't account for it at all. Research has found that when the document containing the correct answer moved from the first position to the tenth in a 20-document context, accuracy on multi-document question answering dropped substantially. That reflects a structural weakness in how transformer attention distributes itself across a long input, not a tuning quirk. The ranking order handed to the model matters almost as much as whether the right document showed up in the candidate set at all.
The mechanism is a U-shaped attention bias: models over-weight tokens near the start and end of a sequence and under-weight the middle, regardless of what that middle content actually says. A 2024 study from MIT and Google Cloud AI, "Found in the Middle," confirmed the pattern and tied it to how positional attention gets calibrated during training. RoPE, the positional encoding scheme behind most modern LLMs, compounds the problem mechanically. It introduces a long-distance decay effect that makes tokens far from either edge harder to retrieve, independent of relevance.
The effect shows up clearest at moderate context lengths, the kind tested up to moderate token counts. Isolating it cleanly gets harder at extreme lengths, at very long lengths, which suggests it shifts rather than vanishes. Either way, the lesson is blunt: which documents make it into the context window is only half the decision, and where they sit inside that window is the other half. The context window behaves like a sequence with a positional structure the model exploits, whether anyone planned for it or not.
The two-stage retrieval architecture that most production systems now use
Most production RAG systems split retrieval into two stages, because no single model handles both jobs well. Forcing one model to do both is where a lot of pipelines quietly lose accuracy; it's the mistake that shows up first when teams try to cut corners on infrastructure.
Stage one is a bi-encoder: a transformer embedding model that encodes queries and documents independently into a shared vector space, then scores similarity with something as simple as cosine distance. Because the encoding happens independently, the entire corpus can be pre-computed offline and searched in milliseconds, even across billions of documents. The tradeoff is architectural, not incidental. The model never sees the query and document together, so it captures broad topical similarity, not the sharper, narrower notion of relevance a specific question demands.
Stage two is a cross-encoder, built to close that exact gap. It processes the query and a candidate document together in a single forward pass, weighing how the two actually interact instead of comparing two static vectors. That joint processing makes cross-encoders considerably more accurate at judging relevance, but running one over an entire corpus isn't practical. It only ever scores the candidate set the bi-encoder already narrowed down, typically the top 50 to 100 results.
Hybrid retrieval has become the standard complement here, and skipping it is a mistake most teams eventually correct the hard way. Combining BM25, a sparse keyword method, with dense embeddings, then fusing the two rankings through Reciprocal Rank Fusion, beats either approach alone on benchmarks like BEIR and MTEB. Layer a cross-encoder on top of that fused set, and the gain in ranking precision on hard cases compounds further. The resulting flow: retrieve broadly (say the top 50 candidates), rerank with a cross-encoder, then hand the model only the top handful, usually three to five documents. Retrieval latency and ranking latency get engineered as separate concerns for good reason: the two stages have entirely different performance profiles and get tuned independently.
Where to position the highest-ranked documents once re-ranking is done
Getting a ranked list is not the same as knowing what to do with it. The intuitive move, dropping documents into the prompt in rank order from one through N, ignores everything the U-shaped attention bias just established. It is also the single most common way teams throw away reranking gains they already paid for in latency and compute.
The better approach places the highest-scored documents at the beginning and end of the context window, leaving lower-ranked material to occupy the middle. This works with the model's primacy and recency tendencies instead of fighting them. Rank score and prompt position are two separate decisions: the reranker decides what matters, and a distinct assembly step decides where that mattering gets expressed inside the prompt. Skip that second step, and the 30-plus point accuracy hit documented by Liu et al. stops being hypothetical and becomes the exact failure mode sitting on the table.
A second lever matters just as much: keeping the final context small, generally three to five documents, limits how much middle exists for anything to get lost in. Fewer documents mean fewer positions where the bias can quietly erase a correct answer. This is one of the least discussed parts of RAG engineering, and it shouldn't be. Teams pour effort into retrieval quality and reranker architecture, then dump the output into the prompt in whatever order the reranker returned it, leaving accuracy on the table for free.
Using LLMs as the re-ranker: listwise, pairwise, and setwise approaches
An LLM can serve as its own reranker, no dedicated cross-encoder required. Prompt the model with a query and a set of candidates, and ask it to reorder them by relevance.
Three prompting patterns dominate. Listwise reranking hands the model the entire candidate set at once and asks for a reordered list. It's conceptually clean but bumps into context limits fast, which is why sliding-window variants exist: they process overlapping subsets iteratively until every candidate has been scored. Pairwise reranking compares documents two at a time, producing more reliable individual judgments but multiplying the number of inference calls needed. Setwise reranking splits the difference, using small batches and binary relevance calls per document, landing closer to listwise quality without listwise's full inference cost.
Here's the irony worth sitting with: LLM-based listwise rerankers suffer from the same lost-in-the-middle problem that motivates reranking for generation in the first place. Positional bias doesn't disappear just because the model doing the reranking happens to be the same kind of model doing the generating.
Cost is the real constraint here, and it deserves to be the first question asked, not an afterthought bolted on later. On a scientific retrieval benchmark, a baseline pipeline with no reranking used 1.01 million tokens at a cost of $0.40, while a sliding-window LLM reranker run on the same benchmark used 9.06 million tokens for $3.62 (a ninefold jump in token consumption for the reranking step alone). Weigh that multiple against the accuracy gain every time; don't assume it away by default. Most teams reaching for LLM reranking by default are overpaying for a job a cross-encoder does cheaper. It earns its cost in specialist domains where cross-encoders have poor coverage, in applications that can absorb the added latency and spend, or in setups where one fine-tuned model already handles both ranking and generation, which is where RankRAG comes in.
RankRAG: what happens when ranking and generation are trained as one task
RankRAG, presented at NeurIPS 2024, takes a different approach entirely. Rather than bolting a separate reranker onto a generation model, it instruction-tunes a single LLM to do both jobs. The same model that writes the final answer also decides which retrieved documents deserve to be in its own context.
The finding here cuts against the two-stage orthodoxy the rest of this piece has been describing, and it deserves to be taken seriously rather than filed away as a footnote. Adding just a small fraction of ranking data into the model's training blend produced a system that outperformed dedicated expert ranking models, including versions of the same base LLM fine-tuned exclusively on large-scale ranking datasets. On Natural Questions using DPR retrieval, Recall@5 climbed from 69.50% before ranking to 77.95% after; Recall@20 moved from 81.00% to 84.56%.
Llama3-RankRAG beat both Llama3-ChatQA-1.5 and GPT-4 across nine knowledge-intensive benchmarks, and it matched GPT-4 on five biomedical RAG benchmarks without any biomedical fine-tuning, a generalization result that's hard to wave off as noise. Ranking and generation are less separable than the two-stage pipeline assumes; training them as one objective beats stitching together two models optimized for different goals, at least on this evidence. The catch is access: this requires a fine-tuning pipeline, which rules it out for teams working strictly through hosted generation APIs. Recent results point the same direction. In the TREC 2025 RAG Track, the top generation-in-the-loop pipelines reached an nDCG@30 of 0.6762, well above learned sparse-only baselines scoring around 0.58 (a gap consistent with exactly the kind of tight integration RankRAG demonstrates).
How agentic retrieval changes the re-ranking problem
Agentic RAG breaks the single-pass model entirely. An agent chooses its retrieval strategy dynamically, runs multiple rounds where each is informed by the last, and interleaves retrieval with reasoning and tool calls in a loop of thought, action, and observation.
That loop means reranking has to happen repeatedly, not once, and this is where a lot of agentic pipelines fall apart in practice. Every retrieval call inside the loop produces candidates that need ranking before they feed the next reasoning step. An agent without disciplined ranking and stopping logic spirals into over-retrieval, calling the retriever far more than necessary, accumulating dozens of chunks that overwhelm the generator, and often producing a worse answer than two well-placed retrievals would have. The fix is procedural: hard budgets on retrieval calls per turn, a faithfulness check that exits the loop once a draft answer is sufficiently supported, and metrics that track retrieve calls against final answer correctness so the tradeoff stays visible instead of hidden inside a black box.
Azure AI Search's agentic retrieval, generally available in 2026, shows the pattern at scale. It decomposes a complex query into focused subqueries, runs them in parallel, semantically reranks each subquery's own results, then merges everything into one unified ranked set before generation. Reranking is embedded at the subquery level throughout, rather than appearing only as a single final step, and that's closer to where this whole discipline is headed.
The AI agent market reached multibillion-dollar scale in 2025 and is projected to reach tens of billions of dollars by 2030, growing rapidly annually. Reranking infrastructure built for a single retrieval pass won't hold up under that kind of load. For agents pulling from the open web specifically, the reranking layer has to weigh source credibility and freshness alongside semantic relevance, something a curated internal index never had to worry about.
What the web retrieval layer must deliver before re-ranking can help
Reranking is a precision layer, and treating it as anything more is among the more expensive mistakes in this pipeline. It can only reorder what actually reaches it. It cannot recover a fact that was never retrieved, was stripped out during scraping, or arrived as a two-line snippet too thin to reason over.
Three upstream failures cap how much reranking can fix, no matter how good the reranker is. Shallow snippets, the kind returned by standard search-results pages, give a human enough to decide whether to click but not enough for a cross-encoder to score relevance reliably, or for an LLM to reason over with any confidence. Scraping fragility is separate: tools that fetch page content at query time break against JavaScript-heavy pages, rate limits, and structural changes, so the candidate set reaching the reranker is incomplete in ways that shift unpredictably from query to query. Staleness closes the list out. Reranking by relevance does nothing to fix a document pulled from a stale cache, since freshness is properly a retrieval-layer concern rather than a ranking one.
Getting the retrieval layer right pays off in a measurable way. On factual recall benchmarks like SimpleQA and multi-hop reasoning benchmarks like FRAMES, web-grounded systems show accuracy gains of 25 to 40 percentage points over ungrounded baselines, though those gains only show up when the retrieved content is complete enough to reason over in the first place. Teams that own their retrieval pipeline end to end can tune document depth, freshness windows, and content structure before a reranker ever touches the data. Seltz, for instance, is a real-time web grounding API built specifically for AI agents and RAG pipelines that delivers content already shaped for this kind of upstream control. Teams leaning on third-party search APIs take whatever those APIs hand back, for better or worse, and no reranker downstream can undo that constraint.
The order of operations follows from all of this, and it's not the order most teams actually use. Audit the retrieval layer before spending more engineering time tuning the reranker. Shallow, fragmented, or stale candidates put a ceiling on what reranker tuning can fix; the remedy belongs upstream of the ranking stage, and the returns on tuning it further diminish fast.
