Context Window Packing Strategies for Retrieved Web Documents

A model does not lose information buried in the middle of a long prompt. It simply stops attending to that information as effectively, which produces the same practical outcome as loss but a different engineering problem entirely. This reframes the context window from a container you fill to a budget you allocate: every system prompt, retrieved document, turn of conversation history, tool output, and generated response draws from the same finite pool of tokens, and how that pool gets divided decides whether the model reasons well or fails with total confidence. The assumption that a bigger context window solves this by giving everything more room breaks down on two fronts at once, cost and quality, because longer contexts are priced higher per token on some platforms even as the marginal tokens contribute less to the answer. Research into effective context limits has found that many models show measurable accuracy degradation by as little as 1,000 tokens, a fraction of their advertised maximums. The real ceiling on performance is what researchers call the Maximum Effective Context Window rather than the number printed in a model's marketing copy. The consequence in production is unforgiving: a frontier model will reason over wrong or diluted context all day without signaling that anything is wrong, because bad packing does not produce obvious errors, it produces fluent, confident, wrong answers. Treating context assembly as an afterthought to retrieval, rather than as its own engineering discipline, is how that failure mode ends up in production systems.

The four ways retrieved content corrupts a context before the model ever sees it

Bad packing fails in patterns, and those patterns are diagnosable, each with its own cause and its own fix, which is the reason understanding them has to come before choosing a packing strategy. The first is context poisoning: an incorrect belief enters the working context and the model reinforces it over subsequent steps, as when an agent retrieves a stale API endpoint, hits an error, and then keeps referencing that same bad endpoint because the failed attempt is now written into its own history. For content pulled from the live web, a freshness failure at the moment of retrieval becomes a reasoning failure that compounds across every turn that follows. The second pattern is context distraction: once context grows past a certain threshold, models lean too heavily on what is sitting in front of them and too little on their own trained knowledge, a dynamic documented in agent workloads where crossing a token threshold caused models to repeat prior actions from history instead of working out a new strategy. The third is context confusion, where irrelevant information actively drags down response quality rather than sitting inert. Research from Berkeley's Function-Calling Leaderboard found models performing worse when handed too many tools to choose from, and narrowing the toolset down to only the relevant options turned an outright task failure into a success. The same mechanism applies to retrieved passages: adding more chunks introduces ambiguity, and the model spends its attention sorting out what is relevant rather than answering the question. The fourth and most subtle pattern is context clash, where contradictory content coexists in the same window. Microsoft and Salesforce research on multi-turn conversations found that early incorrect attempts left sitting in history contaminated final responses, dragging down model accuracy substantially on average. In web retrieval specifically, documents asserting contradictory facts about the same entity degrade output even when each document is accurate taken on its own when two sources disagree. A customer support agent handed the full user manual, a stack of recent tickets, the knowledge base, and the complete API documentation will answer a simple password reset question with a muddled response that mixes OAuth flows into enterprise security policy, while the same question paired only with the relevant documentation section produces a direct and correct answer. All four patterns trace back to the same root cause: packing was treated as a trivial step bolted onto the end of retrieval, rather than as the place where reasoning quality is actually won or lost.

Why retrieval alone cannot solve the packing problem

The standard retrieval pipeline, embed the query, pull the top-k chunks by cosine similarity, and drop them into the prompt, is fast and cheap, and it produces candidates rather than a context ready for reasoning. Bi-encoder similarity measures how close a chunk sits to the query in embedding space, and that proximity correlates only imperfectly with whether the chunk actually helps the model answer the question, so a chunk can score high on similarity while still being the wrong chunk for the task at hand. Part of the problem is structural: when a document gets split into chunks before embedding, each chunk loses its connection to the content around it, so a passage that reads clearly on its own can mislead the model once it is separated from the paragraphs that gave it context. One response to this is a technique called late chunking, which tokenizes the entire document with a long-context embedding model first, so every token already carries information about the document around it, and only splits into chunks afterward; paired with contextual retrieval, this has been shown to cut error rates substantially. Even that technique has a ceiling, though: adding 50 to 100 tokens of surrounding context to a chunk is not always enough to capture how that chunk fits into the larger document it came from. The larger gap that textual similarity alone cannot close is cross-file and cross-document dependency. Research evaluating hybrid retrieval across 180 developer queries found that questions requiring architectural context spanning multiple files made up the majority of the evaluation set, and pure textual similarity missed critical dependencies in most of those cases. The same dynamic occurs in web document packing, where a retrieved page referencing an entity defined on a separate page demands a kind of structural reasoning that cosine similarity was never built to provide. This is why production systems have largely moved to a two-stage pipeline: pull a large pool of candidates with a bi-encoder for recall, then apply a cross-encoder reranker to select the final set, since reranking is more expensive computationally but applies the kind of direct query-document attention that single-vector search cannot replicate. Top-k retrieval, on its own, is only a starting point. Chunk sizes vary, and a fixed top-k without a secondary token budget can quietly blow past the context limit, so the retrieval step has to account for token cost alongside relevance rank rather than treating them as separate concerns. Good retrieval output still arrives needing an active packing layer before it is fit to reason over.

Positional placement: where in the context window content sits determines how much the model uses it

Lost-in-the-middle has a specific mechanical cause; it is not a vague byproduct of scale. Rotary positional embedding, used widely across open-source language models, carries a long-term decay property that biases the model toward tokens near the current position. A large advertised context window does not translate into uniform attention across every token inside it. Research from Stanford and UC Santa Barbara confirmed the practical effect of this bias directly: performance drops significantly when the information a query needs is in the middle of a long context, even though the model technically has full access to it. The consequence for anyone assembling a prompt is concrete and a little counterintuitive: a chunk that scores highest on relevance can end up contributing less to the final answer than a lower-scoring chunk, simply because the high scorer landed in the middle of the window and the lower scorer landed near the edge. The fix is a structural ordering that treats position as a resource rather than an accident. The system prompt stays static and goes first. Retrieved context, led by the chunks judged most important, goes immediately after the system prompt, because that spot is the highest-attention zone of the window. A compressed summary of older history follows, then the most recent conversation turns, and the current user message goes last, where it occupies the second-highest-attention zone the model offers. Instructions belong at the end of the prompt as well, not buried somewhere in the middle where the model's attention is weakest. Retrieval-augmented generation sidesteps the lost-in-the-middle failure almost entirely when implemented this way, because retrieved passages sit at the top of the context rather than accumulating in the middle as a conversation's history grows longer. Explicit structural markers, headers, XML tags, numbered sections, give the model a way to locate information inside a packed context without relying on positional attention alone to do the work. The rule that follows from all of this is a negative one: filling available token space with marginally relevant content because the room exists only pushes genuinely valuable content toward the middle of the window and dilutes the exact signal the model needs to reason correctly.

Token budgeting: treating the context window as a structured, finite resource

Placement determines how much the model uses what is already inside the window. Budgeting determines how much gets let in at all, and it has to be treated as a deliberate policy decision rather than a limit a pipeline bumps into after the fact. System prompts, retrieved documents, conversation history, tool schemas, and the model's own output all draw from one shared token pool, and teams that are careful about retrieval while ignoring the cost of static components routinely discover their context has been blown by elements they never thought to measure. Top-k retrieval without a secondary token budget is structurally unsafe for exactly this reason: chunk sizes vary enough that a fixed top-k count can exhaust the available context without any warning, so the budget has to be enforced after retrieval and not only at the retrieval step itself. A practical pattern that handles this is to fetch roughly twice the target number of candidates and then trim down to a token budget rather than a fixed chunk count, which separates the question of relevance ranking from the question of token accounting and lets each be managed on its own terms. Token counting itself has to match the tokenizer the API actually uses internally, because counting tokens by some other method produces silent overruns where the developer believes the payload fits and the API receives something oversized. The naive assumption that a larger context window is always better breaks down on two axes simultaneously: cost and quality. A sliding window keeps only the most recent N turns and discards the rest, a method that reduces context size by 40 to 70 percent, with turn summarization and RAG-based retrieval each contributing further reductions on top of that, at the cost of discarding earlier context permanently. Turn summarization takes a different approach, compressing older turns into a single block placed at the start of the context so semantic continuity survives without carrying the full token cost of the original exchanges, and that summarization call is a good candidate for a fast, inexpensive model rather than the primary frontier model doing the main reasoning work. For document-heavy workloads, retrieval-augmented generation itself functions as a compression strategy and not merely a knowledge-lookup mechanism: pulling only the semantically relevant chunks for a given query, instead of sending full documents or entire histories, cuts context size substantially while preserving the information the model actually needs to answer. At the scale of an enterprise deployment, gateway-level context policies let these budgets get enforced across an entire system without requiring every individual application team to build its own limits, which matters wherever multiple teams share the same underlying infrastructure.

Chunk filtering and reranking: reducing the candidate set to what supports the inference

A well-tuned pipeline that packs a small number of carefully chosen chunks will routinely outperform a pipeline that stuffs in everything the window can technically hold, even when there is token budget to spare. The production standard for getting to that small, carefully chosen set is the two-stage pipeline already established as the baseline: a large pool of bi-encoder candidates handles recall, and a cross-encoder reranker then selects the final chunks for packing, applying full query-document attention that single-vector search has no mechanism to provide. Beyond reranking, context pruning treats the assembled context as a structured object to be actively shaped rather than a text buffer that simply grows. Tools built for this purpose, such as Provence, can prune documents automatically and achieve high compression rates while retaining the information that actually matters to the query. Selection strategy matters just as much as filtering strength. Research evaluating greedy, file-limited, and submodular approaches to chunk selection across 180 queries found that submodular packing delivered a meaningful diversity advantage over greedy selection while holding comparable citation recall, because it optimizes for coverage and diversity at the same time rather than simply grabbing the highest-scoring chunks one after another. Greedy top-k selection tends to pack redundant material, multiple passages that are highly similar and say roughly the same thing, which consumes token budget without adding any new reasoning surface for the model to draw on. Submodular selection corrects for this by penalizing redundancy directly, spreading the available token budget across chunks that each contribute something the others do not, which is what ultimately turns a filtered candidate set into a context the model can reason over with confidence rather than merely fluency.