Passage Extraction and Boundary Detection in Noisy Web Documents

A retrieval pipeline pulls a passage from a software vendor's documentation page and hands it to a generator. Nothing in that chunk is false. All of it is useless, for structural reasons. Web documents produce this kind of wreckage because extraction pipelines face two distinct, compounding problems: stripping structural noise in a way that corrupts the semantic context around it, and splitting text on token limits in a way that destroys conceptual boundaries. Treating these as one problem produces a system that solves neither.

The pipeline most teams build first follows a simple recipe: fetch the HTML, strip the tags with a regex, split what remains on a fixed character count. It fails because it treats layout artifacts as if they were content and treats content boundaries as if they didn't matter. A vector index built on that stream has no way to tell "Accept All Cookies" from a sentence that actually answers a user's question, because both arrived as plain text with the same formatting, or lack of it.

The deeper mistake is assuming that HTML structure and semantic structure are the same thing, so that removing the markup leaves meaning intact. They are not the same thing, and removing the markup discards the very hierarchy that was carrying the meaning. So when teams convert every retrieved document to Markdown before indexing it, they are working against that finding. That conversion step throws away information the model could have used.

How DOM noise enters the context window

Structural noise is not a cosmetic flaw that shows up at the margins of a retrieval system. It sets a ceiling on how good that system's retrieval can ever be, because a vector index has no built-in mechanism for treating boilerplate as anything other than content equal in weight to the substantive material around it.

That cost lands in two places: the context window itself, and the ranking quality of the retrieval step that precedes generation. The GaRAGe benchmark, published at ACL 2025 Findings, assembles a large collection of annotated passages retrieved from both private document sets and the open web specifically to reflect real-world RAG conditions, and the annotation process itself exposes how often those retrieved passages carry non-content noise that undermines grounding.

Retrieved passages that look topically relevant but contain noise displacing the actual answer are known in the literature as hard negatives, and they are a documented driver of quality loss in long-context RAG systems. It helps to keep two separate failures in view here. A different passage can look just as relevant while it carries noise that actively misleads the generator, and that is a retrieval corruption. Both produce a wrong or incomplete answer, but they arise from different defects and call for different fixes.

The engineering consequence follows directly from that distinction: noise removal cannot be pushed downstream to the generation step, because the damage to retrieval ranking has already happened by the time the model sees the text. So fix it at the extraction layer, upstream of ranking, not inside the prompt the generator eventually reads.

Why token-limit chunking produces semantically broken passages

Splitting text at a fixed token count is a different kind of operation entirely from understanding where a document's ideas actually begin and end, one that can produce a passage that reads as a complete sentence while severing the argument it belongs to. The underlying mechanism is boundary blindness: a chunk may preserve the body of an argument while cutting off the premise it depends on, or preserve a code example while cutting off the function signature that gives its arguments meaning.

When a language model retrieves a passage broken this way, it reasons from an incomplete premise, because the chunk boundary happened to fall across a piece present in the source document but missing from the chunk. Disco-RAG, which appears in a 2026 roundup of advanced RAG approaches, was built against exactly this failure: standard RAG retrieves text that matches a query topically but fails to synthesize evidence that is scattered across passages that depend structurally on one another. In long documents, reports, legal filings, technical specifications, a conclusion can sit several sections away from the evidence that supports it, and a token-limit chunker will split the two apart with no pointer connecting them back.

Making the chunks bigger does not resolve this; it only trades one failure for another. ChunkNorris, published on arXiv in 2025, treats PDF parsing and chunking as a performance and energy problem in its own right, which signals that the field now recognizes chunking as a systems-level engineering concern rather than a preprocessing detail to be handled with a default parameter.

Structure-aware extraction as a counter to both failure modes

The fix for both failure modes sits in the same place: the document's own structure, used first to separate noise from content and second to find where meaning actually breaks. Operating on the HTML tree itself, rather than on the flat text that tag-stripping produces, is a categorically different task from operating on a string. The tree still carries the hierarchy, headings nested under sections, list items grouped under a heading, a table row tied to its header, and that hierarchy tells an extraction system which branches are navigation and which are substance.

Semantic coherence replaces token count as the signal that determines where one passage ends and the next begins. In practice, that means topic transitions, heading boundaries, and discourse markers, contrast, elaboration, conclusion, carry more information about where a passage should break than any fixed token count ever will.

Long documents add a further requirement: a system needs a global view of the document before it can draw local boundaries correctly. MiA-RAG, or Mindscape-Aware RAG, described in a survey of advanced RAG types, builds a hierarchical summary of the entire document, a "mindscape," and uses that summary to guide both retrieval and generation. A pipeline that processes one chunk in isolation, with no visibility into the document surrounding it, will get some fraction of its boundary decisions wrong no matter how carefully it is tuned.

The sufficiency problem in retrieval

Clean extraction solves the problems diagnosed so far, but it does not solve retrieval itself, because relevance and sufficiency are different properties of a passage, and most pipelines check only for the first. A passage can sit close to a query in embedding space, structurally intact and free of boilerplate, and still lack the specific fact or figure you need the generator to answer the question correctly.

Missing key evidence produces an incomplete answer. A passage that looks relevant on every surface metric can still fail to support the claim the system is about to make, so retrieval cannot be treated as verification.

Some recent work builds that verification step directly into the pipeline. ReflectiveRAG, presented at the EACL 2026 Industry Track, is a training-free framework that performs self-reflective retrieval, reformulating a query adaptively rather than on a fixed schedule, alongside contrastive noise removal that denoises the evidence set, and it does both with only 18 milliseconds of added latency. So a production pipeline needs a sufficiency gate between retrieval and generation, a stage whose job is to check that the evidence actually supports an answer before the model can give one.

The "noise helps" finding and sloppy extraction

The strongest argument against everything above comes from inside the research literature itself: some experiments have found that adding random, irrelevant documents to a retrieval set can raise question-answering accuracy. That finding held under specific experimental conditions that do not resemble production systems, and when those conditions are relaxed, the advantage shrinks or vanishes. A 2026 reproduction study re-ran the original experiments using different model families, chat-style instruction templates, and relaxed output length constraints, the kind of settings that actually characterize deployed systems, and found that the apparent benefit of noise did not hold up.

Inference-time design choices, strict output length limits in particular, can manufacture the appearance of a retrieval effect that isn't there, so you need to re-evaluate design guidance built on that original finding under realistic generation settings before you build a pipeline around it.

For extraction infrastructure, the implication is direct: liberal inclusion of marginal or noisy documents, justified by appeal to that original finding, is not a safe choice for a production system. If benchmarks want to offer genuine design guidance, they need to report their prompt templates and max-token settings as explicit experimental variables, because a result that omits them cannot be trusted to generalize past the paper it came from.

Adversarial content as the production consequence of extraction gaps

A pipeline that cannot reliably tell noise from content cannot reliably tell content from adversarial content either, because the two failures share the same root: an extraction layer with no principled basis for excluding text that happens to pass through it. Corpus poisoning exploits that gap directly. Anyone able to contribute content to a source that feeds a RAG corpus gains a channel for influencing what the system retrieves and what it ultimately generates. Email, SharePoint, internal wikis, and support ticket systems all feed enterprise RAG corpora, and each one is a potential injection surface for exactly this reason.

SafeRAG, published in 2025, catalogues four distinct attack classes against RAG systems: silver noise, inter-context conflict, soft ad, and white denial-of-service, and testing against representative production systems found that even simple manipulations under these categories succeed. SafeRAG demonstrated the risk against representative production systems. The extraction layer, not the chat window, was the point of compromise.

The defensive research response has been fast but is still catching up to the scale of the exposure. That density of activity reads less like a solved problem and more like a field racing to catch up with an active threat. The governance conclusion follows from the mechanism, not from caution alone: access controls, metadata tagging, and content classification have to happen before retrieval, because an extraction pipeline that hands unclassified content to a shared index defeats whatever governance controls get layered on afterward.

Where extraction happens in the AI agent stack

In an agentic system, a badly extracted passage does more damage than a wrong answer to one question. It corrupts a reasoning chain that may trigger tool calls, write to memory, and shape what gets retrieved next, all before anyone notices the error.

In the ReAct architecture, the agent produces a thought, executes a tool call, observes the result, and loops back to produce the next thought from that observation. When a noisy or badly bounded passage enters at the observation step, it does not stay contained there. It propagates into the next thought the agent generates and the next action it takes, and it carries the original defect forward through every step that follows. Guides to agent architecture published for 2026 say web grounding is wired directly into the dominant agent design patterns in use, and when HTML or accessibility-tree input gets converted into natural language for the model to reason over, that is the step where extraction quality sets the ceiling on everything downstream.

Agentic RAG, the pattern in which specialized agents handle retrieval and validation as parallel subtasks, is where this problem becomes hardest to catch, because an extraction failure buried in one agent's output can surface only several steps later, in a different agent's decision, far from where the bad passage was first retrieved. RAG gives an agent access to knowledge beyond what it learned in training, but that access is only as good as the passages delivered to it. An agent handed a passage that blends navigation boilerplate with a genuine factual claim has no internal mechanism for telling the two apart, and it will reason from both as if they carried equal weight.

What production-grade extraction infrastructure requires

Taken together, the requirements are specific: structure-aware noise removal, boundary detection grounded in semantics rather than token counts, a sufficiency check before generation, filtering against adversarial content, and delivery fast enough to sit inside an agent's reasoning loop without stalling it. If you compose general-purpose tools built for other jobs, you cannot satisfy all five at once.

You need a separate extraction call to get the full passage back out, and that round-trip adds latency, which becomes a structural liability for an agent that may need to issue several retrieval calls over the course of a single reasoning chain.

General-purpose scraping tools sit on the surface of the HTML and do not own what happens to it afterward. They break under the conditions real websites impose, bot detection, dynamic rendering, schema changes that shift without notice, and what they hand back is raw content that the receiving application still has to clean, parse, and chunk itself. So the team consuming the data ends up carrying the whole burden of noise removal and boundary detection, and what should be infrastructure turns into a maintenance obligation.

Owning the full pipeline, ingestion, parsing, noise removal, boundary detection, ranking, and delivery, as a single designed system is the architecture that can guarantee quality, latency, and control over the data at the same time. Infrastructure for the Agentic Web, published on arXiv in 2025, names this exact gap between what agent architectures need and what current web infrastructure delivers, and the gap persists because you cannot wrap a third-party service built for a different purpose into requirements it was never designed to meet. Filtering, ranking, and shaping content for a language model is its own engineering discipline, and the quality of what reaches the model determines the quality of what the model can do with it far more than the sheer volume of text ever will.