Chunking Strategies for Long-Form Web Documents in RAG

A chunk is a unit the pipeline treats as a retrievable atom, and it only sometimes lines up with a paragraph, the unit a person recognizes as one idea.

Trace what happens to a chunk after it gets created. It's split from the source document, turned into a vector embedding, stored in an index (vector, sparse, or both), matched against a query embedding at retrieval time, then passed, maybe after a reranking pass, into the language model's context window. Every one of those steps depends on where the boundary landed.

Get the size wrong in either direction and something breaks. Too small, and the embedding has nothing to hold onto: a single sentence often lacks the surrounding signal needed for accurate similarity matching against a query. Too large, and the chunk's embedding starts to represent an average of several ideas instead of any one of them clearly, which dilutes the match. There's also the "lost in the middle" effect, where a language model given a long context window tends to underweight information sitting in the middle and favor content near the start or end instead.

A bad boundary doesn't just make a chunk noisy; it makes evidence disappear. Cut a fact in half at a chunk boundary and the model has no way of knowing that fact ever existed in the source document, since it's simply absent from what it sees. RAG has moved from research demos into production systems handling real query volume, and that failure mode has gone from an academic footnote to a line item on somebody's error budget.

Here is the position worth stating up front: there is no single correct chunking strategy, and any team that picks one method and bakes it into a config file has already made the mistake. A web document rarely behaves like a document. It behaves like a junk drawer: navigation links, ad slots, a table buried in the middle, three paragraphs of actual argument, a footer full of legal text nobody wrote to be read. Chunking strategy is a function of document structure, and web document structure is inconsistent by nature, which means the choice has to get revisited document by document, sometimes section by section within the same page.

Fixed-size and recursive character splitting — the baseline and where it breaks

Fixed-size splitting is the simplest option available: cut the document every N tokens, no matter what falls at that boundary. It's cheap to build and cheap to run, and it will split a sentence, a table, or a code block in half without hesitation.

Recursive character text splitting is the more common default across RAG frameworks, and for good reason. It tries a sequence of separators in order, usually double newline first, then single newline, then a plain space, falling back to the next option whenever the preferred one doesn't produce a chunk near the target size. It carries more structural awareness than pure fixed-size splitting, though it's still reading whitespace patterns, not meaning.

In controlled benchmark settings, recursive splitting holds up reasonably well at chunk sizes in the low hundreds of tokens, which is a big part of why so many teams reach for it without much debate. It's a decent trade of cost against performance, right up until the document stops being uniform prose. That's exactly where most production RAG systems built for web content live, and it's why the default is the wrong choice more often than teams admit.

A 400-token split will, sooner or later, land in the middle of a table, cutting the column headers away from the row values and leaving both halves unanswerable on their own. Code blocks fare no better: character-boundary splitting doesn't know what a function signature is, so it can hand back a chunk that isn't even valid syntax. Scraped pages carry boilerplate at the top and bottom, navigation menus, cookie banners, and a fixed-size splitter will happily turn that into a full-price chunk that eats index space and returns nothing useful. Any document that shifts registers, say from a narrative introduction into a structured bullet list, can produce a chunk that straddles both and represents neither.

Overlap, the usual fix, slides the window so adjacent chunks share tokens, which cuts down on boundary failures somewhat. It costs more index space and does nothing about the deeper issue: a splitter that can't tell a paragraph from a table row. Fixed-size and recursive splitting work fine as a baseline for plain, prose-heavy text. Against the structural mess of a real web page, though, they fail in predictable, repeatable ways, and overlap papers over the failure instead of fixing it.

Semantic chunking — what it gains over character splitting and what it costs

Semantic chunking swaps the character count for a meaning signal. Instead of cutting every N tokens, it looks for where the topic actually shifts, usually by comparing embedding similarity between neighboring sentences and placing a boundary where that similarity drops off sharply.

Two flavors show up in practice. LLM-based semantic chunkers hand the text to a language model and ask it to flag topic boundaries directly, which tends to be more accurate and noticeably slower and more expensive. Cluster-based semantic chunkers group sentences by embedding similarity and draw boundaries at cluster edges; cheaper than the LLM route, but it still needs an embedding call for every sentence, and that adds up fast on long documents.

Retrieval benchmarks show both flavors landing above fixed-size baselines on recall. The gap, though, is a matter of a few percentage points, not a multiple, and that's the detail worth sitting with: semantic chunking's advantage is real but bounded, and it doesn't automatically justify its own cost. Whether the extra compute is worth paying for depends on how much throughput the pipeline needs to sustain, and for most high-volume ingestion jobs, the honest answer is no.

Semantic chunking earns its keep on long articles that drift from one topic to the next without ever using a heading to signal the shift, or pages where the HTML has already been stripped to plain text and no structural markers survive. Structured content is another matter entirely. Adjacent rows in a table, or key-value pairs in a spec sheet, often score high on semantic similarity to each other while belonging to completely different retrieval atoms. A semantic chunker reading that similarity as "these belong together" will merge things that should have stayed separate. High-throughput ingestion pipelines hit the same wall from a different angle, since the per-sentence embedding cost doesn't scale. Practitioner writeups from 2024 make the point plainly: semantic chunking's compute bill often isn't justified by the retrieval gain it produces in production.

One wrinkle the benchmarks tend to underreport: chunking strategy interacts with embedding model choice in ways that aren't obvious in advance. HotpotQA testing found that a chunking approach performing well under one embedding model can underperform under another. Strategy correctness is relative to the model reading it, never independent of it, which should make anyone suspicious of a chunking recommendation that doesn't name the embedding model it was tested against.

Hierarchical chunking — indexing the same document at multiple granularities

Nothing says a document has to be chunked one way. Different stages of a retrieval pipeline want different chunk sizes, and hierarchical chunking is the acknowledgment of that fact: index the same content at several granularities and let each stage pull from the one it needs.

Vector search works best on short, tightly focused chunks, since the query itself is usually short too. A reranker needs enough surrounding context to judge relevance properly, while the language model generating the final answer needs a chunk large enough to actually reason over, often a full section rather than a sentence.

A three-tier structure covers this in practice. A fine-grained tier, sentence-level, tens of tokens, is where vector search actually runs. A mid-grained tier, paragraph-level, a few hundred tokens, is what the reranker scores. A coarse tier, section-level, over a thousand tokens, is what finally lands in the LLM's context. The MAL-RAG research pipeline formalizes a version of this idea, indexing at document, section, paragraph, and multi-sentence levels, with each level doing one job instead of one level trying to do all four.

For web content this matters in a specific way. A section heading and its opening paragraph function as a single retrieval unit; split them apart, which single-tier chunking does routinely, and the unit stops making sense. Hierarchical indexing keeps the parent-child link intact: a sentence gets matched at the fine-grained tier, then gets promoted to its enclosing section before it's handed to the model, instead of arriving as an orphaned fragment.

None of this is free. Storage goes up because the same content gets indexed multiple times, and the pipeline has to track chunk-level provenance so a matched child chunk can be traced back to its parent. That's real engineering work, and teams that treat it as a checkbox tend to end up with three copies of the same content and no clear rule for which one gets served.

Anthropic's Contextual Retrieval technique, published in 2024, sits next to this idea rather than replacing it: before embedding a chunk, prepend a short, LLM-written summary of where that chunk sits in the larger document. Anthropic reports substantial reductions in retrieval failures when this is combined with reranking. The technique makes each chunk more self-contained, which pairs naturally with a hierarchical index rather than competing with it.

Agentic and LLM-driven chunking — when the model decides its own boundaries

Push the semantic approach one step further and the model stops scoring similarity and starts making the boundary call outright. Candidate text segments go to an LLM, and the model decides where one idea ends and the next begins, based on its own read of the content rather than a distance threshold or a character rule.

LumberChunker is the concrete example: it aggregates paragraphs into groups, passes them to a language model, and gets back boundary decisions. Evaluated on GutenQA, it outperformed recursive splitting, semantic chunking, and proposition-level chunking alike.

Treating this as a general-purpose upgrade over semantic chunking is the mistake most teams make; the use case here is narrow but real. Legal documents are full of cross-references, conditional clauses, and nested exceptions that no character-based rule will ever honor correctly. The cost per document is steep, in both latency and dollars, but the precision in honoring cross-references, conditional clauses, and nested exceptions justifies it for the right use cases. Long-form narrative or analytical web content, where the topic changes without a heading to announce it, benefits the same way. So does any low-volume, high-stakes retrieval task, due diligence work, compliance review, where getting the chunk right matters more than getting it fast.

Outside that use case, it fails just as clearly. Running an LLM call over every chunk boundary in a bulk web-scraping pipeline is not economically workable at scale. Structured content, tables, specs, product data, needs a rule that respects rows and columns, not a narrative-boundary detector built for prose that changes subject without warning.

Sequential Hierarchical Agglomerative Chunking offers a middle ground worth naming: it merges adjacent sentences based on semantic similarity under a hard structural constraint that preserves narrative flow, without calling an LLM and without requiring the chunk count to be fixed in advance. It buys some of the agentic approach's judgment without paying its full cost. LLM-driven chunking is a ceiling, useful mainly for calibrating how much a cheaper method is leaving on the table, not a default for routine ingestion.

How content type within a web document should govern chunking logic

Every strategy covered so far carries a hidden assumption about the document it was built for. Fixed-size splitting assumes uniform prose, semantic chunking assumes topically coherent text, and agentic chunking assumes something with narrative structure. A long-form web document violates all three assumptions on the same page, sometimes in the same paragraph. Picking one strategy for the whole document is the error here, not any single strategy on its own.

The fix is to stop treating the page as one content type and route by what's actually there. Prose paragraphs go through recursive or semantic chunking with overlap, aiming for coherence at the paragraph level. Tables need to stay whole: splitting a table mid-row destroys the relationship between header and value that makes the table answerable at all, so the boundary belongs at the table's edge, not at a token count. Code blocks should be atomic, never split regardless of length, with the surrounding explanation optionally attached as context. Bulleted and numbered lists need their governing sentence kept with them, since a list item without the sentence that introduces it is often meaningless on its own.

Two more rules matter as much as any of the above. Navigation menus, footers, and boilerplate get filtered out before chunking starts, not cleaned up after, because every token spent on a cookie banner is index space not spent on something a query could actually need. Metadata, author, publish date, schema.org markup, belongs attached to the chunk as metadata, never folded into the embedded text itself.

None of this requires exotic tooling. It requires parsing the HTML structure before stripping it to plain text, since table tags, code tags, and list tags are far more reliable signals than anything inferred after the markup is gone. Each detected segment then routes to the chunking logic built for it, which is conditional logic whose payoff is a measurable drop in retrieval failures caused by structural fragmentation.

Plenty of web content doesn't arrive with clean markup at all, though. Pages rendered by JavaScript, scraped HTML with broken tag nesting, PDFs converted to text with the structure flattened out: all of these show up in real ingestion pipelines, and a system that only works when the markup is clean can't be trusted to run unattended in production. There has to be a fallback path for when the structural signal simply isn't there.

Retrieval goals and query type as inputs to chunk size decisions

Chunk size gets locked in at index time, but the query that would have told the system what size was actually needed doesn't show up until retrieval time, much later, and by then the chunking decision is already made. That mismatch is the central tension in choosing a chunk size, and it never fully resolves. It only gets managed better or worse depending on how well the indexing strategy anticipated the questions the system would eventually face.

A narrow factual query, something like a single spec value or a date, is well served by small, precise chunks, since the answer likely lives in one sentence and a large chunk would only dilute the match. A broader question that asks for a comparison or a synthesis across a section needs a larger chunk, or the hierarchical structure described earlier, so the model has enough surrounding material to construct an answer instead of stitching together disconnected fragments.

Guessing wrong in either direction doesn't produce a graceful failure. It produces a confident-sounding answer built on a chunk too narrow to support it, or a chunk so broad the relevant fact drowned inside it. Betting an entire index on one chunk size chosen in advance is the wrong call for any system that expects both kinds of query, and most production systems do exactly that. Build for more than one granularity, or accept that half the query types the system faces were never really served.