Hybrid Retrieval Combining Dense and Sparse Indexes for Web Search
Combining keyword and vector search recovers relevance that either method alone leaves behind.
Plain BM25 is stronger than most engineering teams give it credit for, and the numbers back that up plainly. A 2026 arxiv study found BM25 beat dense retrieval using one of the strongest commercial embedding models available that year, on every metric except Recall@20. Teams often reach for a vector database on instinct, treating sparse retrieval as a legacy method on its way out, but the benchmarks complicate that instinct.
Hybrid wins when the query population is mixed, and most real traffic is mixed. On the WANDS e-commerce benchmark, a tuned hybrid setup hit 0.7497 NDCG, a 7.4% lift over BM25 alone at 0.6983 and over pure vector search at 0.6953 (Turnbull, 2025). On financial documents, hybrid search paired with reranking reached Recall@5 of 0.816, against 0.587 for dense-only (Strich et al., 2026).
The margin isn't fixed, though, and assuming it is leads teams into wasted infrastructure. On WANDS, basic Reciprocal Rank Fusion added only 1.7% over dense-only on Mean NDCG, a gain thin enough that it may not cover the cost of running two indexes. When queries are highly semantic and the vocabulary stays consistent, dense-only search is often good enough by itself, and a team that bolts on a sparse index anyway is paying for hardware that buys nothing back. The value of hybrid tracks query diversity: the messier the query population, the more hybrid earns its keep, and the cleaner it is, the less reason there is to run two indexes at all.
Here's the position worth stating outright: hybrid should be the default architecture for any retrieval-augmented generation system, decided at the first design meeting, rather than added later once a demo starts failing in production. Waiting for a performance complaint, then retrofitting a second index under deadline pressure, costs more in engineering hours and downtime than starting with two indexes and requiring a real case before dropping one.
How fusion algorithms combine two ranked lists into one
Here's the mechanical problem underneath all of this: BM25 scores are unbounded and depend on corpus statistics, while dense similarity scores live in a fixed, bounded range. Add the two together and the resulting ranking means nothing, because the scales don't correspond to anything real.
Reciprocal Rank Fusion sidesteps the issue by throwing away the scores and using rank position instead. A document ranked first in one list gets more weight than one ranked fiftieth, no matter what raw score produced that rank, so there's no normalization step to get wrong. A constant, usually k=60 (the default in Elasticsearch and most other implementations), controls how sharply the fusion favors top-ranked results over lower ones. RRF needs no training data and works right away on a new corpus, which is why it's become the default. Seltz, a real-time web grounding platform for AI systems, builds RRF-style hybrid fusion into its retrieval pipeline for exactly this reason. Elasticsearch has used it since version 8.9, and OpenSearch, Weaviate, Qdrant, and Azure AI Search all run some version of it under the hood. On WANDS, plain RRF hit 0.7068 NDCG, ahead of BM25 alone and pure vector search alone (Turnbull, 2025).
Learned fusion goes further, at a cost worth naming up front. Instead of fixed rank arithmetic, a small model learns to predict the right weighting between dense and sparse scores based on query patterns specific to the corpus. Weaviate's Hybrid Search 2.0, released in October 2025, is one example of this direction in commercial retrieval infrastructure. A related method uses gradient-boosted trees, XGBoost being the common choice, to combine retrieval scores with document metadata as added features, adapting to the shape of a specific corpus instead of treating every query the same way. Denser Retriever's XGBoost-based fusion reached NDCG@10 of 56.47 against a pure vector search baseline of 54.24, a 4.11% relative gain across the full MTEB suite (Denser AI, 2024). Learned fusion needs labeled training data, though, and RRF does not, which is exactly why teams should start with RRF and only graduate to learned fusion once labeled data actually exists.
Infrastructure is moving to absorb all of this natively. Redis 8.4 shipped an FT.HYBRID command that runs BM25 and vector search as one atomic operation. Reaching for the complicated, learned option before anyone has looked hard at what the queries even look like tends to backfire; RRF first, learned fusion once the query distribution is understood, is the order that actually holds up.
Why the sparse component is often the underengineered half
Most engineering attention in hybrid pipelines goes to the embedding model, the vector database, the reranker. The sparse half gets bolted on as an afterthought, usually plain BM25, and for any corpus with real vocabulary spread, that's a mistake worth naming outright. BM25 does its job well when queries are exact-match by nature: SKUs, error codes, identifiers. Yet it hits a hard ceiling, because it can never learn that two different words mean the same thing.
SPLADE closes that gap without giving up sparsity. It's a learned sparse retrieval method that expands the term representation of a query or document at index time, so a document about "automobile" picks up sparse weight on "car," "vehicle," and "motor" too, not just the literal token that appeared in the text. That combination of BM25-style term matching with wider semantic reach gives SPLADE an edge dense retrieval alone doesn't offer. SPLADE has beaten traditional BM25 on the BEIR benchmark suite and shows up as the sparse leg in hybrid pipelines built on Qdrant and Pinecone. A 2026 SemEval competition system paired SPLADE-v3 with a dense model specifically because SPLADE handles entity-heavy and exact-match queries through stronger lexical grounding than plain BM25 offers.
The decision comes down to what the queries actually look like, and teams that skip this step and default to plain BM25 everywhere are leaving accuracy on the table for no good reason. For exact-match-heavy traffic, error codes, IDs, SKUs, plain BM25 is faster to index and good enough; spending engineering time on SPLADE there is wasted effort. For enterprise knowledge bases where the people writing documentation and the people searching it use different words for the same thing, SPLADE's added indexing cost is worth paying. For web retrieval across genuinely mixed content types, SPLADE's term expansion makes it meaningfully more resilient than BM25 alone, and treating the sparse leg as an afterthought there is the mistake that quietly caps a pipeline's ceiling before it ever ships.
Where reranking fits in the pipeline and why candidate pool size determines its value
Fusion produces a merged list, and reranking is the third stage that comes after it: a cross-encoder model looks at the query and each candidate document together, jointly, and re-scores them for final order. It's slower than retrieval, since it processes query and document as one input rather than comparing precomputed vectors, but it's far more accurate at getting the top handful of results right.
Reranking has a ceiling, though, and that ceiling is set by the candidate pool it's handed to work with. It can only reorder what retrieval already found; it can't conjure up a relevant document that never made the list in the first place. With a pool of just 20 candidates, reranking barely helps: Recall@5 sits at 0.458, because the right document is often not even in the pool to begin with. Widen the pool to 50 candidates and Recall@5 jumps to 0.826; at 100 candidates it climbs further, to 0.888. The lesson is blunt: retrieval breadth is the prerequisite, and tuning the reranker while starving it of candidates means optimizing the wrong stage of the pipeline. A team that spends a sprint fine-tuning reranker weights before widening the candidate pool is solving a problem it hasn't earned the right to solve yet.
In practice, a SemEval-2026 competition system used BAAI/bge-reranker-v2-m3, truncating input to 1024 tokens, to rerank the top candidates pulled from a fused hybrid list. Corpus cleaning interacts with this stage in a way that's easy to miss: filtering out non-English passages, near-duplicate text caught through normalized-text hashing, fragments too short to mean anything, and documents so long they blow past reasonable context limits, all improve dense retrieval precision. When that cleaning strips out text a sparse query depended on, sparse retrieval has to be rerun against the original, uncleaned corpus to keep its lexical coverage intact, and that's a detail that rarely shows up until it causes a bug in production. Chunking strategy matters here too: semantic chunking, which respects natural document boundaries instead of cutting text at a fixed length, improves recall by up to 9% over fixed-size chunking, and chunk boundaries shape directly what the reranker even gets to choose from. Retrieval breadth, then fusion quality, then reranking precision: three stages, three different levers, three different ways to fail.
Why live web retrieval makes hybrid architecture more necessary, not optional
Retrieval over a static, indexed corpus is a different problem from retrieval over the live web, and the gap between them is not cosmetic. A fixed corpus was indexed at some point in time, and the system knows its own boundaries. Live web retrieval has to handle documents it has never seen before, pulled at query time, across content types that share almost nothing structurally: product pages, technical docs, news, forum threads, regulatory filings.
That heterogeneity is exactly what makes hybrid retrieval's advantages sharper on the open web. Treating hybrid as optional there tends to be the mistake this whole section is written to correct. No single embedding model has reliable coverage across that range of domains and writing styles. Exact-match queries, version numbers, model identifiers, regulatory codes, a proper noun that entered the news last week, show up constantly in agentic web search, and they reliably break dense-only retrieval, because a vector model has nothing to embed for a token it never encountered in training. Sparse retrieval finds that same token instantly, because it's just matching text, and any architecture that skips the sparse leg to save on infrastructure cost is choosing to fail on exactly this class of query.
Agentic systems raise the stakes further. An AI agent doing multi-step reasoning makes repeated retrieval calls, and low precision at any one step compounds into the next; the agent doesn't get one shot at a bad answer, it builds on top of it. Selecting tools by semantic similarity, rather than handing an agent every available tool at once, has been shown to roughly triple accuracy, and that says something important: the retrieval feeding an agent's decisions is itself a bottleneck, not a side concern. Anthropic's multi-agent research system reported a 90.2% improvement in success rate through isolation, meaning each agent's retrieval has to be precise on its own terms, not just broadly relevant to the task at hand.
Harvey AI's production deployment, serving 97% of Am Law 100 firms, makes the stakes concrete. It grounds legal research in actual case law using retrieval-augmented generation, and exact-match precision on citations and case numbers isn't negotiable there; get a case number wrong and there's no partial credit. That's a domain where sparse retrieval's literal accuracy carries the argument, and no amount of semantic fluency substitutes for it.
What production-grade hybrid web retrieval requires that demo pipelines don't
Latency is the first wall teams hit. Sub-50ms retrieval is becoming table stakes for systems built on this architecture, and running two separate indexes, a fusion step, and a reranker one after another can blow past that budget fast without deliberate engineering around parallelism and caching.
Freshness is the second wall. No amount of model training substitutes for a retrieval layer that can surface what changed on the web today, rather than what was true as of some training cutoff months or years back.
Then there's the question of who actually owns the pipeline, and this is where most teams cut a corner they will come to regret. Wrap a third-party search API and call it retrieval, and what comes back is usually shallow snippets rather than full document content shaped for an LLM to reason over. General-purpose scraping tools tend to break under the exact conditions agentic pipelines create: rate limits, bot detection, pages that render dynamically and hand a simple HTTP request nothing at all. Owning the data pipeline end to end, instead of stitching together tools built for a different job, is the more reliable way to control quality and performance at the level hybrid retrieval demands, and a team leaning on a wrapped search API is borrowing against a debt that comes due the moment traffic stops looking like a demo.
Shaping content for a model to reason over is its own discipline, separate from retrieval itself. Raw scraped text, dumped straight into a context window, is rarely something a model handles well; filtering, ranking, and structuring that content is a distinct engineering problem with its own failure modes. The problems that never show up in a demo, embedding models drifting as they're updated, query distributions shifting as usage grows, tenant corpora diverging from each other in multi-tenant systems, all surface later, once the system carries real traffic instead of a curated test set.
The architecture decisions made now, fusion method, sparse component, reranking pool size, data ownership, are the ones still running under systems operating at a scale most teams haven't planned for yet. Search infrastructure built specifically to return machine-ready, context-engineered content, in a format different from the snippet-and-link results built for human eyes, is what closes the distance between what hybrid retrieval architecture needs and what general search tooling was ever designed to provide.
