Crawl Frequency and Freshness Guarantees in Web Search Indexes
Search indexes don't guarantee freshness, limiting what AI systems can reliably retrieve.
Web search indexes do not guarantee the freshness of any page they serve. For AI engineers grounding production systems in live-index retrieval, that structural reality defines what those systems can and cannot deliver. The gap between what a search index provides and what a deployed language model requires is not a quality problem to be tuned away. It is a design consequence of systems built for a different consumer.
How web search indexes decide what to crawl and when
Crawl frequency is not a clock. It is a prioritization algorithm. Google's crawler operates from a continuously replenished queue of addresses, and that queue grows through link-following, submitted sitemaps, and redirect signals. Publishers do not push content into the queue. They wait for the crawler to arrive.
The rate at which any particular page gets recrawled is determined by a combination of factors, none of which a publisher directly controls. The engine weighs how often the page has changed historically, how much link authority points to it, how quickly the host server responds, and what the engine's own systems estimate the page's value to be. Those inputs yield a practical range that is astonishingly wide, and deliberately so. Google's documentation states that it may recrawl news homepages every few minutes while leaving pages that have not changed in years for a month or more between visits. Both ends of that range are operating as intended, not as exceptions.
The latency is structural, not incidental. Google says crawling "can take anywhere from a few days to a few weeks." Internal estimates Gary Illyes presented in October 2026 put the typical discovery time for a new URL at about 20 hours, with indexing itself taking roughly 1.5 hours after that. Crawl budget is finite and shared across every URL on a host, and it is governed by two inputs: crawl capacity, which reflects how responsive the server is, and crawl demand, which reflects how valuable the engine estimates the pages to be.
Why crawling and indexing are separate failure points, each with its own delay
The gap between a crawl event and an index update is a second, independent source of latency. Crawling and indexing are distinct pipeline stages with distinct failure modes. A page can be fetched by Googlebot and still never enter the index. A page already in the index can go unrecrawled for weeks while its live content drifts away from what the index holds. The two stages compound their uncertainty rather than resolving it.
The distinction matters because it defines what "being in the index" actually promises. Crawling decides whether Google gets to see the content of a URL. Indexing decides whether Google stores it and keeps it ready for queries. Only a fraction of what is crawled enters the index at all. Roughly ten percent of crawled pages get indexed, with the engine filtering aggressively for duplicates, low-value pages, and content that fails quality thresholds. A page that fetches successfully is nowhere near guaranteed to become retrievable.
The index itself is not a single uniform store, which compounds the opacity. Gary Illyes described in 2021 that Google organizes its index in tiers by storage type. The names Base, Zeppelins, and Landfills come from the 2024 leak. Different pages sit in different tiers, with different retrieval and refresh characteristics. A retriever querying "the index" is actually querying a stratified store whose tiers it cannot see.
What "freshness" actually means inside a search index, and what it does not mean
Freshness in a search index refers to the currency of the indexed representation of a page relative to the live page. It is a property that is always decaying between crawl events. A page's indexed state is a snapshot taken at the moment of the last successful crawl, and between that moment and the next one, any change to the live page is invisible to the index and therefore invisible to any system retrieving from it. Freshness is not a state a page holds. It is a distance that grows until the next fetch resets it.
The most important distinction for reasoning about this is the separation between recency and freshness. Google formalizes a related concept, Query Deserves Freshness, which identifies query classes where users expect current information and weights when content was last meaningfully changed rather than when it was originally published. That is an acknowledgment that freshness is a per-query property, not a per-page constant.
Crawl frequency is a prerequisite for freshness, not a guarantee of it. Frequent crawling helps only insofar as updates get reflected faster when indexing succeeds. The chain from live change to retrievable update has multiple links, and each link can break independently.
Why the index's crawl tiers produce a two-speed web with no freshness floor
The crawl schedule is not one system applied uniformly. It is a differentiated allocation in which a small fraction of the web receives near-continuous attention and the large majority waits in a long tail of infrequent visits. There is no mechanism to override this from outside, and no published guarantee for any tier.
The consequence is a freshness distribution, not a freshness floor. What any given retrieval returns depends on where that page sits in the crawl priority stack, which is opaque to the retriever at query time. A person looking at a search result can notice a date, compare it with other knowledge, and discard or verify what seems wrong.
For a system retrieving from this index programmatically, an AI agent or a RAG pipeline, tolerability is not the standard. For AI systems that require deterministic freshness windows rather than probabilistic crawl schedules, this allocation model presents a fundamental constraint: general search indexes cannot promise that any specific page will be visited on a known cadence. Seltz, which operates its own crawl infrastructure, engineers refresh guarantees aligned with production AI system demands rather than accepting the variable latency of shared consumer-grade indexes.
How stale index retrieval fails AI systems in ways it does not fail human searchers
When an AI system retrieves from an index, a stale result produces a confident, fluent, incorrect answer. That is a failure mode qualitatively different from a human user receiving an outdated link they can recognize and discard. The staleness is visible to them, and the reader is the error-correction layer. An LLM grounded on a retrieved document has no such external check. It treats the retrieved content as current evidence and generates output whose confidence is uncorrelated with the document's actual age relative to the live web.
The failure pattern is concrete and familiar. There is no indication that the source is stale, because the pipeline has no way to check. This failure is invisible by design. The pipeline continues to answer every query. It simply grounds some answers on outdated evidence with no signal to the downstream system or the end user that the indexed document no longer matches the live source.
What is missing is the temporal reasoning a human performs without effort. The intuition that an article from eight months ago may no longer reflect reality is not present in a cosine-similarity retrieval that selects the most semantically relevant chunk regardless of when it was indexed. The retrieval mechanism optimizes for semantic fit and is blind to time. AI-focused infrastructure that owns its full data pipeline, from fetch through ranking to delivery, eliminates this gap, ensuring that what an LLM retrieves for reasoning has actually been freshly validated against the source.
How data type determines how fast staleness causes harm
Not all stale content fails equally. The harm scales with how fast the underlying data moves and how consequential an incorrect answer is, which means different data types require different refresh cadences to stay within an acceptable error window. Product documentation and API references are frequently workable on a weekly refresh, though version-specific queries remain dangerous on stale docs, particularly when a version change is the substance of the query.
The gradient across these categories is measured in orders of magnitude. Since freshness is a decaying property between crawl events rather than a guaranteed state, production AI applications cannot rely on publication date alone to reason about currency. Systems built specifically for AI agents must operationalize freshness as continuous validation, knowing not just when a page was last seen but when it was last checked and what changed, so the LLM has the grounding to assess its retrieved facts.
Why freshness is structurally downstream of access reliability, not the other way around
Increasing crawl frequency is not a sufficient solution to the freshness problem, because every refresh begins with successfully fetching the source, and reliable access to live web pages is not guaranteed on the modern web. JavaScript-rendered content, the AI overviews, dynamic carousels, and expandable modules that increasingly carry the substantive information, does not appear in a static HTML fetch. A crawl that succeeds at the network level may still return an incomplete or misleading representation of the page.
A SERP API that parses static HTML misses substantial portions of the page depending on query class, because the content a human searcher actually sees is increasingly not in the initial document. A cached index is only as fresh as the ability to replace it, which requires dependable access on demand. The question "how fresh is my data" is downstream of the question "can I reliably get the data at all."
How production teams handle the latency-freshness trade-off in practice
In production AI systems, the dominant pattern is a hybrid approach that matches the retrieval strategy to the volatility of the underlying data. Cached retrieval under a managed freshness TTL serves stable content. Most production systems occupy the middle, with the boundary between the two approaches determined by how fast the source data moves and what a stale answer costs in that context.
An emerging best-practice pattern, sometimes called Temporal RAG, treats freshness as an engineering property to be actively managed rather than assumed. The central insight this pattern encodes is that a completed crawl is not the same as a fresh RAG corpus. Freshness is achieved only when a validated source change has propagated into retrieval and the obsolete version can no longer win at query time.
Enterprise security teams add a constraint that live-fetch approaches must address. Pipelines that bypass access controls to achieve freshness can introduce new attack surfaces, particularly prompt injection through retrieved web content. Data provenance is a security surface, not just a data-quality concern.
What standard SERP APIs and scraping tools structurally cannot provide for AI retrieval
SERP APIs and scraping tools were designed to return results as a human search session would see them: shallow snippets, rendered for display. A SERP API returns the URL, a title, and a short excerpt drawn from the indexed version of the page. It does not return the full, structured, machine-readable content that a language model can reason over without extensive post-processing. Scraping tools that parse static HTML miss the script-rendered layer of modern pages, where the substantive information increasingly lives, producing incomplete or misleading source material before freshness is even considered.
Neither approach was designed for the depth, latency predictability, or reliability that autonomous agents and high-throughput LLM pipelines require. Scraping tools retrieve whatever HTML the server returns. Neither selects, filters, ranks, or shapes content for AI reasoning as a first-class design concern. The result is that AI teams using these tools must build substantial post-processing layers themselves: parsing, deduplication, relevance filtering, freshness assessment. That work belongs in the retrieval infrastructure, not in the application layer, and its absence from standard tools is not an oversight but a reflection of what those tools were built for.
What AI-native web retrieval infrastructure has to do differently, and where Seltz fits
The structural limitations of standard indexes, SERP APIs, and scraping tools converge on a single requirement. AI agents and production LLM systems need retrieval infrastructure that owns the full data pipeline end to end, delivers machine-ready content, and treats freshness and access reliability as first-class engineering concerns rather than downstream optimizations. And it has to deliver predictable latency and reliability under real production load, with data provenance and access controls that satisfy enterprise security requirements without requiring the AI team to build a separate compliance layer.
Seltz is built around exactly this gap. It is a purpose-built Web Knowledge API for AI agents, LLM applications, and RAG pipelines that owns its full data pipeline end to end, delivering context-engineered, machine-ready web content selected, filtered, ranked, and shaped specifically to maximize usefulness for AI reasoning rather than formatted for human eyes. The web is increasingly consumed by machines, and the infrastructure serving it needs to be rebuilt with that reality as the starting point, not retrofitted from tools designed for human search.
