Caching: what to cache, for how long, and what it does for crawling

Search cares mostly about the middle row. A crawler is usually a first-time visitor from somewhere far away, so a warm edge cache is what it experiences, and the time to first byte it measures is the one your visitors get too.

Any file whose name contains a hash of its content can be cached as long as you like, because a change produces a new name and therefore a new URL.

# /_next/static/chunks/main.6f3c9a2b.js, /assets/logo.4b21e0.svg
Cache-Control: public, max-age=31536000, immutable

• A year is the conventional maximum, and immutable tells the browser not to revalidate even on a reload.
• This applies to hashed JavaScript, CSS, fonts and images produced by your build.
• It does not apply to files with stable names, such as /logo.svg or /favicon.ico. Those get a shorter cache.
• Most frameworks and platforms set this correctly by default. Check rather than assume.

HTML has one URL forever, so it cannot be cached like an asset. The aim is to avoid rebuilding it for every request without serving yesterday's page after you publish.

# A page that changes occasionally
Cache-Control: public, max-age=0, must-revalidate
ETag: "a1b2c3"

# The same page, cached at the edge and revalidated in the background
Cache-Control: public, s-maxage=600, stale-while-revalidate=86400

• max-age governs the browser, s-maxage governs the CDN. They can differ, and usually should.
• An ETag or a Last-Modified header lets a cache ask "has this changed?" and receive a small 304 instead of the whole page.
• must-revalidate means the cached copy may be used only after checking, which is what you want for pages that matter.
• Never send no-store for public pages. It forbids caching anywhere and every visitor pays full price.

With stale-while-revalidate the edge serves the cached page immediately and fetches a fresh copy in the background. The visitor who triggers the refresh never waits for it.

• The first visitor after the window gets the old page instantly, and the next one gets the new page.
• Your origin handles one request per window per edge location rather than one per visitor.
• For a marketing site or a library of guides, a window of minutes with a long stale period is usually right.
• Pair it with a purge on deploy so a correction is live immediately rather than at the end of the window.

This is the difference between a server-rendered site that feels slow and one that feels like static files, without turning it into static files.

A CDN in front of the origin removes most of the time to first byte for visitors far from your server, because the response comes from a machine near them rather than from one continent away.

• The distance: the round trips of the handshake happen close to the visitor.
• The origin's work: on a cache hit, your server is not involved at all.
• The variance: a warm cache answers in a consistent time, which matters because field data is measured at the 75th percentile of real loads.
• Compression and modern protocols usually arrive with it: Brotli for text, HTTP/2 and HTTP/3 on the connection.

Time to first byte is not itself a Core Web Vital, but Largest Contentful Paint inherits every millisecond of it. Fixing the first byte is the cheapest way to move a page that is close to the 2.5 second threshold.

The Crawl stats report in Search Console shows the response codes, the download sizes and the average response time over ninety days. A rising response time there is usually a caching problem before it is a code problem.

• No cache headers at all, so every visitor and every crawler pays for a full render.
• Caching a logged-in page at the edge, which serves one person's dashboard to everyone. Mark private responses private and vary correctly.
• Caching a page that varies by cookie without telling the cache, which mixes up sessions.
• Hard-caching unhashed assets, so a deploy leaves visitors on the old stylesheet.
• no-store on public marketing pages, usually copied from an authenticated route.
• No purge on deploy, so a fix you shipped is invisible for the length of the window.
• Caching a 404 or a 500 for a long time, which keeps an error alive after you repaired it.

Check your live site rather than your config: request a page and read the Cache-Control and ETag headers that actually come back. Platforms override what you set more often than you would think.