Internal link checker

The tool fetches the home page, then looks for the sitemap where robots.txt says it is, or at /sitemap.xml. When there is one, it takes up to sixty of its addresses spread evenly across the file, so the sample spans the site rather than its first folder. When there is none, it follows the home page's own links, then the links of those pages, until it has sixty. Each page is fetched with a plain crawler's user agent, six at a time, and every internal link on it is read.

From those links it builds the graph of the sample and computes, for each page: how many other pages of the sample link to it, how many distinct internal addresses it links to, and how many clicks it sits from the home page along the sampled links. Then it reports:

• Orphans in the sample: pages no other sampled page links to.
• Pages more than three clicks from the home page, or with no path from it at all.
• The ten least linked and the ten most linked pages.
• The average number of internal links a page carries.
• Pages that did not answer 200: redirects, 404s, timeouts.

A sitemap tells a crawler a page exists. A link tells it the page matters, and what it is about, and hands it some of the standing the linking page has. A page reachable only from the sitemap is crawled less often, indexed later, and holds no standing of its own; Search Console files it under Discovered, currently not indexed, and the fix is never the sitemap.

Depth works the same way. Pages one click from the home page are crawled first and most; pages four clicks down, found through a category, then a listing, then a pagination page, are crawled rarely. On a small site nothing should be more than three clicks from home, and the pages that earn, the pricing page, the best guides, the comparisons, should be one or two.

The verdict names the worst finding: orphans first, depth second, then the good case. Under it, the size of the sample and where it came from, the average links per page, then the tables. Each row gives the page's path, how many sampled pages link to it, how many internal links it carries, and its distance from home in clicks, or no path when the sampled links never reach it.

A worked example. yourproduct.com has a sitemap of 140 addresses; the tool samples sixty. Four guides under /guides/ are linked from nothing but the guides hub, which itself is three clicks from home because the navigation links to /blog and the hub is reachable only from a footer link on the blog. Verdict: no orphans, six pages more than three clicks from home. The fix is one navigation link to /guides, which pulls the whole family two clicks closer; then two body links to each of the four guides from the guides on the same subject.

1. 1

   Give every orphan two links from its own subject

   Not from the footer: from the body of the two pages closest to it, with an anchor that says what the page is. A hub page for the family gives a third.

2. 2

   Bring the earning pages within two clicks

   Pricing, the best guides, the comparisons: a link from the navigation, the home page or a hub. Depth is a decision about what matters.

3. 3

   Add a read-next block to templates that dead-end

   Guides, changelog entries and docs pages that link nowhere are the usual dead ends. Three related links at the foot of each, chosen by subject, and every page becomes a door.

4. 4

   Clean the sitemap of what did not answer 200

   Redirects and dead pages in the sitemap spend the crawl for nothing. List the final address of every page that exists, and only those.

5. 5

   Run the check again in a month

   New pages arrive as orphans by default. Make the check part of publishing: a page is not done until two others link to it.

• A sample of sixty pages, not the whole site. An orphan in the sample may be linked from a page outside it; a page that looks well linked may be linked only by pages inside it. On a site of a few hundred pages the sample is a fair read; on a site of ten thousand it is a sketch.
• Links are read from the HTML as sent. Links drawn by JavaScript after load are not seen, which is also true of the AI crawlers and of Googlebot's first pass.
• Fetching stops after forty seconds, six pages at a time; a slow site gets a smaller sample, and the result says so.
• Five checks an hour per visitor and a shared daily budget, because a check fetches up to sixty pages of someone's site. Nothing is stored.