Ahrefs vs Screaming Frog: an index against a crawler

Ahrefs runs its own crawl of the web and sells you queries against the result: who links to a domain, what it ranks for, which pages earn its traffic. You are asking about the outside world, including your rivals.

Screaming Frog runs on your own machine and fetches the pages you point it at, one by one, then puts everything it found in tables: status codes, titles, canonicals, headings, links, redirects. You are asking about one site, usually yours, in detail.

• Every URL on the site, including the ones nobody links to and no search engine has bothered to index.
• The exact status code of each, and the chain when one redirects to another that redirects again.
• Titles, descriptions, canonicals, robots directives and headings for all of them in one table you can sort.
• Internal links in and out per page, which is how you find orphans and count click depth.
• Whatever you tell it to extract: a price, a schema field, a missing element, through custom extraction on a licence.
• The site as it is right now, not as an index remembers it from its last crawl.

The free version crawls up to 500 URLs, which covers a small product site completely. A licence removes the limit and adds scheduling, custom extraction and connections to other APIs.

• Who links to a site, and to which page, with the anchor text. Your own crawler cannot see inbound links at all.
• What a rival ranks for, and which of their pages earns it.
• Search volumes and difficulty estimates for the queries you are considering.
• History: what a domain looked like a year ago, and which links it lost.
• The same picture for any domain on the web, without access to it.

None of that is derivable from crawling your own site, which is why the two tools sit in different parts of a workflow rather than competing for the same slot.

Ahrefs has a crawler too: Site Audit runs against your site from their servers on a schedule and reports the same families of problem. It is comfortable, it needs no install, and it keeps a history you can compare.

• A cloud audit is easier to keep running, and easier to share with someone else.
• A desktop crawl is faster to iterate with: change something, re-crawl a folder, look again in a minute.
• The desktop tool goes further into the odd cases, because you control the configuration and can extract anything from the HTML.
• Ahrefs Webmaster Tools gives the audit free on sites you verify, which covers a small site's technical checks without a licence.

A product site of eighty pages has a short technical list and a long list of questions about the outside world. That asymmetry decides the order.

• Month one: crawl the site once with the free crawler, fix the twenty things it finds, and move on. The technical work is finite.
• Month two onwards: the questions are about rivals, searches and links, which need an index.
• Keep the crawler for the day after a migration, a redesign or a URL change, when the technical list refills.

Neither tool tells you what your own pages actually earn. That lives in Search Console, free, and it is the first thing to connect whichever of these you buy.

1. 1

   Crawl your site

   Point the crawler at the home page, let it finish, and sort by status code. Fix the 404s and the redirect chains, then sort by title and find the duplicates and the empties.

2. 2

   Export the pages that matter

   Your twenty pages with the most internal links. That is the list you will compare against the outside.

3. 3

   Look up two rivals in the index

   Their top pages by traffic, the searches those pages hold, and who links to them.

4. 4

   Cross the two lists

   Searches where a rival ranks and you have a page, but it is weak, are this month's work. Searches where you have no page at all are next month's.

5. 5

   Write down which tool answered which question

   After one afternoon you will know which of the two you would miss, and that is the one to pay for.