An automated report recently scored a client site 86 out of 100 and stated it had zero errors and zero redirects. A full crawl of the same site, a few hours later, reached 4,758 URLs and found broken links, five two-hop redirect chains, contact buttons returning 404, and placeholder text live on the pricing page.
Neither report was lying. They were measuring different things. Here is where the gap comes from, and how to close it.
1. Sampling is not crawling
Many tools cap a crawl at 25, 100 or 500 pages. On a site with thousands of URLs, that sample is usually the homepage, the main navigation and the newest posts: the best-maintained pages on the site. Problems live in the long tail: old location pages, county resources, archived posts and the links between them.
If a report doesn't tell you how many URLs it crawled and how many it found, you can't tell what it missed.
2. Sitemap URLs are not the whole site
Fetching every URL in the XML sitemap is a good start, and it will catch sitemap entries that redirect or are marked noindex. But a sitemap only lists the pages you meant to publish. Broken internal links, redirect chains and query-string variants only show up when you follow links from page to page.
On the site above, every one of the 2,120 sitemap URLs returned 200. The problems were in the links: a footer phone button pointing to a broken web address on every page, two county contact buttons whose ?location= parameters produced a 404, and discontinued location pages redirecting to the homepage.
3. Tooling bugs create false positives and false negatives
Crawlers are software, and they have bugs. Two we have hit in practice:
- robots.txt fetched with the wrong client. A 403 on
robots.txtmade one crawler conclude the whole site was blocked. The site was fine; the request library was being rejected. - URL rebuilding errors. Putting a query string into the wrong slot when rebuilding a URL produced hundreds of fake 404s and fake duplicate pages.
Any finding that would change a client's priorities should be re-checked live, by hand or with a second tool, before it goes in a report.
4. CDN caching hides origin problems
When pages are served from a CDN cache, every request gets a fast, clean 200. Requests that reach the origin server (uncached pages, query strings, new posts) can behave very differently. We have seen origins return 403 to specific crawlers and serve a "One moment, please..." interstitial under load, while the cached homepage looked perfect.
5. Content errors are invisible to status codes
A page that says "Confirm before publish" where a price should be returns HTTP 200. So does a legal guide quoting a law that changed two years ago. No crawler flags these on its own; you have to read the pages that matter most to customers: pricing, services, locations and legal content.
A checklist for a report you can trust
- The crawl covers every sitemap URL and follows internal links, with the URL counts stated.
- Redirects are reported with hop counts and final destinations, not just "301".
- Query-string and tracking-parameter variants are tested, not ignored.
- Critical findings are re-checked live, with the date.
- Pricing, service and legal pages are read by a person.
- Speed is tested on mobile, on more than one template.
Read the full case study for the numbers behind this article.