Googlebot Crawl Data Analysis: Where Crawl Budget Goes
Googlebot crawl data study: where crawl budget goes on real sites, the share of hits on parameters, redirects, 404s and assets, and which waste to fix first.
Kazi Khalid Hasan Tamim · BugViso Team
11 min read
Quick answer: Googlebot crawl data analysis starts by classifying every Googlebot request in your server logs: indexable pages, redirects, 404s, server errors, parameter URLs and static assets. Anything that isn't an indexable page or a needed resource is waste. The usual culprits are redirecting internal links, removed pages returning
200, parameter URLs and slow responses.
Search Console tells you how much Google crawls. Your logs tell you what it crawls, request by request.
This guide shows how to turn raw access logs into a waste report, what real sites leak, and the fixes in order of payoff. For how often Google crawls and how to raise it, see the companion guide how often Google crawls your site.
Where Crawl Requests Go Wrong: Real Site Data
We crawled randomly sampled sites (Tranco ranks 1,001–50,000) the way a search crawler does, following internal links from the homepage, and checked their sitemaps. October 2026:
| Crawl-waste source | Finding | Sample |
|---|---|---|
| Internal links that redirect | 56.6% of homepages | 219 sites |
| Redirect chains (2+ hops) among homepage links | 189 links | 17,435 links checked |
Link-followed requests that hit 404 | 1.2% of requests | 11,691 requests, 76 sites |
Requests answered 429 Too Many Requests | 0.7% of requests | 11,691 requests, 76 sites |
| Sitemaps listing redirecting URLs | 25% | 36 sitemaps |
Sitemaps listing 404/410 URLs | 14% | 36 sitemaps |
A crawler that follows your links and your sitemap meets every one of these. Each redirect is an extra request before the real page; each 404 in the sitemap is a request for nothing; each 429 tells Google's crawler to slow down, which Google says lowers your crawl capacity.
What Google Says Wastes Crawling
From Google's crawl budget documentation:
| Issue | Google's words |
|---|---|
| Soft 404s | "soft 404 pages will continue to be crawled, and waste your budget." |
| Removed pages | "Return a 404 or 410 status code for permanently removed pages … a 404 status code is a strong signal not to crawl that URL again." |
| Redirect chains | "Avoid long redirect chains, which have a negative effect on crawling." |
| Slow or failing servers | If the site "responds with server errors (5xx) or rate-limiting signals (such as HTTP 429), the limit goes down." |
Step 1: Get the Data
Start with three data sources: Search Console's Crawl stats, server or CDN access logs, and a site crawler. The table shows what each one gives you and where it falls short.
| Source | What it gives you | Limits |
|---|---|---|
| Search Console → Settings → Crawl stats | Daily requests, response codes, file types, purpose, average response time (90 days) | Aggregated; sample URLs only |
| Server or CDN access logs | Every request: URL, status, timestamp, user-agent, IP | You must keep and parse them |
| A site crawler | What links and sitemaps expose to crawlers | Simulates Googlebot; doesn't show Google's choices |
Logs are the ground truth. Keep at least 30 days, and make sure your CDN logs are included if the CDN answers some requests itself.
Step 2: Classify Every Row of Googlebot Crawl Data
This standard-library script reads combined-format logs and buckets every request with "Googlebot" in its user-agent:
"""Classify where Googlebot's requests go in your access logs, and how many are waste.
Usage: python3 crawl_waste.py access.log [more.log ...]
Counts requests whose user-agent contains "Googlebot" (verify IPs separately for
security decisions) and buckets them as: indexable page (200 HTML path),
redirect, not found, server error/rate limit, parameter URL, static asset,
or robots/sitemap. Then lists the top wasted URLs and the busiest sections.
"""
import re, sys
from collections import Counter
LINE = re.compile(r'"(?:GET|HEAD) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*Googlebot[^"]*)"')
ASSET = re.compile(r"\.(js|css|png|jpe?g|gif|webp|avif|svg|ico|woff2?|ttf|map|json)$", re.I)
def bucket(url, status):
path = url.split("?")[0]
if path in ("/robots.txt",) or path.endswith(".xml"):
return "robots/sitemap"
if ASSET.search(path):
return "static asset"
if status.startswith("3"):
return "WASTE: redirect"
if status in ("404", "410"):
return "WASTE: not found"
if status.startswith("5") or status == "429":
return "WASTE: server error / rate limit"
if "?" in url:
return "WASTE?: parameter URL (check canonical)"
return "indexable page" if status == "200" else f"other {status}"
def main(paths):
buckets, wasted, sections = Counter(), Counter(), Counter()
for path in paths:
for line in open(path, errors="replace"):
m = LINE.search(line)
if not m:
continue
url, status = m.group(1), m.group(2)
b = bucket(url, status)
buckets[b] += 1
if b.startswith("WASTE"):
wasted[(status, url)] += 1
sections["/" + url.lstrip("/").split("/")[0].split("?")[0]] += 1
total = sum(buckets.values())
if not total:
print("No Googlebot requests found.")
return
print(f"{total} Googlebot requests\n")
for b, n in buckets.most_common():
print(f" {n:>7} {100 * n / total:5.1f}% {b}")
waste = sum(n for b, n in buckets.items() if b.startswith("WASTE"))
print(f"\nPotential waste: {waste} requests ({100 * waste / total:.1f}%)")
print("\nTop wasted URLs:")
for (status, url), n in wasted.most_common(10):
print(f" {n:>6} {status} {url}")
print("\nBusiest sections:")
for s, n in sections.most_common(8):
print(f" {n:>6} {s}")
if __name__ == "__main__":
main(sys.argv[1:])Example output on a small test log:
9 Googlebot requests
3 33.3% indexable page
1 11.1% WASTE: redirect
1 11.1% WASTE: not found
1 11.1% WASTE?: parameter URL (check canonical)
1 11.1% static asset
1 11.1% robots/sitemap
1 11.1% WASTE: server error / rate limit
Potential waste: 4 requests (44.4%)
Top wasted URLs:
1 301 /blog/post-b?utm=x
1 404 /old-page
1 200 /shop?color=red&size=m
1 503 /api/searchAnyone can send a "Googlebot" user-agent. For security or blocking decisions, verify IPs with reverse and forward DNS; the companion post's script does that.
Step 3: Read the Buckets
Compare each bucket with a healthy share: most requests should hit indexable pages, and server errors should sit near zero. The table shows what each bucket should look like and what to do when it runs high.
| Bucket | Healthy share | If it's high |
|---|---|---|
| Indexable page | The large majority of page requests | — |
| Redirect | Small | Update internal links and sitemap to final URLs; collapse chains |
| Not found | Small, falling over time | Remove links to dead URLs; keep real 404/410 |
| Server error / rate limit | Near zero | Fix capacity; don't rate-limit verified Googlebot |
| Parameter URL | Depends on the site | Canonicalise, stop linking parameter variants, consolidate facets |
| Static asset | Normal | Needed for rendering; don't block CSS/JS in robots.txt |
Step 4: Fix in Order of Payoff
Fix the buckets in order of payoff, starting with server errors and 429s because they lower the crawl capacity limit for everything else. The numbered list gives the full order.
- Server errors and
429s. They lower the crawl capacity limit for everything else. - Soft 404s. Return real
404/410for removed content. - Redirecting internal links and sitemap entries. Point them at final URLs.
- Redirect chains. Make every redirect a single hop.
- Parameter sprawl. Canonicals, consistent internal linking, and not linking to filter combinations you don't want indexed.
Our guides to redirect chains and loops and URL parameters and crawl budget cover steps 4 and 5.
How BugViso Finds Crawl Waste Before Googlebot Does
A BugViso audit crawls your site from its links and sitemap and reports the site-side causes of crawl waste: broken internal links, links that redirect, redirect chains with every hop, canonical conflicts, noindex pages, duplicate content clusters and orphan pages. It can't see Google's own crawl decisions (only your logs and Search Console can), but fixing what it finds removes most of what the buckets above flag.
You can see every rule BugViso applies in its crawl-based website audit.
FAQ
How do I analyse Googlebot crawl data?
Export at least 30 days of server or CDN access logs, filter verified Googlebot requests, and classify them by status code and URL type. Compare with Search Console's Crawl Stats report.
What wastes crawl budget?
Redirects (especially chains), soft 404s, server errors and rate limiting, parameter URL variants and duplicate content. Google's documentation names soft 404s, long redirect chains and server errors explicitly.
Do I need to worry about crawl budget on a small site?
Google's guide is aimed at sites with 1M+ pages, or 10k+ pages changing daily. Smaller sites still benefit from removing waste, because it gets new and updated pages crawled sooner.
Should I block CSS and JavaScript from Googlebot to save crawl budget?
No. Google needs them to render pages. Blocking them can hurt how your pages are understood.
New to crawling and indexing? Start with our beginner's guide to technical SEO. To go deeper on the logs themselves, read log file analysis for SEO.
The Takeaway
Crawl budget is wasted on your side of the request, in redirects, dead URLs and slow responses, so classify your logs and fix what a BugViso crawl audit flags.
Written by
Kazi Khalid Hasan Tamim · BugViso Team
Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.
Summarise with AI
Open this guide in your assistant.
Follow BugViso
New guides and audit data as we publish them.
AI visibility tracker
Is your brand in AI answers?
See how often AI assistants mention you, cite your site and rank you against your competitors.
- ChatGPT
- Gemini
- Perplexity
- Claude
- AI Overviews
See where your site stands
Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.