How Often Does Google Crawl Your Site? How to Measure It
How often does Google crawl a site? There is no fixed schedule. See how size, change rate and links drive it, how to measure yours and how to raise it.
Kazi Khalid Hasan Tamim · BugViso Team
11 min read
Quick answer: How often Google crawls a site has no fixed schedule. Google crawls each URL based on demand (how popular it is and how often it changes) and capacity (how fast and reliably your server responds). Popular, frequently updated pages can be crawled several times a day; static, rarely linked pages may wait weeks.
Generic averages ("small sites get crawled every X days") don't tell you much, because crawl frequency varies by URL even within one site. What you can do is measure it for your own pages and remove the things that slow it down.
This guide covers how Google decides, how to measure, and what to fix.
How Google Decides How Often to Crawl
Google's crawl budget documentation describes two forces:
| Factor | What Google says | What it means for you |
|---|---|---|
| Crawl capacity limit | "If the site responds consistently and its response times … remain stable or improve, the limit goes up … If the site slows down … or responds with server errors (5xx) or rate-limiting signals (such as HTTP 429), the limit goes down." | A fast, stable server can be crawled more |
| Crawl demand (popularity) | "URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems." | Well-linked pages get revisited more |
| Crawl demand (staleness) | Google tries to keep its copy of pages fresh, so pages that change are worth revisiting | Pages that change often get recrawled more |
The same documentation says the crawl budget guide is for "large sites (1 million+ unique pages) with content that changes moderately often (once a week)" and "medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)". For most sites, crawl budget isn't a constraint; crawl efficiency (not wasting requests) still is.
What Wastes Google's Crawls: Data
We can't see Google's logs for other people's sites, but we can measure the conditions Google says reduce crawling. From our studies of randomly sampled sites (Tranco ranks 1,001–50,000), October 2026:
| Signal Google says affects crawling | Finding | Sample |
|---|---|---|
| Slow server responses (TTFB over 800 ms on throttled mobile) | 49.5% of homepages | 200 sites |
| Sitemaps listing URLs that redirect | 25% of sitemaps | 36 sitemaps |
Sitemaps listing URLs that return 404/410 | 14% of sitemaps | 36 sitemaps |
| Internal links pointing at redirects | 56.6% of homepages | 219 sites |
Homepage <title> changed in 5 days (a freshness signal) | 5.5% of homepages | 183 sites |
Every redirecting sitemap entry or internal link costs Googlebot an extra request to reach the real page. Google is explicit about two related cases: "soft 404 pages will continue to be crawled, and waste your budget," and "avoid long redirect chains, which have a negative effect on crawling."
How Often Does Google Crawl a Site? How to Measure Yours
1. Search Console Crawl Stats
Open Settings → Crawl stats in Search Console. It shows total crawl requests per day for the last 90 days, average response time, and breakdowns by response code, file type, purpose (discovery vs refresh) and Googlebot type. Watch for:
- A drop in requests at the same time as a rise in average response time (capacity limit falling).
- A high share of
301,404or5xxresponses (wasted crawls). - "Discovery" crawls staying low after you publish new pages (weak internal linking or sitemap).
2. URL Inspection
For a single page, URL Inspection → Page indexing shows the last crawl date. It's the quickest answer to "when did Google last see this page?"
3. Server logs (the only per-URL, per-day answer)
Your access logs record every Googlebot request. This script counts verified Googlebot hits and shows when each section was last crawled. It verifies each IP with reverse and forward DNS, as Google's Googlebot verification guide describes, because many bots fake the Googlebot user-agent.
"""How often does Google crawl my site? Answer it from your own access logs.
Usage: python3 googlebot_log.py access.log [access.log.1 ...]
Reads combined-format logs (nginx/Apache default), keeps requests whose
user-agent claims Googlebot, verifies each IP with reverse + forward DNS as
Google documents (googlebot.com / google.com / googleusercontent.com), then
reports verified hits per day, status codes, most-crawled paths and how many
days since each section was last crawled.
"""
import re, socket, sys
from collections import Counter, defaultdict
from datetime import datetime
LINE = re.compile(r'^(\S+) \S+ \S+ \[([^\]]+)\] "(?:GET|HEAD|POST) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')
GOOGLE_DOMAINS = (".googlebot.com", ".google.com", ".googleusercontent.com")
_verified = {}
def is_googlebot(ip):
if ip not in _verified:
try:
host = socket.gethostbyaddr(ip)[0]
ok = host.endswith(GOOGLE_DOMAINS) and ip in socket.gethostbyname_ex(host)[2]
except OSError:
ok = False
_verified[ip] = ok
return _verified[ip]
def main(paths):
per_day, statuses, urls, fake = Counter(), Counter(), Counter(), Counter()
last_seen = defaultdict(lambda: datetime.min)
for path in paths:
for line in open(path, errors="replace"):
m = LINE.match(line)
if not m or "Googlebot" not in m.group(5):
continue
ip, ts, url, status = m.group(1), m.group(2), m.group(3), m.group(4)
if not is_googlebot(ip):
fake[ip] += 1
continue
when = datetime.strptime(ts.split()[0], "%d/%b/%Y:%H:%M:%S")
per_day[when.date()] += 1
statuses[status] += 1
urls[url.split("?")[0]] += 1
section = "/" + url.lstrip("/").split("/")[0].split("?")[0]
last_seen[section] = max(last_seen[section], when)
total = sum(per_day.values())
if not total:
print("No verified Googlebot requests found.")
return
days = len(per_day)
print(f"Verified Googlebot requests: {total} over {days} day(s), {total / days:.1f}/day")
print(f"Unverified 'Googlebot' requests (fake UAs): {sum(fake.values())} from {len(fake)} IP(s)\n")
print("Status codes:", ", ".join(f"{k}: {v} ({100 * v / total:.0f}%)" for k, v in statuses.most_common()))
print("\nMost-crawled URLs:")
for u, n in urls.most_common(10):
print(f" {n:>6} {u}")
newest = max(last_seen.values())
print("\nDays since last crawl, by section:")
for section, when in sorted(last_seen.items(), key=lambda kv: kv[1]):
print(f" {(newest - when).days:>4} {section}")
if __name__ == "__main__":
main(sys.argv[1:])Example output on a small test log:
Verified Googlebot requests: 4 over 3 day(s), 1.3/day
Unverified 'Googlebot' requests (fake UAs): 1 from 1 IP(s)
Status codes: 200: 2 (50%), 301: 1 (25%), 404: 1 (25%)
Most-crawled URLs:
1 /blog/post-a
1 /blog/post-b
1 /pricing
1 /old-pageRun it on a month of logs. The 301 and 404 shares show how much of Google's attention goes to URLs that aren't your real pages.
How to Get Important Pages Crawled More Often
Link to important pages from pages Google already visits often, keep the sitemap clean, and cut wasted requests such as redirect chains and soft 404s. The table lists each action and why it works.
| Action | Why it works |
|---|---|
| Link to new and updated pages from pages Google already visits often (home, hubs, category pages) | Raises crawl demand through internal popularity |
Keep the sitemap to final, indexable URLs with honest <lastmod> | Cuts wasted requests; lastmod that changes on every build teaches Google to ignore it |
| Fix redirect chains and point internal links at final URLs | One request per page instead of two or three |
Return real 404/410 for removed pages | Google stops recrawling them; soft 404s keep being crawled |
Keep server response times stable and avoid 5xx/429 | Raises the crawl capacity limit |
| Update content meaningfully | Changed pages are worth revisiting; cosmetic date changes aren't |
For sitemap specifics, see our XML sitemap guide. For finding pages Google can't reach through links, see orphan pages.
How BugViso Helps With Crawl Efficiency
A BugViso audit finds the crawl-wasting issues on your side: broken internal links, internal links and sitemap entries that redirect, redirect chains with their full hop list, noindex and canonical conflicts, orphan pages from the internal link graph, and slow TTFB. It doesn't read Google's crawl logs (only Search Console and your server can), but it shows what Googlebot runs into when it arrives.
More detail is on the crawl-based website audit feature page.
FAQ
How often does Google crawl a website?
It varies per URL. Popular, frequently changing pages may be crawled daily or more; static, rarely linked pages may wait weeks. Check Crawl Stats in Search Console for your site's actual numbers.
How do I check when Google last crawled a page?
Use URL Inspection in Search Console, or search your server logs for verified Googlebot requests to that URL.
Can I make Google crawl my site more often?
Indirectly. Link important pages from frequently crawled pages, keep your sitemap clean, fix redirects and soft 404s, keep the server fast, and update content meaningfully. For a single page, you can request indexing in URL Inspection.
Does a sitemap make Google crawl more?
A sitemap helps Google discover URLs and notice updates via <lastmod>, but it doesn't raise crawl capacity. A sitemap full of redirects and 404s works against you.
Do I need to worry about crawl budget?
Google says crawl budget mainly matters for very large sites (1M+ pages changing weekly) or medium sites (10k+ pages) changing daily. Smaller sites should focus on crawl efficiency instead.
Crawl frequency is one slice of technical SEO for beginners. For a request-by-request view of what Googlebot does with each visit, see Googlebot crawl data analysis and crawl budget explained.
The Takeaway
Crawl frequency is earned through popularity, freshness and a healthy server, so measure yours in Crawl Stats and your logs, and clear the redirects and dead URLs a BugViso audit finds.
Written by
Kazi Khalid Hasan Tamim · BugViso Team
Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.
Summarise with AI
Open this guide in your assistant.
Follow BugViso
New guides and audit data as we publish them.
AI visibility tracker
Is your brand in AI answers?
See how often AI assistants mention you, cite your site and rank you against your competitors.
- ChatGPT
- Gemini
- Perplexity
- Claude
- AI Overviews
See where your site stands
Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.