How Often Does Google Crawl Your Site? How to Measure It

How often does Google crawl a site? There is no fixed schedule. See how size, change rate and links drive it, how to measure yours and how to raise it.

· BugViso Team

11 min read

Quick answer: How often Google crawls a site has no fixed schedule. Google crawls each URL based on demand (how popular it is and how often it changes) and capacity (how fast and reliably your server responds). Popular, frequently updated pages can be crawled several times a day; static, rarely linked pages may wait weeks.

Generic averages ("small sites get crawled every X days") don't tell you much, because crawl frequency varies by URL even within one site. What you can do is measure it for your own pages and remove the things that slow it down.

This guide covers how Google decides, how to measure, and what to fix.


How Google Decides How Often to Crawl

Google's crawl budget documentation describes two forces:

FactorWhat Google saysWhat it means for you
Crawl capacity limit"If the site responds consistently and its response times … remain stable or improve, the limit goes up … If the site slows down … or responds with server errors (5xx) or rate-limiting signals (such as HTTP 429), the limit goes down."A fast, stable server can be crawled more
Crawl demand (popularity)"URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems."Well-linked pages get revisited more
Crawl demand (staleness)Google tries to keep its copy of pages fresh, so pages that change are worth revisitingPages that change often get recrawled more

The same documentation says the crawl budget guide is for "large sites (1 million+ unique pages) with content that changes moderately often (once a week)" and "medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily)". For most sites, crawl budget isn't a constraint; crawl efficiency (not wasting requests) still is.


What Wastes Google's Crawls: Data

We can't see Google's logs for other people's sites, but we can measure the conditions Google says reduce crawling. From our studies of randomly sampled sites (Tranco ranks 1,001–50,000), October 2026:

Signal Google says affects crawlingFindingSample
Slow server responses (TTFB over 800 ms on throttled mobile)49.5% of homepages200 sites
Sitemaps listing URLs that redirect25% of sitemaps36 sitemaps
Sitemaps listing URLs that return 404/41014% of sitemaps36 sitemaps
Internal links pointing at redirects56.6% of homepages219 sites
Homepage <title> changed in 5 days (a freshness signal)5.5% of homepages183 sites

Every redirecting sitemap entry or internal link costs Googlebot an extra request to reach the real page. Google is explicit about two related cases: "soft 404 pages will continue to be crawled, and waste your budget," and "avoid long redirect chains, which have a negative effect on crawling."


How Often Does Google Crawl a Site? How to Measure Yours

1. Search Console Crawl Stats

Open Settings → Crawl stats in Search Console. It shows total crawl requests per day for the last 90 days, average response time, and breakdowns by response code, file type, purpose (discovery vs refresh) and Googlebot type. Watch for:

  • A drop in requests at the same time as a rise in average response time (capacity limit falling).
  • A high share of 301, 404 or 5xx responses (wasted crawls).
  • "Discovery" crawls staying low after you publish new pages (weak internal linking or sitemap).

2. URL Inspection

For a single page, URL Inspection → Page indexing shows the last crawl date. It's the quickest answer to "when did Google last see this page?"

3. Server logs (the only per-URL, per-day answer)

Your access logs record every Googlebot request. This script counts verified Googlebot hits and shows when each section was last crawled. It verifies each IP with reverse and forward DNS, as Google's Googlebot verification guide describes, because many bots fake the Googlebot user-agent.

python
"""How often does Google crawl my site? Answer it from your own access logs.

Usage: python3 googlebot_log.py access.log [access.log.1 ...]
Reads combined-format logs (nginx/Apache default), keeps requests whose
user-agent claims Googlebot, verifies each IP with reverse + forward DNS as
Google documents (googlebot.com / google.com / googleusercontent.com), then
reports verified hits per day, status codes, most-crawled paths and how many
days since each section was last crawled.
"""
import re, socket, sys
from collections import Counter, defaultdict
from datetime import datetime

LINE = re.compile(r'^(\S+) \S+ \S+ \[([^\]]+)\] "(?:GET|HEAD|POST) (\S+)[^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')
GOOGLE_DOMAINS = (".googlebot.com", ".google.com", ".googleusercontent.com")
_verified = {}

def is_googlebot(ip):
    if ip not in _verified:
        try:
            host = socket.gethostbyaddr(ip)[0]
            ok = host.endswith(GOOGLE_DOMAINS) and ip in socket.gethostbyname_ex(host)[2]
        except OSError:
            ok = False
        _verified[ip] = ok
    return _verified[ip]

def main(paths):
    per_day, statuses, urls, fake = Counter(), Counter(), Counter(), Counter()
    last_seen = defaultdict(lambda: datetime.min)
    for path in paths:
        for line in open(path, errors="replace"):
            m = LINE.match(line)
            if not m or "Googlebot" not in m.group(5):
                continue
            ip, ts, url, status = m.group(1), m.group(2), m.group(3), m.group(4)
            if not is_googlebot(ip):
                fake[ip] += 1
                continue
            when = datetime.strptime(ts.split()[0], "%d/%b/%Y:%H:%M:%S")
            per_day[when.date()] += 1
            statuses[status] += 1
            urls[url.split("?")[0]] += 1
            section = "/" + url.lstrip("/").split("/")[0].split("?")[0]
            last_seen[section] = max(last_seen[section], when)
    total = sum(per_day.values())
    if not total:
        print("No verified Googlebot requests found.")
        return
    days = len(per_day)
    print(f"Verified Googlebot requests: {total} over {days} day(s), {total / days:.1f}/day")
    print(f"Unverified 'Googlebot' requests (fake UAs): {sum(fake.values())} from {len(fake)} IP(s)\n")
    print("Status codes:", ", ".join(f"{k}: {v} ({100 * v / total:.0f}%)" for k, v in statuses.most_common()))
    print("\nMost-crawled URLs:")
    for u, n in urls.most_common(10):
        print(f"  {n:>6}  {u}")
    newest = max(last_seen.values())
    print("\nDays since last crawl, by section:")
    for section, when in sorted(last_seen.items(), key=lambda kv: kv[1]):
        print(f"  {(newest - when).days:>4}  {section}")

if __name__ == "__main__":
    main(sys.argv[1:])

Example output on a small test log:

text
Verified Googlebot requests: 4 over 3 day(s), 1.3/day
Unverified 'Googlebot' requests (fake UAs): 1 from 1 IP(s)

Status codes: 200: 2 (50%), 301: 1 (25%), 404: 1 (25%)

Most-crawled URLs:
       1  /blog/post-a
       1  /blog/post-b
       1  /pricing
       1  /old-page

Run it on a month of logs. The 301 and 404 shares show how much of Google's attention goes to URLs that aren't your real pages.


How to Get Important Pages Crawled More Often

Link to important pages from pages Google already visits often, keep the sitemap clean, and cut wasted requests such as redirect chains and soft 404s. The table lists each action and why it works.

ActionWhy it works
Link to new and updated pages from pages Google already visits often (home, hubs, category pages)Raises crawl demand through internal popularity
Keep the sitemap to final, indexable URLs with honest <lastmod>Cuts wasted requests; lastmod that changes on every build teaches Google to ignore it
Fix redirect chains and point internal links at final URLsOne request per page instead of two or three
Return real 404/410 for removed pagesGoogle stops recrawling them; soft 404s keep being crawled
Keep server response times stable and avoid 5xx/429Raises the crawl capacity limit
Update content meaningfullyChanged pages are worth revisiting; cosmetic date changes aren't

For sitemap specifics, see our XML sitemap guide. For finding pages Google can't reach through links, see orphan pages.


How BugViso Helps With Crawl Efficiency

A BugViso audit finds the crawl-wasting issues on your side: broken internal links, internal links and sitemap entries that redirect, redirect chains with their full hop list, noindex and canonical conflicts, orphan pages from the internal link graph, and slow TTFB. It doesn't read Google's crawl logs (only Search Console and your server can), but it shows what Googlebot runs into when it arrives.

More detail is on the crawl-based website audit feature page.


FAQ

How often does Google crawl a website?

It varies per URL. Popular, frequently changing pages may be crawled daily or more; static, rarely linked pages may wait weeks. Check Crawl Stats in Search Console for your site's actual numbers.

How do I check when Google last crawled a page?

Use URL Inspection in Search Console, or search your server logs for verified Googlebot requests to that URL.

Can I make Google crawl my site more often?

Indirectly. Link important pages from frequently crawled pages, keep your sitemap clean, fix redirects and soft 404s, keep the server fast, and update content meaningfully. For a single page, you can request indexing in URL Inspection.

Does a sitemap make Google crawl more?

A sitemap helps Google discover URLs and notice updates via <lastmod>, but it doesn't raise crawl capacity. A sitemap full of redirects and 404s works against you.

Do I need to worry about crawl budget?

Google says crawl budget mainly matters for very large sites (1M+ pages changing weekly) or medium sites (10k+ pages) changing daily. Smaller sites should focus on crawl efficiency instead.

Crawl frequency is one slice of technical SEO for beginners. For a request-by-request view of what Googlebot does with each visit, see Googlebot crawl data analysis and crawl budget explained.


The Takeaway

Crawl frequency is earned through popularity, freshness and a healthy server, so measure yours in Crawl Stats and your logs, and clear the redirects and dead URLs a BugViso audit finds.

Written by

· BugViso Team

Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.

Found this useful? Share it.

Summarise with AI

Open this guide in your assistant.

Follow BugViso

New guides and audit data as we publish them.

AI visibility tracker

Is your brand in AI answers?

See how often AI assistants mention you, cite your site and rank you against your competitors.

  • ChatGPT
  • Gemini
  • Perplexity
  • Claude
  • AI Overviews
See how it works

See where your site stands

Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.