Best robots.txt Tester Tools 2027: 6 Checked vs RFC 9309
The best robots.txt tester tools for 2027, and a test that matters: Python's built-in parser got 7 of 12 cases wrong against RFC 9309. Free script inside.
Kazi Khalid Hasan Tamim · BugViso Team
12 min read
The best robots.txt tester for 2027 depends on which crawler you care about: Search Console's robots.txt report and URL Inspection tool for Googlebot, Google's open-source robots.txt library for exact Google behaviour, and an RFC 9309-compliant tester such as BugViso or the script below for AI crawlers like GPTBot and ClaudeBot. A tester is only as good as its matching rules.
That last point is not academic. We ran 12 robots.txt cases through Python's built-in urllib.robotparser, the parser many home-made checkers use. It got 7 of them wrong compared with RFC 9309, the robots.txt standard. Every one of those mistakes would tell you a page is open when it is blocked, or blocked when it is open.
By the end you will know which tester to trust for each crawler, the five rules most testers get wrong, and how to test any URL against any bot with a free script.
Data used: 12 robots.txt test cases written for this post, each checked against the matching rules in RFC 9309, then run through Python 3.12.10's
urllib.robotparseron 2026-11-16. Tool facts come from each vendor's own documentation, fetched the same day. We make BugViso, one of the tools listed.
What a robots.txt Tester Has to Get Right
A robots.txt tester answers one question: may this crawler fetch this URL? To answer it correctly, it must follow four rules from RFC 9309, published by the IETF in 2022.
- Pick the right group. A crawler obeys the group that names its product token (for example
GPTBot), and falls back toUser-agent: *only when no group names it. - Merge duplicate groups. If two groups name the same crawler, their rules combine.
- Longest match wins. When an
Allowand aDisallowboth match a path, the rule with the longer pattern applies. Order in the file doesn't matter. - Support
*and$.*matches any run of characters, and$anchors the end of the path.
Google's robots.txt specification follows the same rules for Googlebot.
Python's Parser Got 7 of 12 Cases Wrong
Python's standard library robots.txt parser reads rules in file order and ignores wildcards, so it disagrees with RFC 9309 in most real-world files. Here are the 12 cases.
| Case | RFC 9309 answer | urllib.robotparser | Right? |
|---|---|---|---|
Allow: /blog/ inside Disallow: / | allow | block | No |
Longer Allow listed after a Disallow | allow | block | No |
Disallow: /*.pdf$ on a PDF | block | allow | No |
Disallow: /*.pdf$ on .pdf?x=1 | allow | allow | Yes |
Wildcard in the middle (/*/private/) | block | allow | No |
| Two groups for the same bot | block (merged) | allow | No |
A named group beats * | allow | allow | Yes |
| Bot name in a different case | block | block | Yes |
Empty Disallow: | allow | allow | Yes |
Allow and Disallow of equal length | allow | block | No |
| Full user-agent string passed in | block | allow | No |
Path in a different case (/Admin/ vs /admin/) | allow | allow | Yes |
The pattern behind the failures is simple. Python's parser takes the first rule that matches in file order, where RFC 9309 takes the longest. It treats * and $ as literal characters. It stops at the first group for a bot rather than merging. And it matches the user agent by the part before the first /, so passing a full browser-style string like Mozilla/5.0 (compatible; ClaudeBot/1.0) finds no group at all.
The Python documentation for urllib.robotparser doesn't claim RFC 9309 support, so this isn't a bug report. It is a warning about any tester built on it.
The 6 Best robots.txt Testers
| Tool | Best for | Crawlers it models | Price |
|---|---|---|---|
| Search Console robots.txt report | Seeing what Google fetched and any parse errors | Googlebot | Free |
| Search Console URL Inspection | Testing one URL against Google's live view | Googlebot | Free |
| Google's open-source robots.txt library | Exact Google matching in your own tests | Googlebot | Free (C++) |
| BugViso AI Search Readiness | AI crawler access across a whole site | GPTBot, ClaudeBot, PerplexityBot and more | First report free |
| The script below | Any bot, any path, from the command line | Any product token | Free |
| Your own logs | Proof of what crawlers really did | Every bot that visited | Free |
Search Console robots.txt report: what Google actually fetched
Search Console's robots.txt report shows which robots.txt files Google found for the top 20 hosts on your site, when they were last crawled, and any warnings or errors. It also lets you request a recrawl after you fix the file. Google's robots.txt report help page notes it works only for domain-level properties.
The report no longer tests individual URLs. For that, Google points to URL Inspection.
Search Console URL Inspection: one URL, Google's view
URL Inspection tells you whether Googlebot may crawl a specific URL on your verified site, using the robots.txt Google has on file. It is the most authoritative answer for Google, but it only covers Googlebot, one URL at a time, on sites you own. See Google's URL Inspection documentation.
Google's open-source robots.txt library: exact Google behaviour
Google publishes the C++ parser it uses in Search as an open-source robots.txt library. Build it into your test suite if you need byte-for-byte Google behaviour, for example before shipping a large robots.txt change. It models Googlebot's rules, so treat its answers for other bots as a guide.
BugViso AI Search Readiness: AI crawlers across a whole site
BugViso checks your robots.txt against the AI search and training crawlers with RFC 9309 matching, as part of a site audit. It reports per crawler which ones can reach your pages, so a stray rule that blocks OAI-SearchBot or Claude-SearchBot shows up as a finding next to the rest of the audit. Where others go further: it tests the robots.txt you have now; it doesn't simulate an edited file before you deploy it.
The script below: any bot, any path
For quick checks of draft rules or of a bot no tool lists yet, the RFC 9309 script in this post tests any user agent against any paths. It passes all 12 cases in the table above.
Your server logs: what really happened
A tester predicts behaviour; logs prove it. If a crawler fetched a path your tester says is blocked, either the crawler ignores robots.txt or your tester is wrong. Check both before you blame the bot.
The robots.txt Mistakes Testers Help You Catch
Most robots.txt mistakes come from assuming rules run top to bottom. These five cause the most damage.
- A named group silently drops the
*rules. When you addUser-agent: GPTBot, that bot stops reading the*group entirely. Repeat any private-path rules inside the named group, as our own robots.txt guide explains. - An
Allowyou expected to win loses on length.Allow: /shopdoesn't beatDisallow: /shop/checkoutfor checkout pages; the longer rule wins. - Wildcards that match more than you meant.
Disallow: /*?blocks every URL with a query string, including tracking parameters on landing pages. - Case mistakes in paths. Paths are case-sensitive:
Disallow: /Admin/doesn't block/admin/. - Blocking a search crawler while meaning to block training. OpenAI, Anthropic and Google use separate tokens for training and for search, covered in how to block AI training but allow AI search.
Run It Yourself: An RFC 9309 robots.txt Tester
This script downloads a site's robots.txt and tests any user agent against any list of paths using RFC 9309 rules: group matching, merging, longest match and wildcards. It uses only Python's standard library.
#!/usr/bin/env python3
"""Test URLs against a robots.txt the way RFC 9309 says crawlers must: longest match wins.
Usage: python3 robots_check.py <site-or-robots.txt-url> <user-agent> <path> [more paths ...]
Example: python3 robots_check.py https://example.com GPTBot /blog/post /admin/
"""
import re
import sys
import urllib.request
def parse(text):
"""Group rules by user-agent; RFC 9309 merges every group that names the same agent."""
groups, agents, in_rules = {}, [], False
for raw in text.splitlines():
line = raw.split("#", 1)[0].strip()
if ":" not in line:
continue
key, value = (s.strip() for s in line.split(":", 1))
key = key.lower()
if key == "user-agent":
if in_rules: # a user-agent line after rules starts a new group
agents, in_rules = [], False
agents.append(value.lower())
groups.setdefault(value.lower(), [])
elif key in ("allow", "disallow") and agents:
in_rules = True
for a in agents:
groups[a].append((key == "allow", value))
return groups
def to_regex(pattern):
# '*' matches any run of characters, a trailing '$' anchors the end of the path.
anchored = pattern.endswith("$")
body = re.escape(pattern[:-1] if anchored else pattern).replace(r"\*", ".*")
return re.compile(body + ("$" if anchored else ""))
def allowed(groups, agent, path):
agent = agent.lower()
# The most specific group that matches the crawler's product token; "*" is the fallback.
names = [n for n in groups if n != "*" and n in agent]
rules = groups[max(names, key=len)] if names else groups.get("*", [])
best = None # (length, allow)
for allow, pattern in rules:
if pattern == "": # an empty Disallow allows everything
continue
if to_regex(pattern).match(path):
cand = (len(pattern), allow)
# Longest pattern wins; on a tie the least restrictive (Allow) wins.
if best is None or cand[0] > best[0] or (cand[0] == best[0] and allow):
best = cand
return True if best is None else best[1]
if __name__ == "__main__":
site, agent, paths = sys.argv[1], sys.argv[2], sys.argv[3:]
url = site if site.endswith("robots.txt") else site.rstrip("/") + "/robots.txt"
req = urllib.request.Request(url, headers={"User-Agent": "robots-check/1.0"})
with urllib.request.urlopen(req, timeout=20) as r:
groups = parse(r.read().decode("utf-8", "replace"))
for p in paths:
print(f"{'ALLOW' if allowed(groups, agent, p) else 'BLOCK'} {agent} {p}")A real run against bugviso.com, unedited:
$ python3 robots_check.py https://bugviso.com GPTBot / /api/v1/scan /blog/
ALLOW GPTBot /
BLOCK GPTBot /api/v1/scan
ALLOW GPTBot /blog/GPTBot isn't named in our file, so it falls back to the * group: the site is open and the API is blocked, which is what we intended. Two limits: the script reads the file you have live, not a draft, and it treats a missing robots.txt (404) as an error rather than "allow all", which is what RFC 9309 says crawlers should do. Catch that case before you rely on the output.
How BugViso Tests Crawler Access Automatically
BugViso's AI Search Readiness engine reads your robots.txt with RFC 9309 matching and reports, crawler by crawler, which AI search and training bots can reach your pages. It runs in the same audit as the technical SEO, speed and accessibility checks, and its findings land in the ranked fix list and the PDF.
The same engine validates llms.txt if you have one, checks how extractable your content is for AI answers, and rolls the results into a 0–100 GEO citability score. A blocked search crawler is usually the cheapest AI visibility fix on a site, which is why it sits at the top of the list when it happens. The AI search readiness feature page describes every check.
Frequently Asked Questions
Does Google still have a robots.txt tester?
Search Console now has a robots.txt report instead. It shows which robots.txt files Google found for your top hosts, when they were last crawled and any errors, and lets you request a recrawl. To test whether a specific URL is blocked for Googlebot, Google points to the URL Inspection tool.
How do I test robots.txt for GPTBot or ClaudeBot?
Use a tester that follows RFC 9309 and lets you choose the user agent, such as the script in this post or BugViso's AI crawler check. Google's tools only model Googlebot. Enter the bot's product token, for example GPTBot, not a full browser user-agent string.
Does the order of rules in robots.txt matter?
Not for crawlers that follow RFC 9309, including Googlebot. The most specific rule, meaning the longest matching pattern, wins regardless of order. Some older parsers, including Python's built-in one, still use file order, which is why testers built on them can give the wrong answer.
Is robots.txt case-sensitive?
Paths are case-sensitive, so /Admin/ and /admin/ are different. User-agent names are matched without regard to case, so gptbot and GPTBot name the same crawler. The field names, such as Disallow, are not case-sensitive either.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling, not indexing. A blocked URL can still appear in results if other pages link to it, usually without a description. To keep a page out of the index, allow crawling and use a noindex tag instead.
The One robots.txt Test Worth Running Today
Test your three most important pages against every crawler you care about, Googlebot and the AI search bots, using a tester that applies longest-match rules, not file order. For the whole site at once, run a BugViso audit and read the AI crawler access section first.
Written by
Kazi Khalid Hasan Tamim · BugViso Team
Kazi writes BugViso’s guides on website audits, Core Web Vitals, technical SEO, accessibility and AI search. The data in them comes from audits BugViso runs in a real Chromium browser, including our 219-site study of website health.
Summarise with AI
Open this guide in your assistant.
Follow BugViso
New guides and audit data as we publish them.
AI visibility tracker
Is your brand in AI answers?
See how often AI assistants mention you, cite your site and rank you against your competitors.
- ChatGPT
- Gemini
- Perplexity
- Claude
- AI Overviews
See where your site stands
Run a free BugViso audit for SEO, speed, accessibility and AI search readiness — with fixes you can ship today.