AI Search Readiness Data Study: 219 Websites Tested (2026)

Key finding: Of 219 randomly sampled websites, only 22.4% passed all eight core AI search readiness checks (reachable, AI crawlers allowed, content and key tags in the raw HTML, structured data). Adding a valid llms.txt file drops that to 7.3%. The most common failures weren't AI-specific at all: missing JSON-LD (67.6% of sites), no canonical tag (40.6%) and no H1 in the raw HTML (35.6%).

AI answer engines can only cite what their crawlers can fetch and read. Most of those crawlers don't run JavaScript, so the raw HTML response is what counts. We measured how many ordinary websites pass the basic checks that decide whether ChatGPT, Claude, Perplexity and Google's AI features can use a page at all.

Method

Third-party sites in the sample aren't named. A site that didn't return 200 to the raw request fails every HTML check, because a crawler would get nothing to read.

Results: 9 AI Search Readiness Checks

What the Data Says

1. Bot protection blocks more sites than robots.txt does

15.1% of sites didn't return 200 to a plain HTTP request; most of those (24 sites) returned 403. Only 6.4% blocked an AI search crawler in robots.txt. Firewall and bot-management rules that challenge non-browser clients are the bigger, quieter problem, because they don't show up when you read your own robots.txt.

2. Most sites never mention AI crawlers

189 of 214 readable robots.txt files had no rule for GPTBot at all. Silence means "allowed" under the Robots Exclusion Protocol, which is fine if that's your intent. Where sites did write rules, they mostly blocked: GPTBot was blocked by 12 sites and explicitly allowed by 7.

Eight sites blocked GPTBot but not OAI-SearchBot, the pattern OpenAI documents for "don't train on my content, but do show me in ChatGPT search". Our guide to blocking AI training while allowing AI search shows the rules.

3. JavaScript-only content still hides about one site in four

26.9% of sites failed the content check, either unreachable or with less than half their text in the raw HTML. Among reachable sites, 5.8% were near-empty shells. We break this down by framework in do AI crawlers render JavaScript?

4. Structured data is the weakest basic signal

Only 32.4% of sites had any JSON-LD in their raw HTML. Structured data isn't required to be cited, but it's the most direct way to state facts like organisation name, product, price and author in a form machines don't have to infer.

5. llms.txt is still rare

12.8% served a valid llms.txt. It's a proposal, not a standard, and no major AI provider has said it uses the file for ranking or citation. Treat it as cheap and optional; fix checks 1–8 first. Our llms.txt guide covers the format.

Readiness Priorities, Based on the Data

Run the Same 9 Checks on Your Site

This standard-library Python script runs the study's checks against any URL:

"""Nine-point AI search readiness check for one site (standard library only).

Usage: python3 ai_ready_check.py https://example.com/
Same checks as the BugViso 219-site study: reachable, AI search crawlers
allowed in robots.txt, content in raw HTML, title, meta description, H1,
canonical, JSON-LD in raw HTML, and a valid /llms.txt.
"""
import re, sys, urllib.request
from urllib.parse import urlsplit

UA = {"User-Agent": "Mozilla/5.0 (ai-ready-check)"}
AI_SEARCH = ["oai-searchbot", "chatgpt-user", "gptbot", "claudebot", "perplexitybot"]


def get(url):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=20) as r:
            return r.status, r.read(3_000_000).decode("utf-8", "replace")
    except urllib.error.HTTPError as e:
        return e.code, ""
    except Exception:  # noqa: BLE001
        return None, ""


def blocked_bots(robots):
    """Bots whose own group disallows '/' (RFC 9309 groups; simplified)."""
    groups, cur, agent_line = [], None, False
    for line in robots.splitlines():
        line = line.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, val = (s.strip() for s in line.split(":", 1))
        if key.lower() == "user-agent":
            if not agent_line:
                cur = {"agents": [], "disallow_root": False, "allow_root": False}
                groups.append(cur)
            cur["agents"].append(val.lower())
            agent_line = True
        else:
            agent_line = False
            if cur and key.lower() == "disallow" and val == "/":
                cur["disallow_root"] = True
            if cur and key.lower() == "allow" and val in ("/", "/*"):
                cur["allow_root"] = True
    return [b for b in AI_SEARCH for g in groups
            if b in g["agents"] and g["disallow_root"] and not g["allow_root"]]


def main(url):
    s = urlsplit(url)
    base = f"{s.scheme}://{s.netloc}"
    status, html = get(url)
    _, robots = get(base + "/robots.txt")
    l_status, llms = get(base + "/llms.txt")
    body = re.sub(r"(?is)<(script|style|noscript|template|svg)\b.*?</\1>", " ", html)
    words = len(re.findall(r"\w+", re.sub(r"<[^>]+>", " ", body)))
    blocked = blocked_bots(robots)
    first_llms_line = next((l.strip() for l in llms.splitlines() if l.strip()), "")
    checks = [
        ("Page returns 200", status == 200, status),
        ("AI search crawlers allowed", not blocked, ", ".join(blocked) or "none blocked"),
        ("Content in raw HTML (>=150 words)", words >= 150, f"{words} words"),
        ("<title> in raw HTML", bool(re.search(r"<title[^>]*>\s*\S", html, re.I)), ""),
        ("Meta description", bool(re.search(r'<meta[^>]+name=["\']description["\'][^>]*content=["\']\s*\S', html, re.I)), ""),
        ("H1 in raw HTML", bool(re.search(r"<h1[\s>]", html, re.I)), ""),
        ("Canonical tag", bool(re.search(r'<link[^>]+rel=["\']canonical', html, re.I)), ""),
        ("JSON-LD in raw HTML", "application/ld+json" in html, ""),
        ("Valid /llms.txt", l_status == 200 and first_llms_line.startswith("# "), l_status),
    ]
    for name, ok, detail in checks:
        print(f"{'PASS' if ok else 'FAIL'}  {name:<36} {detail}")
    print(f"\n{sum(ok for _, ok, _ in checks)}/9 AI readiness checks passed")


if __name__ == "__main__":
    main(sys.argv[1])

Output for a site that passes everything, and for a bare page:

PASS  Page returns 200                     200
PASS  AI search crawlers allowed           none blocked
PASS  Content in raw HTML (>=150 words)    2185 words
PASS  <title> in raw HTML
PASS  Meta description
PASS  H1 in raw HTML
PASS  Canonical tag
PASS  JSON-LD in raw HTML
PASS  Valid /llms.txt                      200

9/9 AI readiness checks passed

FAIL  JSON-LD in raw HTML
FAIL  Valid /llms.txt                      404

3/9 AI readiness checks passed

The script uses a word-count threshold rather than a rendered comparison, so it doesn't need a browser. For the raw-vs-rendered comparison, use the Playwright script in our JavaScript crawler study.

How BugViso Scores AI Search Readiness

The BugViso AI readiness audit covers every check in this study and goes further: it parses robots.txt for 17 AI and search crawlers with RFC 9309 longest-match rules, sends live requests with AI crawler user-agents to catch CDN blocks, compares raw HTML with the rendered DOM, validates llms.txt and the links inside it, and scores content extractability and E-E-A-T signals such as author and dates.

The results roll into a 0–100 AI readiness score with the failing checks listed first.

More detail is on the AI search readiness checker feature page.

Limitations

• One homepage per site; inner pages can differ.
• One request from one network; some sites treat other locations or IP ranges differently.
• User-agent tests approximate real crawlers, which some CDNs also verify by IP.
• Passing these checks makes a page citable. It doesn't guarantee citations, which depend on relevance and authority.

FAQ

What percentage of websites are ready for AI search?

In our sample, 22.4% passed all eight core checks, and 7.3% also had a valid llms.txt.

What's the most common AI search readiness problem?

Missing JSON-LD structured data (67.6% of sites), followed by missing canonicals (40.6%) and H1s that aren't in the raw HTML (35.6%).

Do most websites block AI crawlers?

No. Only 6.4% blocked an AI search crawler in robots.txt, but 15.1% didn't serve their page to a plain HTTP request at all, which typically blocks AI crawlers too.

Is llms.txt required for AI search?

No. It's an optional proposal with low adoption (12.8% in our sample). Crawler access and content in the HTML matter far more.

The Takeaway

Most sites that fail AI search readiness fail on plain HTML basics, not AI-specific tricks, and a BugViso AI readiness scan checks all of them on your own pages.