Do AI Crawlers Render JavaScript? Data From 219 Websites

Quick answer: Mostly no. OpenAI's GPTBot, Anthropic's ClaudeBot and PerplexityBot fetch your HTML (and sometimes your JavaScript files as text) but don't execute JavaScript. Googlebot renders JavaScript, and Google's AI features use Google's own crawling infrastructure; Applebot also renders. If your content only appears after JavaScript runs, most AI answer engines see an empty page.

That makes the raw HTML response, the bytes your server sends before any script runs, the version of your page that ChatGPT, Claude and Perplexity actually read.

We measured how much of each page exists in that raw response for 219 randomly sampled websites. Most were fine. A meaningful minority were invisible, and the risk concentrated in client-rendered React and Vue sites.

Which AI Crawlers Render JavaScript?

Vercel's December 2024 analysis of AI crawler traffic found that "none of the major AI crawlers currently render JavaScript". ChatGPT's and Claude's crawlers did request JavaScript files (11.50% and 23.84% of their requests) but didn't execute them. Crawler behaviour can change, so treat this as the current baseline, not a permanent rule. OpenAI documents its crawlers on its bots page, and Perplexity on its bots guide.

💡 Rule of thumb: If curl https://yoursite.com/page doesn't contain your main text, assume ChatGPT, Claude and Perplexity can't quote it.

What 219 Websites Show Non-Rendering Crawlers

Method. We took the same random sample we use across our data studies: 219 valid homepages from the Tranco list (ranks 1,001–50,000, seed 20261003). For each one we fetched the raw HTML with a plain HTTP request, as a non-rendering crawler would, and rendered the page in headless Chromium. We compared visible word counts, H1s and links, and read each site's robots.txt. Measured on 8 October 2026. 173 sites returned 200 to the raw request and had at least 50 rendered words.

How much of the page is in the raw HTML

The raw HTML often holds more words than the rendered view (median ratio 1.12), because it includes menus and hidden text that innerText skips. That's fine. The problem is the bottom rows.

Other signals missing from raw HTML

A crawler that doesn't render can't follow links it never sees, so a JavaScript-only navigation also hides every page behind it.

By front-end framework

Frameworks that server-render by default (Next.js, Nuxt, Astro, WordPress) almost always shipped their content in HTML. Plain React and Vue apps, which render on the client unless you add SSR or prerendering, made up most of the empty shells. Small samples for some frameworks, so read the direction rather than the decimals. Our Vue.js SEO guide covers the fix for Vue.

Blocked before JavaScript even matters

Two other things stop AI crawlers regardless of rendering:

• 24 of 219 sites (11.0%) returned 403 to a plain HTTP request from our browser user-agent. Bot protection that blocks simple clients usually blocks AI crawlers too.
• 14 of 214 readable robots.txt files (6.5%) disallowed at least one AI search crawler (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot or PerplexityBot). ClaudeBot (6.1%) and GPTBot (5.6%) were the most blocked. Most sites didn't mention AI crawlers at all: 189 of 214 had no GPTBot rule.
• 28 of 219 (12.8%) served a valid llms.txt file.

How to Check What AI Crawlers See on Your Site

The 10-second check

# Does the raw HTML contain a sentence from your page?
curl -s https://yoursite.com/pricing | grep -c "per month"

# What does the server return to an AI crawler user-agent?
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://yoursite.com/pricing

A 0 from the first command, or a 403 from the second, means AI crawlers can't use the page. User-agent tests only approximate the real crawlers, since some CDNs verify bots by IP address as well.

The full comparison

This script compares raw HTML with the rendered DOM and tests three AI crawler user-agents:

"""Compare what a non-rendering AI crawler sees with what a browser sees.

Usage:
  pip install playwright httpx beautifulsoup4 && python -m playwright install chromium
  python raw_vs_rendered.py https://example.com/
Prints word counts, H1s and links in the raw HTML vs the rendered DOM, plus
the HTTP status the page returns to GPTBot, ClaudeBot and PerplexityBot user-agents.
"""
import re, sys
import httpx
from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

BROWSER_UA = ("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) "
              "Chrome/121.0.0.0 Safari/537.36")
BOT_UAS = {
    "GPTBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot",
    "ClaudeBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com",
    "PerplexityBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot",
}
WORD = re.compile(r"\w+(?:['’-]\w+)*")


def raw_view(url):
    r = httpx.get(url, headers={"User-Agent": BROWSER_UA}, follow_redirects=True, timeout=20)
    soup = BeautifulSoup(r.text, "html.parser")
    h1 = [h.get_text(" ", strip=True)[:60] for h in soup.find_all("h1")]
    links = len(soup.find_all("a", href=True))
    for t in soup(["script", "style", "noscript", "template", "svg"]):
        t.decompose()
    return {"words": len(WORD.findall((soup.body or soup).get_text(" "))), "h1": h1, "links": links}


def rendered_view(url):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        page = browser.new_page(user_agent=BROWSER_UA)
        page.goto(url, wait_until="networkidle", timeout=45000)
        data = page.evaluate("""() => ({
            text: document.body.innerText,
            h1: [...document.querySelectorAll('h1')].map(h => h.innerText.trim().slice(0, 60)),
            links: document.querySelectorAll('a[href]').length })""")
        browser.close()
    return {"words": len(WORD.findall(data["text"])), "h1": data["h1"], "links": data["links"]}


def main(url):
    raw, rendered = raw_view(url), rendered_view(url)
    ratio = raw["words"] / max(rendered["words"], 1)
    print(f"{'':14}{'raw HTML':>12}{'rendered':>12}")
    print(f"{'words':14}{raw['words']:>12}{rendered['words']:>12}")
    print(f"{'links':14}{raw['links']:>12}{rendered['links']:>12}")
    print(f"{'H1 count':14}{len(raw['h1']):>12}{len(rendered['h1']):>12}")
    verdict = "OK" if ratio >= 0.5 else "WARNING: most content needs JavaScript" if ratio >= 0.1 else "FAIL: empty shell without JavaScript"
    print(f"\nraw/rendered word ratio: {ratio:.0%}  ->  {verdict}")
    for bot, ua in BOT_UAS.items():
        try:
            code = httpx.get(url, headers={"User-Agent": ua}, follow_redirects=True, timeout=20).status_code
        except httpx.HTTPError as e:
            code = type(e).__name__
        print(f"{bot:14} HTTP {code}")


if __name__ == "__main__":
    main(sys.argv[1])

Output for a client-rendered app from our sample:

                  raw HTML    rendered
words                    0         639
links                    0          75
H1 count                 0           1

raw/rendered word ratio: 0%  ->  FAIL: empty shell without JavaScript
GPTBot         HTTP 403
ClaudeBot      HTTP 403
PerplexityBot  HTTP 403

That site is invisible to all three engines twice over: blocked at the edge, and empty even if it weren't.

How to Fix a JavaScript-Only Page for AI Crawlers

Google calls serving different HTML to bots "dynamic rendering" and describes it as a workaround, not a recommended long-term solution. Prefer SSR or SSG so every client gets the same HTML.

Don't forget links. Use real <a href> elements in the HTML for navigation; a <div onclick> menu hides your site structure from every crawler, including Google's link discovery.

How BugViso Checks AI Crawler Visibility

The BugViso AI search readiness audit runs the checks above on every scan:

• Raw vs rendered parity: it fetches your raw HTML and compares it with the rendered DOM, flagging pages where under 25% of the content exists without JavaScript and raising an error below 10%.
• Live crawler probes: it requests your page with real AI crawler user-agents to catch CDN or WAF 403 blocks that robots.txt doesn't show.
• robots.txt matrix: it parses robots.txt with RFC 9309 longest-match rules for 17 crawlers, including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended.
• llms.txt: it validates the file and tests the links inside it.

These roll into a 0–100 AI readiness score, and the remediation section tells you which of the fixes above applies.

You can see every rule BugViso applies in its GEO audit for AI search.

Common Mistakes

• Testing in a browser. Your browser runs JavaScript. Test with curl or "View Page Source", not DevTools Elements.
• Assuming Google's rendering covers everyone. Googlebot renders; most AI crawlers don't.
• Blocking AI crawlers by accident. CDN "block AI bots" toggles can block search crawlers like OAI-SearchBot, not just training crawlers. See how to block AI training but allow AI search.
• Injecting JSON-LD with a tag manager. Non-rendering crawlers never see it.

FAQ

Do AI crawlers execute JavaScript?

GPTBot, ClaudeBot and PerplexityBot don't, based on Vercel's published crawler analysis. Googlebot and Applebot do render JavaScript.

Can ChatGPT read client-side rendered content?

Not through its crawlers. If the text only exists after JavaScript runs, ChatGPT's search crawler sees an empty shell.

Does Google AI Overviews see JavaScript content?

Google's AI features rely on Google's own crawling and indexing, and Googlebot renders JavaScript. Rendering can be delayed, so server-rendered content is still safer.

How do I know if my site is client-side rendered?

View the page source (Ctrl+U). If the body is mostly <div id="root"></div> or <div id="app"></div> plus script tags, it's client-rendered.

Is prerendering enough for AI search?

Yes, if every route you want cited is prerendered and the HTML includes the full text, links, title and structured data.

The Takeaway

For AI search, your raw HTML is your page: if the content isn't there without JavaScript, most answer engines can't cite it, and a BugViso AI readiness scan shows the raw-vs-rendered gap for your own site.