Site Audit API: Build an Automated Website Audit Endpoint

A site audit API takes a URL and returns structured findings (status, redirects, indexability, metadata, performance, accessibility) as JSON, so audits can run from code instead of a dashboard. You can assemble one from Google's free APIs, self-host Lighthouse and axe-core, or write a small endpoint for the SEO checks. The hard parts are bot protection, JavaScript rendering and time per URL.

We measured all three. In a sample of 219 random homepages, 28 (12.8 %) refused a plain HTTP client but loaded in a real browser. On a 60-site benchmark, a browser render took a median 3.2 s against 1.8 s for a raw fetch, with a slow tail to 9.1 s at the 90th percentile.

By the end you'll know which free APIs cover which checks (and their quotas), what a useful audit response contains, and how to run the working endpoint below without opening an SSRF hole in your network.

Data used: Two measurements on the same population: 219 valid homepages from the Tranco list 94GG2 (ranks 1,001–50,000, seed 20261003). (1) Raw HTTP vs rendered fetch of all 219 on 8 October 2026, with a Chrome user agent. (2) A latency benchmark on 60 of them (drawn with the same seed) on 25 October 2026: httpx GET to the full body, and Playwright Chromium 1366×900 to the load event, then up to 10 s for network idle, 3 pages at a time. Sites are not named.

What a Site Audit API Is (and the Three Ways to Get One)

A site audit API is any service you can call with a URL and get audit findings back in a machine-readable format. Teams use one to audit pages after each deploy, to audit thousands of client URLs on a schedule, or to feed findings into their own dashboards and tickets.

There are three routes, and most production setups mix them:

The rest of this guide covers the first two, because they're what you can build on today without a contract.

Google's Free Audit APIs and Their Quotas

Google's free APIs cover performance and index status well, but none of them crawls a site or checks on-page SEO for you. Use them for the data only Google has, and build the rest.

Two details matter when you design around them. Google says it plans to stop including CrUX field data in the PageSpeed Insights API and recommends the CrUX API instead, so don't build field-data features on the PSI response. And URL Inspection only works for properties you've verified, so it fits auditing your own or your clients' sites, not arbitrary URLs.

# Field data for one URL from the CrUX API (needs an API key)
curl -s "https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=$CRUX_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com/", "formFactor": "PHONE"}'

What the Audit Response Should Contain

A good site audit API response separates facts from findings. Facts are what the page is (status, final URL, title, canonical). Findings are what's wrong with it, each with a stable code and a severity. Clients can then filter, count and diff findings without parsing messages.

Sort issues by severity. Make critical mean "can remove the page from search" (a non-200 status, noindex, robots.txt blocking Googlebot), so a caller can alert on that one value.

Why Raw HTTP Audits Fail on One in Eight Sites

The first surprise of running audits from a server is how often the server gets blocked. A plain HTTP client sends the request a crawler would, and bot protection treats it like one.

Of the 28 refusals, 21 were 403 responses, typically a bot challenge in front of the site. An audit API that only does raw HTTP reports these sites as broken when they're fine for visitors. A rendering one gets further, but some challenges block headless browsers too. Detect the challenge page and say so, rather than scoring it.

The rendering gap is smaller than people expect: 85.5 % of the 173 readable homepages had at least 80 % of their text in the raw HTML. But the 5.8 % that didn't are the sites where a raw audit is meaningless, and they're mostly client-rendered apps. Our study of whether AI crawlers render JavaScript covers the same split from the crawler side.

How Long Each Audit Takes: Raw Fetch vs Browser

A rendered audit takes about 2.1× as long as a raw fetch at the median, and the tail is what breaks synchronous APIs. Measured on 60 homepages:

6 of the 49 rendered pages never went network-idle within 10 seconds of load. Analytics beacons, chat widgets and long-polling keep connections open. Waiting for networkidle without a cap hangs your workers. Use load plus a fixed settle time (we use 3 s in our accessibility studies) and a hard timeout.

The two methods also fail on different sites. The browser timed out at 45 s or got a 429 on 5 sites whose raw fetch returned 200, while 9 sites refused the raw fetch but rendered fine. An audit API that falls back from one method to the other reaches more sites than either alone. (All 60 raw requests started at once, so the raw timings include some network contention; the real gap is, if anything, wider.)

These numbers set the architecture. A raw check fits in a normal HTTP request. A rendered audit of one page sometimes doesn't, and a crawl of 100 pages never does. So rendered audits belong behind a queue: POST /audits returns a job ID, workers render pages a few at a time, and GET /audits/{id} returns progress, then the result.

A Working Site Audit API in One File

This endpoint runs the raw-HTML checks synchronously and returns the response shape above. It needs only the Python standard library. Treat it as the fast tier of an audit API, the one you'd call after every deploy, and put rendered checks behind a queue.

#!/usr/bin/env python3
"""site_audit_api.py: a minimal site audit API. GET /audit?url=... returns JSON findings.

Usage:  python3 site_audit_api.py [port]            # default 8787, binds 127.0.0.1
        curl -s 'http://127.0.0.1:8787/audit?url=https://example.com/' | python3 -m json.tool

Standard library only. Raw HTML (no JavaScript rendering). Per request it follows redirects
hop by hop, refusing any hop that resolves to a private, loopback, link-local or reserved
address (an audit API fetches URLs strangers send it, which makes it an SSRF target), then
checks status, redirects, timing, title, meta description, H1, canonical, noindex, lang,
viewport, hreflang, JSON-LD, word count, HSTS and robots.txt access for Googlebot and
OAI-SearchBot. Not production-hardened: pin the resolved IP per connection to stop DNS
rebinding, add auth and rate limits, and queue rendered audits instead of running them inline.
"""
import ipaddress, json, re, socket, ssl, sys, time, urllib.error, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urljoin, urlsplit

UA = "Mozilla/5.0 (compatible; site-audit-api/1.0)"
CTX = ssl.create_default_context()
MAX_BYTES, TIMEOUT, MAX_HOPS = 3_000_000, 15, 5


class Refused(Exception):
    pass


class NoRedirect(urllib.request.HTTPRedirectHandler):
    def redirect_request(self, *a, **k):
        return None


OPENER = urllib.request.build_opener(NoRedirect, urllib.request.HTTPSHandler(context=CTX))


def guard(url):
    """Allow only http(s) on 80/443 to hosts whose every address is public."""
    s = urlsplit(url)
    if s.scheme not in ("http", "https") or not s.hostname:
        raise Refused("only http and https URLs")
    if s.port not in (None, 80, 443):
        raise Refused("only ports 80 and 443")
    try:
        infos = socket.getaddrinfo(s.hostname, s.port or (443 if s.scheme == "https" else 80))
    except socket.gaierror:
        raise Refused("host does not resolve")
    for info in infos:
        ip = ipaddress.ip_address(info[4][0])
        if not ip.is_global or ip.is_multicast:
            raise Refused(f"{s.hostname} resolves to non-public address {ip}")


def get(url):
    """Follow redirects manually, re-checking each hop. Returns (chain, status, headers, html, seconds)."""
    chain, t0 = [], time.perf_counter()
    for _ in range(MAX_HOPS + 1):
        guard(url)
        req = urllib.request.Request(url, headers={"User-Agent": UA, "Accept": "text/html,*/*"})
        try:
            with OPENER.open(req, timeout=TIMEOUT) as r:
                body = r.read(MAX_BYTES).decode("utf-8", "replace")
                return chain + [(r.status, url)], r.status, r.headers, body, time.perf_counter() - t0
        except urllib.error.HTTPError as e:
            loc = e.headers.get("Location")
            chain.append((e.code, url))
            if e.code in (301, 302, 303, 307, 308) and loc:
                url = urljoin(url, loc)
                continue
            return chain, e.code, e.headers, "", time.perf_counter() - t0
    raise Refused(f"more than {MAX_HOPS} redirects")


def robots_allows(base, bot):
    try:
        _, status, _, txt, _ = get(base + "/robots.txt")
    except Refused:
        return None
    if status != 200:
        return True
    groups, cur, last = {}, [], False
    for line in txt.splitlines():
        k, _, v = line.split("#")[0].partition(":")
        k, v = k.strip().lower(), v.strip()
        if k == "user-agent":
            cur = cur if last else []
            cur.append(v.lower()); groups.setdefault(v.lower(), []); last = True
        elif k in ("allow", "disallow"):
            for a in cur:
                groups[a].append((k, v))
            last = False
    rules = groups.get(bot.lower(), groups.get("*", []))
    return not (("disallow", "/") in rules and ("allow", "/") not in rules)


def find(pattern, html, flags=re.I | re.S):
    m = re.search(pattern, html, flags)
    return re.sub(r"\s+", " ", m.group(1)).strip() if m else None


def audit(url):
    chain, status, headers, html, secs = get(url)
    final = chain[-1][1]
    issues = []
    add = lambda sev, code, msg: issues.append({"severity": sev, "code": code, "message": msg})
    if status != 200:
        add("critical", "status", f"final status {status}")
    if len(chain) > 2:
        add("warning", "redirect_chain", f"{len(chain) - 1} redirects before the final URL")
    title = find(r"<title[^>]*>(.*?)</title>", html)
    desc = find(r'<meta[^>]+name=["\']description["\'][^>]*content=["\']([^"\']*)', html)
    canon = find(r'<link[^>]+rel=["\']canonical["\'][^>]*href=["\']([^"\']+)', html)
    robots = (find(r'<meta[^>]+name=["\']robots["\'][^>]*content=["\']([^"\']+)', html) or "") + \
             " " + (headers.get("X-Robots-Tag", "") if headers else "")
    h1 = len(re.findall(r"<h1\b", html, re.I))
    body = re.sub(r"(?is)<(script|style|noscript|svg|template)\b.*?</\1>", " ", html)
    words = len(re.sub(r"<[^>]+>", " ", body).split())
    ld = re.findall(r'"@type"\s*:\s*"([^"]+)"', " ".join(
        re.findall(r'(?is)<script[^>]+application/ld\+json[^>]*>(.*?)</script>', html)))
    if "noindex" in robots.lower():
        add("critical", "noindex", "page is noindex (meta robots or X-Robots-Tag)")
    if not title:
        add("error", "title_missing", "no <title>")
    elif not 15 <= len(title) <= 65:
        add("notice", "title_length", f"title is {len(title)} characters")
    if not desc:
        add("warning", "description_missing", "no meta description")
    if h1 != 1:
        add("warning", "h1_count", f"{h1} <h1> elements in raw HTML")
    if not canon:
        add("warning", "canonical_missing", "no rel=canonical")
    elif urljoin(final, canon).rstrip("/") != final.rstrip("/"):
        add("warning", "canonical_other", f"canonical points to {urljoin(final, canon)}")
    if not re.search(r"<html[^>]+\blang=", html, re.I):
        add("notice", "lang_missing", "no lang attribute on <html>")
    if not re.search(r'<meta[^>]+name=["\']viewport', html, re.I):
        add("warning", "viewport_missing", "no viewport meta tag")
    if words < 150:
        add("warning", "thin_raw_html", f"{words} visible words before JavaScript runs")
    if final.startswith("https://") and headers and not headers.get("Strict-Transport-Security"):
        add("notice", "hsts_missing", "no Strict-Transport-Security header")
    base = "{0.scheme}://{0.netloc}".format(urlsplit(final))
    for bot in ("Googlebot", "OAI-SearchBot"):
        if robots_allows(base, bot) is False:
            add("critical" if bot == "Googlebot" else "error", "robots_blocks", f"robots.txt blocks {bot} from /")
    order = {"critical": 0, "error": 1, "warning": 2, "notice": 3}
    return {
        "url": url, "final_url": final, "status": status, "seconds": round(secs, 2),
        "redirects": [{"status": c, "url": u} for c, u in chain[:-1]],
        "page": {"title": title, "description": desc, "canonical": canon, "h1_count": h1,
                 "words_raw_html": words, "jsonld_types": sorted(set(ld)),
                 "hreflang_count": len(re.findall(r'hreflang=["\']', html, re.I))},
        "issues": sorted(issues, key=lambda i: order[i["severity"]]),
    }


class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        s = urlsplit(self.path)
        url = (parse_qs(s.query).get("url") or [""])[0]
        if s.path != "/audit" or not url:
            return self.reply(404, {"error": "use GET /audit?url=https://example.com/"})
        try:
            self.reply(200, audit(url))
        except Refused as e:
            self.reply(400, {"error": f"refused: {e}"})
        except Exception as e:  # network errors, timeouts
            self.reply(502, {"error": f"{type(e).__name__}: {e}"})

    def reply(self, code, payload):
        body = json.dumps(payload, indent=1).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


if __name__ == "__main__":
    port = int(sys.argv[1]) if len(sys.argv) > 1 else 8787
    print(f"site audit API on http://127.0.0.1:{port}/audit?url=...")
    ThreadingHTTPServer(("127.0.0.1", port), Handler).serve_forever()

A real request, through the two-hop redirect on our own www host:

$ curl -s 'http://127.0.0.1:8787/audit?url=http://www.bugviso.com/pricing'
{
 "url": "http://www.bugviso.com/pricing",
 "final_url": "https://bugviso.com/pricing",
 "status": 200,
 "seconds": 1.21,
 "redirects": [
  {"status": 308, "url": "http://www.bugviso.com/pricing"},
  {"status": 308, "url": "https://www.bugviso.com/pricing"}
 ],
 "page": {
  "title": "Website Audit Pricing: Plans from $19/mo | BugViso",
  "canonical": "https://bugviso.com/pricing",
  "h1_count": 1,
  "words_raw_html": 2302,
  "jsonld_types": ["BreadcrumbList", "FAQPage", "Offer", "Organization", "Product", "..."],
  "hreflang_count": 0
 },
 "issues": [
  {"severity": "warning", "code": "redirect_chain", "message": "2 redirects before the final URL"}
 ]
}

(The description and the full JSON-LD list are trimmed for width.) The one finding is real: http://www. takes two hops to reach the apex host.

Guard the Endpoint Against SSRF

An audit API fetches URLs that strangers send it, which makes it a server-side request forgery target by design. Without a guard, ?url=http://169.254.169.254/latest/meta-data/ asks your server to read the cloud metadata service and hand back the credentials.

The script refuses these before connecting, and it re-checks every redirect hop, because a public URL can redirect to a private one. These are real responses from the script:

$ curl -s 'http://127.0.0.1:8787/audit?url=http://169.254.169.254/latest/meta-data/'
{"error": "refused: 169.254.169.254 resolves to non-public address 169.254.169.254"}

$ curl -s 'http://127.0.0.1:8787/audit?url=http://localhost/'
{"error": "refused: localhost resolves to non-public address 127.0.0.1"}

# A public URL that 302s to the metadata IP is refused at the redirect hop
$ curl -s 'http://127.0.0.1:8787/audit?url=https://httpbin.org/redirect-to?url=http%3A%2F%2F169.254.169.254%2F'
{"error": "refused: 169.254.169.254 resolves to non-public address 169.254.169.254"}

The OWASP SSRF prevention cheat sheet adds the step this script leaves out: validate the resolved IP, then make the HTTP client connect only to that address. Otherwise a second DNS lookup between check and connect (DNS rebinding) gets around the guard. In production, pin the IP per connection, and run the fetchers in a network segment that can't reach internal services anyway.

Running Audits on a Schedule With BugViso

BugViso doesn't offer a public API today. It runs the rendered, multi-page side of this design for you, on a schedule, and exports the results as JSON and CSV. That's the part of an audit pipeline that needs browsers, queues and crawl logic.

Each audit crawls from your homepage links and sitemap. It renders the entry page and any page that needs JavaScript in Chromium, reads server-rendered pages from their HTML, and runs 25 engines: Core Web Vitals with Slow and Fast 3G re-loads, console errors, axe-core WCAG checks, canonical and structured-data validation, the internal link graph, security headers and TLS, and AI search readiness. When a site sits behind a bot challenge, the scan stops and says so rather than scoring the challenge page. Agency plans schedule daily, weekly or monthly runs, and every run lands in your audit records with a PDF and the JSON/CSV export. Crawl limits and exports are on the site crawl and reports page.

If you need audits inside your own code today, the pattern that works is the script above for per-deploy checks, plus scheduled full audits for the rendered, site-wide picture.

Mistakes That Make Audit APIs Unreliable

• Treating a 403 as a broken site. One in eight homepages in our sample refused a plain client. Report "blocked" as its own state.
• Waiting for network idle. 6 of 49 pages never got there within 10 s. Cap every wait.
• One request per audit, synchronously. Fine for raw checks. For rendered pages, return a job ID.
• Mixing raw and rendered results without saying which. A 40-word page may be a client-rendered app, not thin content. Put rendered in the response.
• Free-text findings only. Without stable codes, nobody can alert on or count anything. Add a code to every issue.
• Running Lighthouse once. Lab scores move between runs. For performance, take the median of 3 runs before you alert. Our Lighthouse CI alternatives guide covers the CI side.

Frequently Asked Questions

Is there a free site audit API?

Yes, partly. The PageSpeed Insights API runs Lighthouse for free (Google recommends a key for automated use), the CrUX API returns field Core Web Vitals at 150 queries per minute, and the Search Console URL Inspection API returns index status for verified sites at 2,000 queries per day. None of them crawls a site or checks on-page SEO, so you'd add your own checks, like the script above.

What should a site audit API check?

At minimum: final status and redirect chain, noindex in meta robots and X-Robots-Tag, robots.txt access for Googlebot, title, meta description, H1, canonical, viewport and word count. Then add performance (Lighthouse or field data), accessibility (axe-core) and structured data. Return each finding with a severity and a stable code.

Can a site audit API render JavaScript?

Only if it runs a real browser. A raw HTTP audit reads the server response, which on 5.8 % of the homepages we measured contained under a tenth of the visible text. Rendering costs time: a median 3.2 s to the load event against 1.8 s for a raw fetch. That's why rendered audits usually run behind a queue.

How do I audit a whole website through an API?

Crawl first, then audit each URL. Seed the crawl from the sitemap and the homepage's links, limit it by page count and depth, and run audits a few pages at a time. A synchronous endpoint can't hold a 100-page crawl open, so use a job API: create the job, poll its status, fetch the result.

Does BugViso have an API?

Not a public one today. BugViso runs scheduled, rendered multi-page audits and exports each one as JSON and CSV, plus a PDF report. For per-deploy checks in your own pipeline, the standard-library endpoint in this guide is a working starting point.

The Tier Split That Makes an Audit API Work

Run the cheap raw checks synchronously on every deploy, and push rendered, multi-page audits through a queue with hard timeouts. Never accept a URL without an SSRF guard. For the rendered, site-wide tier, a scheduled BugViso audit gives you the same checks without running the browsers yourself.