Automated Accessibility Testing: What It Catches and Misses

Automated accessibility testing reliably catches failures a machine can verify from the DOM: low color contrast, images with no alt attribute, links and buttons with no accessible name, unlabeled form fields, invalid ARIA, a missing page language, disabled zoom and undersized touch targets. It can't judge whether alt text is accurate, whether keyboard focus moves in a logical order, whether captions match the audio, or whether an error message actually helps.

The line between those two groups decides how far you can trust a green dashboard. Deque Systems, the company that maintains the open-source axe-core engine, has published research estimating that automated testing finds roughly 57% of accessibility issues by volume. Measured by WCAG success criteria, the coverage is smaller still. Of the 55 Level A and AA criteria in WCAG 2.2, only a handful can be checked end to end by a machine.

If you've already run a scan, our breakdown of the most common WCAG violations shows how to fix the failures automation does find.

This guide splits accessibility testing into what automation decides, what it flags for a human, and what it never sees. It uses axe-core results from 219 real homepages and ends with a script that reports all three buckets.

The Three Buckets Every axe-core Run Returns

Most dashboards show one number. Under the hood, axe-core sorts every check into separate result lists:

The incomplete bucket is the one teams skip. When text sits on a background image or gradient, for example, axe can't calculate one background color. Rather than guess, it puts the element in "needs review". A tool that reports only violations will show that element as passing.

⚠️ "0 violations" doesn't mean accessible. It means no rule that could run, failed. Everything in "needs review" and everything outside the rule set is still unknown.

What Automation Found on 219 Real Homepages

We drew 420 domains at random from Tranco list 94GG2 (ranks 1,001–50,000), loaded each homepage in Chromium at 1366×900 on 3 October 2026, and kept the 219 that served a real page. We ran axe-core 4.10.2 with the WCAG 2.0, 2.1 and 2.2 A/AA rule tags, the same configuration BugViso uses.

The most frequently failed rules show where automation is strong:

Every one of these is a property of the DOM: a ratio, a missing attribute, a size, a role mismatch. Automation is excellent at that kind of check, and these issues are also among the most common, which is why they dominate any automated report.

The needs-review bucket is bigger than the violations bucket

We ran a second pass on the same homepages (209 completed) and recorded axe-core's incomplete results as well. The scale surprised us:

Contrast dominates: it accounted for 92% of all needs-review elements. Text over images and gradients is common in hero sections and banners, exactly where a low-contrast headline costs the most. Automation flags those elements and steps back. Someone has to look.

The Capability Matrix: Automated vs Manual

The pattern is consistent. Automation answers "is something there?" A person answers "does it work for someone using it?"

Why Some Failures Are Invisible to Automation

Meaning can't be computed. alt="Image" and alt="Chart showing sales doubled in Q3" both satisfy "has alt text". Only one of them is useful, and a rule engine can't tell which. Our alt text writing guide is about exactly that gap.

Behaviour needs interaction. A keyboard trap only shows up when you tab into a widget and can't tab out. A modal that doesn't return focus to its trigger only fails after you close it. A single-page scan never presses those keys.

Context changes the rule. The same red text passes as a decorative flourish and fails as an error message. "Click here" is a fine link inside a sentence that explains it, and a failure in a list of twelve identical links.

Some content isn't in the DOM at test time. Content behind tabs, accordions, infinite scroll, login walls and multi-step forms never reaches a homepage scan. A clean scan of the homepage tells you nothing about your checkout.

A Manual Testing Checklist for What Automation Misses

Run these on your key templates (homepage, a content page, a form, checkout or signup) after every major release:

1. Keyboard only. Unplug the mouse. Tab through the whole page, Shift+Tab back. Can you reach, operate and leave every control? Is focus always visible, and never hidden behind a sticky header?
2. Zoom to 200% and 400%. Does text reflow without horizontal scrolling at 320 CSS px wide? Do overlays still fit?
3. Screen reader pass. With NVDA (Windows) or VoiceOver (macOS/iOS), navigate by headings, then by landmarks, then by links. Does the outline make sense? Are images described usefully?
4. Forms. Submit empty and invalid. Is each error tied to its field, announced, and specific enough to fix?
5. Dynamic updates. Add to cart, filter results, load more. Is the change announced (live region) or at least discoverable?
6. Media. Play videos with sound off. Are captions accurate? Is there a transcript for audio?
7. Motion and timing. Do carousels pause? Can session time-outs be extended?

This list takes about an hour per template. Paired with a clean automated scan, it covers most of what an external auditor will test first.

Run axe-core and See All Three Buckets

This script runs axe-core in Chromium and prints violations and needs-review results separately, plus how many rules passed or didn't apply.

#!/usr/bin/env python3
"""Run axe-core against a live page and separate what automation decided from what it couldn't.

axe-core returns three buckets that most dashboards collapse into one number:
  violations   – rules that definitely failed
  incomplete   – "needs review": axe found the element but cannot decide (a human must)
  passes       – rules that ran and passed
Everything axe has no rule for (keyboard traps, focus order, alt-text *quality*,
caption accuracy…) never appears in any bucket — see the manual checklist in the post.

Usage:  pip install playwright && playwright install chromium
        python3 axe_audit.py https://example.com
        python3 axe_audit.py https://example.com --json > report.json
"""
import argparse, asyncio, json, sys
from playwright.async_api import async_playwright

AXE_CDN = "https://cdnjs.cloudflare.com/ajax/libs/axe-core/4.10.2/axe.min.js"
TAGS = ["wcag2a", "wcag2aa", "wcag21a", "wcag21aa", "wcag22aa"]


async def main(url, as_json):
    async with async_playwright() as pw:
        browser = await pw.chromium.launch()
        # bypass_csp: sites with a strict Content-Security-Policy would block the axe <script> tag
        page = await browser.new_page(viewport={"width": 1366, "height": 900}, bypass_csp=True)
        await page.goto(url, wait_until="load")
        await page.wait_for_timeout(2000)
        await page.add_script_tag(url=AXE_CDN)
        res = await page.evaluate("""async (tags) => {
            const r = await axe.run(document, {runOnly: {type: 'tag', values: tags}});
            const slim = (list) => list.map(v => ({id: v.id, impact: v.impact, help: v.help,
                nodes: v.nodes.length, sample: (v.nodes[0] && v.nodes[0].target.join(' ')) || ''}));
            return {violations: slim(r.violations), incomplete: slim(r.incomplete),
                    passes: r.passes.length, inapplicable: r.inapplicable.length};
        }""", TAGS)
        await browser.close()

    if as_json:
        json.dump(res, sys.stdout, indent=2)
        return
    print(f"axe-core 4.10.2 on {url}\n")
    for bucket, label in (("violations", "VIOLATIONS (fix these)"), ("incomplete", "NEEDS REVIEW (a human must decide)")):
        rows = sorted(res[bucket], key=lambda v: -v["nodes"])
        print(f"== {label}: {len(rows)} rules, {sum(v['nodes'] for v in rows)} elements")
        for v in rows:
            print(f"  [{v['impact'] or '-':8}] {v['id']:28} x{v['nodes']:<4} {v['help']}")
            print(f"             e.g. {v['sample'][:90]}")
        print()
    print(f"Rules passed: {res['passes']}  |  rules not applicable to this page: {res['inapplicable']}")


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("url")
    ap.add_argument("--json", action="store_true")
    a = ap.parse_args()
    asyncio.run(main(a.url, a.json))

Here's real output from a large open-source CMS homepage that scores zero violations:

== VIOLATIONS (fix these): 0 rules, 0 elements

== NEEDS REVIEW (a human must decide): 2 rules, 5 elements
  [serious ] link-in-text-block           x3    Links must be distinguishable without relying on color
  [serious ] color-contrast               x2    Elements must meet minimum color contrast ratio thresholds

Rules passed: 27  |  rules not applicable to this page: 35

A violations-only dashboard would show a perfect result. The honest reading is: 27 automated checks passed, and 5 elements still need a person to look at them.

How BugViso Uses Automated Testing (and Where It Stops)

BugViso runs a self-hosted axe-core engine with the WCAG 2.0, 2.1 and 2.2 A/AA tags on every audited page, after the page has fully rendered in Chromium. Each violation is reported with its rule, WCAG criterion, impact and failing elements. For contrast failures it shows the measured foreground/background colors and ratio. A separate mobile device-emulation pass adds tap-target sizing against WCAG 2.5.8 and checks for horizontal overflow and viewport problems.

On a multi-page crawl, BugViso counts a rule that fails in a shared header once per site rather than once per page, so one template bug doesn't look like 200 separate problems. Findings appear in the web report and the PDF report with remediation guidance.

BugViso doesn't claim that an automated pass makes a site compliant. It clears the machine-verifiable failures quickly and repeatedly, so your manual testing time goes to the judgement calls listed above. Start with a free BugViso accessibility scan to see which bucket your issues fall into.

Common Misconceptions

• "Our score is 100, so we're compliant." A perfect automated score covers only the criteria a machine can test. Legal standards are written against all of WCAG.
• "Overlays fix what scanners find." In our own tests, overlay widgets rarely changed axe-core results at all. See our analysis of whether accessibility overlays work.
• "Manual testing is only for big companies." The checklist above takes about an hour per template, and it finds the problems users actually complain about.
• "Automated tools produce lots of false positives." Modern engines like axe-core are designed to avoid them, which is exactly why they move uncertain cases to "needs review" instead of failing them.

FAQ

What percentage of accessibility issues can automated testing find?

Estimates vary with how you count. Deque's published research puts it at roughly 57% of issues by volume, because common failures like contrast and missing names are highly automatable. Measured by WCAG success criteria, the share that can be fully automated is much smaller, since most criteria need human judgement.

Is axe-core better than other accessibility scanners?

axe-core is the open-source engine behind many commercial and browser tools, and it's built to keep false positives low. Every engine has the same fundamental limit, though: none can judge meaning, behaviour over time or context.

How often should I run automated accessibility tests?

On every deploy, or at least every release, because regressions are cheap to catch early. Run manual testing on key templates each quarter and whenever a template changes substantially.

Do I still need an accessibility audit if my scans are clean?

Yes, if you need to claim conformance. A clean automated scan removes the most common failures, but an audit by someone using keyboard and screen readers is the only way to evaluate the criteria automation can't reach.

Why do two tools give different results for the same page?

They use different rule sets, different WCAG versions, different timing (before or after scripts load) and different rules for what goes to "needs review". Compare tools by the rules they report, not by their headline score.

Conclusion

Automated testing is the fastest way to find the failures a machine can verify, and the least reliable way to prove a site is accessible, so treat its violations as a to-do list and its "needs review" items as a human's job. A BugViso scan handles the first part on every page it crawls.