AI Visibility Benchmark: Honest Competitor Comparisons

An honest AI visibility benchmark compares how often AI engines name a brand against named competitors, on the same buyer questions, with an interval around every rate and a paired test for every gap. It never uses an "industry average", because no shared prompt set exists to average over. On our own 280 answers, 5 of the 15 gaps between six brands weren't real.

That's the number to remember before a client pitch. By the end of this post you'll know how to build the benchmark, which differences you can state as fact, which you have to call a tie, and how many answers a claim needs. The script that runs the test is included.

Data used: 280 answers to 35 unbranded buyer prompts (ChatGPT, Gemini, Perplexity and Claude × 2 runs on 2026-10-20, web search on) from BugViso's AI Visibility Tracker for bugviso.com, against five competitors anonymised as A–E. A brand counts as named when the answer mentions it by name. Intervals are 95 % Wilson intervals; pair tests are exact McNemar tests.

What an AI Visibility Benchmark Can and Can't Claim

A benchmark is only as comparable as its inputs: the same prompts, asked of the same engines, at the same time, for every brand in the table. Change any of those and the rates measure different things.

That rules out the benchmark clients ask for most often: "how do we compare to the industry?" An industry average would need everyone's prompts, engines and dates to match. They don't, and nobody publishes them, so any average quoted in a pitch is either invented or measured on someone else's question set. Neither belongs in a client deck.

What you can claim is narrower and stronger:

The Benchmark on 280 Real Answers

Six brands, one prompt set, one pair of runs: the leader was named in 25.4 % of unbranded answers and our brand in 7.5 %.

The intervals overlap all the way down the table. Overlapping intervals don't prove a tie, but they're the warning sign that the ranking might be partly noise. To decide, you need the right test.

We use the Wilson score interval for each rate because it behaves well near 0 % and at small samples, where the textbook formula breaks (binomial proportion confidence intervals). Our guide on how often to track AI visibility covers how the intervals shrink as runs pool.

Which Gaps Are Real: Compare Brands on the Same Answers

Every brand is measured on the same 280 answers, so the comparison is paired, and a paired test is much sharper than comparing two intervals. What matters is the discordant answers: the ones that name one brand and not the other. If A is named in 30 answers without C and C in 11 without A, that imbalance is the evidence. McNemar's test turns it into a p-value.

Here are all 15 pairs, ranked by the gap:

Ten gaps are real. Five aren't, including the one every pitch would lead with: A versus B, the top two. A was named 71 times and B 63, but on the 54 answers that named only one of them, the split was 31 to 23. That's well within what chance produces.

So the honest version of this benchmark has tiers, not ranks: A and B lead; C, D and E form a middle group with no reliable order; BugViso trails D but is level with E. That's a less exciting slide, and a more defensible one.

💡 Don't eyeball it: C versus E split 39 to 23, which looks decisive, and still came out at p = 0.056. Run the test on every pair you plan to show.

Per-Engine Benchmarks Are Too Thin to Rank

At 70 answers per engine, most per-engine rankings are noise. The headline benchmark pools 280 answers. Split by engine and each brand's interval roughly doubles:

The Perplexity row is the trap. Our brand had the most mentions on Perplexity, and it would be tempting to tell a prospect "you're #1 on Perplexity". The paired test on 19 discordant answers (11 versus 8) gives p = 0.65. It's a tie.

The Claude row shows the opposite trap. Zero mentions in 70 answers is a real finding, and still the interval runs up to 5.2 %. "Claude never names you" means "in these 70 answers"; our post on why Claude skips small brands looks at why.

How Many Answers a Benchmark Claim Needs

To state a mention rate within ±5 points, you need roughly 150 answers at a 10 % rate and 390 at 50 %. Rarer outcomes need fewer answers for the same absolute margin:

One run of a 50-prompt set with 35 unbranded prompts gives 140 unbranded answers across four engines. That's enough for one pooled rate at ±5 points if the brand sits near 10 %, and not enough for any per-engine rate. Pool runs before you quote engines.

Pooled rates were stable between our two runs (A: 35 then 36 of 140; B: 31 then 32; BugViso: 11 then 10), even though individual answers changed a lot. Totals are reliable at this size; individual prompts and engines are not.

Test Your Own Benchmark

This standard-library script reads an answers export (one row per answer, with the answer text and optionally the engine), counts which brands each answer names, and prints rates with Wilson intervals plus an exact McNemar test for every pair. Use --engine to test one engine.

#!/usr/bin/env python3
"""benchmark_compare.py: which AI visibility gaps between brands are real, and which are noise.

Usage: python3 benchmark_compare.py answers.csv "Brand A" "Brand B" "Brand C" [--engine chatgpt]

answers.csv: one row per AI answer with columns answer (the text) and, optionally, engine and
branded (0/1). Branded answers are skipped. Standard library only.
"""
import csv, math, re, sys
from itertools import combinations


def wilson(x, n, z=1.96):
    if n == 0:
        return 0.0, 0.0
    p = x / n
    d = 1 + z * z / n
    c = (p + z * z / (2 * n)) / d
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    return 100 * max(0.0, c - h), 100 * min(1.0, c + h)


def mcnemar_exact(b, c):
    """Two-sided exact p-value: binomial test on the discordant pairs."""
    n = b + c
    if n == 0:
        return 1.0
    tail = sum(math.comb(n, i) for i in range(min(b, c) + 1)) / 2 ** n
    return min(1.0, 2 * tail)


def main():
    args = sys.argv[1:]
    engine = args[args.index("--engine") + 1] if "--engine" in args else None
    args = [a for i, a in enumerate(args) if a != "--engine" and (i == 0 or args[i - 1] != "--engine")]
    if len(args) < 3:
        sys.exit(__doc__)
    path, brands = args[0], args[1:]
    rows = [r for r in csv.DictReader(open(path, newline=""))
            if r.get("branded", "0") in ("0", "", "false", "False") and (not engine or r.get("engine") == engine)]
    pats = {b: re.compile(r"(?<![\w-])" + re.escape(b) + r"(?![\w-])", re.I) for b in brands}
    named = [{b: bool(p.search(r.get("answer") or "")) for b, p in pats.items()} for r in rows]
    n = len(named)
    print(f"{n} unbranded answers{' on ' + engine if engine else ''}\n")
    rate = {b: sum(a[b] for a in named) for b in brands}
    for b in sorted(brands, key=lambda b: -rate[b]):
        lo, hi = wilson(rate[b], n)
        print(f"  {b:<16} {rate[b]:>4}/{n}  {100 * rate[b] / max(n, 1):5.1f}%   95% CI {lo:5.1f}-{hi:5.1f}%")
    print("\npairwise (exact McNemar on the same answers):")
    for a, b in combinations(sorted(brands, key=lambda b: -rate[b]), 2):
        only_a = sum(x[a] and not x[b] for x in named)
        only_b = sum(x[b] and not x[a] for x in named)
        p = mcnemar_exact(only_a, only_b)
        verdict = f"{a} ahead" if p < 0.05 and only_a > only_b else f"{b} ahead" if p < 0.05 else "tie"
        print(f"  {a:<14} vs {b:<14} only first {only_a:>3}  only second {only_b:>3}  p={p:.3f}  {verdict}")


if __name__ == "__main__":
    main()

Real output on our Perplexity answers:

$ python3 benchmark_compare.py answers.csv BugViso "Competitor A" --engine perplexity
70 unbranded answers on perplexity

  BugViso            12/70   17.1%   95% CI  10.1- 27.6%
  Competitor A        9/70   12.9%   95% CI   6.9- 22.7%

pairwise (exact McNemar on the same answers):
  BugViso        vs Competitor A   only first  11  only second   8  p=0.648  tie

Name matching here is plain word-boundary search, which reproduced our tracker's counts exactly on this data. For brands with common-word names or many aliases, use proper brand mention detection before you count.

How BugViso Builds the Competitor Benchmark

BugViso's AI Visibility Tracker measures you and at least three competitors on the same answers, so every benchmark it shows is paired by construction. It asks ChatGPT, Gemini, Perplexity, Claude and Google AI Overviews your 50 buyer prompts with web search on, and records which tracked brands each answer names and in what order.

The leaderboard ranks every brand on the visibility index and shows mention rate, share of voice, position and citations side by side, with a 95 % Wilson interval on each brand's visibility. Engine, buyer-stage and topic segments under 20 answers are flagged low confidence rather than ranked silently. Runs on a daily, weekly or monthly schedule pool into the day, week and month views, which is how the intervals get narrow enough to quote. See the AI visibility tracker for the full report.

Benchmark Mistakes That Lose Client Trust

• Quoting an industry average. There isn't one. Benchmark against named competitors on your own prompt set.
• Ranking ties. A and B were 71 versus 63 and statistically level. Show tiers, not a podium.
• Selling per-engine wins. "#1 on Perplexity" was a 12-versus-9 tie. Pool engines or runs first.
• Comparing intervals instead of testing pairs. Overlapping intervals can still hide a real paired difference, and separated ones can come from different answer sets. Test on the same answers.
• Mixing branded prompts in. Branded questions name the client by definition. Benchmark on unbranded prompts only.
• Treating p < 0.05 as size. The American Statistical Association's statement on p-values is blunt: a p-value doesn't measure the size of an effect. Report the rates and the gap, not just "significant".

Frequently Asked Questions

What is an AI visibility benchmark?

An AI visibility benchmark compares how often AI engines name your brand against named competitors on the same buyer questions. A defensible one reports each brand's mention rate with an interval and tests each gap on the same answers. In our data, the leader was named in 25.4 % of 280 unbranded answers and our brand in 7.5 %.

Is there an industry average for AI visibility?

No. Mention rates depend entirely on the prompt set, the engines and the date, and there's no shared prompt set that companies measure against. Any "industry average" either comes from someone else's questions or is made up. Compare against named competitors on your own prompts instead.

How do I know if a competitor is really ahead in AI answers?

Count the answers that name one brand and not the other, and run an exact McNemar test on those two counts. If p is below 0.05, the gap is unlikely to be chance. In our benchmark, 10 of 15 brand pairs passed, and the top two brands didn't.

How many prompts do I need for an AI visibility benchmark?

Count answers, not prompts. To pin a mention rate near 10 % within ±5 points, you need about 150 answers; near 50 %, about 390. One run of 35 unbranded prompts across four engines gives 140, so pool several runs before comparing engines or topics.

Can I say my client ranks #1 on ChatGPT?

Only if the gap to second place survives a paired test on the same answers. At 70 answers per engine that's rare. In our data, our brand had the most Perplexity mentions (12 versus 9), but the test gave p = 0.65, so the honest statement is "level with the leader on Perplexity".

Tiers, Intervals, and the Questions Behind Them

Present a benchmark as tiers backed by paired tests, show the interval on every rate, and name the prompt set it came from. A client who hears "A and B lead, you're level with E" can act on it. A client sold "#1 on Perplexity" will notice when the next run says otherwise. Build the comparison on paired runs with BugViso's AI visibility tracking.