AI Visibility Week Over Week: Prompts Won, Gained and Lost

Week-over-week AI visibility compares which buyer prompts each AI engine named your brand on this week against last week: prompts won, gained and lost. It's the most readable change report you can send a client, and the easiest to misread. Between two runs 23 minutes apart, with nothing changed, our brand "gained" 4 prompt-engine pairs and "lost" 5. Only 6 of 11 wins survived.

By the end of this post you'll know how the won rule works, why single-sample weeks flip constantly, how many samples a week make a gain believable, and how to label each change as clear or within noise. A script does the labelling for you.

Data used: 35 unbranded buyer prompts × ChatGPT, Gemini, Perplexity and Claude × 2 runs on 2026-10-20 (280 answers, web search on), started 23 minutes apart, collected by BugViso's AI Visibility Tracker for bugviso.com (website-audit software) against five anonymised competitors. Flip probabilities are computed from the binomial distribution, not simulated.

How "Won", "Gained" and "Lost" Are Defined

A prompt is won on an engine when the brand is named in at least half of that week's successful answers to it. The unit is a prompt-engine pair: 35 prompts on 4 engines give 140 pairs. Each week, every pair is either won or not.

• Gained: a pair that wasn't won last week and is won this week.
• Lost: a pair that was won last week and isn't this week.
• Skipped: a pair with no successful answer in either week. It's unknown, not lost.

This is the rule in BugViso's weekly wrap-up, which pools all daily runs of the week on the same prompt version. If the prompt set changed between the two weeks, the wrap-up writes a note instead of a comparison, because a gain on an edited prompt isn't a gain. Our post on prompt set versioning explains why that matters.

The 50% threshold is a choice. It makes "won" mean "named more often than not", which a client understands without a statistics lesson. The cost is that pairs near 50% sit on a knife edge.

Two Runs, Nothing Changed: 9 Prompts Moved

When each "week" holds one answer per pair, won and lost are coin flips for any prompt that isn't a sure thing. We treated our two same-day runs as two periods and applied the won rule:

Every brand moved. Competitor C "lost" 10 pairs and "gained" 6, a net drop of 4 won prompts. Read as a weekly report, that's a story about Competitor C losing ground. In reality the two runs were 23 minutes apart. Our study of AI answer consistency found the same instability at the level of single answers.

For our own brand, the flips sat on two engines:

"ChatGPT dropped us on 4 of 5 prompts" would be a headline in a client email. It's what one sample per pair produces. The pooled mention rate barely moved: 11 of 140 answers, then 10 of 140.

How Often a Prompt Flips by Chance

The chance that a won/lost status flips between two weeks with no real change depends on the prompt's true mention rate and the samples per week. If a brand's true rate on a pair is p, the probability of a flip is 2 × P(won) × (1 − P(won)), where P(won) comes from the binomial distribution.

Two things stand out. Daily runs pooled weekly cut the flip chance for clear-cut prompts (true rate 20% or 80%) from about 1 in 3 to about 1 in 16. And a prompt where the brand's true rate is near 50% flips half the weeks no matter how many samples you take, because the won rule is a threshold sitting on its true value.

Even sample counts have a quirk: ties count as won. With 2 samples a week, a pair at 35% flips 49% of weeks. Seven samples, one per day, avoid ties.

💡 Rule of thumb: a gained or lost prompt from a single weekly run is a rumour. With daily runs pooled into the week, it's a lead worth checking. Only a clear change in the per-pair counts is a finding.

When a Gained Prompt Is Real

A gain is real when the two weeks' counts differ by more than chance would allow. For a 2×2 table of named/not named before and after, the right test at these sample sizes is Fisher's exact test. It's exact for small counts, where approximations break.

Here's the smallest change that reaches p < 0.05, by samples per week:

With 7 samples, going from 1 of 7 to 6 of 7 is clear. Going from 3 of 7 (not won) to 4 of 7 (won) never is, even though the wrap-up calls it a gain. That's the pair most weekly reports celebrate.

So report changes in two buckets:

1. Clear changes: gained or lost pairs whose counts pass Fisher's test. Name them, show the counts, and connect them to work you did.
2. Moved within noise: the rest. List them for transparency, with the counts, and don't explain them.

The headline number needs the same care. Compare pooled mention rates with their intervals, as covered in how often to track AI visibility. Our 11 of 140 to 10 of 140 gives Fisher p = 1.00: no change.

Watch for Regression on Last Week's Wins

Prompts you just gained are the ones most likely to be lost next week, even if nothing changes. A pair that crossed into "won" by chance had a lucky week. Next week its count tends to fall back towards its true rate. Statisticians call this regression toward the mean.

It shows in our data. Of the 11 pairs our brand won in run 1, 5 were lost in run 2. Most of them were ChatGPT pairs where we were named once in one answer. They weren't strongholds, just single hits.

Two practices keep regression out of client reports:

• Count a pair as a stable win only after two consecutive won weeks. Report "new wins" separately until they hold.
• Judge an action by the pooled rate on its target prompts over several weeks, not by one week's gains. BugViso's impact tracking compares up to 7 daily runs before an action with up to 7 after it, and calls a change below the run-to-run spread "no clear change".

Label Week-Over-Week Changes Yourself

This script reads an answer export with a period label in the run column, applies the won rule per prompt-engine pair, lists gained and lost pairs with their counts, and tags each with a Fisher p-value. It also compares the pooled mention rates. Standard library only.

#!/usr/bin/env python3
"""Week-over-week AI visibility: prompts won, gained and lost, with a noise check per change.

A (prompt, engine) pair is won when the brand is named in >= 50% of that period's answers.
Each gained/lost pair gets a two-sided Fisher exact p-value, so a 1-of-1 -> 0-of-1 flip
isn't reported as a real loss.

Usage: python3 wow_changes.py answers.csv --brand Acme --before run1 --after run2
answers.csv columns: run (period label), engine, prompt_id, prompt, answer, branded (1/0).
"""
import argparse, csv, re
from collections import defaultdict
from math import comb

def fisher(a, b, c, d):
    """Two-sided Fisher exact p for [[a, b], [c, d]] (hits/misses before, hits/misses after)."""
    n, r1, c1 = a + b + c + d, a + b, a + c
    pr = lambda x: comb(r1, x) * comb(n - r1, c1 - x) / comb(n, c1)
    p0 = pr(a)
    lo, hi = max(0, c1 - (n - r1)), min(r1, c1)
    return min(1.0, sum(pr(x) for x in range(lo, hi + 1) if pr(x) <= p0 * (1 + 1e-9)))

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("csv"); ap.add_argument("--brand", required=True)
    ap.add_argument("--before", required=True); ap.add_argument("--after", required=True)
    a = ap.parse_args()
    rx = re.compile(r"(?<![\w-])" + re.escape(a.brand) + r"(?![\w-])", re.I)
    cnt = {a.before: defaultdict(lambda: [0, 0]), a.after: defaultdict(lambda: [0, 0])}
    text = {}
    for r in csv.DictReader(open(a.csv, newline="", encoding="utf-8")):
        if r.get("branded") == "1" or r["run"] not in cnt:
            continue
        k = (r["prompt_id"], r["engine"])
        cnt[r["run"]][k][0] += bool(rx.search(r["answer"])); cnt[r["run"]][k][1] += 1
        text[k[0]] = r["prompt"]
    b, f = cnt[a.before], cnt[a.after]
    won = lambda h: h[0] / h[1] >= 0.5
    pairs = sorted(set(b) & set(f))  # pairs without data in either period are skipped
    changes = [(k, b[k], f[k]) for k in pairs if won(b[k]) != won(f[k])]
    hits = lambda d: sum(d[k][0] for k in pairs); n = lambda d: sum(d[k][1] for k in pairs)
    print(f"{len(pairs)} prompt x engine pairs | won before {sum(won(b[k]) for k in pairs)}, "
          f"after {sum(won(f[k]) for k in pairs)}")
    print(f"pooled mention rate: {hits(b)}/{n(b)} -> {hits(f)}/{n(f)} "
          f"(Fisher p = {fisher(hits(b), n(b) - hits(b), hits(f), n(f) - hits(f)):.2f})\n")
    for label, want in (("GAINED", True), ("LOST", False)):
        rows = [(k, x, y) for k, x, y in changes if won(y) == want]
        print(f"{label}: {len(rows)}")
        for (pid, eng), x, y in rows:
            p = fisher(x[0], x[1] - x[0], y[0], y[1] - y[0])
            tag = "clear" if p < 0.05 else "within noise"
            print(f"  {eng:<11}{x[0]}/{x[1]} -> {y[0]}/{y[1]}  p={p:.2f} {tag:<13}{text[pid][:60]}")
    clear = sum(1 for _, x, y in changes if fisher(x[0], x[1] - x[0], y[0], y[1] - y[0]) < 0.05)
    print(f"\n{len(changes)} changes, {clear} clear. Report the clear ones; list the rest as 'moved within noise'.")

if __name__ == "__main__":
    main()

On our two runs it prints:

140 prompt x engine pairs | won before 11, after 10
pooled mention rate: 11/140 -> 10/140 (Fisher p = 1.00)

GAINED: 4
  perplexity 0/1 -> 1/1  p=1.00 within noise recommend a tool to audit a prospect's website and send a br
  ...
LOST: 5
  chatgpt    1/1 -> 0/1  p=1.00 within noise is a $19 a month website audit plan worth it for an indie de
  ...

9 changes, 0 clear. Report the clear ones; list the rest as 'moved within noise'.

For weekly use, label each answer's run with its ISO week (2026-W43) and pass two weeks as --before and --after.

How BugViso Builds the Weekly Wrap-Up

BugViso's AI Visibility Tracker builds a weekly wrap-up when the daily run on the brand's weekly day finishes. It pools that week's daily runs on the newest prompt version and applies the won rule per prompt and engine: named in at least half of the week's successful samples. It then compares with the previous week and lists prompts gained and lost, plus index and rank movement, the change in each index part, new and lost cited domains, competitor movers and the sentiment shift. Pairs with no data in either week are skipped, and a changed prompt version produces a note rather than a comparison.

Pooled visibility carries a 95% Wilson interval, and segments with fewer than 20 answers are flagged low confidence. Reports export as PDF or CSV; the CSV has one row per prompt and engine with the successful samples and brand mentions, so you can run the fisher() check above on two weekly exports. The weekly report sits in the tracker's report list; the AI visibility tracker page describes what each section contains.

AI Visibility Week-Over-Week Mistakes That Mislead Clients

Weekly change from weekly runs. One sample per pair means every non-certain prompt is a coin flip. Use daily runs pooled into the week, or report monthly.

Explaining every lost prompt. A loss that's within noise has no cause to find. Inventing one ("a competitor published a new page") trains the client to expect a story every week. The same discipline applies to daily numbers, as covered in what an AI visibility email report should say.

Netting gains against losses. "Net +1 prompt" from 9 gained and 8 lost hides that both lists are mostly noise. Report clear changes and the pooled rate.

Comparing across prompt edits. A prompt reworded on Wednesday isn't the same question. Compare only weeks on the same prompt version.

Ignoring the coin-flip prompts. Pairs where the brand's rate sits near 50% will flip half the weeks forever. Flag them once ("contested prompts") rather than reporting them as gained and lost each week.

Frequently Asked Questions

What does "prompt gained" mean in AI visibility tracking?

A prompt is gained on an engine when the brand wasn't named in at least half of last week's answers to it and is this week. It's a threshold crossing, so check the counts behind it before treating it as progress.

Why did my brand lose prompts this week when nothing changed?

Because AI answers vary between runs. In our data, two runs 23 minutes apart produced 4 gains and 5 losses for one brand. With one sample per prompt a week, any prompt that isn't a sure thing flips often.

How many samples per week do I need for week-over-week tracking?

Daily runs pooled into the week (7 samples per prompt and engine) are a practical minimum. With 7 samples, a change from 1 of 7 to 6 of 7 is clearly real, while 3 of 7 to 4 of 7 never is.

Should I report prompts won or mention rate?

Both, for different readers. Prompts won is easy to explain. Mention rate with an interval is the more reliable trend. Lead with the rate and use won prompts as examples.

What is a good week-over-week change in AI visibility?

One that clears noise: a pooled mention rate whose interval no longer overlaps last week's, or prompt-level changes that pass Fisher's exact test. A change of a few prompts in either direction usually isn't one.

Report Clear Changes, List the Rest

A won, gained and lost list is only as honest as the samples behind each pair: pool daily runs into the week, tag every change as clear or within noise, and lead with the pooled rate. That's how BugViso's AI visibility tracking builds its weekly wrap-up, with the per-pair counts exported so you can check them.