Are AI Answers Consistent? 200 Prompt Pairs, 23 Min Apart

No, not fully. We asked ChatGPT, Gemini, Perplexity and Claude the same 50 buyer questions twice, 23 minutes apart, with web search on. When an answer named at least one of the six tracked brands, the second answer named exactly the same brands only 60% of the time. The cited sources matched exactly in 41 of 200 pairs.

That has a direct consequence for anyone who reports AI visibility. A screenshot of one ChatGPT answer is a single sample, not a measurement. Asking the same question again can flip the outcome.

This post shows the data, explains why engines drift, and gives you a script to measure drift on your own brand.

Method in one line: 50 prompts × 4 engines × 2 runs on 2026-10-20 for one brand (ours) in the website-audit software category, tracked against five anonymised competitors. All numbers below come from those 400 real answers.

How We Tested Answer Consistency

We ran BugViso's own AI Visibility Tracker on bugviso.com. The tracker had drafted 50 buyer questions from the site; 35 are unbranded ("which website audit tools let unused scans roll over") and 15 are branded ("how do BugViso scan quotas work"). Five established audit platforms were tracked as competitors. We call them Competitor A–E, because naming rivals isn't the point of the study.

The first run started at 14:47 UTC and the second at 15:10 UTC. Nothing changed on the site in between.

Google AI Overviews was tracked as well. No overview was captured for these queries, so it's left out of every table.

A pair means the same prompt sent to the same engine in both runs. We have 200 pairs: 50 prompts times 4 engines.

The Drift Benchmark: Same Prompt, 23 Minutes Later

The table below is the output of the script further down, run on the dataset. "Same brands" means the set of tracked brands named in the answer was identical in both runs. "Same #1" means the first-named tracked brand matched. A brand flip means our brand appeared in one run and not the other. Citation overlap is the median Jaccard similarity of cited domains.

engine       pairs  same brands  same #1  brand flips  citation overlap
chatgpt         50          76%      78%            4             0.40
claude          50          64%      78%            0             0.25
gemini          50          74%      82%            1             0.33
perplexity      50          80%      86%            4             0.29

Those percentages flatter the engines. 67 of the 200 pairs named no tracked brand in either run, and two empty answers count as "the same". Remove those, and the picture changes:

Claude was the least stable: in half of its non-empty pairs, the shortlist changed. Perplexity was the most stable, yet one in three of its shortlists still changed.

How big the changes were

Most changes were small. Counting brands that appeared in one answer but not the other (the symmetric difference of the two sets):

So a shortlist rarely gets rebuilt from scratch, but it often gains or loses a name. If that name is yours, a single-run report swings from "mentioned" to "invisible".

That happened 9 times for our brand: 4 on ChatGPT, 4 on Perplexity, 1 on Gemini and 0 on Claude. Claude never named us in either run, so it had nothing to flip.

Citations drift more than brands

Only 41 of 200 pairs cited exactly the same set of domains. The median overlap ranged from 0.25 (Claude) to 0.40 (ChatGPT). Answer length moved too: the median pair changed length by 15%.

💡 Rule of thumb: treat a single AI answer like a single poll respondent. It tells you what is possible, not what is typical.

Why the Same Prompt Gets a Different Answer

Four mechanisms stack on top of each other. You can't switch them off, because consumer chat apps run the same way.

1. Sampling in the language model

Language models pick each next token from a probability distribution. Unless temperature is fixed at zero (it isn't in chat products), two runs take different paths through the same distribution. A sentence that starts "Popular options include…" can continue with a different third name.

2. The engine rewrites your question into search queries

Search-enabled engines don't send your prompt to an index as typed. They generate queries first. In our data, ChatGPT ran 3.26 search queries per answer on average, Perplexity 2.77 and Claude 1.57. Different wording on a second run fetches different pages, and the answer follows the pages.

3. Retrieval results move

The pages behind those queries change between calls: index refreshes, freshness boosts, personalisation-free ranking ties. Gemini's grounding is Google Search (how grounding works), so small ranking shifts change which pages it reads. Gemini's API returned no query list through our provider, so its query count isn't shown.

4. The answer is written from the retrieved set

Even with identical sources, the model chooses what to quote. That explains why Claude, which names 1.40 tracked brands per unbranded answer, drifts more than ChatGPT, which names 0.54: more names means more chances for one to change.

Where Drift Is Worst: Buyer Intent

We split pairs by the intent stage the tracker assigns to each prompt. "Same brands" includes empty pairs, so read it as the overall chance that a re-run tells the same story.

Learn-stage questions are stable mainly because they rarely name vendors at all. Consider-stage questions are where engines build shortlists, and they're the least stable. Unfortunately, that's also where buyers form their shortlist.

Measure Answer Drift on Your Own Brand

You need two runs of the same prompts, saved as CSV with one row per answer. The script below is standard-library Python. It compares the runs per engine and prints the same table as above.

#!/usr/bin/env python3
"""Measure how much AI answers drift between two runs of the same prompts.

Usage:
  python3 ai_answer_drift.py answers.csv --runs run1 run2 --brand "Acme" --competitor "Globex" ...
answers.csv columns: run, engine, prompt_id, answer, cited_urls (space-separated URLs).
"""
import argparse, csv, re, statistics
from collections import defaultdict
from urllib.parse import urlparse

def names_in(text, names):
    found = []
    for n in names:
        m = re.search(r"(?<![\w-])" + re.escape(n) + r"(?![\w-])", text, re.I)
        if m:
            found.append((m.start(), n))
    return [n for _, n in sorted(found)]

def domains(urls):
    out = set()
    for u in urls.split():
        host = (urlparse(u).hostname or "").lower()
        out.add(host[4:] if host.startswith("www.") else host)
    return out - {""}

def jaccard(a, b):
    return len(a & b) / len(a | b) if a | b else 1.0

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("csv"); ap.add_argument("--runs", nargs=2, required=True)
    ap.add_argument("--brand", required=True); ap.add_argument("--competitor", action="append", default=[])
    a = ap.parse_args()
    names = [a.brand, *a.competitor]
    by = defaultdict(dict)
    with open(a.csv, newline="", encoding="utf-8") as f:
        for r in csv.DictReader(f):
            if r["run"] in a.runs:
                by[(r["engine"], r["prompt_id"])][r["run"]] = r
    stats = defaultdict(lambda: {"pairs": 0, "same_set": 0, "same_first": 0, "brand_flip": 0, "jac": []})
    for (engine, _pid), runs in by.items():
        if len(runs) != 2:
            continue
        x, y = runs[a.runs[0]], runs[a.runs[1]]
        nx, ny = names_in(x["answer"], names), names_in(y["answer"], names)
        s = stats[engine]
        s["pairs"] += 1
        s["same_set"] += set(nx) == set(ny)
        s["same_first"] += (nx[:1] == ny[:1])
        s["brand_flip"] += (a.brand in nx) != (a.brand in ny)
        dx, dy = domains(x["cited_urls"]), domains(y["cited_urls"])
        if dx or dy:
            s["jac"].append(jaccard(dx, dy))
    print(f"{'engine':<12}{'pairs':>6}{'same brands':>13}{'same #1':>9}{'brand flips':>13}{'citation overlap':>18}")
    for engine, s in sorted(stats.items()):
        p = s["pairs"]
        med = statistics.median(s["jac"]) if s["jac"] else float("nan")
        print(f"{engine:<12}{p:>6}{100 * s['same_set'] / p:>12.0f}%{100 * s['same_first'] / p:>8.0f}%"
              f"{s['brand_flip']:>13}{med:>17.2f}")

if __name__ == "__main__":
    main()

Run it with your brand first and each competitor after it:

python3 ai_answer_drift.py answers.csv --runs run1 run2 \
  --brand "Acme" --competitor "Globex" --competitor "Initech" --competitor "Umbrella"

A few notes on the matching:

• The look-around pattern (?<![\w-])…(?![\w-]) works like a word boundary that also respects hyphens, so "Acme" doesn't match inside "Acme-like". Python's re module docs cover look-behind syntax.
• Citation overlap uses the Jaccard index: shared domains divided by all domains seen in either run.
• www. is stripped so www.example.com and example.com count as one source.

What Drift Means for Your Reporting

Drift isn't a reason to stop measuring. It's a reason to measure the way pollsters do: many samples, pooled, with an error bar.

How many samples you need, and how often to run them, is a sample-size question. We worked it out in how often to track AI visibility. The short version: the number of answers matters more than the calendar.

If you track citations, apply the same rule. The AI citation tracking guide covers which citation metrics to keep. Given that only 20% of pairs cited identical sources, report citation rates, not citation lists.

Drift or Real Change? How to Tell Them Apart

Every tracking report eventually shows a drop, and someone asks whether it's real. Drift and real change leave different fingerprints:

Two checks settle most cases. First, put the old and new rates through an interval: if they overlap, you don't have evidence of a change yet. Second, open the cited sources of the answers that flipped. A real loss usually traces back to a source: a listicle that dropped you, a page you removed, a competitor's new comparison page. Our guide to how AI answer engines pick sources explains why one page can move several answers at once.

How BugViso Handles Answer Drift

The AI Visibility Tracker is built around the assumption that any single answer is noisy:

• A fixed, versioned prompt set. Each brand gets a set of buyer prompts (50 by default), drafted from its own site and reviewed before tracking starts. Every run asks every prompt on every engine, so samples stay comparable.
• Raw answers are stored. Every answer keeps its text, citations and search queries, so you can open the exact answer behind any number.
• Pooled weekly scores. The weekly wrap-up pools all daily runs of the week on the same prompt version. A prompt counts as won on an engine only when the brand appears in at least 50% of that week's samples, which absorbs a single flip.
• Uncertainty is shown. Pooled visibility carries a 95% Wilson interval, and any segment (engine, intent, topic) with fewer than 20 answers is flagged low confidence instead of being presented as a trend.
• Branded prompts are scored separately, so prompts that name the brand by construction don't inflate visibility.

You can see the per-engine scorecard, the interval and the answers behind it on the AI visibility tracker page.

Traps and Edge Cases

Counting empty pairs as agreement. Two answers that name no tracked brand are "identical" but tell you nothing. Report consistency on non-empty pairs, as in the second table.

Changing the prompt between runs. Even a one-word edit changes the search queries. Any edit starts a new prompt version, and versions must never be compared as if they were the same instrument.

Matching only the brand name. Engines also write your domain ("acme.com") and product names. Missing an alias reads as a flip that never happened. Our brand mention detection guide covers aliases and overlapping names.

Treating refusals as misses. An empty or refused answer isn't a lost mention. Exclude it from the denominator, or a provider outage will look like a visibility crash.

Running one engine at a time, hours apart. Index changes then mix with drift. Run all engines in the same window so differences between engines aren't timing artefacts.

FAQ

Are ChatGPT answers the same every time?

No. In our test, ChatGPT named the same set of tracked brands in 76% of repeat pairs. Excluding answers that named none, the figure was 59%. Its first-named brand matched 78% of the time.

Which AI engine gives the most consistent answers?

Perplexity, narrowly. It kept the same brand set in 80% of all pairs and 68% of non-empty ones. Claude was least consistent at 49% of non-empty pairs, partly because it names more brands per answer.

How long does it take for an AI answer to change?

In our test, 23 minutes was enough to change 40% of shortlists that named a tracked brand. The changes come from sampling and live search, not from the web changing, so they can happen on any re-ask.

Does turning on web search make answers more consistent?

It changes the cause of drift rather than removing it. With search on, the engine's query rewriting and changing results add variation on top of model sampling. All four engines here had search on.

How many times should I ask the same prompt?

Enough to put an interval around the rate you report. Our sample-size post shows that a 10% mention rate measured on 200 answers is ±4.2 points. Over a pooled week it's ±1.6 points.

Conclusion

AI answers drift enough that a single run can't tell you whether a brand is visible, so measure with a fixed prompt set, pool many answers and report intervals, which is how BugViso's AI visibility tracking turns noisy answers into a trend you can defend.