AI Visibility API vs Scraping: Which Measurement to Trust

Track AI visibility through the engines' official APIs with web search on, and use manual checks in the consumer apps to calibrate, not to measure. APIs give repeatable, permitted, structured answers with citations at the volume tracking needs. Scraping chat apps gets closer to what one signed-in user sees, but most terms forbid it, it breaks when the interface changes, and one user's view is a sample of one.

Neither method sees "the" answer, because there isn't one: in our data, the URLs an engine cited for the same prompt 23 minutes apart overlapped by a median of 14% to 29%. By the end of this post you'll know what each method captures, the documented gaps between API and app answers, what our own API runs showed per engine, and how to describe the method honestly in a client report. A script checks the health of your own API answers.

Data used: 400 answers collected through official APIs with web search on (50 buyer prompts × ChatGPT, Gemini, Perplexity and Claude × 2 runs on 2026-10-20, 23 minutes apart) by BugViso's AI Visibility Tracker for bugviso.com. Gemini was called through a gateway that doesn't return its search queries. Vendor behaviour comes from each provider's documentation, linked below, as of October 2026.

AI Visibility API vs Scraping, Side by Side

An API call asks the model directly, with tools and parameters you set; scraping drives the consumer app as a browser would and reads the page. They differ on almost every property that matters for tracking:

The terms point is not a footnote. Anthropic's consumer terms, for example, prohibit accessing its services through automated means except via an API key, and prohibit crawling or scraping them. Read the terms of every app you'd automate before you build anything.

What Official APIs Expose

APIs return the answer plus the evidence: which searches ran and which sources were used. Each engine does it differently, and the differences shape what you can measure:

• ChatGPT (OpenAI Responses API): the web search tool returns inline URL citations and the search actions, and accepts an approximate user_location.
• Claude (Anthropic Messages API): the web search tool returns cited results, takes a max_uses limit on searches and an approximate user_location.
• Gemini: grounding with Google Search returns grounding chunks and the queries it ran; source URLs arrive as redirect links that need resolving.
• Perplexity: the Agent API's web search tool returns the search results it used, which can be the citation list itself.

Our post on parsing AI citations covers each format in detail. The practical point is that an API tells you why an answer looks the way it does. A scraped page shows the result and hides the process.

What 400 API Answers Showed

Here's the health table from the script below, run on our two runs:

Three findings change how to read any AI visibility number.

The model decides whether to search

With search offered as a tool, Claude ran no search on 17 of 100 prompts and answered from its training data, with no citations on 18. ChatGPT skipped search on 3, Perplexity on 4. The same choice happens inside the apps; there you just can't see it. An answer with no search can't cite your new page, however good it is, which is part of why small brands fare worse on some engines, as we found when looking at why Claude names big brands.

Citation counts measure the format, not the engine's opinion

Perplexity attached 14.4 sources per answer because its API returns the search results it used; 96 of its 100 answers took their citations from that list rather than from inline markers. ChatGPT cited 2.1 per answer. Comparing raw citation counts across engines compares output formats.

Answers vary run to run on both methods

The median overlap of cited URLs between two runs of the same prompt, 23 minutes apart, ranged from 0.14 on ChatGPT to 0.29 on Gemini. That instability is the engine, not the API. A scraped screenshot is one draw from the same distribution, which is why a single app check proves nothing about visibility. Our study of AI answer consistency measured how much the brands named change too.

Where API and App Answers Can Differ

API and app answers come from the same model families but not always the same setup. The documented and structural reasons:

1. Model routing. An app may route a question to a different model or mode than the one you call through the API.
2. Memory and history. Apps can use account memory and earlier messages. API calls are stateless unless you send history.
3. Location and language. The API takes an approximate location you choose; the app infers it from the user.
4. Search settings. The API lets you limit or configure searches; the app's settings aren't yours to set.
5. Interface elements. Product cards, maps and images in an app have no equivalent in API text.

None of these makes the app the "true" answer. It makes it one user's answer. For tracking, you want a fixed, documented setup that measures the same thing every day, then a periodic manual check to confirm that what the API says matches what a buyer sees in broad strokes.

💡 Rule of thumb: measure with APIs, calibrate with people. Once a month, have someone ask five tracked prompts in each app, signed out or in a fresh account, and compare the brands named with the API answers from the same week.

How to Describe the Method in a Report

State the method in one paragraph, in plain words, before any number. A client who understands it won't be surprised when their own app check differs. Suggested wording:

We ask each engine your buyer questions through its official API with web search turned on, set to your market. Answers vary from run to run and can differ from what a signed-in user sees in the app, because apps may use memory, a different model or the user's location. We report rates over many answers rather than single replies.

Then report what you can defend: mention rates with intervals, pooled over runs, with the number of answers behind each.

Check the Health of Your API Answers

This script reads an answers export and prints, per engine: answers, how often the model ran no web search, how often it cited nothing, citations per answer, median latency, and the overlap of cited URLs between the first two runs of each prompt. It flags engines whose API never reports search queries. Standard library only.

#!/usr/bin/env python3
"""Health check for AI answers collected through official APIs: per engine, how often the model
skipped web search, answered with no citations, how many sources it cited, how long calls took,
and how much the cited URLs overlap when the same prompt is asked again.

Usage: python3 api_answer_health.py answers.csv
answers.csv columns: run, engine, prompt_id, cited_urls (space-separated),
                     search_queries (count, optional), latency_ms (optional).
"""
import csv, sys
from collections import defaultdict
from statistics import median

def main():
    if len(sys.argv) != 2:
        sys.exit(__doc__)
    rows = list(csv.DictReader(open(sys.argv[1], newline="", encoding="utf-8")))
    by = defaultdict(list)
    for r in rows:
        by[r["engine"]].append(r)
    print(f"{'engine':<12}{'answers':>8}{'no search':>11}{'no cites':>9}{'cites/ans':>10}{'median s':>9}{'URL overlap':>12}")
    for e, rs in sorted(by.items()):
        n = len(rs)
        q = [int(r["search_queries"]) for r in rs if r.get("search_queries", "").isdigit()]
        cites = [len(r["cited_urls"].split()) for r in rs]
        no_search = f"{sum(x == 0 for x in q)}" if q else "n/a"
        if q and not any(q) and any(cites):  # citations but never a query: the API doesn't report them
            no_search = "unreported"
        lat = [int(r["latency_ms"]) / 1000 for r in rs if r.get("latency_ms", "").isdigit()]
        runs = defaultdict(dict)  # prompt -> run -> set of URLs
        for r in rs:
            runs[r["prompt_id"]][r["run"]] = {u.split("?")[0].rstrip("/") for u in r["cited_urls"].split()}
        overlaps = []
        for d in runs.values():
            sets = list(d.values())
            if len(sets) >= 2 and (sets[0] | sets[1]):
                overlaps.append(len(sets[0] & sets[1]) / len(sets[0] | sets[1]))
        ov = f"{median(overlaps):.2f}" if overlaps else "n/a"
        print(f"{e:<12}{n:>8}{no_search:>11}{sum(c == 0 for c in cites):>9}{sum(cites) / n:>10.1f}"
              f"{(median(lat) if lat else float('nan')):>9.1f}{ov:>12}")
    print("\nno search = answers where the model ran no web search (answered from training data);"
          "\nURL overlap = median Jaccard of cited URLs between the first two runs of the same prompt.")

if __name__ == "__main__":
    main()

On our export:

engine       answers  no search no cites cites/ans median s URL overlap
chatgpt          100          3        3       2.1      8.5        0.14
claude           100         17       18       4.1      9.7        0.25
gemini           100 unreported       11       4.6      5.8        0.29
perplexity       100          4        4      14.4      8.7        0.20

URLs are compared without query strings, because some engines add tracking parameters to cited links. A "no search" rate that jumps between runs usually means a model or tool change on the provider's side; check the provider's changelog before blaming your prompts.

How BugViso Collects AI Answers

BugViso's AI Visibility Tracker asks ChatGPT, Gemini, Perplexity and Claude through their official APIs with web search enabled, sets each brand's country as the location where the API supports it, and stores every answer with its citations and, where the API returns them, the searches the engine ran. Google AI Overviews has no answer API, so the tracker reads the overview from a search-results data provider for the brand's country and language, and records "no overview" when Google shows none. An engine that fails is recorded as failed; another engine's answer is never substituted.

Every report includes method notes and run health: prompts, engines, runs, usable answers, failed or skipped calls, and a note that API answers can differ from what users see in the apps. The method is described on the AI visibility tracker page.

Method Mistakes to Avoid

Treating an app screenshot as ground truth. It's one answer, for one account, at one moment. Compare it with rates, not with a single API reply.

Scraping without reading the terms. Automated access to consumer apps is often prohibited, and accounts used for it can be suspended mid-contract.

Comparing engines on raw citation counts. Perplexity's 14.4 and ChatGPT's 2.1 sources per answer reflect output formats more than how much each engine trusts the web.

Ignoring no-search answers. If an engine answers 17% of prompts without searching, fresh content can't reach those answers. Report the rate per engine.

Changing the API setup mid-period. A new model or search setting is a new instrument. Note the change and start a new baseline, as with any prompt set version.

Frequently Asked Questions

Are AI visibility tools accurate if they use APIs instead of ChatGPT itself?

They're accurate about what they measure: answers from the engine's API with web search on, under a fixed setup. App answers can differ because of memory, model routing and location. Good tools say so in their method notes and report rates over many answers, which hold up better than any single reply.

Is it legal to scrape ChatGPT or Claude for brand tracking?

It's a terms-of-service question first. Anthropic's consumer terms, for example, prohibit automated access except via an API key. Check each provider's terms; using the official APIs avoids the issue.

Why does my brand appear in ChatGPT but not in my tracking report?

Your app session may use memory, history or a different model, and any single answer varies. Ask the same prompt several times in a fresh session and compare with the tracked mention rate for that prompt over several runs.

Do AI APIs use web search like the apps do?

Only when you enable the search tool, and even then the model decides whether to search. In our runs, Claude ran no search on 17 of 100 prompts; ChatGPT on 3 and Perplexity on 4.

How should I check AI visibility manually?

Once a month, ask five tracked prompts in each app in a signed-out or fresh session, note the brands named and the sources shown, and compare with the API results from the same week. Use it to sanity-check the trend, not to replace it.

Measure With APIs, Calibrate With People

Official APIs give you the repeatable, permitted, citation-rich answers that tracking needs; a monthly manual check in the apps keeps that measurement honest. BugViso's AI visibility tracking uses the official APIs with web search on and states the API-versus-app caveat in every report.