Measure AI visibility without fooling yourself (2026)

Why a single AI answer is noise

A single AI answer is noise because the same prompt returns a different set of brands and sources almost every time. Across independent studies, two runs of the same prompt share roughly 40–60% of the brands they name and 25–50% of the sources they cite (strong evidence: several studies agree).

SparkToro and Gumshoe (Jan 2026, 2,961 runs on ChatGPT, Claude and Google's AI features): the same brand list came back less than 1% of the time, and the same list in the same order about 0.1% of the time.
Don't Measure Once (arXiv, Apr 2026, 45 days, four engines): same-day reruns shared 38–49% of brands by Jaccard similarity. Day-to-day change was about as large as same-day randomness, so most apparent movement was sampling noise.
Ahrefs (Oct 2026, 20 days, five tools): about 60% of ChatGPT and AI Mode citations were gone the next day, and only 4.0–6.8% lasted all 20 days on AI Overviews, AI Mode and ChatGPT.

Setting temperature to zero does not fix it. Hosted models give different outputs for identical requests because server load changes how the computation is batched: 1,000 identical completions produced 80 distinct outputs (Thinking Machines, Sep 2025, confirmed). The noise is built into the systems, so the method has to absorb it.

Mention rate pooled over many runs is the unit to report

The defensible unit of AI visibility is mention rate: the share of answers that name the brand, pooled over many runs, prompts and days. Individual answers churn, but the rate is fairly stable. In the SparkToro headphones test, real people wrote 142 very different prompts, yet the top brands each appeared in 55–77% of 994 responses.

The same logic applies to recommendation rate (answers that actively suggest the brand) and citation coverage (answers that cite the brand's own pages). Report each with its sample size and a confidence range. For citations, the site or domain is the stable unit. Individual URLs churn too fast to be more than diagnostics.

optmz reports mention, recommendation and citation rates with the sample size and a 95% range next to each number.

How many samples you need: AI visibility sample size and confidence ranges

You need about 100 answers for a range of ±10 points and about 400 for ±5 points, at a mention rate near 50%. That is standard binomial math, set out for AI tracking by Peec AI (Sep 2026): the 95% half-width is 1.96 × √(p(1−p)/n).

For small samples, and for rates near 0% or 100%, use a Wilson interval instead of that simple formula. The table shows 95% Wilson ranges we computed for a few cases.

Answers (n)	Observed rate 50%	Observed rate 20%	Observed rate 0%
10	24–76%	6–51%	0–28%
30	33–67%	10–37%	0–11%
100	40–60%	13–29%	0–4%
400	45–55%	16–24%	0–1%

Three rules follow from the math.

Detecting a change needs about twice the runs of measuring a level (Peec AI, 2026).
For a portfolio number, adding prompts helps more than adding runs per prompt. For detecting change, total runs are what count.
Per-prompt daily numbers from 1–5 runs are noise. Pool across prompts, runs and days. Don't Measure Once found the standard error of a brand's detection rate fell below 0.05 only after about 24 days of daily runs.

Why rank position in AI answers is meaningless

Rank position in an AI answer is close to meaningless because the order of brands is not reproducible. SparkToro found the same ordered list less than 0.1% of the time and described rank position as “close to meaningless”. List length also varies from two or three brands to more than ten, so “position 3” means different things in different answers.

Causal tests point the same way. In CITECHOICE (arXiv 2026), raw correlations between position and citation overstated the causal effect of position by about 5x. Report whether the brand was mentioned or recommended, not where it appeared. An average position hides the real signal, which is how often the brand shows up at all.

API vs app: two different AI surfaces, not a noisy copy

AI visibility measured through an API is a different surface from the consumer app, not a noisy version of it. Brand overlap between the two is about 20–30% and page overlap about 5–25% (strong evidence across studies).

ChatGPT app vs the Responses API with web search: the API's full source list covered about 23% of the pages the app cited (NUS and Xi'an Jiaotong, arXiv, Oct 2026). The app never searched with the user's exact wording.
API vs scraped app answers: 21–32% brand overlap after normalizing names, across 1,000 prompts (Surfer with cloro, Oct 2026; a vendor study using single runs). ChatGPT's API answers named more brands and cited fewer sources than the app.
Perplexity's own FAQ says its API uses the same search system with differences in configuration.

The main driver is probably the consumer system prompt and settings, which the API doesn't reproduce. Reasoning effort matters too: on one model, high reasoning raised the citation rate from 50% to 68% (Growth Memo, plausible). Whether API trends follow app trends over time is untested. Label API data as a proxy, record the model and settings, and check it against a sample of real app answers. The glossary entry for API vs app has the short version.

Google's AI surfaces need their own sampling too. AI Overviews had a 70% chance of changing between observations of the same keyword, with 45.5% of citations turning over per change, yet the meaning of the answer stayed almost the same (semantic similarity 0.95; Ahrefs, 43k keywords, Nov 2025, confirmed). AI Mode and AI Overviews cite different sources about 87% of the time (Ahrefs, strong). The Gemini app grounds only some answers in search: 41% in July 2026 and about 68–71% from August to October (cloro, plausible). Treat each surface as its own measurement, not as one Google number.

Paraphrases, language and country change AI answers more than reruns

Rewording a prompt moves AI answers more than rerunning it. In API tests, reruns of the same prompt shared 50–61% of recommended brands, cosmetic rewrites only 29%, and rewrites that added constraints such as a region only 14% (arXiv 2605.27440, May 2026, vendor authors, confirmed from the paper).

So a single prompt is a fragile thing to track. Group 3–10 wordings of the same buyer question into a paraphrase group and report the group's rate. Keep the wording fixed over time, so that changes in the data are not changes in the question.

Language and country are large effects. Query language explained 26.5% of response variance against 1.5% for brand identity across 12,933 answers, and the country a request came from shifted which country's brands appeared, independently of language (Żatuchin, arXiv 2026, small samples, strong). Treat language and country as dimensions of every measurement.

Personalization is smaller but real. History-primed accounts in the UK showed about 6.6 points more UK and EU brands (Clark and da Silva, 2026, press release only, plausible). Logged-out and API data can't see memory effects, so call that data an unpersonalized baseline.

Before vs after: every AI visibility test needs a control group

A before/after comparison of AI visibility without a control group mostly measures platform drift. Engines change models, citation volumes and favored sources every month. Ahrefs' schema study is the standard warning: AI Mode citations rose 43% after pages added schema, but matched control pages rose almost as much, and the net effect was about +2% and not significant (Ahrefs, 1,885 vs 4,000 pages, May 2026, confirmed).

The fix is a difference-in-differences test: net lift equals the change in the treated prompts or pages minus the change in comparable untouched ones. Keep the prompt set, the engine version and the locale fixed for the whole test, and note model changes, which can happen mid-series without warning. The methodology section on control groups shows how optmz sets them up.

Rolling windows and a minimum-data rule for AI visibility reports

Rolling windows and a minimum-data rule keep AI visibility reports from reacting to noise. A rolling window pools the last two to four weeks of answers into one rate, so each report rests on enough data and one odd day can't swing it. Don't Measure Once recommends 2–4 week windows.

A minimum-data rule says when not to show a number. Below a threshold, the confidence range is so wide that any single figure misleads: at 10 answers, an observed 50% is consistent with anything from 24% to 76%. Show “collecting data” instead, and never declare zero visibility from a handful of runs. One vendor analysis suggests about 60 runs before “zero” is defensible (Thrivestack, plausible).

optmz uses rolling 4-week windows and shows “Collecting” when a cell has fewer than 20 answers, as described in the methodology section on low data.

How AI visibility tools sample in 2026

Most daily AI visibility trackers collect one answer per prompt per engine per day, judging by their own documentation and published reviews. The pattern is strong across tools, though the detail for any single tool is plausible at best. One answer a day is below every statistical recommendation in the research, which ranges from 7 or more runs per prompt per day to 60–100 runs pooled.

One answer a day is not useless. Pooled over many prompts and a month, it can give a usable portfolio rate: Peec AI's worked example puts 50 prompts run once a day at about ±6 points on a monthly rate. The problem is per-prompt daily charts, rank positions and week-to-week movement read from single runs.

Tools also differ in what they measure. Some scrape the consumer apps, some use APIs, some mix both, and at least one large index re-tests chatbots monthly on synthetic prompts and calls the result potential exposure. When you compare tools, ask four questions.

How many answers per prompt per engine, per day and per reporting period?
API or app, logged in or out, which country and which language?
Are sample sizes and confidence ranges shown next to each number?
Are the prompts real buyer questions, and can you see them?

The prompt set deserves the same scrutiny as the sampling. In OpenAI's own study of 1.1M ChatGPT messages, only 2.1% were about purchasable products (OpenAI and NBER, Sep 2025, confirmed). Commercial “best X” prompts, which most trackers sample, are a small slice of real use, so a brand's exposure inside how-to and informational answers may matter more than the tracked prompts suggest. That last step is an inference, not a measured result.

Checklist: measuring AI visibility without fooling yourself

Measuring AI visibility honestly comes down to a short list of habits.

Write 10–30 buyer questions, each as a paraphrase group of 3–10 wordings, and freeze them.
Set the language and country for every question.
Collect several answers per question per engine each week, and pool them into rolling 2–4 week windows.
Report mention, recommendation and citation rates with the sample size and a 95% range. Drop rank position.
Show “collecting data” below a minimum sample instead of a number.
Label every number with its method: API or app, logged out, model version.
Mark model changes on charts.
Test every fix against control questions or pages, and report the net change.
Read citations at the site level and treat single URLs as diagnostics.

Sources