Measuring AI Search Answers Without Fooling Yourself: Sampling, Drift and Evidence Trails for LLM-Answer Data Collection

A single AI answer tells you almost nothing about whether your brand shows up in AI search, which is why AI search answer data needs sampling before it needs a dashboard. Ask the same question three times and you get three different sets of sources. In the tracking tests Rankability cites, only 9.2% of URLs repeated across three identical AI Mode runs, and over 60% of domains differed between responses (Rankability).

So if your dashboard says "cited: yes" or "cited: no", you are probably looking at one draw from a noisy process. This post covers how to collect AI search answer data so the numbers survive scrutiny: how to sample, how to control what you can, and what to store so every figure can be checked later.

Why is AI search answer data so unstable?

Classic rank tracking works because a results page is fairly stable for a fixed query, location and device. Generative answers are not. Google’s own documentation says AI Mode may use a "query fan-out" technique, issuing multiple related searches across subtopics and data sources while it builds a response, and that AI Mode and AI Overviews use different models and techniques, so "the set of responses and links they show will vary" (Google Search Central). The page does not describe separate AI Mode reporting: sites shown there are counted in the Performance report under the "Web" search type, so Search Console won’t give you this data either.

Practitioners add that AI Mode behaves more like a dialogue than a page: each follow-up can change the answer set, the cited domains and the brands that appear (Rankability).

The variability isn’t only a Google quirk. Atil et al. tested five LLMs on eight tasks over 10 runs with settings meant to be deterministic and still saw accuracy vary by up to 15% across naturally occurring runs; none of the models was consistently repeatable across all tasks (arXiv:2408.04667). That study measures task accuracy, not search citations, so read it as evidence that repeat runs differ even when you try to hold everything still, not as a figure for AI Mode.

Two things follow. First, a yes/no flag per prompt throws away the very thing you need, which is how often something appears. Second, a monthly check of five prompts is closer to an anecdote than a dataset.

A useful mental model: treat each prompt as a small experiment with a hidden appearance probability. Your job is to estimate it, with an honest sense of the error bar.

What should you control before you sample?

Rankability’s guidance is blunt: "uncontrolled settings create false volatility". If part of the movement you see comes from your own setup, you can’t tell it from real change in the engine.

Fix and log these for every run:

• Location and language
• Device type
• Logged-in or logged-out state (logged out where possible)
• Session state, for example a fresh incognito session

The same source recommends incognito mode and keeping location, language and device constant across runs (Rankability). If you run collection through browsers or proxies, this also means pinning the egress region and browser profile in config, not leaving them to whatever the pool hands you. Locale matters for another reason: it changes who your numbers describe. Results from one country say nothing about another.

How do you sample: repeated runs and follow-up turns?

Run each prompt several times and record every run. Then add one or two realistic follow-up prompts, because follow-ups often surface extra brands, deeper comparisons and citations that don’t appear in the first response (Rankability).

A simple design:

Element	Suggestion
Prompt set	20-30 targeted prompts as a baseline (the figure Rankability gives for monthly manual audits)
Runs per prompt	Several repeats, more for prompts that flip between runs
Turns	Base prompt plus one or two follow-ups
Schedule	Same cadence, same time window, same settings
Output	Appearance rate with run count, for example 7 of 10

The run counts above are our suggestion, not a figure from a source, and you should adjust them against your own variance. The point is to report "appeared in 7 of 10 runs" rather than "appears".

Here is a minimal record and stability calculation in Python:

from dataclasses import dataclass, field
from datetime import datetime, timezone
@dataclass
class AnswerRun:
prompt: str
turn: int                  # 0 = base prompt, 1.. = follow-ups
engine: str
geo: str
language: str
device: str
logged_in: bool
answer_text: str           # full text, not a cropped excerpt
cited_urls: list[str] = field(default_factory=list)
captured_at: str = field(
default_factory=lambda: datetime.now(timezone.utc).isoformat()
)
def url_repeat_rate(runs: list[AnswerRun]) -> float:
"""Share of cited URLs seen in more than one run of the same prompt."""
seen: dict[str, int] = {}
for r in runs:
for u in set(r.cited_urls):
seen[u] = seen.get(u, 0) + 1
if not seen:
return 0.0
return sum(1 for n in seen.values() if n > 1) / len(seen)
def appearance_rate(runs: list[AnswerRun], brand: str) -> tuple[int, int]:
hits = sum(1 for r in runs if brand.lower() in r.answer_text.lower())
return hits, len(runs)

Track url_repeat_rate per prompt over time. If it sits near the 9.2% figure from the cited tests, individual URLs are too unstable to report. Domains and brand mentions are steadier units, though even domains differed in over 60% of cases in those tests, so test your own variance before choosing the unit.

What belongs in the evidence trail?

If someone challenges a number, you should be able to show the answer that produced it. Rankability recommends saving the prompt, full text, screenshots and all cited URLs, and logging geo, language, device and session conditions with each run (Rankability). Full text matters because mentions often appear mid-answer or late, so a cropped screenshot of the opening lines can miss them.

In pipeline terms, this is the same discipline as any other collection job: keep the raw capture immutable, derive metrics from it, and version the extraction logic that turns answer text into "brand mentioned" or "domain cited". When your brand-matching rule changes, you can recompute history instead of arguing about it. It is the same reasoning behind change detection in incremental crawling: store what you saw, then decide what it means.

How do you validate the tools you buy?

Many teams buy monitoring rather than build it. That is reasonable, but treat the output as a claim to be tested. Rankability advises preferring tools that capture the front-end answer over partial API output, and spot-checking tool results against manual runs each month (Rankability).

Ask any vendor, including us, the same questions: how many runs per prompt, which locale and device, are follow-ups captured, and can you export raw answers with cited URLs? If the raw answers can’t be exported, you can’t audit the metric.

Also be careful with headline statistics in this area. When we opened Ahrefs’ blog index, it listed headlines such as a study of 15 million ChatGPT queries and AI Overviews appearing for 83% of branded searches, but the index page gives no methodology (Ahrefs blog). We did not read the underlying studies, so we don’t rely on those numbers here. That is a good habit generally: a percentage without a sampling description is a headline, not a measurement.

Where does this fit with the rest of your web data stack?

AI answer capture is one more source with unstable output and unclear access terms. We did not find sources on the terms of service or rate limits of the engines involved, so check those for each one before building collection at scale. Decide per source whether an official route, a search API or your own collection is the right fit; our posts on a hybrid sourcing strategy and on search API, crawl API or your own fleet cover that choice.

Conclusion

Measuring AI answers is a sampling problem with an auditing requirement attached. Run prompts repeatedly, hold the environment constant, include follow-up turns, report rates with run counts, and keep the full raw capture. Then your charts show how the engine behaves rather than how noisy your setup was. As of October 2026 the sources we reviewed are mostly practitioner and vendor write-ups, so validate any method against your own variance before basing decisions on it.

If your team needs AI-answer or SERP-style data with the collection dates, source list and legal basis documented for each record, that is how Dataizi datasets are delivered. See our datasets page or talk to us at dataizi.net/contact.

Sources

• How to Track Brand Mentions in Google AI Mode (2026 Guide), Rankability, opened 2026-10-28
• AI features and your website, Google Search Central, opened 2026-10-28
• Non-Determinism of "Deterministic" LLM Settings, Atil et al., arXiv:2408.04667, opened 2026-10-28
• Ahrefs blog index, opened 2026-10-28 (headlines only; no methodology)