DeepSmith

Nov 26 · AEO & AI Visibility

19 min read

Why AI Answers Change From Run to Run and How Trackers Handle It

Avinash Saurabh
Avinash Saurabh · CO-Founder & CEO
Dark monochrome illustration of one search box fanning out to several slightly different answer cards, with the cover line Why AI Answers Keep Changing.

If you have asked ChatGPT or Google the same question twice and got two different lists of brands, you have run into what people search for as "AI answers inconsistent", and it is nothing you did wrong. An AI answer can change because the model does not always pick the same words each time, and because search-enabled tools can pull in different material or use different systems on different runs. The practical fix is to run a fixed prompt many times and count how often your brand shows up, which turns one noisy answer into an estimated rate.

This piece walks through why the answers move (the research calls it LLM response variability), what the repeated-run studies actually found, and how a tracker turns all that movement into something you can use. It does not cover differences that come from who is asking or where they are, and it does not set the point at which a change is big enough to act on. Those are separate questions.

Why does the same prompt return different brands?

The short answer is that there are a few different things going on at once, and nobody outside the engines can say how much each one contributes to any single answer. Here is what the documentation and research do support.

The model can choose different words each time

A language model writes an answer one small piece of text at a time, and at each step it picks from several likely options. Temperature is the setting that controls how adventurous that choice is. A higher value lets the model pick more varied continuations, and a lower value keeps it closer to the most probable one. Google's Gemini documentation also describes top-p and top-k, which are two more ways to limit which options the model can pick from.

Those are controls for developers using a model through an API. They are not necessarily something you can see or change in the chat product you type into, so you should not assume every engine exposes them.

It is tempting to think that turning the randomness all the way down gives you the same answer every time. It does not, at least not as a promise. Anthropic says that identical inputs can still produce different outputs across API calls even at temperature zero. Google says a fixed seed makes a best effort to return the same response, but it does not guarantee it. An older OpenAI cookbook example says much the same about its seed setting. That cookbook page is archived, so it makes sense to read it as background and not as a current rule for consumer ChatGPT.

There is also a more technical reason. A NeurIPS 2025 paper on numerical sources of nondeterminism shows that small rounding differences in how a model runs can pile up and change the output, under specific hardware, batch size, and precision conditions. That result comes from a controlled test. It does not tell you why a particular consumer product recommended one brand over another, so treat it as one more reason that identical inputs do not always give identical outputs.

Search-grounded answers can pull in different material

Many AI answers are not written from the model's memory alone. They use retrieval-augmented generation, which means the system looks up relevant material when you ask, and hands it to the model as part of the question. If the material it finds is different from one run to the next, the answer can be different too.

Google describes one version of this for AI Overviews and AI Mode. It says they may use query fan-out, which means running several related searches across subtopics and data sources to build one answer. Google also says the two features may use different models and techniques, so the responses and the links can vary. That is a documented mechanism for those two Google features, and it would be a stretch to say every AI engine works the same way.

The research we reviewed does not say how much of any one brand's appearance or absence comes from retrieval and how much comes from the model's word choices. It is fair to say both can matter. It is not fair to say which one mattered for your brand today.

Mentions and citations are two different things

One more thing is worth keeping straight before we go on. A brand mention means the answer names your company. A citation means the answer links to a page as a source. You can have one without the other: a brand can be named with no link to its site, and a page can be cited without the brand being named in the text. Ahrefs' Brand Radar documentation also separates sources an AI feature found from sources it visibly cites. Our AEO glossary defines both terms, and our explainer on AI brand mentions covers linked and unlinked mentions. For measurement, it helps to track mentions and citations as two separate numbers and not use one as a stand-in for the other.

Comparing across days adds more changes

Two runs a few minutes apart tell you about answer-to-answer variation under similar conditions. Runs that are days or weeks apart can also reflect changes in the model setup, the retrieval, the sources available, or how the data was collected. OpenAI's archived reproducibility example describes a backend fingerprint that changes when the serving setup changes, and its current API guidance suggests pinning production apps to a model snapshot when you need consistency. Those are developer tools. They do not mean that the answers a customer sees in a consumer product can be pinned in place.

So when someone says one cause explains all the changes, be a little careful. The studies below measure how much the outputs moved. They do not separate out how much came from word choice, retrieval, or backend changes.

What do repeated-run studies show?

Three public studies on LLM response variability are worth knowing about. They each ran the same prompts again and again and looked at what changed. They used different prompts, time periods, platforms, and definitions of "the same," so the numbers are examples and not a shared benchmark. Please do not line their percentages up side by side as if they measured one thing.

Detailed: more than 70,000 answers over 28 days

Glen Allsopp's Detailed study of AI volatility reports more than 70,000 responses. It checked more than 1,300 prompts every day across ChatGPT and Google AI Overviews for 28 days, and the prompts were mostly commercial questions adapted from traditional search keywords.

For a strict exact-match test, the study reports that randomly picked pairs of days gave an identical set of brands or cited domains less than 3% of the time. It also reports that 41% of ChatGPT prompts and 30% of AI Overview prompts never returned the same brand set twice during the study.

That number is easy to misread. It does not mean a brand had less than a 3% chance of being mentioned. It measures whether the entire set of brands was identical, which is a much stricter test than whether one brand showed up. The study also finds that the top brand for a prompt can appear on most days even when the full list changes. The list can be unstable while one brand's showing up is comparatively steady, and that gap is the reason repeated sampling works at all.

SparkToro: the same recommendation question, many times

SparkToro worked with Patrick O'Donnell of Gumshoe.ai on research into how consistent AI recommendations are. It asked 600 volunteers to run 12 prompts through ChatGPT, Claude, and Google's AI Overview or AI Mode, for 2,961 runs in total, with each prompt run roughly 60 to 100 times. The article text we retrieved does not give a publication date.

The authors report that ChatGPT or Google AI had less than a one-in-100 chance of giving the same brand list in two responses, and that matching the same order was rarer still, around one in 1,000 runs. They also tested 142 human-written headphone prompts and looked at 994 responses. Bose, Sony, Sennheiser, and Apple appeared in 55% to 77% of those responses.

The takeaway is not that every brand has a steady rate. It is that lists can vary a lot and individual brands can still show up with a measurable frequency. The authors call their work non-exhaustive and say more data and larger samples are needed, so it is better read as an illustration than a final estimate. Their whole-list and ordering results also should not be confused with a mention rate for one brand.

Ahrefs: cited pages move more than brand names

Ahrefs' article on how stable AI citations are, dated October 29, 2026, reports tracking about 37,500 citations across five AI tools: Gemini, Copilot, Google AI Mode, ChatGPT, and Google AI Overviews. It asked the same questions daily for 20 days and followed the websites that were cited on the first day to see whether they stayed cited or came back later.

It reports that 50% to 83% of citations disappeared the next day, and that about 75% of those missing citations returned within 20 days. It also reports that about 50% to 70% of cited websites appeared on only one day, compared with about one in five brand names. So a specific cited page or website moved around more than a brand name did. The article also includes a separate analysis of an Ahrefs brand using 22 questions, and the authors caution that those findings may not generalize. The research comes from Ahrefs, and its author is an Ahrefs marketing researcher and educator, so keep that in mind when you weigh it.

What the studies agree on

They measured different things: exact lists, named-brand appearance, and how long citations last. What they share is the practical point that a single answer is a fragile measure. Any one run is a sample, and you should treat it that way.

How does repeated sampling turn answers into a rate?

This is the part that makes tracking work, and it is simpler than it sounds. You pick one prompt, one platform, and one definition of your brand, and you run it many times over a set window. Each answer that comes back successfully counts as one observation.

For each observation you ask one yes or no question: did the brand appear at least once in this answer? If yes, it is a 1. If no, it is a 0. Then the calculation is:

Mention rate = successful answers that mention the brand ÷ all successful answers

Here is a made-up example, not a result from any study. A tracker collects 25 successful answers for one prompt, and the brand is named in 10 of them. The observed mention rate for that prompt on that platform is 10 out of 25, or 40%. If an answer says the brand name four times, it still counts once, because the question is whether the brand was present, and several tracker methodologies work this way.

Citation rate is a separate number. It is the share of eligible answers where the brand's page or domain gets credit as a source. What counts as eligible, and what the denominator is, differs between trackers, so check each one's method. Some AI surfaces also show richer source information than others, which makes it harder to compare across them. If you want the full list of what to track and why, our guide to AI visibility metrics goes through them.

Say how you combine prompts

A rate for one prompt answers a narrow question: how often was the brand present in the answers collected for this prompt, on this platform, in this period? It is not a promise about every user's answer, every possible wording, or next month.

When you track many prompts and platforms, you also have to decide how to combine them, and say what you chose. One published method, from refd, averages the samples inside each prompt and platform pair first, and then gives every eligible pair equal weight. That keeps a pair with more successful samples from quietly outweighing another buyer question. The other option is to pool every answer together, which gives more weight to the pairs that have more samples. The two approaches can give different overall rates because they estimate different things, and neither one is the universal rule.

A tracker should also keep the details so you can look back at them: the prompt set, the platform, the collection date, the answer itself, the brand names and aliases it matched, and the counting rules. Failures should be kept separate from successful answers that did not mention you. If a fetch fails, that is missing data, not evidence that the brand was absent. Some features do not return an overview or answer for every prompt, and a tracker has to decide whether that counts as a coverage outcome or as part of the visibility denominator. Methods differ on that.

Matching the brand name matters too. People abbreviate, and some brand names are also ordinary words. It is better to define the aliases carefully and keep the same rules the whole time. A negative description still counts as a mention, so measure sentiment as its own thing instead of treating a bad mention as no mention.

Why more runs help, and where they stop

If each run behaves like an independent coin flip with a steady chance of naming your brand, then the estimated rate is the number of mentions divided by the number of runs. Call that p̂, so p̂ = k/n, where k is the number of answers that mention the brand and n is the number of successful answers. The usual rough measure of how much that estimate could wobble is the standard error:

SE ≈ √[p̂(1 − p̂) / n]

In the example above, p̂ is 0.40 and n is 25, which gives a standard error of about 9.8 percentage points. With the same estimated rate and 100 independent samples, it drops to about 4.9 percentage points. In general, taking four times as many samples cuts that wobble roughly in half. This is an approximation, and it does not promise your measured rate will land within that many points of the truth.

There are some limits. The formula assumes a fixed probability and independent runs. Repeated calls can be related to each other, and results collected over several days may span backend or retrieval changes, so the process may be changing underneath you. In those cases the formula can understate how uncertain you are. For proportions, it is better to use a proper confidence interval, and NIST's guidance on estimating instrument performance discusses Wilson intervals and warns that common approximate intervals can be unreliable when the sample is small. Also, zero mentions in a small sample is not proof that the real chance is zero.

More runs improve precision for the sample you defined. They cannot fix a badly chosen prompt set, missed brand aliases, inconsistent rules, failed collections, or a gap between how the tracker reaches the engine and what your customers see. We cover that last point in our piece on how AI visibility tools collect data, and the effect of who is asking in our piece on personalization and location. Repeated sampling gives you a usable estimate of AI visibility tracking accuracy you can defend. It does not give you a full picture of all AI visibility.

What can trackers actually establish?

Trackers differ in how often they collect, how many samples they take, how they reach each platform, how they match names, what they use as the denominator, and how they weight prompts. None of this is standardized.

Two published examples show how different the methods can be. Refd says its hosted service takes one sample per prompt and platform in each scheduled run, treats each completed run as an observation, and compares results over time, while operators who want more can request extra samples for focused runs. AppearIn describes weekly scores built from successful answers, with repeated mentions in one answer counted once and share of voice split equally among the distinct direct brands. These are vendor-specific methods and not a standard.

Cadence differs too. OtterlyAI's help page, dated August 5, 2026, describes daily prompt monitoring, says prompts do not run at a fixed hour, and says users cannot manually trigger a run. That is one provider's way of doing it. Daily collection is also not the same as taking many independent samples of one prompt at one moment, so it is worth asking which one a tool is doing.

When you look at a dashboard, these are the questions to ask:

  • Does it show results for each prompt or only pooled results, and are the prompts weighted equally?
  • Does it report mentions, citations, or both, and how is each denominator defined?
  • How does it handle duplicate names, aliases, and negative mentions?
  • How does it treat failed fetches and answers that never appeared?
  • Can you open the original answers and see the collection dates?
  • Were the same prompts and brand definitions used in the periods you are comparing?

If you want to test a tool's numbers yourself, our complete guide to AI visibility measurement walks through auditing and tracking. Our roundup of the best AI visibility tools is a good place to see how the products in this space differ.

How should you read the numbers?

A one-run result is a snapshot, not a verdict. A trend across many repeated observations is more useful, but a trend by itself cannot tell you what caused a change. It could be your content, a model update, a retrieval change, or just noise, and the number alone will not say which.

This piece does not give a threshold for when a change is large enough to act on, because that depends on your sample size, your design, and the decision you are making. If you want to think that through, our piece on alert thresholds for AI visibility changes is the place to start, and our look at day-to-day noise in AI visibility scores covers what that noise means for the scores you see.

Here is a simple way to use all this on a regular week. Pick the ten or so prompts your buyers really ask and keep the wording fixed. Look at the rate over a stretch of weeks, not a single day. Write down how you count a mention, and keep the same rule each time. If a number jumps, look at the saved answers before you decide what it means. It helps to compare against competitors on the same prompts, and the beginner's guide to share of voice explains how that works.

DeepSmith's AI Visibility module runs your tracked prompts on a schedule and reports per-prompt mention and citation rates, with the full answer history kept so you can read the answers behind each number. If you want to see how that looks on your own prompts, you can start a free 7-day trial.

The short version

The changing list is the raw observation. The repeated, consistently scored answer is the measurement. When a tracker explains what it sampled, how it counted a mention or citation, and how it averaged the answers, you can judge AI visibility tracking accuracy for yourself and read its numbers with a clear view of what they do and do not say. No single run is the definitive answer, and that is the main thing to take from this.

Frequently asked questions

Why do AI answers change when I run the same prompt again?

The model can choose different words each time, and search-grounded tools may retrieve different material or use different system setups. The evidence we have does not let anyone assign one cause to every changed answer.

Does setting temperature to zero make an AI answer identical every time?

No. Provider documentation says identical API inputs can still give different outputs at temperature zero, and seed settings are described as best-effort. Consumer chat products also may not let you change these settings at all.

What is an AI brand mention rate?

For one prompt on one platform, it is the share of successful answers that name the brand at least once. It is separate from citation rate, which asks whether a page or domain tied to the brand was cited as a source.

How many runs are enough?

There is no single number in the evidence we reviewed. More independent runs generally reduce sampling error, but how many you need depends on how precise you want to be, how steady the collection process is, and whether you care about one prompt or a whole set of them.