DeepSmith

Nov 26 · AEO & AI Visibility

19 min read

How to Audit the Data Quality of an AI Visibility Tool

Avinash Saurabh
Avinash Saurabh · CO-Founder & CEO
Abstract monochrome diagram of a stored record and a live answer card joined by lines and a checkmark, centered under a short cover line.

If you use an AI visibility tool to report on your brand, at some point someone will ask whether the numbers are right. This guide is for the marketing lead who has to answer that. It walks through a simple audit in eight steps, so you can check AI visibility data quality yourself by sampling prompts, comparing the tool's records with live answers, and repeating a few runs to see how stable the results are. By the end you will have a written method to verify AI tracking results and a short memo you can show anyone who asks.

One thing to say up front. This AI tracker accuracy check is a practical protocol, not an industry standard. We did not find any agreed sample size, accuracy threshold, or metric formula for this, so the audit tells you how well the tool matched what you could observe for the prompts, engines, settings, and dates you tested. It does not prove anything about prompts you did not check.

This guide does not cover how tools collect their data or whether a vendor's marketing claims hold up. Those are separate questions. Here the focus is only on the evidence in the records and in the answers.

Step 1: Define exactly what the audit will test

Start by deciding what "correct" means before you look at anything. Pick the engine or engines, the date range, the tracked prompt set, and the fields you will check. It helps to separate two kinds of checks. Record-level checks ask whether one stored answer, with its mention and citation labels, matches reality. Aggregate checks ask whether a displayed rate, like a mention rate, matches the records underneath it.

Then write down the unit you are testing, which is usually one answer for one prompt, on one engine, in one run. Decide in advance what counts as a brand mention, an alias, a competitor mention, and a citation to your site. A workable rule is that a mention is present when the answer text names the brand or an agreed variant of its name, and a citation is present when a source link in the answer points to an agreed brand domain. Keep the two labels independent, because an answer can mention you without citing your site, cite your site without naming you, do both, or do neither. If the difference is new to you, the piece on AI citations vs brand mentions covers it in more detail.

You are done with this step when you have a short rule sheet that gives the date window, the engine, the prompt version, the labels, and the denominator. A second reviewer should be able to apply the labels from that sheet without guessing.

The most common mistake here is treating any link to your domain as a brand-name mention, or treating a mention as proof that you were cited. They are different events, and a tool can get one right and the other wrong, so you want them scored separately from the start.

Step 2: Freeze a representative prompt sample

Next, build your sample from the prompts the tool already tracks, and choose a manageable subset before you look at any live answers. Choosing first matters, because otherwise it is easy to pick the prompts that confirm what you already think.

Use groups that matter to your business, such as buyer stage or prompt intent. Make sure the subset includes prompts where the tool has reported a brand mention, prompts where it has not, and prompts where competitors have appeared. These are choices about how to design the audit, not a published list of required categories, but they give you a chance to find both kinds of error.

Write each prompt exactly as it is tracked and give it a stable ID. Save the wording and a version number. Don't paraphrase a prompt between the tool record and the live test. If you need to change a prompt, treat it as a new version instead of quietly swapping it in.

It helps to estimate the workload now. As an example of size only, reviewing 20 prompts across two engines gives 40 prompt and engine pairs if you select one stored answer for each pair. Three live repeats for each pair would give 120 live answers. That is a workload illustration, not a recommended sample size, and it says nothing about engines you did not sample.

You are done when the prompt list, the selection method, the IDs, the versions, and the engine list are all saved in one place and not changed afterward.

A common mistake is sampling only the prompts where the tool already reports a win for your brand. That can test the positives, but it can never show you a mention or citation the tool missed, because those would sit among the records labeled negative.

Step 3: Save the tool's records before you run the live check

Before you open a single live answer, export or copy what the tool says for every selected pair. For each one, save the stored answer, the date or timestamp, the mention and citation labels the tool reports, and any cited URLs or source names it shows. Also keep a copy of the exact prompt and the tool's aggregate figures for the same period. If a record is missing or shows an error, mark it that way. Don't quietly drop it, because removing it changes the denominator and makes the tool look better than the evidence supports.

Saving first is what keeps the audit honest. If you look at the live answer first, you will tend to read the tool's record in light of it.

DeepSmith keeps this kind of history for you. It runs your tracked prompts on a schedule, stores the full answers, and tracks mentions and citations separately, with answers viewable by engine and over time. If you work through the MCP server, the get_prompt_answers tool takes a prompt ID and a range, with optional platform and date inputs (dates in YYYY-MM-DD format) and a full input, and returns answers newest first, up to 200 per call. The get_prompt_analytics tool gives one prompt's rates, trends, per-platform results, and the competitors mentioned, and get_visibility_setup shows the brand name, domain, variants, and tracked competitors the tool is using. The 200 figure is a limit on one call, not on how much history exists. The MCP server is read-only, so it is a safe way to pull records for an audit without changing anything in the workspace.

Whatever tool you put through an AI tracker accuracy check, use the interface or documented access path it gives you. Don't assume an export or field exists if the documentation doesn't mention it.

You are done when every selected pair has either a saved record or an explicit missing or error status, and the snapshot includes the date range and the metric labels as the tool displays them.

A frequent mistake is comparing a live answer with a record for a different prompt version or a different date, and then calling every difference a data error. Most of those differences are a mismatch in the test, not in the tool.

Step 4: Run matched live checks with controlled settings

Now enter the frozen prompt wording in the same engine the record came from. For each run, write down the date and time, the locale or region where it applies, whether you were signed in, whether you used a search or answer mode, and any visible personalization setting. Start a fresh conversation for every prompt so earlier turns don't add context. Save the answer word for word, and save the source links exactly as they appear.

Settings matter more than people expect, so here are a few specifics. For ChatGPT, make sure the test really uses web search if you are comparing against a search answer. OpenAI's help page on searching the web with ChatGPT notes that search can use location information, including an approximate location from your IP address, and that saved memory may change how a search query is rewritten. A Temporary Chat can be set to Unpersonalized before the first message, which means it doesn't use memory, custom instructions, or plugins. That setting can't be changed once the conversation starts, and OpenAI notes that limited safety-related context may still be used. So write that down as a limitation instead of promising a result free of any outside influence.

For Google AI Mode, previous searches and activity saved in Search Services History can be used for personalization when history and personalized recommendations are turned on. Google's help page on how AI Mode personalizes responses describes this experience as available in the US and in English, and you can opt in or out through the personalized recommendations setting. Record whether it was on for your test, and don't assume a result from one region holds in another.

For Perplexity, answers include numbered citations that link to the original sources, so capture those links and open them, instead of relying on the summary text alone. If you want more background on why settings like these change what you see, the guide on why AI answers change goes further.

You are done when the live answer, the citations, the prompt, the engine, the timestamp, and the settings are saved for every run.

A common mistake is comparing runs that used different modes, locations, dates, account states, or follow-up context without noting the differences. A mismatch found under mismatched conditions isn't a clean test of whether the record was accurate.

Step 5: Label the live answers on your own

Review each saved live answer and its links without looking at the tool's label first. Apply your rule sheet. For each answer, record whether the brand is explicitly named, whether an alias is named, whether a competitor is named, whether a brand-domain source is cited, the exact cited URL, and whether the answer or citation could not be checked. If you can, ask a second person to label the unclear cases, and settle any disagreement by going back to the written rules instead of arguing it out.

Citations are worth recording at two levels. One is whether any page on your domain was cited. The other is which exact page was cited. A domain-level match doesn't show that page-level attribution is right. A link with a different path can still belong to your domain, so decide before you score how you will treat redirects, canonical pages, and subdomains.

You are done when each answer has your own label, with the answer text or link that supports it, and the uncertain ones are marked as uncertain instead of being pushed into a yes or no.

The usual mistake is counting an answer as correct just because a relevant source shows up somewhere in it. Check that the link is actually shown as a citation for that answer, and that it resolves to the domain or page your rule names.

Step 6: Reconcile the record labels and the displayed rates

With both sets of labels in hand, compare them. Build one table for mentions and a second one for citations, since they are separate labels.

Your audit says yesYour audit says no
Tool says yesTrue positive (TP)False positive (FP)
Tool says noFalse negative (FN)True negative (TN)

From the counts you can work out a few common measures. Accuracy is (TP + TN) divided by all labeled cases. Precision is TP divided by (TP + FP), which is the share of the tool's positive labels that are positive under your rules. Recall is TP divided by (TP + FN), which is the share of your audit's positives that the tool found. F1 is the harmonic mean of precision and recall, when both are defined.

Here is a made-up example, to show the arithmetic only. Say a 100-record mention audit has 18 true positives, 2 false positives, 6 false negatives, and 74 true negatives. Accuracy is 92%, precision is 90%, recall is 75%, and F1 is about 81.8%. These numbers are not a benchmark and not a customer result. The example also shows why accuracy alone can hide problems. The tool looks 92% accurate, yet it missed a quarter of the mentions you found, and when positives are uncommon that gap is easy to overlook. Report precision and recall alongside accuracy.

After the record-level check, calculate your own mention rate as the audit-positive answers divided by the eligible audited answers, and calculate the citation rate on its own using the citation rule. Only then compare your figures with the tool's metrics, and only after matching its date range, platform, sample, and documented denominator. Tools do not all define these metrics the same way. One metrics guide defines visibility as the percentage of responses where a brand appears and separates brand visibility from source visibility. Another counts citations at the page or domain level, while its share of impressions is based on prompt search volumes. A third describes share of voice as a relative measure where a brand both mentioned and cited in one run counts once. These are examples of different implementations, not a universal standard, and share of voice in particular can be calculated in more than one way, which the walkthrough on calculating share of voice across AI answers explains. The overview of AI visibility metrics and KPIs is also useful for lining up definitions.

If the tool doesn't document enough to reproduce a number, say so in your memo: the metric could not be independently reconciled from the sampled records. Don't guess at the denominator or the weighting.

You are done when your report has record-level disagreement counts and your own calculated rates, with any difference in denominator or aggregation described plainly.

A common mistake is comparing a manual per-answer rate with a tool metric that weights prompts by search volume, combines engines, or counts citations per page. The label can be the same while the calculation is not. If you want a deeper walkthrough of the arithmetic itself, the guide on measuring citation rate shows how it is usually built.

Step 7: Repeat a subset and report the variability

A single live answer is one dated observation, so to verify AI tracking results you need to repeat part of your sample. Re-run a subset of the frozen prompts more than once, in separate fresh sessions, and keep the prompt, engine, locale, mode, and other recorded settings the same each time. One DeepSmith article on building a repeatable prompt set recommends three rolls and a majority vote as the smallest honest measurement. That is that article's practical recommendation, and it isn't a validated statistical standard, so treat it as a reasonable starting point. Save each run on its own. A majority label is fine as a short summary, but it shouldn't replace the underlying answers.

Then compare the runs. Look at the mention labels, the citation labels, the cited pages, and the content of the answers. You don't need identical wording, because for this audit what matters is whether the labels and sources you measure stay stable under the same test conditions. Report how many prompts were stable across all runs, how many changed, and what kinds of changes you saw.

OpenAI's guidance on reproducible outputs is about API completions. It says outputs are non-deterministic by default, and that a fixed seed can make them mostly the same when the seed, the request parameters, and the system fingerprint all match, though this is best effort and not a guarantee. That doesn't mean you can make the consumer ChatGPT app give identical answers, so don't claim it does. It also means variation across runs doesn't by itself prove a stored record is wrong, and the article on why AI citations keep changing is a good companion when you need to explain that to a stakeholder. If you want to know whether day-to-day noise could explain a gap you found, the piece on whether AI visibility scores are reliable looks at it directly.

You are done when the audit shows per-run results and says which findings were stable, which varied, and which could not be compared because the conditions changed.

The mistake to avoid is treating one live response as the final truth, or averaging away a disagreement without reporting that the answer changed. A live result is a dated observation and not a permanent ground truth for the engine.

Step 8: Summarize the findings and set a recheck plan

Finish with a short memo that another analyst could follow, so the AI visibility data quality findings can be reviewed. State the sample, the engines, the date window, the prompt versions, the number of stored records and live runs, the label rules, the missing records, and the denominator you used. Sort the findings into groups: answer-record mismatches, mention-label errors, citation-label or URL errors, metric arithmetic mismatches, and run-to-run variability. Put the issues that affect your decisions first, such as a pattern of missed brand mentions or a misleading aggregate. Don't invent pass or fail thresholds, since none were established, and say instead what level of disagreement would change your decision about the tool.

If you want to show how uncertain a percentage is, a proportion interval can help, though it doesn't fix a biased sample or make repeated answers independent. The NIST handbook section on confidence intervals for proportions recommends methods such as the Wilson interval and notes that the simple normal approximation can be inadequate for very small samples or very few failures. An exact binomial interval is another option. As a calculation example only, 90 positives in 100 independently sampled records gives an approximate two-sided 95% Wilson interval of 82.6% to 94.5%. That is not an observed product result. If your records cluster by prompt or come from repeated runs, don't present a simple interval as though every observation were independent.

You are done when someone else could reproduce your sample, your labels, and your arithmetic, and the memo separates real errors from expected variation and from runs where the conditions didn't match.

A last common mistake is reporting a discrepancy as an error before checking whether the prompt, date, settings, or platform behavior were actually matched. Keep those two categories apart in the memo.

What to do next

Keep the audit sample as a saved, versioned set and recheck the same prompts on a regular schedule, so you can see whether the tool's data quality holds up over time. When you need broader coverage, add a fresh sample instead of editing the old one. If your findings point to problems you can't explain, the list of AI visibility tool red flags is a sensible next read. For the bigger picture on tools in this space, the roundup of AI visibility tools is the place to start. It also helps to read the guide to measuring AI visibility manually, since the same habits make this audit faster.

If you want a tracker that stores full answers and keeps mentions and citations separate, you can start a free DeepSmith trial and run this audit on your own prompts. DeepSmith doesn't certify its own numbers, which is why the audit above is built to be run by you.

Frequently asked questions

How many prompts do I need to audit?

None of the sources we reviewed set a universal minimum. Choose a subset that covers the prompt types, the labels, and the engines you need to test. Report the sample size, and avoid implying it represents prompts or platforms you didn't test.

Should live answers exactly match the tool's stored answers?

Only compare answers for the same prompt, engine, and mode, with sufficiently matched time and settings. AI outputs can vary, so compare the mention and citation labels under explicit rules, and keep each run as evidence.

What is the difference between a mention rate and a citation rate?

A mention rate concerns whether the answer names the brand. A citation rate concerns whether the answer links to the brand's page or domain. Track them separately, and state the denominator and the domain or page rule you used.

What should I do when a metric does not match my calculation?

First line up the prompt set, the date range, the engines, the eligible-answer denominator, and any weighting. If the tool's definition or its records don't give you enough to reproduce the figure, report that limitation in your memo and don't guess at the formula.