DeepSmith

Nov 26 · AEO & AI Visibility

19 min read

How AI Visibility Tools Collect Data: APIs, Scraping and Prompt Runs

Avinash Saurabh
Avinash Saurabh · CO-Founder & CEO
Abstract monochrome diagram of an API pipe and a browser window feeding data nodes into a stack of answer cards, with the cover line How AI Tools Collect Data.

AI visibility tools collect answers in one of two ways. They send your tracked prompts through an AI provider's API, or they automate the provider's user interface in a browser, and then they repeat those checks on a schedule and analyze the answers and citations that come back. That is the short version of how AI visibility tools work, and the rest of this piece fills in what each step involves and what it means for the coverage and cost of whatever tool you end up looking at.

If you've ever wondered whether a tool is reading real ChatGPT conversations, it isn't. Nobody outside a platform gets a feed of what its users type. A tool makes its own fresh requests, using the questions you picked, and records what the engine says back.

What does LLM monitoring data collection actually involve?

LLM monitoring data collection means getting an AI engine to answer a question and saving the answer, along with any sources it shows. Everything you see in a dashboard (mention rate, citation rate, share of voice) is calculated from those saved answers. The tool doesn't observe the AI engine from the outside. It asks it questions, the same way you would, only on a schedule and at a much larger scale.

Three terms come up whenever vendors describe how they do this, and they don't sit at the same level, so it helps to separate them first.

  • Official API collection. The tool sends a configured question to a provider's developer API, sometimes with web search turned on, and stores the generated answer plus whatever source data comes back.
  • Browser or interface collection, often called scraping. The tool automates a browser, types the question into the AI product's own web interface, and extracts what is displayed: the answer and, where they appear, the citations and other elements of the response.
  • Scheduled prompt runs. The tool runs a list of questions again and again, on a set cadence, against the engines you selected. Sometimes the run also varies by language or location.

The first two describe how an answer is obtained. The third describes when and how often that happens. Scheduling is a layer on top of a collection method, so a tool that uses APIs and a tool that uses a browser can both run prompts daily. That's why "AI tracking API vs scraping vs prompt runs" is a slightly misleading way to line them up, even though the three terms do show up together in vendor pages. A better way to see it is that a tool picks a capture method for each engine, and then a schedule that decides how often to use it.

This piece is about the mechanics and the coverage and cost that follow from them. It doesn't try to decide which method gives more accurate results, which is its own topic and is covered in the comparison of API answers and scraped answers. It also doesn't explain why the same question can produce different answers from one run to the next, which is covered in why AI answers change from run to run.

How does API collection work?

With API collection, the tool sends a request to the provider's developer interface, receives a generated answer, and stores it. The request is made for the monitoring job, so it has no connection to anyone's private conversation. What comes back depends on the provider and on which options the tool turns on.

What an OpenAI response can contain

OpenAI's Responses API can use a web search tool to ground an answer in current web results. According to OpenAI's documentation on its web search tool, a response can contain a record of the search action, a message with the generated answer text, and citation annotations that carry source URLs and related details. There is also a sources field that lists the fuller set of URLs the model consulted, and that list can be longer than the citations shown inline in the answer.

That last point matters for anyone reading a citation report. A page that was retrieved during the search is not necessarily a page the answer cited. A tool that keeps only the inline citations and a tool that keeps the full source list are measuring two different things, even if both call the number "citations."

A few other details from the documentation shape what a tool can do:

  • Search can be optional when the tool choice is automatic, so a developer who wants a search on every request has to require it.
  • The search can be set to use live web access or cached and indexed content.
  • Approximate country, city, region and timezone can be passed in as location context.
  • The Responses API allows domain filters, with up to 100 domains to allow or block.
  • Some of these controls, such as the full source list and the live-access settings, are not necessarily available on the specialized search models in the older Chat Completions format.

What a Perplexity API response can contain

Perplexity's current documentation splits things into two products, and it's worth keeping them apart because they return different things.

The Agent API generates a model response. A request supplies a model and an input, and web search is opt-in: you add a web_search tool to turn it on. The response exposes the generated answer text separately from the search results, which can include queries and details like IDs, URLs, titles, snippets and dates. Inline citation markers inside the answer can depend on the prompt, so a caller that needs markers should ask for them directly.

The standalone Search API is different. It returns ranked web results as structured data and does not write an answer. It has no model-token charge, and it is not a substitute for an answer-and-citations response. If a vendor says it uses "Perplexity data," it's fair to ask which of the two it means, because a list of ranked search results and a model-written answer with sources are not the same record.

One more thing to know about Perplexity's documentation. Older pages describe a POST /v1/sonar endpoint and say it would be supported until September 27, 2026, with users directed to the Agent API. That date has passed, so the Sonar endpoint and its pricing tables are best read as legacy documentation, not a description of how a current integration works.

What API collection costs

API pricing is the one part of this topic where the public documents give real numbers, so it's the easiest place to see how cost builds up. The charges below come from OpenAI's and Perplexity's own pricing documentation, and they are per-call charges for the search tool, not full costs.

API search optionTool-call chargeAdditional charges
OpenAI standard web search$10 per 1,000 callsSearch-content tokens and model tokens at the selected model's rates
OpenAI web-search preview with reasoning models$10 per 1,000 callsSearch-content tokens and model tokens at the selected model's rates
OpenAI web-search preview with other models$25 per 1,000 callsModel token charges may still apply; search-content tokens are free
Perplexity Agent API web_search, standard$2.50 per 1,000 invocationsModel tokens billed separately
Perplexity Agent API web_search, Fast Search$1.00 per 1,000 invocationsModel tokens billed separately
Perplexity Search API$5 per 1,000 successful requestsNo model or token charge, but returns results, not an answer
Perplexity Search API, Fast Search$1 per 1,000 requestsReturns search results, not an answer

Search-content tokens are the retrieved search material that gets passed to the model, and they're billed separately from the tool-call fee. For example, OpenAI documents a fixed block of 8,000 input tokens per call as the search-content charge for gpt-4o-mini and gpt-4.1-mini on the non-preview web search tool. So the tool-call rate is never the total price of an answer.

Perplexity's Agent API token prices depend on the model you choose, and its models documentation lists them model by model, so there isn't one universal price for "a Perplexity answer." For the Search API, one successful request is one billing unit, no matter that a single request can hold up to five queries. Invalid, rate-limited and upstream-failed requests aren't billed, while a successful response is billed even when it returns no results.

One more cost point is easy to get wrong. A ChatGPT subscription doesn't ordinarily include API usage. API usage is billed separately, so paying for a consumer plan doesn't mean a team can run monitoring through the API for free.

How does browser and interface collection work?

With browser collection, automation software opens an AI product's website, enters the question, waits for the answer and extracts what is shown. The saved record can include the prompt, the answer, the citations and other visible parts of the response. This is a different path from asking an API to generate an answer, even when the same question goes in.

Vendors describe their own practices in public, and they don't all do the same thing. Three examples show how much it varies, and each is what that vendor says, not a rule for the whole industry.

  • Scrunch says it uses several methods, including browser automation and official platform APIs, and that the right method depends on the platform. It names ChatGPT and Perplexity among the engines it covers, but it doesn't say which method it uses for which engine, so there's no engine-by-engine mapping to report.
  • Promptwatch says it automates browser activity, submits the prompt, and then collects the prompt, the response and the citations. It says it repeats this daily across the configured models and languages, and that interface collection lets it observe things like language, region, certain product enrichments, and whether an answer used images or tables.
  • Peec's documentation describes UI scraping, lets a project choose an IP address for running prompts, and lets users pick which AI models to track. Its quickstart suggests starting with three to five questions.

What this means for coverage is that browser collection is configured around the interfaces, models, languages and locations a vendor chooses to support. The sources don't show that every browser-based tool supports every location, language, feature or account state, so a vendor's claim about one of them doesn't carry over to the next vendor. If a particular surface matters to you, such as a certain engine or a certain country, ask the vendor to name it and to show what its collected record contains for that surface.

What it means for cost is less clear, and it's better to say so than to guess. The sources reviewed for this piece don't give a standard per-run price for browser automation or a general figure for what it costs to maintain. A monitoring subscription may simply bundle collection into the plan. Because of that, a subscription price and an API tool-call fee shouldn't be set side by side as if they bought the same thing. One is a package that includes collection and reporting, and the other is a single line item in a build-it-yourself setup.

Why some vendors say they use both

Since each method returns something a little different, it isn't unusual for one tool to use more than one. A vendor might use an API for one engine and a browser for another, depending on what each platform offers. A plain "we use APIs" or "we use scraping" claim says less than it sounds like, so the useful question is which method applies to each engine you care about.

There's also a terms-of-service angle that you may run into in vendor conversations. OpenAI's Terms of Use prohibit automatically or programmatically extracting data or output from its services and prohibit circumventing restrictions. Perplexity's consumer terms prohibit automated scraping, extraction and monitoring, and also describe limits on manual collection. Those statements apply to the services the terms cover, and the pages don't say what terms govern a particular vendor's own integration. So this isn't something to draw a legal conclusion from about any one vendor. It's just a reason to ask a vendor how it gets its data and to get that answer in writing if it matters to your buying decision.

What do scheduled prompt runs add?

A prompt run is one execution of a configured question against one selected engine, which is the basic unit of prompt tracking. Schedules matter because a single answer tells you very little, and what a dashboard reports is built from many of them over time.

The number of observations a tool collects grows with four things: the number of prompts, the number of engines, the number of setting combinations such as languages or locations, and the number of run dates. Doubling any one of them doubles the total, if everything else stays the same.

Cadence varies, and there's no standard to assume. The documented examples are these:

  • Promptwatch says it repeats its collection daily across the configured models and languages.
  • Peec describes running prompts daily across AI platforms.
  • Scrunch's data FAQs say prompts are monitored daily for the first two weeks and then collected every 72 hours after that. That's one vendor's schedule, not a general rule.

So a daily run is one documented cadence, and it would be a mistake to treat it as what every tool does. When you compare plans, check how often the prompts actually run and whether that changes by plan or by platform.

A tracked prompt set is a sample

A prompt portfolio measures the questions that were selected, not every question people ask and not the activity of real users. Peec's material on prompt tracking says this directly, describing portfolio visibility as an approximation of visibility across all possible daily prompts. That's a fair way to put it for every tool. If you track 50 questions, the numbers describe those 50 questions, on those engines, in those settings, on those dates.

So the useful coverage question is which prompts, models, languages, locations and dates are included, and not how many responses the tool has stored. A tool with a big pile of responses to a poorly chosen question set can still tell you little about the questions your buyers ask. If you're still choosing yours, how to find the AI-search prompts to track walks through where to look, and how many AI-search prompts you should track covers sizing the set.

How do coverage and cost add up?

Cost and coverage come from the same few numbers, so it helps to see them together. Here's a worked example using the documented tool-call prices from the table above. It's an arithmetic illustration, not a quote.

Say a team schedules 100 prompts on two engines, once a day, for 30 days. That is 6,000 scheduled attempts in total, or 3,000 for each engine. If every attempt makes exactly one search-tool call and everything is charged at the standard documented rate, the tool fees come out like this:

  • OpenAI web-search tool fees: 3,000 calls at $10 per 1,000 is $30, plus any search-content and model-token charges.
  • Perplexity Agent API standard web-search fees: 3,000 calls at $2.50 per 1,000 is $7.50, plus model-token charges.
  • Combined tool fees in this example: $37.50, plus the token charges.

A few warnings go with that number. It assumes exactly one search call per attempt. It leaves out model tokens and search-content tokens, which are billed on top. It leaves out the monitoring platform's subscription fee, and it doesn't account for retries or other implementation choices. It also doesn't mean a tool costs $37.50 a month to run. Adding a second location would double the attempts if each prompt, engine and location is run on its own.

What the example does show is how the pieces multiply. Prompts times engines times locations times days gives you the run count, and the run count is what drives both usage-based API fees and, in a subscription, the prompt and engine limits a vendor puts on each plan. Public documentation doesn't give a general per-run price for a full AI-visibility subscription, and plan prices can't be worked out from API call prices, so the comparison to make is between plan limits and your own run count.

What should you check before trusting a dashboard?

A few questions about how the data was collected will tell you most of what you need to know. They're worth asking in a demo or finding in the documentation.

  1. Is the unit a unique tracked prompt, a prompt-platform combination, a run, or an answer returned?
  2. How many engines, languages, locations or settings combinations count against the limit?
  3. How often are prompts run, and can the cadence differ by plan or platform?
  4. Are the answers and citations kept as history, or only summarized as metrics?
  5. Does the listed price include collection, API usage and model tokens, or only access to the reporting platform?
  6. When a vendor says "Perplexity data," does that mean raw Search API results or a model-written answer with sources?
  7. If a vendor says it uses both APIs and browser collection, which method applies to each engine?

The first two are the ones people most often skip, and they change the meaning of a plan's limit. A limit of 100 prompts can mean 100 questions, or it can mean 100 question-and-engine pairs, which would be 50 questions on two engines. Those are very different amounts of coverage for the same number on the pricing page.

The answers you get also tell you how far to trust what the dashboard reports. An API-based result isn't automatically a copy of what a person saw in a logged-in consumer session, because the API and the consumer interface are different paths to an answer. And as noted earlier, the citations in an answer and the wider set of sources a model looked at are two different lists, so it's worth knowing which one a tool counts.

Teams that want to export or inspect the data they've already collected usually have an API of their own to work with. DeepSmith, for example, has a read-only REST API for pulling workspace data into other tools, which is useful for reporting. Using an AEO platform API for custom reporting covers that side. That API reads data already in the workspace, and it isn't how answers get collected from ChatGPT or Perplexity in the first place.

What to take from this

The way a tool collects data decides what it can show you. An API gives a structured response with citations and source lists that depend on the provider's options. A browser gives you what the interface displays. A schedule decides how many of those observations you get and how often, and your prompt set decides which questions they're about. The same pattern holds for tracking the same prompts across ChatGPT, Gemini and Perplexity, where the engine and the method both change what you see, and for engines outside the chat assistants, such as Google AI Overviews tracking.

If you're comparing tools, the practical next step is to write down the prompts, engines and locations you need, count the runs they imply, and put the vendor questions above to each tool. A broader look at what's on the market is in our guide to the best AI visibility tools, and once you have a prompt set, setting up a repeatable prompt set keeps the checks consistent. If you'd like to see scheduled tracking in action, you can start a DeepSmith free trial, define the questions you care about, and review mention and citation results by platform.

Frequently asked questions

How do AI visibility tools collect data from ChatGPT and Perplexity?

They submit configured prompts through a supported provider API or automate the platform's user interface, and then they capture the returned answer and any available citations on a schedule. Some vendors use one approach, and some say they use both depending on the platform.

Does Perplexity's Search API return an AI-written answer?

No. It returns ranked web results as structured data. If the output you need is a model-written response with sources, that comes from the Agent API instead.

Does a daily prompt tracker show every question people ask?

No. It reports on the prompts and settings that were selected for its monitoring portfolio, so the results describe that set and not all user activity.

What does API-based tracking cost?

It depends on the tool-call charges, the model and search-content tokens, the search option chosen, the number of prompt and engine runs, and any monitoring subscription on top. Call fees alone aren't a total-cost figure.