You opened your logs, saw a wave of hits from GPTBot and ClaudeBot, and thought: great, the AI engines are finally reading my site. Then a small doubt crept in. How do you know those were real? The honest answer is that you do not, not yet, because anyone can type GPTBot into a header. This guide shows you how to verify GPTBot and every other major AI crawler, so you can tell a genuine visit from a spoofed AI crawler and stop making decisions on traffic that was never what it claimed to be.
If that gap feels unsettling, that is a normal reaction. Most teams have never checked. By the end of this walkthrough you will have a repeatable way to confirm AI bot identity using three signals: the user-agent string, a reverse DNS check, and the operator's own published list of IP addresses. Take it one step at a time. You are closer to a clean, trustworthy picture of your bot traffic than you think.
Step 1: Treat the user-agent string as a claim, not an ID
Here is the single idea the whole guide rests on. The HTTP user-agent string is a self-declared label. It travels in plain text, and any client can set it to any value in one line of code.
That means a user-agent is not a credential. It cannot prove a request came from the operator whose name sits inside it. So when you see a line that reads Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, you are looking at what the visitor typed, not who the visitor is. Spotting a fake user agent AI bot starts with accepting that the string alone tells you nothing you can trust.
Why does this matter so much right now? Because the impersonation is real and measurable. In a two-week analysis published in September 2025, HUMAN Security found that roughly 1 in every 17 to 18 requests claiming to be an AI crawler was actually spoofed, around 5.7 percent overall. For some bots it is far worse. If you block, bill, or count training access on the user-agent alone, you will be wrong a meaningful share of the time.
So what do you do about it? You add two checks that a spoofer cannot fake. That is the rest of this guide.
Common mistake: blocking or allowing traffic by user-agent with no verification behind it. That lets spoofers wear a real bot's name while you accidentally deny the legitimate crawler, which hurts both your analytics and your AI search visibility.
Step 2: Capture the real source IP first
Every verification method below depends on one thing: the actual IP the request came from. Get this wrong and everything after it is built on sand.
Read the IP from the TCP peer, the real connecting address. In PHP that is REMOTE_ADDR, in Node it is the request's .remoteAddress, and on most stacks it is the connection-level value your server already has.
Do not read it from the X-Forwarded-For header unless you control the proxy that set it. That header is easy to forge, and trusting it is the same trap as trusting the user-agent. A visitor can put any address they like in there.
How do you know this step is done? You can point to one field in your logs and say with confidence, "this is the address that actually opened the connection." If you cannot say that yet, fix it before you go further.
There is a wrinkle here worth naming early. If you sit behind a CDN like Cloudflare, Fastly, or Akamai, your origin server sees the CDN's IP, not the bot's. That is not a failure, it just means the verification happens at the edge instead of at your origin. We will come back to that in Step 6, so park it for now.
Step 3: Run a forward-confirmed reverse DNS check
This is the method Google, Apple, and Common Crawl all recommend, and it is the gold standard for the operators that support it. It is called forward-confirmed reverse DNS, or FCrDNS. The name sounds heavy. The idea is simple, and it is four small moves.
- Take the source IP you captured in Step 2.
- Run a reverse DNS (PTR) lookup on it. The hostname you get back should live under the operator's published domain.
- Run a forward DNS (A or AAAA) lookup on that hostname.
- Confirm the forward lookup returns the original IP. If it matches, the IP genuinely belongs to the operator. If it does not, the request is unverified.
Why both directions? Because a one-way check is forgeable. Anyone who controls a reverse DNS zone can point an IP at whatever hostname they choose, so a PTR-only check can be gamed. The forward step closes the loop. A hostname under googlebot.com cannot resolve back to an IP outside Google's network unless someone has compromised Google's own DNS, which is not a realistic threat.
For a running reverse DNS GPTBot lookup you would follow the same four steps, but here is the honest catch: OpenAI does not currently publish a stable reverse-DNS suffix, so FCrDNS is not the primary check for GPTBot. That is what Step 4 is for. FCrDNS is the right tool for the operators that do publish a suffix.
As of July 2026, the published reverse-DNS suffixes are:
| Operator | Reverse-DNS suffix | Notes |
|---|---|---|
| Google crawlers | *.googlebot.com, *.google.com, *.search.google | Covers Googlebot and its siblings |
| Applebot | *.applebot.apple.com | Powers Spotlight, Siri, Safari Suggestions |
| Common Crawl (CCBot) | AWS host containing "crawl" or "commoncrawl" | IPv4 only, no IPv6 FCrDNS support |
OpenAI, Anthropic, Perplexity, and Meta do not publish a stable suffix. For those, you verify against their published IP list instead, which is exactly where we go next.
Pro tip: you do not have to build the DNS plumbing by hand. Libraries like dnspython, node-dns, and php-dns run PTR and forward lookups for you, so the whole check fits in a small function.
Step 4: Check the IP against the operator's published range
For the AI crawlers that do not offer a reverse-DNS suffix, the operators hand you something better: a live list of the exact IP ranges their bots use, published as JSON. Your job is to confirm the source IP appears in the right file for the bot named in the user-agent. This is how you verify GPTBot with confidence.
Match on the operator token in the user-agent (GPTBot, ClaudeBot, PerplexityBot, and so on), not the full string, because operators change version numbers over time. Then check the captured IP against the matching JSON. Here is the per-operator playbook.
OpenAI
OpenAI runs four distinct crawlers as of mid-2026, and each has its own IP file:
- GPTBot crawls for training data. IPs at
https://openai.com/gptbot.json. - OAI-SearchBot surfaces pages inside ChatGPT's search results. IPs at
https://openai.com/searchbot.json. - ChatGPT-User fetches a page only when a ChatGPT user clicks a link or asks it to read a URL. IPs at
https://openai.com/chatgpt-user.json. - OAI-AdsBot crawls for ad relevance. IPs at
https://openai.com/adsbot.json.
To confirm AI bot identity for any of these, check whether the source IP appears in the JSON for the specific bot named in the user-agent.
Anthropic
Anthropic runs three bots, ClaudeBot (training), Claude-User (user-triggered fetch), and Claude-SearchBot (search quality), and they share one IP file at https://claude.com/crawling/bots.json. Worth knowing: Anthropic's own documentation says IP blocking alone is unreliable because the list can change, and points to robots.txt as the primary control. So treat the IP list as a fresh, short-lived signal, not a permanent allowlist.
Every Google crawler verifies through the FCrDNS check in Step 3. You can also cross-check the IP against Google's published ranges at https://developers.google.com/static/search/apis/ipranges/googlebot.json. One thing that trips people up: Google-Extended is not a crawler. It is a robots.txt token that controls whether content Googlebot already fetched can be used for Google's generative AI. There is no Google-Extended user-agent and no separate traffic to hunt for.
Apple
Applebot verifies through FCrDNS (Step 3) against *.applebot.apple.com, with an optional cross-check against https://search.developer.apple.com/applebot.json. Same footnote as Google: Applebot-Extended is not a separate crawler. It is a robots.txt token about training use, with no user-agent of its own.
Perplexity
Perplexity runs PerplexityBot (indexing) and Perplexity-User (end-user fetches), each with its own file: https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json. Confirm the IP against the right one.
Common Crawl
CCBot builds the open Common Crawl corpus that many labs train on. Verify it with FCrDNS to an AWS host containing "crawl" or "commoncrawl," then cross-check the IP against AWS's ranges at https://ip-ranges.amazonaws.com/ip-ranges.json. Note that Common Crawl does not support reverse-DNS verification over IPv6, so verify IPv6-claimed CCBot traffic against the AWS list instead.
One rule ties all of these together: these lists move. Operators add and drop ranges, so fetch the JSON fresh and cache it for at most about an hour. Hard-coding a snapshot is a slow way to start rejecting real bots.
Where to go next once your bots are real
Here is the quieter question hiding under all of this. Once you have verified which crawlers are genuinely reading your site, the next thing worth knowing is whether those engines are actually citing you when someone asks a question in your space. DeepSmith tracks which prompts mention your brand, which of your pages AI cites, and which competitor pages are winning those citations, so you can produce content built to be lifted cleanly by the bots you just verified. Start a free DeepSmith trial when you are ready to see where you show up.
Step 5: Flag the bots that publish nothing
Not every operator gives you a way to verify. Some publish neither a reverse-DNS suffix nor an IP list, and pretending otherwise would set you up to trust traffic you cannot check.
As of mid-2026, these have no published verification mechanism:
- Bytespider (ByteDance / TikTok): no suffix, no IP list.
- Meta-ExternalAgent and Meta-ExternalFetcher: no official verification method.
- Cohere-AI and its training crawler: no published IP list.
- MistralAI-User and MistralAI-ExtractionBot: no published IP list.
- DuckAssistBot and DuckDuckBot: limited docs, and spoofing has been documented.
- Third-party data providers like YouBot, TimpiBot, WebzioBot, Omgili, and Diffbot: mostly nothing to verify against.
For any of these, the only real signal is the user-agent plus behavior: how the client's TLS fingerprint looks, the order of its headers, its request pacing, and whether it respects robots.txt. That is softer evidence, so the safe default is to treat these as unverified and decide by policy, not by the name in the header.
How do you know you have handled this step? You have a short list of "cannot verify by network identity" tokens, and your system labels them as such instead of quietly counting them as real.
Step 6: Handle the edge cases that break simple checks
This is where a lot of well-built systems still slip, so read it slowly. None of these are hard once you see them coming.
You are behind a CDN. Your origin sees the CDN's IP, not the bot's, so you cannot run FCrDNS yourself. Lean on the CDN's verified-bot feature instead. Cloudflare stamps a cf-verified-bot signal, and Akamai Bot Manager does the equivalent, so let the edge do the verification and trust its result.
Shared egress IPs. Cloud NAT gateways and corporate proxies collapse many different senders behind one address. IP-only checks break here, because one IP no longer means one operator. You need the original client IP from a header source you actually control.
IPv6. Common Crawl has said plainly that reverse-DNS verification is not yet supported on IPv6, and several operators publish minimal or no IPv6 ranges. Do not read "no IPv6 ranges published" as "no IPv6 traffic." Treat IPv6-only AI-bot traffic as unverified by default.
A real-looking user-agent from the wrong hands. A legitimate string does not prove a legitimate request. A third party can scrape your site while wearing a real bot's name, which is the whole reason you pair the user-agent with an IP or reverse DNS check every time. This is the subtlest kind of fake user agent AI bot, because the header looks perfect and only the IP gives it away.
User-agent drift. Operators bump version numbers, so match on the token, not the full string, and keep an eye out for new versions when a bot suddenly stops verifying.
A spoofed PTR record. An attacker who controls a reverse zone can produce a PTR that looks right. The forward-confirmation step from Step 3 is exactly what defeats that, so never skip it.
Step 7: Log every verdict and clean your analytics
You did the hard part. Now make it pay off by writing down what you decided, every single time.
For each request that claims to be an AI bot, store the source IP, the user-agent, the reverse DNS result, whether the IP matched the published range, and your final verdict. That record becomes two things at once: a clean dataset for your bot analytics, and an evidence trail for the day something looks off.
With that log in hand, filtering gets simple. Real, verified crawler visits stay in your reporting. Requests you could not verify get separated out, so a spoofed AI crawler never inflates your "the AI engines love us" numbers. This is the payoff of the whole workflow. You stop guessing and start reporting on traffic you have actually checked.
And do not treat "AI bot" as one bucket. GPTBot crawling for training, OAI-SearchBot feeding ChatGPT search, and ChatGPT-User fetching on demand each mean something different for your SEO, your analytics, and your content decisions. Splitting them apart is where the real insight lives.
What to do next
You do not need to verify every bot on the internet this week. Pick the crawlers that matter most to you, likely GPTBot, ClaudeBot, PerplexityBot, and Googlebot, and get those three checks working: capture the real IP, run FCrDNS where it is supported, and match the IP against the published JSON everywhere else. That covers the operators most likely to be impersonated, and it is a genuinely good start.
Momentum matters more than perfection here. One verified bot today beats a perfect system you never ship. Once your traffic is clean, the natural next move is to ask what those verified engines are doing with your content, and whether they are citing you or a competitor. That is the question worth chasing next.
