You searched your brand in ChatGPT last week and wondered: is it even reading my site? Here is the honest answer. Your analytics will not tell you, because most AI crawlers do not run JavaScript, so tools like Google Analytics never record them. The raw server logs AI crawlers leave behind are the only complete record of what actually reached your origin.
Good news: you do not need a new tool to find out. You need one file and a handful of commands. By the end of this guide you will know which AI bots crawl you, how often, and which pages they touch. That is crawler log analysis, and it is calmer than it sounds.
Two numbers show why this is worth an hour. Cloudflare's network data found GPTBot traffic up 305 percent year over year, with total crawler traffic up 18 percent over the same period. And a publisher-network measurement reported by WIRED found roughly 1 in 31 visits, about 3.2 percent, now come from AI bots. Your site is almost certainly part of that shift.
One caveat before we start, because it shapes everything. A line in your log proves a request arrived. It does not prove the visitor is who it claims to be. Any client can label itself GPTBot. We will read identity as claimed, not confirmed, and I will show you exactly where that line sits.
Find your access log first
You cannot analyze what you cannot open, so let's start there. The file you want is the server access log, the plain-text record your web server writes for every single request.
Here is where it usually lives:
- Apache on Debian or Ubuntu:
/var/log/apache2/access.log - Apache on RHEL or CentOS:
/var/log/httpd/access_log - nginx:
/var/log/nginx/access.log - IIS:
%SystemDrive%\inetpub\logs\LogFiles\(W3C extended format, so the field order differs)
Older entries rotate into files like access.log.1 and access.log.2.gz, so a full month may span several files. If you are on a CDN like Cloudflare, Fastly, or CloudFront, your origin server still writes its own log, and that origin log is usually the better source. CDN logs can be sampled and only show edge activity.
No shell access? You are not stuck. On managed WordPress or shared hosting, your control panel (cPanel, Plesk, RunCloud) almost always has a "Raw Access Logs" or "Access Log Downloader" section. Download the file and you can do everything below on your laptop.
These files are your AI crawler traffic logs in raw form, unfiltered and complete. That is exactly why they beat any dashboard for this job: nothing has been sampled away or hidden behind a bot filter. You know this step is done when you can run head -1 access.log and see one real request line.
Read one line to learn the format
Before you count anything, look at a single line. The format decides which commands will work, and two minutes here saves you a confused afternoon.
Run this:
head -1 access.log
Most sites use one of two formats. Common Log Format has seven fields and no user-agent:
127.0.0.1 - frank [01/May/2025:07:20:10 +0000] "GET /index.html HTTP/1.1" 200 9481
Combined Log Format adds two quoted fields at the end, the referer and the user-agent:
127.0.0.1 - frank [01/May/2025:07:20:10 +0000] "GET /index.html HTTP/1.1" 200 9481 "https://google.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 ..."
Combined is the default when Apache combined logging is on and the default "combined" format in nginx. You want that last quoted field, the user-agent, because that is where a bot names itself.
The most robust way to pull it out is to split on the double-quote character. The user-agent lives inside quotes, so awk -F'"' '{print $6}' access.log isolates it cleanly. Here is the field map when you split on the quote:
$1client IP$3authenticated user (usually a dash)$4timestamp with brackets$5the request line (method, path, protocol)$6the user-agent$7the referer
Common mistake to avoid: if your head -1 output has no user-agent field, someone overrode the default LogFormat (Apache) or log_format (nginx). Restore combined logging, or note the real field position and adjust the index below. Do not skip this check and guess.
Know which AI bot tokens to look for
This is the roster. Each AI operator publishes a stable product token that shows up in the user-agent string. The server logs AI crawlers touch will list these tokens verbatim, so learning the names is half the work. Pin your filters to the token, not the version digit, because the digit changes and the token holds.
It helps to know one distinction as you read. Some bots crawl proactively to build an index or a training set, and some fetch a single page on demand when a user asks their assistant something live. GPTBot and ClaudeBot are the first kind. ChatGPT-User and Perplexity-User are the second. A spike of on-demand fetches often means real people are asking about you right now, which is worth noticing.
Here are the ones worth knowing:
- OpenAI:
GPTBot(training crawl),OAI-SearchBot(surfaces pages into ChatGPT search),ChatGPT-User(on-demand fetch when a user triggers it),OAI-AdsBot(validates ad landing pages). - Anthropic:
ClaudeBot(training crawl),Claude-User(on-demand fetch),Claude-SearchBot(search-grounded retrieval). - Google:
Googlebot(the classic indexer, which also grounds some Gemini surfaces). Note thatGoogle-Extendedis not a separate crawler. It is a robots.txt token that controls whether already-fetched content can train Gemini or Vertex AI. The actual request still arrives asGooglebot. - Apple:
Applebot, which powers Apple Intelligence grounding alongside Spotlight and Siri.Applebot-Extendedis a robots.txt token, not a crawler. - Microsoft and Bing:
bingbot, which also carries Copilot grounding requests. There is no widely published separate Copilot crawler. - Perplexity:
PerplexityBot(indexing) andPerplexity-User(live user query). - Meta:
meta-externalagent(indexing) andmeta-externalfetcher(on-demand). - ByteDance:
Bytespider, often high volume, with inconsistent robots.txt behavior. - Amazon:
Amazonbot. Cohere:cohere-aiandcohere-training-data-crawler. Common Crawl:CCBot, which feeds many training pipelines and does not honor robots.txt.
You will also see long-tail names like DuckDuckBot, YouBot, Diffbot, ImagesiftBot, and SeekportBot. Keep them in mind, but you do not need them all on day one.
One trap that trips people up: strings like HeadlessChrome, python-requests, curl, and Go-http-client are scrapers, monitoring tools, or bots of another kind. They are not AI crawlers. Do not fold them into your count or your numbers will lie to you.
Count the AI crawler hits and their share
Now the satisfying part. Let's find out how much of your traffic is AI bots. First, get your denominator, the total request count for the period:
wc -l access.log
Then count the AI bot hits with one grep. This pattern anchors on documented tokens:
grep -Eic 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|cohere-ai|cohere-training-data-crawler|CCBot|Applebot|Bytespider|Amazonbot' access.log
The -c flag returns a count and -i makes it case-insensitive. Divide that number by your wc -l total and you have your AI bot share of traffic. This single number is the headline of your whole AI bot log analysis: it tells you, in one figure, how present these crawlers are on your site.
Want just one operator? To track GPTBot visits on their own, narrow the pattern:
grep -c 'GPTBot' access.log
You know this step is done when you can say a sentence like "AI bots were 4 percent of my requests last month, and GPTBot was most of it." That sentence is the reason you opened the file.
Break the traffic down by bot
A single total is a start, but the interesting story is the mix. Which bots are actually working your site, and in what proportion?
Pull the user-agent field, match it against the roster, and tally:
awk -F'"' '{print $6}' access.log \
| grep -Eio 'GPTBot|ChatGPT-User|OAI-SearchBot|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|meta-externalagent|meta-externalfetcher|cohere-ai|cohere-training-data-crawler|CCBot|Applebot|Bytespider|Amazonbot' \
| sort | uniq -c | sort -rn
Your output ranks each bot by hit count, something like 12418 GPTBot, 7002 ClaudeBot, 1889 PerplexityBot, 643 Applebot, 411 CCBot. Now you can see the shape of it: a heavy training crawler, a couple of search bots, a long tail.
Here is the thing to hold onto. A bot crawling you is not the same as that bot's product citing you. GPTBot reading every page does not mean ChatGPT names your brand when a buyer asks. Crawl presence and citation presence are two different measurements, and the log only shows the first one.
That gap is exactly what an AI search visibility tool is for. DeepSmith's AEO module checks the questions your buyers actually ask across engines like ChatGPT, Perplexity, and Gemini, then reports your mention and citation rates. Pair your bot breakdown with that, and you can see whether the platforms crawling you are the ones actually surfacing you. The log tells you who reads; the AEO data tells you who repeats you.
See which pages each bot crawled
Bots do not read your whole site evenly. Knowing which pages they favor tells you what they think you are about. Filter to one bot and rank the paths it requested:
grep 'GPTBot' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -25
Swap GPTBot for any token to repeat it. The $7 field holds the request path in Apache combined logs, so confirm with your head -1 layout if your format differs.
Read the output like a map. Are the bots hitting your cornerstone content, or burning requests on tag pages and old archives? Are your newest, most important pages showing up at all? If a page you care about never appears, the bots may not know it exists yet.
Once you know which pages get crawled, the next question is competitive: on the topics those pages cover, who is actually winning the AI answers? DeepSmith's competitor tracking watches what your rivals publish on adjacent queries and which of their pages earn AI citations. That turns a list of crawled URLs into a plan: here is what the bots read on my site, and here is where a competitor is still beating me on the same topic.
Read timing, status codes, and bandwidth
You have the who and the what. Now the how, which is where waste and problems hide.
Start with timing. A burst of hits from one bot in a tight window is a training-crawl pass, not a casual visit:
grep 'GPTBot' access.log | awk '{print $1, $4}' | sort | uniq -c | sort -rn | head
Then check status codes per bot, because a bot collecting 404s or 403s is wasting its requests and your bandwidth:
grep -E 'GPTBot|ClaudeBot|PerplexityBot|Applebot|Bytespider' access.log \
| awk '{print $9, $14}' | sort | uniq -c | sort -rn | head -30
A wall of 200s means you are feeding the bot cleanly. A pile of 403s on a bot you never meant to block is a misconfiguration worth fixing. And if a bot you did block still shows up here with 200s, your block is leaking somewhere upstream, so it is worth a second look.
Finally, bandwidth, the number to bring to a budget conversation. Bytes served to bots are real egress you pay for:
grep -E 'GPTBot|ClaudeBot|PerplexityBot|Applebot|Bytespider' access.log \
| awk '{arr[$14]+=$10} END {for (k in arr) print arr[k], k}' | sort -rn
Adjust $10 and $14 to your format. If awk keeps tripping on rotated or custom logs, drop into a short Python loop that splits each line on the quote character and matches the user-agent with a regex. It is slower to type but far more forgiving of messy formats.
Quick privacy note, one paragraph and then we move on. Access logs contain IP addresses, and under GDPR an IP can be personal data. If you plan to share a log excerpt, redact the IP first, truncate it or hash it, and strip any embedded emails or session tokens. Read your own logs freely; just clean them before they leave your machine.
Turn it into a repeatable weekly routine
A one-time look is a snapshot. The value is in the trend, because AI crawl patterns shift fast and a single GPTBot pass can pull a site's worth of pages in hours.
Append a dated line to a small CSV each week:
echo "$(date -I),$(grep -c GPTBot access.log),$(grep -c ClaudeBot access.log),$(grep -c PerplexityBot access.log)" >> ai-bot-history.csv
After a month you can chart which bots are growing and which are fading. That is the whole point: not a number, but a direction. If your only goal this week is to track GPTBot visits and watch their trend, that CSV is enough on its own. Add the other bots when you are ready.
Consistent crawler log analysis, even five patient minutes a week, beats an occasional deep dive you never repeat. Small and regular wins here.
When grep starts to feel small, reach for a tool. GoAccess builds a real-time terminal or HTML dashboard from a log file, though you must run it with --ignore-crawlers=false because it hides known bots by default. Matomo's log-import mode tags bot hits against its own database and keeps them separate from real-user metrics. Screaming Frog's Log File Analyser flags AI bots by name and is strong for correlating hits with crawl paths. For tens of millions of requests a day, a pipeline into ELK, Grafana Loki, or ClickHouse earns its keep, but almost no one needs that on week one.
Your AI crawler traffic logs will keep changing, so recheck them on a schedule. Weekly is plenty for most sites, daily if you are actively making allow-or-block decisions.
What to do next
Here is where you are now. You can open your log, read its format, count AI bot hits, break them down by crawler, see which pages they touch, and track the trend week over week. That is the full loop of AI bot log analysis, and you built it with tools you already had.
The logs answer one question well: who is crawling me, and how much? They cannot answer the next one: am I actually showing up when buyers ask AI about my space? That is a different measurement, and it is the one that ties crawl activity to pipeline.
If you want both sides in one place, the crawl and the citation, that is what DeepSmith is built for. It tracks where you appear in AI answers, finds the gaps, and produces the on-brand content to close them. You can start a free trial and see your real data before you pay.
