You checked your logs, saw a wave of AI bots pulling your pages, and now you want to decide who gets in. That is a healthy instinct, and the good news is you are closer than you think. A robots.txt AI crawlers policy is just a short text file with a few rules, and by the end of this guide you will know the exact user-agent token for every major bot and precisely what each rule allows or blocks.
This is a hands-on walkthrough for anyone who runs a site and wants control over GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. You do not need to be a developer. You need a text editor, access to your web root, and about twenty minutes.
One thing to set expectations. robots.txt is a voluntary standard. Compliant crawlers honor it, and the four vendors here all say they do. It is a polite gate, not a locked door, so we will also flag where it stops and where you need something stronger.
Meet the AI crawlers you can control
Before you write a single line, it helps to see the whole field. Four vendors matter most, and each one runs more than one bot. The pattern is consistent once you spot it: every vendor separates the training crawler from the user-initiated fetcher, and sometimes from a search crawler too. Each is controlled independently in robots.txt. That is the quiet power of a robots.txt AI crawlers policy. You are not flipping one switch for all AI, you are setting a rule per bot, and the bots are more granular than most people expect.
Here is the reference grid, accurate as of July 2026. Vendor token lists change, so treat this as a snapshot and confirm against each vendor's own bot page before you finalize.
| Vendor | Bot token | What it does | Trains models? | Honors robots.txt? |
|---|---|---|---|---|
| OpenAI | GPTBot | Open-web crawler that collects training data for foundation models | Yes | Yes |
| OpenAI | OAI-SearchBot | Crawls sites to surface and link them as sources in ChatGPT search | No | Yes |
| OpenAI | ChatGPT-User | Fetches a URL when a ChatGPT user asks for it | No | Yes |
| OpenAI | OAI-AdsBot | Checks landing pages of ads served inside ChatGPT | No | Yes |
| Anthropic | ClaudeBot | Web crawler that collects content that can contribute to training | Yes | Yes |
| Anthropic | Claude-User | Fetches a site when a Claude user asks Claude to look at it | No | Yes |
| Anthropic | Claude-SearchBot | Crawls pages to improve search results inside Claude | No | Yes |
| Perplexity | PerplexityBot | Crawls sites so Perplexity can cite and link them in answers | No | Yes |
| Perplexity | Perplexity-User | Fetches a page when a Perplexity user triggers retrieval | No | Usually ignores robots.txt |
| Google-Extended | A product opt-out token for Gemini and Vertex training, not a crawler | Signal only | Yes, as a separate block |
Two more tokens are worth a line each. Applebot-Extended is Apple's opt-out for using web content in Apple Intelligence training, and it works exactly like Google-Extended. Amazonbot is Amazon's general crawler, with an Alexa-training subset, and it respects robots.txt too.
Notice the split in that table. Blocking the training crawler is a different decision from blocking the fetcher a real person triggers. We will come back to that, because it is where most people trip.
Step 1: Decide your policy for each bot
Start here, before you touch any file. For each bot, you are making one small choice: allow it everywhere, block it everywhere, or allow it on some paths and block it on others. The decision is independent for every bot, so you never have to solve it all at once.
Write one plain sentence per bot. For example: "Allow GPTBot on the blog, block it on /gated/." "Block ClaudeBot site-wide." "Allow PerplexityBot so we stay citable." That is your whole plan.
Done when: you have a one-line policy for each of the four named bots, written down where you can see it.
Common mistake: assuming Google-Extended controls whether you appear in AI Overviews. It does not, and we will unpack why in a later step. If you block Google-Extended expecting it to change AI Overviews, you will be disappointed.
Step 2: Find or create robots.txt at your root
Open a browser and go to https://yourdomain.com/robots.txt. One of two things happens.
If you see a text file, you already have one and you will add to it. If you see a 404, you need to create a plain-text file named robots.txt and place it at your web root.
The location is not negotiable. The file has to sit at the root of the host. A file at /blog/robots.txt or in any subfolder is simply ignored. It is also host-scoped, which means the file on www.example.com does not cover example.com or shop.example.com. If you run subdomains, each one needs its own.
Done when: curl https://yourdomain.com/robots.txt returns a 200 with a text/plain content type.
Common mistake: putting the file in a subfolder, or behind a login. Crawlers only read it at the root, served publicly. If they cannot fetch it, they assume no rules apply.
Step 3: Write one user-agent block per bot
Now the fun part. A rule group starts with a User-agent: line naming the bot, followed by the Allow: and Disallow: lines that apply to it. The group ends at the next User-agent: line or the end of the file.
To block one bot from your whole site, you name it and disallow everything. Here is the GPTBot robots.txt block:
User-agent: GPTBot
Disallow: /
Swap the token to control a different vendor. To block ClaudeBot everywhere, the shape is identical:
User-agent: ClaudeBot
Disallow: /
The PerplexityBot user agent follows the same pattern, and so does the Google-Extended robots.txt token. You can stack all four in one file, each in its own group:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Each group stands alone. A User-agent: * block elsewhere in the file applies to every other bot and does not touch these four.
The tokens matter more than anything else here. Match the vendor's exact spelling. Matching is case-insensitive, so gptbot works, but a typo does not. GPT-Bot, Claude_bot, or a bare Perplexity will match nothing, and the bot you meant to control walks right in. A GPTBot robots.txt group with a misspelled token is not a block at all, it is just a comment the crawler skips.
Done when: each bot you chose in Step 1 has its own group, and every token matches the reference grid above.
Common mistake: misspelling a token and assuming the block works. Read every User-agent: line back against the table before you move on.
Step 4: Add path rules inside each block
Blocking a whole site is the blunt version. Most of the time you want something finer, and that is where a few syntax rules earn their keep.
Default matching is path-prefix. Disallow: /private/ blocks /private/ and everything beneath it. Two special characters give you precision:
*matches any run of characters.$anchors a pattern to the end of the URL path.
So to keep ClaudeBot away from your PDFs and Word docs across the whole site, you anchor the file extension:
User-agent: ClaudeBot
Disallow: /*.pdf$
Disallow: /*.docx$
The $ makes /reports/whitepaper.pdf?v=2 still match, because the anchor sits at the end of the path, right before the query string.
When two rules could both apply, the longest match wins, and on a tie Allow beats Disallow. That is how you carve an exception. This lets GPTBot into one public folder while the rest of a directory stays closed:
User-agent: GPTBot
Disallow: /drafts/
Allow: /drafts/public-preview/
/drafts/public-preview/ is a longer match than /drafts/, so the allow wins underneath it.
Done when: every Disallow: value starts with /, and any file-type pattern ends with $.
Common mistake: forgetting the anchor. Disallow: /private without the $ also blocks /private-stuff/ and /privateer/, because prefix matching does not stop at a word boundary. If you mean one exact path, write Disallow: /private$.
Step 5: Separate the training crawlers from the user fetchers
This is the step that trips almost everyone, so take a breath, because once it clicks you will never confuse it again.
The bot that gathers training data and the bot that fetches a page for a live user are two different bots with two different names. Blocking one does not block the other.
Picture the most common goal: you are fine being cited in answers, but you do not want your content used to train models. For OpenAI, that means allowing the training crawler's absence and blocking the live fetcher, or the reverse, depending on your goal. An empty Disallow: explicitly allows everything, so this pairing lets the training crawler through while blocking the user fetcher:
User-agent: GPTBot
Disallow:
User-agent: ChatGPT-User
Disallow: /
Flip which line carries Disallow: / to reverse the policy.
Perplexity has the same shape, and it comes with a sharp edge. The PerplexityBot user agent is the crawler you name to control the citation index. Blocking the PerplexityBot user agent removes your pages from that index, but it does not stop Perplexity-User from fetching your page when a person asks. Perplexity states that Perplexity-User generally does not honor robots.txt, because that fetch is user-driven. So this block is honest about its limits:
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Disallow: /
Google is the odd one out. Google-Extended is not a crawler at all. It is a product opt-out token, and this is the single most misunderstood point in the whole topic. A Google-Extended robots.txt disallow removes your pages from Gemini and Vertex model training, plus certain Vertex AI grounding uses. It does not affect Google Search, Discover, News, Images, or AI Overviews. Those are built from the standard web index that Googlebot crawls. If you want to influence AI Overviews, you use the same controls that govern Google Search: Googlebot rules, noindex, and the rest.
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Disallow:
Done when: for each vendor where you care about the training-versus-fetch difference, you have a separate group for the training bot and the user bot, each with its own rule.
Pro tip: if your only goal is "keep my content out of model training," you mostly want the training tokens (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and can leave the search and user bots alone. Match the tokens to the outcome you wrote in Step 1.
Step 6: Add your sitemap line
This one is quick and easy, and it belongs in your robots.txt regardless of your bot policy.
A Sitemap: directive points crawlers to your XML sitemap. It is global, so it lives outside every User-agent: group, and it takes an absolute URL:
Sitemap: https://yourdomain.com/sitemap.xml
You can list more than one sitemap line if you have several. Drop them at the bottom of the file where they are easy to find.
Done when: your Sitemap: lines sit outside any bot group, and each URL returns a 200.
Common mistake: tucking Sitemap: inside a User-agent: GPTBot group. The directive is site-wide, not per-bot, and burying it in a group is confusing even if parsers forgive it.
Step 7: Deploy, validate, and watch what actually happens
You wrote the file. Now make it real and confirm it works, because a rule that never deploys protects nothing.
Upload the file to your web root and fetch the live URL with curl to confirm what is actually being served. Local edits do not count, and a stale copy cached at your CDN edge can serve the old file for minutes after you deploy. Then run it through Google's robots.txt parsing check in Search Console to catch syntax errors on Google's side.
Here is the patience part. Most compliant crawlers cache robots.txt for up to around 24 hours. OpenAI says as much for its search bot. So after you deploy, wait a day before you judge the effect, and resist the urge to revert a change that was quietly working the whole time.
Then watch. Check your server logs or analytics for the user-agent strings of the four bots and confirm the ones you allowed still appear and the ones you blocked drop off. This is also where knowing your outcome pays off. If your goal was to stay citable while opting out of training, the real question is whether your pages keep earning citations in AI answers after the change.
That last question is hard to answer from server logs alone, because logs tell you who crawled, not who cited. This is where a visibility layer helps. DeepSmith tracks which of your pages AI engines actually cite across ChatGPT, Gemini, Perplexity, Claude, and Google AI Mode, so after your crawler policy goes live you can watch citation rate and see whether your footprint held or shifted. You can absolutely do this piece by hand, but seeing it in one place makes the cause and effect obvious.
Done when: 24 hours have passed, the fetched file matches what you wrote, and the expected bots (or their absence) show up in your logs.
Common mistake: assuming instant effect and reverting good changes before the cache clears.
robots.txt versus noindex: what each one actually does
A quick clarification, because these two get mixed up and they solve different problems.
A robots.txt Disallow tells a compliant crawler not to fetch the URL. It does not remove a URL the crawler already has in its index. A noindex directive, set as a meta tag or an X-Robots-Tag header, tells the crawler not to include the page in its index at all.
Here is the catch that surprises people. If you block a URL in robots.txt, a compliant crawler never fetches the page, so it never sees a noindex tag sitting on it. The two do not stack the way you might expect. If your goal is "crawl it but keep it out of the index," you allow it in robots.txt and add noindex on the page. If your goal is "never fetch it," robots.txt alone does that.
And remember the ceiling on all of this. robots.txt governs crawling by well-behaved bots. Non-compliant scrapers can ignore it entirely. For anything you truly need locked down, use authentication, IP controls, or rate limiting at the edge, not a text file that asks nicely.
What to do next
You now have everything you need: the tokens, the syntax, and the one distinction (training bot versus user fetcher) that most guides skip. Start small. Pick the one bot whose behavior bugs you most, write its block, deploy, and check your logs tomorrow. Momentum beats a perfect file you never ship.
Once your policy is live, the natural next question is whether it changed how AI answers talk about you. If you would rather see your AI citations in one dashboard instead of piecing it together from logs, DeepSmith tracks mention and citation rates across the major engines and shows which pages earn them. You can start a free trial and watch the effect of your new rules with real data.
