You checked your logs, saw three different OpenAI user-agents, and now you are not sure which one to block, which one to welcome, and whether any of them can even read your JavaScript. That is a completely normal place to be. OpenAI runs more than one bot, each does a different job, and the rules for controlling them are scattered across a few docs. Here is the good news: once you can name the three ChatGPT crawlers and what each one wants, the decisions get simple.
This guide is for the marketing or content lead who owns AI search visibility but does not want to guess at server config. By the end you will be able to tell GPTBot vs OAI-SearchBot vs ChatGPT-User apart, answer the question "does ChatGPT run JavaScript" for your own site, and control exactly what each of the OpenAI bots access, training versus live retrieval, with one file you probably already have.
Take it one step at a time. You are closer than you think.
Step 1: Sort the three OpenAI bots by the job each one does
Start here, because everything downstream depends on it. OpenAI documents three separate crawlers that hit your site (plus one ad-validation bot that is out of scope for us). They are not interchangeable, and treating them as one is the single most common mistake operators make.
Here is the plain-English version of each.
GPTBot is the training crawler. It crawls content that may be used to train OpenAI's future foundation models. Think of it as gathering raw material for the model's general knowledge, not for any one answer.
OAI-SearchBot is the search crawler. It surfaces your pages inside ChatGPT's search features, so it is the bot that decides whether you can show up as a cited source in a ChatGPT answer. Per OpenAI, search indexing does not feed model training. Different bot, different purpose.
ChatGPT-User is the live fetcher. It only fires when a real person triggers it: someone pastes your URL into ChatGPT, or a Custom GPT action calls out to your page. It is request-driven, not a proactive crawl.
Here is the part people miss. Each bot gets its own decision. Blocking only GPTBot still leaves OAI-SearchBot and the ChatGPT-User bot free to fetch you. Allowing only OAI-SearchBot still lets ChatGPT-User pull your page the moment a user asks for it.
You know this step is done when you can say, out loud, which bot handles training, which handles search citations, and which handles live user fetches. That mental map is the whole foundation.
Where people go wrong: they read "GPTBot," assume it covers everything OpenAI does, block it, and then wonder why they still are not showing up as a cited source. GPTBot was never the search bot.
Step 2: Learn to spot each bot in your server logs
You cannot control what you cannot see. So before you touch any config, learn to recognize each ChatGPT crawler in your access logs. There are two signals: the user-agent string and the IP range.
Each bot announces itself with a user-agent that names the bot and points to an OpenAI URL. GPTBot carries a GPTBot token and a +https://openai.com/gptbot contact link. OAI-SearchBot carries an OAI-SearchBot token and +https://openai.com/searchbot. ChatGPT-User carries a ChatGPT-User token and +https://openai.com/bot.
One detail saves you real confusion. The version number after the bot name (like /1.4) floats and changes over time, so do not match on it. Match on the bot name and the path in the contact URL instead. When OAI-SearchBot fetches your robots.txt specifically, OpenAI may add an extra robots.txt marker into the user-agent, which lets you tell a rules check apart from a content fetch even when your logs drop the request path.
The second signal is the IP address. OpenAI publishes a separate JSON file of IP prefixes for each bot: one for GPTBot, one for OAI-SearchBot, one for ChatGPT-User, and one for the ad bot. Each file lists a creationTime and an array of prefixes. This is how you verify a request claiming to be GPTBot actually came from OpenAI and is not a spoofer wearing its name.
Pro tip: the published lists are lopsided in a telling way. GPTBot and OAI-SearchBot each span a couple dozen prefixes because they crawl at scale, while ChatGPT-User currently publishes only a small handful, because it fires one page at a time on human demand. If you see a flood of "ChatGPT-User" traffic from outside those ranges, treat it as suspect.
You know this step is done when you can filter your logs to each of the three bots by name and confirm the source IPs fall inside OpenAI's published prefixes.
Where people go wrong: they hard-code today's IP list into a firewall rule and forget it. OpenAI says the ranges change as its infrastructure changes, so a stale list quietly starts blocking the real bot. Re-fetch the JSON files on a schedule.
Step 3: Answer "does ChatGPT run JavaScript" for your own site
This is the question that keeps content leads up at night, so let's be direct about it. For the three main crawlers, the answer to "does ChatGPT run JavaScript" is no.
GPTBot, OAI-SearchBot, and ChatGPT-User all fetch your raw HTML over HTTP, parse the markup, and stop. There is no headless browser step, no waiting for the page to hydrate, no running of client-side scripts. What is in your initial HTML response is what they see. What your JavaScript paints in afterward, they do not.
Sit with what that means for a second. If your page is a single-page app that hydrates its content into the DOM after load, if your key copy arrives through a useEffect fetch, if your body only fills in on scroll or after a "load more" click, then that content is effectively invisible to every one of the three ChatGPT bots. They came, they read an empty shell, they left.
There is one real exception, and it is narrow. Two newer OpenAI surfaces do run a real browser. ChatGPT Agent (the automation mode launched in July 2025) and ChatGPT Atlas (OpenAI's own browser, launched October 21, 2025, on macOS first) both operate inside a real Chromium environment, so they can execute JavaScript. But these are user-driven sessions, not the proactive crawlers that build training and search data. You cannot count on them to index your site. We will come back to how they identify in Step 6.
Common mistake: assuming that because your page looks perfect in Chrome, it looks the same to the bots. Chrome runs your JavaScript. GPTBot does not. The fix is to ship the content in the HTML itself: server-side rendering, static generation, or dynamic rendering that puts the real text in the first response. If you run a JavaScript-heavy stack, this is the highest-leverage change you can make for AI visibility.
You know this step is done when you can view the raw HTML source of a key page (not the rendered DOM, the actual source) and see your headline, body copy, and answers already present as text.
Step 4: Decide what each OpenAI bot can access with robots.txt
Now the fun part: you get to choose. The one control surface OpenAI documents is robots.txt, and because each bot honors its own user-agent token, you can give each of the OpenAI bots access rules that match your strategy.
The most common stance for a brand that wants AI visibility is "do not train on me, but do cite me." You express it by blocking the training crawler while allowing the search and live-fetch bots.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Want the opposite, a full opt-out where you are invisible to all three? Disallow all three by name. Want maximum discoverability? Allow all three, which is also the default if you say nothing. Path-level rules work too, so you can, for example, keep your /drafts/ folder out of OAI-SearchBot while leaving the rest open.
A few honest limits, because false comfort helps no one. Robots.txt is advisory: compliant bots obey it, and OpenAI's do, but it is not a wall. Blocking GPTBot going forward does not pull your content out of a model already trained on it. And changes take time to land: OpenAI documents that a robots.txt update can take up to about 24 hours to propagate through OAI-SearchBot's systems. The propagation lag for GPTBot and ChatGPT-User is not documented, so give any change a day before you judge it.
You know this step is done when your robots.txt names each bot you care about explicitly and the rules match your actual intent, not a copied snippet you half-remember.
Where people go wrong: they block "AI bots" with one vague rule and accidentally lock OAI-SearchBot out, which quietly removes them from citable sources in ChatGPT answers. If you want the citations, keep OAI-SearchBot allowed on purpose. Getting this right is a piece of the same technical foundation that decides whether AI can retrieve you at all.
Step 5: Add IP allowlisting as a second layer, not a replacement
Robots.txt tells well-behaved bots what to do. IP verification proves who is actually knocking. Use the second to harden the first, and never lean on IP rules alone.
Here is the workflow. Pull OpenAI's per-bot JSON files, cache the prefixes, and check incoming requests that claim to be an OpenAI bot against the matching list. A request that carries the GPTBot user-agent but comes from an IP outside gptbot.json is a spoof wearing GPTBot's name, and you can treat it accordingly.
Keep two cautions in mind. First, the lists move. OpenAI is explicit that ranges change with its infrastructure, so schedule a refresh rather than trusting a one-time snapshot. Second, IP-only enforcement is brittle on its own. Real traffic that routes through shared egress, a CDN edge, or a proxy can look wrong, and you can end up blocking legitimate fetches. That is why this is a hardening layer on top of robots.txt, not a substitute for it.
You know this step is done when a spoofed "GPTBot" request from a random IP gets caught, while genuine OpenAI fetches pass because their IPs match the published prefixes.
Where people go wrong: they treat the IP list as the whole strategy and forget robots.txt, or they never refresh the list. Both failure modes are quiet. You do not notice the stale list until your real citations start slipping and you cannot see why.
Step 6: Account for ChatGPT Agent and ChatGPT Atlas
Remember the JavaScript exception from Step 3? Here is how to think about it without over-rotating.
ChatGPT Agent is a user-driven automation mode. When it fetches a page on someone's behalf, it uses the same ChatGPT-User user-agent as any other user-triggered fetch. So your existing ChatGPT-User rules already cover it. Nothing new to configure.
ChatGPT Atlas is different, and this is worth understanding so you do not chase a control that does not exist. Atlas is a full browser built on Chromium, with ChatGPT sitting beside the page as an assistant. Because it is a real browser, its requests carry a standard Chrome user-agent with no OpenAI-specific token. To your server, an Atlas visitor looks like an ordinary Chrome visitor. You cannot reliably separate the two by user-agent alone.
So what can you actually do? Accept that Atlas browsing is human browsing and treat it that way. The only reliable server-side signals are IP matches to the ChatGPT-User ranges for fetches a user explicitly triggered through a ChatGPT surface, plus behavioral heuristics that are not documented controls. Do not build a robots.txt rule for "Atlas." There is no user-agent for it to match.
You know this step is done when you have stopped looking for an Atlas bot to block and instead confirmed your key pages render their content server-side, so both a bot and a real browser get the full page.
Common mistake: assuming a new OpenAI surface means a new bot to manage. Agent reuses ChatGPT-User. Atlas is a browser, not a crawler. Your three-bot map from Step 1 still holds.
Step 7: Verify what ChatGPT actually sees, then keep watching
You have made your decisions. Now confirm they produce the outcome you wanted, because config that looks right and behavior that is right are not the same thing.
Do three checks. Fetch your own robots.txt and read it back as each bot would, confirming the rules resolve the way you intended. View the raw HTML of your top pages and confirm your real content is present without JavaScript. Then watch your logs for a week and confirm you are seeing OAI-SearchBot and ChatGPT-User fetches on the pages you want cited, and no GPTBot fetches on the paths you disallowed.
This is also where measurement beats guessing. Server logs tell you a bot arrived. They do not tell you whether ChatGPT actually cites the page in answers your buyers are asking. This is the layer where a platform like DeepSmith earns its keep: its AEO module tracks your citation rate across ChatGPT and shows which of your pages get cited and for which prompts, so you can connect a crawl-access change to a real visibility change instead of hoping. When you fix a rendering problem or open OAI-SearchBot back up, you want to see the citation curve move, not just the log line appear.
DeepSmith also produces content that ships its answers in clean, server-rendered HTML with the structure AI engines can lift, which closes the loop from "the bot can reach me" to "the bot cites me."
You know this step is done when your logs, your raw HTML, and your citation tracking all agree with the strategy you set in Steps 4 and 5.
Where people go wrong: they change robots.txt, see nothing happen the same afternoon, and revert. Give it the documented propagation window of about a day for OAI-SearchBot, and judge the result on citations over a couple of weeks, not on the first log line.
What to do next
You do not have to fix everything today. If you only change one thing this week, make it Step 3: confirm your most important pages ship their real content in raw HTML, because that single fix decides whether any of the three ChatGPT crawlers can read you at all. Then set your robots.txt intent in Step 4, add IP verification in Step 5 when you have time, and start watching citations so you can see the payoff.
You have already done the hardest part, which is understanding that GPTBot vs OAI-SearchBot vs ChatGPT-User are three different jobs with three different controls. The rest is just applying it, one page at a time.
If you would rather see where you already show up in ChatGPT, which pages get cited, and what to fix first, start a free DeepSmith trial and let the data show you where to point your effort.
