What Cloudflare Pay-Per-Crawl Means for Your AI Search Visibility

You just found out your site might be blocking the AI crawlers that decide whether you show up in ChatGPT. That is a lot to sit with. Take a breath, because you are closer to fixing this than you think.

Here is the short version. Cloudflare pay per crawl lets you put a price on every AI bot that visits your pages, or wave it through for free, or shut it out entirely. That single choice quietly decides whether an answer engine can ever cite you. Get it wrong and you disappear from AI answers without a single error message.

By the end of this, you will understand how the mechanism works, what each setting does to your ability to get cited, and exactly how to decide. No jargon, one step at a time.

What Cloudflare pay per crawl actually does

Think of it as a toll booth for bots.

Cloudflare pay per crawl sits inside a product called AI Crawl Control. It launched on July 1, 2025, in a move Cloudflare branded "Content Independence Day." The idea is simple: for the first time, the publisher sets the price, not the crawler.

When an AI bot requests one of your pages, Cloudflare checks your policy for that bot. If your policy is to charge, Cloudflare answers with an HTTP 402 Payment Required response and a header stating your price per request. If the bot is willing to pay and can prove who it is, Cloudflare serves the page and bills the bot's company. Cloudflare acts as the go-between: it collects the money, takes a cut, and pays you.

You get three choices for every bot.

• Allow. The bot fetches your page for free, like normal.
• Charge. The bot gets a 402 and only sees the page if it pays.
• Block. The bot is turned away and sees nothing.

That is the whole model. You are deciding, bot by bot, who gets in, who pays, and who stays out. This all happens at Cloudflare's edge, before your page even loads, so it works whether or not the bot respects the rules in your robots.txt file.

One catch worth knowing now: this only works if your site runs on Cloudflare. Pay per crawl lives on Cloudflare's network, so no Cloudflare, no toll booth.

Why Cloudflare built this in the first place

You might be wondering why any of this exists. It helps to know, because the reason explains the risk.

Cloudflare looked at how much AI bots take versus how much they give back, and the numbers were stark. As of mid-2025, Anthropic's crawler fetched roughly 38,000 pages for every one visitor it sent back to a site. OpenAI sat around 750 pages per referral. Perplexity was near 194 to 1. Compare that to Google's traditional search bot, which sends back roughly one visit for every page it crawls.

So AI engines were reading the open web at industrial scale and returning almost none of the traffic that used to pay for it. That is the crawl-to-click gap, and it is why Cloudflare decided publishers deserved a lever.

There was a second problem. Cloudflare found that many AI crawlers ignore robots.txt entirely, fetching pages you explicitly told them to leave alone. A polite "please don't" was not holding the line.

So Cloudflare changed the default. Since July 1, 2025, any new domain added to Cloudflare blocks known AI crawlers unless the owner opts in. Blocking became the new normal, and pay per crawl became the escape hatch that lets a willing, paying bot back in. Since Cloudflare sits in front of roughly 20% of all web traffic, that default shift is a big deal for the whole web, not just for you.

The one rule that decides your visibility

If you remember nothing else, remember this: block equals no citation. There is no second channel.

Let's walk through what each setting does to your AI search visibility, because this is where marketing leads get surprised.

Allow. The bot fetches your page, reads it, and can quote or paraphrase you in its answers. Citations are possible. Referral clicks are possible. You are giving the content away, but you stay fully visible. This is the right call when reach matters more than revenue.

Charge. A bot that has signed on to Cloudflare's framework can still reach your page if it pays. Your citation eligibility stays intact, and you earn a little per fetch. That sounds like the best of both worlds, and often it is. But here is the trap. If a bot has not signed the framework, your charge policy behaves exactly like a block for that bot. It pays nothing, sees nothing, and cannot cite you.

Block. The bot cannot fetch the page at all. It cannot quote you, paraphrase you, or link to you. A block is also a citation block, full stop. There is no setting that hides your page from a bot while still letting it cite you.

Read that middle one again, because it is the part people miss. When you charge AI crawlers, you are not just flipping a monetization switch. You are also gating your visibility on whether each bot has agreed to pay. That is what pay per crawl AI visibility really comes down to: every dollar decision is also a citation decision.

Here is a simple way to hold all three in your head.

The default that quietly makes you invisible

Now for the part that keeps me up at night on your behalf.

The default for any new Cloudflare domain is block. Not allow. Block.

When you add a domain, Cloudflare's onboarding asks whether to allow or block AI bots, and block is the option sitting there waiting. Plenty of people click through without a second thought. Cloudflare has reported that more than one million customers turned on the AI block toggle in its first year alone.

So think about what that means for you. If nobody on your team ever opened the AI Crawl Control dashboard, there is a real chance you are blocking the exact search bots that feed ChatGPT and Perplexity. You would have no error, no warning, no traffic drop you could point to. Just a slow, silent absence from AI answers.

The absence of a decision is a decision. And for anyone who wants to be found in AI search, it is the wrong one.

The fix is small. Open the dashboard, look at the list of bots hitting your site, and choose allow, charge, or block on purpose. That is it. You do not need to be an engineer. You need to spend twenty minutes making choices that are currently being made for you.

You are in good company here

If this still feels like uncharted territory, it helps to know who has already walked it.

Some of the largest publishers on the web have signed on to charge AI crawlers through Cloudflare's framework. The named partners include Condé Nast, the Associated Press, The Atlantic, Dotdash Meredith, Gannett's USA TODAY Network, Reddit, Quora, Stack Overflow, TIME, and Ziff Davis. Stack Overflow, in a February 2026 post, is the first major publisher to describe its configuration in public detail.

On the other side of the table, Cloudflare says OpenAI, Google, Anthropic, Apple, and Mistral have agreed in principle to the framework. "Agreed in principle" is an honest phrase to sit with. It means the plumbing exists, not that every AI company pays every publisher a set rate. The per-crawl prices are private, and no AI company has published what it pays.

So when you decide to monetize AI bots, you are not running an experiment nobody has tried. You are joining a path that major newsrooms and the biggest model makers are actively building out. You are just doing it at your scale, on your terms.

Search, agent, and training bots are not the same

Here is where you can stop treating "AI bots" as one scary blob. Cloudflare sorts them into three kinds, and each one affects you differently. Once you see the split, the right policy gets a lot clearer.

Search bots fetch your content so an answer engine can index it and cite it later. Think of bots like OAI-SearchBot, PerplexityBot, and Claude-SearchBot. These are your citation lever. If a search bot cannot reach your page, that engine cannot cite you. This is the group you almost never want to block.

Agent bots fetch a page live, in the moment, because a user just asked the model to do something. ChatGPT-User is the classic example, firing when ChatGPT needs to look something up right now. There may be no referral click, but showing up in that real-time answer is still a win. Allowing these is usually the easy call.

Training bots ingest your content to train or fine-tune a model. GPTBot and ClaudeBot live here. There is no citation and no referral. Your page just goes into the model, and you get nothing back in the moment. This is the group where charging or blocking makes the most sense, and it matters more than you might guess: training crawls make up roughly 80% of all AI bot activity.

See the pattern? Training bots are the volume problem. Search bots are the citation lever. You want to be generous with the bots that can cite you and firm with the ones that only take. When you monetize AI bots, start with the training crawlers, because that is where the traffic is heaviest and the return is lowest.

One thing to clear up while we are here, because it worries everyone. Blocking an AI crawler does not hurt your Google rankings. Regular Googlebot, the one that powers classic search, is not treated as an AI crawler by Cloudflare. The AI one is Google-Extended, and that is a separate, opt-in signal for AI features and training. You can be firm with AI bots and leave your traditional SEO completely untouched.

How to decide, one step at a time

You do not need a grand strategy today. You need a short, honest audit. Here is the sequence I would follow.

Step 1: Look before you touch anything. Open AI Crawl Control and see which bots are actually hitting your site and how often. This baseline is the whole game. Without it, you are guessing.

Step 2: Decide by intent, not by name. Set your policy for search, agent, and training as categories first. You can fine-tune individual bots later. Starting broad keeps you from drowning in a list of names.

Step 3: Keep the search bots open. For the bots that drive citations, default to allow, or charge if you want a revenue line and you trust the framework to deliver. Blocking a search bot is a citation forfeit, and you rarely mean to do that.

Step 4: Allow the agent bots. These enable the live lookups that put you in real-time answers. Let them in.

Step 5: Charge or block the training bots. They take the most and give back the least. This is the natural place to draw a firmer line, and it is where you can block AI crawlers at edge without giving up a single citation.

Step 6: Come back to it every quarter. Bot behavior changes. Cloudflare's defaults change. The list of partners changes. What was right in July 2025 may not be right six months later. A quick quarterly check keeps you from drifting back into invisibility.

That is the whole framework. Audit, decide by category, protect your search bots, and revisit. You can do the first pass this week.

Where pay per crawl stops and your content begins

Here is a gentle reality check, so you set your expectations right. Pay per crawl controls who can fetch your page. It does not, by itself, create a single citation.

Letting the right bots in is table stakes. It gets you into the room. But once a search bot can reach your page, whether it actually cites you still depends on the same things it always did: clear structure, crisp answers near the top of each section, schema, and content genuinely worth quoting. Even when you allow everything, AI engines cite far fewer sources than a traditional results page ever did. Access is necessary. It is not sufficient.

This is the part where knowing your real numbers helps. Once your Cloudflare settings are deliberate, the next question is simple: are you actually getting cited now? That is exactly the gap DeepSmith was built to close. It tracks how often AI engines mention and cite your brand across ChatGPT, Perplexity, Gemini, and more, so you can see whether your access changes are translating into real visibility, then produce the on-brand content that earns those citations. You fix the toll booth once; you watch the results continuously.

A couple more myths worth retiring, since they cause bad decisions:

• Content already used for training stays in the model. Pay per crawl is forward-looking only. Charging today does not claw back what was learned yesterday.
• Turning on charge is not a quiet money faucet. If a bot has not signed the framework, charge behaves like block, and you lose that citation. Money and visibility move together here.

You are more in control than you were an hour ago

Let's land this. An hour ago, "AI crawlers" probably felt like weather, something that happens to you. Now you know it is a set of switches you own.

Cloudflare pay per crawl gives you three levers per bot: allow, charge, block. Allow keeps you visible for free. Charge keeps you visible if the bot pays. Block makes you invisible to that bot, and there is no way around that. The whole of pay per crawl AI visibility lives in those three choices. You can allow the search bots that cite you and still block AI crawlers at edge for the training-only bots that just take. The default for new domains is block, which is why so many sites are quietly missing from AI answers without knowing it.

Your next step is small and specific. Open AI Crawl Control, check what is set, and make sure your search bots can reach you. That one visit could be the difference between showing up in tomorrow's AI answers and staying invisible.

When you are ready to see whether those changes are actually earning you citations, and to produce the content that wins them, start a free DeepSmith trial and watch your AI visibility in real numbers. You have already done the hard part, which was understanding it. The rest is just clicks.