How Perplexity Crawls and Fetches Your Pages: PerplexityBot, Live Fetching, and Why Raw HTML Matters

You searched your own core topic in Perplexity, and there it was: a competitor cited, footnoted, sitting in the answer where your page should be. That stings. It also raises a fair question, because you have good content, so why did Perplexity skip it?

Here is the reassuring part. Once you understand how Perplexity crawls and fetches pages, most of the mystery goes away. It is not luck. It is a system, and systems can be worked with.

By the end of this guide you will know the two ways Perplexity reaches your site, whether it honors your robots.txt, and the one technical detail that quietly makes a lot of good pages invisible to it. Take it one section at a time. You are closer to fixing this than you think.

Perplexity Uses Two Bots, Not One

Start here, because everything else follows from it. Perplexity does not run a single crawler. It runs two, and they behave differently.

The first is PerplexityBot. Think of it as the librarian. It discovers your pages, follows your links, and builds an index over time so Perplexity can recall your content in future answers. It works the way a traditional search crawler works.

The second is Perplexity-User. Think of it as a runner sent out the moment someone asks a question. When Perplexity needs a specific page to ground an answer right now, Perplexity-User goes and fetches it live. This is the Perplexity live fetch path, and it happens at query time, not on a schedule.

Why does this matter to you? Because the two systems run independently. A page indexed months ago can be cited today without being re-crawled. A page PerplexityBot has never touched can still show up in an answer if Perplexity-User pulls it live. One page, two very different doors, and each door has its own rules.

Keep this two-bot model in your head. Every question you have about crawling, blocking, and citations has a different answer depending on which bot you mean.

How Perplexity Crawls: PerplexityBot and the Index

PerplexityBot is the part that behaves most like the search crawlers you already know. It identifies itself with a clear user-agent string, so you can spot it in your logs:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

It also publishes its current IP ranges as a JSON file at perplexity.com/perplexitybot.json. That file carries a timestamp showing when the list was last updated. If you ever allow-list Perplexity by IP, read that file on a schedule rather than freezing a copy, because the ranges change and a stale list will quietly break.

So how Perplexity crawls, at the index level, is not exotic. PerplexityBot fetches your pages, reads the links, and files what it finds so the answer engine can recall it later. The pages it indexes feed long-term memory. That is the slow, steady half of the story.

Here is the good news for you: because PerplexityBot acts like a normal crawler, the habits that help it are habits you may already have. Clean internal linking. Real topical depth. Pages that are easy to reach and easy to parse. Nothing here asks you to reinvent your site.

One caution before you move on. The index is memory, and memory can lag. A page PerplexityBot filed away months ago can still surface in an answer today, even if you have since rewritten it or taken the claim down. That cuts both ways. It means a strong page keeps earning citations long after it was crawled, and it means an outdated page can keep showing up until the crawler comes back around. So getting into the index is not a one-time task. It is a relationship you keep fresh by publishing, updating, and linking so the librarian has a reason to return.

Does Perplexity Respect robots.txt? Yes and No

This is the question site owners ask first, and the honest answer is: it depends on which bot you mean. The Perplexity robots.txt story is genuinely split down the middle.

PerplexityBot, the indexer, respects robots.txt. Perplexity's own documentation says the bot crawls in compliance with your directives. So when you write an Allow or Disallow stanza for the PerplexityBot user-agent, that is the behavior you should expect.

Perplexity-User, the live fetcher, is the other half. Perplexity's crawler documentation states plainly that because a user requested the fetch, this fetcher generally ignores robots.txt rules. The reasoning is that the person already chose to look at the content, so the bot is acting as their proxy. You do not have to love that framing. It is the documented policy, and you should plan around it.

What does the split mean in practice? A few things worth sitting with:

• Blocking only PerplexityBot does not stop live fetching. A page you disallow for the indexer can still be pulled and cited through the Perplexity live fetch path.
• Blocking only Perplexity-User does not stop indexing. A page you disallow for the fetcher can still enter the index and surface in answers built from recall.
• A real opt-out means addressing both user-agents, and as you will see in a moment, sometimes more than robots.txt alone.

This asymmetry is not an accident. It is the core of the two-bot design, and it is the single thing most people get wrong when they assume one robots.txt line will shut Perplexity out.

So when you write your Perplexity robots.txt rules, write for both agents by name. One stanza for PerplexityBot, one for Perplexity-User, each spelled exactly as the bot reports itself. A rule aimed at a name Perplexity does not use will simply do nothing, and you will think you are covered when you are not. If you only remember one line from this section, remember that your Perplexity robots.txt has two audiences, and they do not follow the same rules.

Perplexity JavaScript Rendering: It Only Reads Raw HTML

If you remember one technical detail from this whole guide, make it this one. It is the quiet reason so many good pages never get cited.

PerplexityBot fetches raw HTML and parses it. That is all. It does not execute JavaScript. It does not spin up a headless browser. It does not wait for hydration, lazy-load triggers, or client-side data calls. Perplexity JavaScript rendering, in short, does not happen. If the answer to "what does this page actually say" is not in the initial HTML response, PerplexityBot cannot see it.

Let that land, because it has sharp edges. Picture a modern single-page app built in React, Vue, Angular, or Svelte. It ships an almost-empty shell, something like an empty <div id="root">, and then JavaScript paints the real content into the page in your browser. To you, it looks finished. To PerplexityBot, that page is close to blank.

Here is what tends to disappear:

• Content in a client-rendered SPA that only appears after scripts run.
• Anything behind a "Load more" button, infinite scroll, or intersection-observer lazy loading, unless it is also in the initial payload.
• Content fetched at runtime by client-side calls after the page loads.
• Personalized or A/B-tested content, visible only if the default version is already in the raw HTML.

And here is what still works beautifully, so you can breathe:

• Server-side rendering, where your server sends the full markup. Next.js with the App Router, traditional server-rendered stacks, and similar setups are fully visible.
• Static site generation, where pages are pre-built as complete HTML. Astro, Hugo, Eleventy, and static exports all qualify.
• Hybrid rendering, where the initial HTML already holds the critical content and scripts only layer interactivity on top. The crawler reads the pre-rendered markup and is happy.

Not sure which camp your site is in? There is a quick gut check. Open one of your important pages, view the raw page source (not the inspector, the actual source), and search it for a sentence you can see on screen. If the words are there, PerplexityBot can read them. If the source is a thin shell and your content is missing, that is your answer, and your next project.

Why This Bites Harder on Perplexity Than on Google

You might be thinking: my JavaScript site ranks fine on Google, so what is the problem? Fair. The difference is real.

Googlebot runs a headless browser and will, eventually, render JavaScript-heavy pages, even if it queues them and takes its time. Bing offers partial JavaScript execution. PerplexityBot does neither. So a site that leans on client-side rendering can rank perfectly well on Google and still be functionally invisible to Perplexity. Same page, two completely different outcomes. That gap is exactly where good content quietly loses citations.

Index or Live Fetch: How an Answer Actually Comes Together

It helps to see the whole pipeline once, because it ties the two bots together and shows where your page can enter an answer.

When someone asks Perplexity a question, roughly this happens:

1. The question gets interpreted, rewritten, and expanded.
2. Perplexity checks its own index, the one PerplexityBot built, for pages that match.
3. For fresh or specific needs, Perplexity-User is sent to fetch pages live in real time.
4. The retrieved passages get scored and reranked for relevance, recency, and source authority.
5. A language model writes the answer and embeds citations to the sources it used.
6. The interface shows those citations as numbered footnotes.

Notice the two on-ramps. The index feeds long-term recall. The Perplexity live fetch feeds fresh, per-query evidence. Both can put your page in the final answer, which is why a page does not have to be indexed to be cited, and why an indexed page can still go uncited if nothing selects it that day.

For you, this splits neatly into two jobs. To help the index, treat PerplexityBot like a search crawler: clean HTML, good structure, internal links, real depth. To win the live fetch, make sure your answer-ready facts, your definitions, your named entities, your dated statistics, sit right there in the initial HTML where the runner can grab them fast. Do both and you are covered on both on-ramps.

There is a subtle payoff in seeing the pipeline this way. Getting cited is not the same as ranking, and it is not one single achievement to unlock. It is retrieval, then selection, then synthesis, and your page has to survive each step. The reranker weighs relevance, recency, and authority before a single word of the answer gets written. So the definitions and dates you place up top are not decoration. They are the evidence a machine uses to decide, in a fraction of a second, whether your page is worth quoting. Write for that reader too, and you stop leaving citations on the table.

The Cloudflare Stealth-Crawling Story, August 2025

There is one more chapter you should know, because it complicates the tidy "just use robots.txt" answer. It is worth telling plainly, with the date attached.

On August 4, 2025, Cloudflare published an investigation alleging that Perplexity used undeclared crawlers to get around no-crawl directives. In their tests, they set up domains with a robots.txt that disallowed all crawling. The official PerplexityBot and Perplexity-User agents obeyed the block. So far, so good.

Then, Cloudflare reported, after they blocked the official agents at the network layer, Perplexity's product kept surfacing detailed content from those same domains. The traffic, they said, came from generic Chrome-on-macOS user-agent strings, originating from infrastructure outside Perplexity's published IP ranges, and rotating across multiple networks to dodge simple IP blocks.

Perplexity's response was that the crawlers in question were user-driven agents acting on someone's behalf, not autonomous indexers, and therefore not bound by robots.txt. Cloudflare, for its part, removed PerplexityBot from its verified-bot list and added rules to catch the pattern. Press coverage across outlets like The Verge and AppleInsider treated it as a break from long-standing crawler norms.

You do not need to pick a side in that dispute to take the practical lesson. If your goal is a guaranteed block, robots.txt is necessary but not sufficient. This episode is the clearest reason why. It also explains why the two-bot design exists in the first place: Perplexity treats a user-requested fetch differently from an indexer crawl, and the line between them is not always visible from where you sit.

What to Do Next, Depending on What You Want

You want one of two things here. Either you want Perplexity to cite you more, or you want it to leave you alone. Let's take each, and keep the first step small.

If you want to get cited by Perplexity:

• Render your critical content server-side or as static HTML. This is the highest-leverage fix, because it undoes the invisibility problem entirely.
• Put your definitions, statistics, named entities, and dates in the initial HTML, not behind a script.
• Use a clean heading hierarchy and structured data so the machine understands what each page is.
• Keep pages fast and light. AI crawlers are not patient with slow, bloated pages.
• Allow PerplexityBot in robots.txt so your pages can enter the index at all.
• Support authority with clear authorship, dates, and cited sources, which the reranker leans on.

If that list feels long, do not do it all this week. Pick the rendering check first. It is the one that unlocks the rest.

If you want to block Perplexity:

• Disallow both PerplexityBot and Perplexity-User in robots.txt. That handles the polite, declared bots.
• Add firewall or WAF rules on Perplexity's published IP ranges from the JSON endpoints.
• Use a bot-management system to catch undeclared crawlers that use stealth patterns.
• Consider JavaScript-based challenges on protected paths, since bots that skip JavaScript will fail them.

Neither path is complicated once you know which bot you are talking to. That is the whole reason we started with the two-bot model.

Where DeepSmith Fits

Here is the honest tension. Knowing how Perplexity crawls is one thing. Watching whether it actually cites you, week after week, across every prompt your buyers ask, is another job entirely, and it is not one you want to do by hand.

That is the gap DeepSmith is built for. It tracks how AI engines, including Perplexity, answer the questions that matter in your space: your mention rate, your citation rate, and which of your pages get cited versus which competitor pages win instead. When you can see exactly where you are invisible, the fixes in this guide stop being guesswork and start being a short, prioritized list. And because the same platform produces on-brand, answer-ready content designed to be readable in raw HTML, closing those gaps becomes a step in your workflow instead of a project you keep postponing.

You do not need to overhaul everything at once. Start by seeing where you stand.

Want to know if Perplexity is citing you or your competitor right now? Start a free DeepSmith trial and find out.