The markdown vs HTML AI question has become a recurring one for content and marketing teams building a citation strategy, and the honest answer is narrower than most of the advice around it suggests. Every major AI engine consumes HTML today. Markdown endpoints, the newer option promoted through features like Cloudflare's edge conversion and /page.md routes, reduce token cost and produce cleaner extraction, but no published evidence shows they cause a page to be cited more often. Clean HTML is the floor that determines whether a page is parseable at all. Markdown delivery is an optimization layered on top, valuable in specific situations and neutral in most others.
Framed plainly, the best format for AI crawlers today is well-structured HTML, with Markdown as a situational supplement rather than a replacement. This comparison covers delivery format for machine parsing only. It does not address how to structure on-page content for citability, nor how individual engines handle JavaScript rendering, both of which are separate decisions that matter independently of the wire format a server returns.
The three delivery options at a glance
There are three practical ways to hand content to AI crawlers, and the differences between them are operational rather than about which produces better answers. The table below summarizes the tradeoffs before the sections that follow examine each one.
| Delivery option | What crawlers get | Token cost | Citation evidence | Engineering effort | Best suited to |
|---|---|---|---|---|---|
| Clean HTML only | HTML, parsed by each engine's extractor | Baseline (highest) | None needed; this is the consumed format | None; status quo | Every site, as the required baseline |
| Markdown only | Markdown returned by default at a URL | Roughly 60 to 85 percent lower | No proven citation lift | Moderate; routing or build changes | Documentation and developer sites |
| Both, via content negotiation | HTML by default, Markdown on request | Lower only for requests that ask | No proven citation lift | Moderate to high; caching care needed | Mid-market and enterprise sites with dev resources |
Two points from the table deserve emphasis before the detail. First, the token savings attached to Markdown are real and large, but they accrue to the engine running inference, not to the publisher measuring citations. Second, no row carries citation evidence in Markdown's favor, because the determinants of citation sit in content quality and authority rather than in the serialization a server chooses.
What AI crawlers actually consume today
The short answer is HTML, across every major engine. The AI crawler content format that reaches production retrieval systems in mid-2026 is the same HTML a browser receives, fetched through standard HTTP GET requests, identified by User-Agent string, and subject to robots.txt directives. None of the major crawlers sends Accept: text/markdown as part of its default request profile, which means a Markdown representation is retrieved only when a URL returns Markdown by default or when a bot's developer has explicitly programmed it to request that media type.
OpenAI
OpenAI operates distinct bots for distinct purposes. OAI-SearchBot fetches pages for inclusion in ChatGPT search results, ChatGPT-User handles user-initiated browse actions, and GPTBot collects training data with an opt-out available through a User-agent group in robots.txt. All of them parse HTML through a standard content extractor. A Markdown version reaches these bots only when the endpoint serves Markdown as its default response.
Anthropic
Anthropic runs ClaudeBot for indexing, which the company describes as retrieval rather than training, alongside Claude-SearchBot for search results and Claude-User for browse actions triggered inside Claude. The consumed format is HTML. Anthropic engineers have publicly characterized HTML as their preferred input for agent workflows on the grounds that it is the most expressive option available, a position that qualifies the enthusiasm around Markdown-only delivery.
Perplexity
Perplexity uses PerplexityBot for indexing and surfacing, and Perplexity-Search for live answer generation. The company has stated that both are for search-result linking rather than foundation-model training. The format consumed is HTML.
Google AI Overviews, AI Mode, and Gemini
Google's AI features draw from the same Search index that powers classic web search, so HTML is parsed identically to standard ranking, and JSON-LD structured data is used to ground answers. There is no Markdown-specific behavior in this pipeline. Gemini as a consumer product relies on the same index plus extensions; the fact that the Gemini API can accept Markdown passed directly into a prompt is a developer-product detail, not web-crawler behavior, and it does not change what the crawler retrieves.
Microsoft Copilot and Common Crawl
Microsoft Copilot is backed by Bing's index and consumes HTML on the same terms as web search. Common Crawl fetches HTML at scale, and its downstream pipelines extract clean text from that HTML for use in training corpora. In neither case is Markdown a factor in what gets fetched.
The consistent pattern across every one of these systems is that the operative AI crawler content format is HTML. Markdown endpoints are additive infrastructure a publisher can choose to expose, not a channel any major engine currently requires or requests by default.
Why teams advocate for Markdown endpoints
The case to serve markdown to LLMs rests on three arguments, each defensible on its own terms and each with a specific boundary on how far it extends.
Token economics
Markdown strips the wrapper markup that inflates HTML: the nested div structures, class attributes, inline scripts, and navigation chrome that carry no semantic content for a retrieval system. Cloudflare's own measurement on one of its blog posts recorded roughly 16,180 tokens for the HTML version against roughly 3,150 for the Markdown equivalent, a reduction near 80 percent, with a typical range of 60 to 85 percent depending on page complexity. A reduction of that size lets a retrieval pipeline hold several times more pages in the same context window and lowers per-query inference cost proportionally. The benefit is material at consumer-search scale and incremental at the scale of a single content team, and in both cases it is a saving realized by the party running inference rather than by the publisher.
Semantic clarity
Markdown is structural by design. Headings, lists, tables, and emphasis are first-class tokens rather than styled div and span elements a parser must interpret. For embedding models that depend on finding clean content boundaries, the absence of wrapper noise can improve how reliably a passage is isolated. This argument overlaps with the reasons semantic HTML and heading hierarchy matter for parsing; Markdown simply enforces that discipline mechanically rather than relying on an author to apply it.
Resilience to rendering gaps
Many crawlers do not execute JavaScript, so content that depends on client-side rendering can be invisible to them. Server-rendered HTML addresses this, but it still carries the surrounding noise. A pre-rendered Markdown endpoint sidesteps both problems: it is always available without a hydration step and it contains only the content. For sites where JavaScript rendering has historically undermined LLM visibility, a static Markdown route removes an entire class of failure.
Why the case is not yet proven
Against those arguments sit four constraints that keep Markdown delivery in the category of optimization rather than requirement.
There is no demonstrated citation lift
As of mid-2026, no peer-reviewed or vendor-published study demonstrates that serving Markdown rather than HTML causes any AI engine to cite a page more frequently. Citation rates are governed by content quality, source authority, structured data, and the clarity of the answer a page provides, and the mechanics of how LLMs select and extract citations operate on the extracted text regardless of the format it arrived in. Any claim that Markdown improves citations should be treated as marketing until it is backed by comparative data, which is the position this article takes on the available evidence.
Crawlers are not asking for Markdown
Because no major engine includes Accept: text/markdown in its default headers, a Markdown endpoint that responds only to content negotiation will rarely be exercised by the crawlers a publisher cares about. The endpoint helps only when a bot has been specifically built to request Markdown, which remains uncommon, or when the URL returns Markdown by default to every client, which raises separate duplicate-content questions.
Conversion fidelity is lossy
Automated HTML-to-Markdown conversion is heuristic, and it degrades on exactly the structures that often carry the most value: complex tables with merged cells or nested headers, captioned figures, definition lists, embedded charts and widgets, and ARIA roles that encode accessibility and meaning. Media is preserved as references only. Where a page's substance lives in those structures, a mechanical Markdown conversion is a downgrade rather than an improvement, and the token saving comes at the cost of information the engine would otherwise have received.
Divergence creates cloaking risk
If a Markdown representation differs materially from the HTML version, whether through added keywords, hidden sections, or altered content, every major engine treats the divergence as cloaking, which carries the risk of ranking penalties and citation exclusion. The mitigation is straightforward in principle: derive both representations from the same source so they differ only in markup, never in substance. It is nonetheless a real operational constraint that any dual-format setup has to enforce deliberately.
How to serve Markdown correctly, if you choose to
For teams that decide to serve markdown to LLMs alongside their HTML, four implementation patterns exist, and they differ mainly in engineering cost and in where the conversion happens. Content negotiation is the technical foundation beneath most of them: text/markdown was registered as a media type in RFC 7763 in March 2016, and any server that switches representation based on the Accept header must emit a Vary: Accept header so that caches and CDNs do not serve HTML to a client that requested Markdown or the reverse.
Edge conversion (Cloudflare Markdown for Agents)
Cloudflare's Markdown for Agents, launched in February 2026, is a network-level toggle that intercepts a request carrying Accept: text/markdown, fetches the HTML from origin, converts it at the edge, and returns it with the correct content type and Vary: Accept header. It requires no origin changes and, during its beta, is free for paid Cloudflare zones. The tradeoff is that the conversion is the heuristic kind described above, so it suits text-heavy content such as articles and knowledge bases better than structure-heavy pages.
Origin-level content negotiation
A server can read the Accept header in middleware and branch its serialization accordingly, the pattern some API-driven products use to return either HTML or a structured format from the same route. This gives the publisher full control over the Markdown output rather than delegating it to a heuristic converter, at the cost of building and maintaining the routing logic and the caching discipline that content negotiation demands.
Static Markdown routes
Sites built with static generators can author in Markdown, render HTML for browsers, and publish the raw Markdown at a parallel /page.md route. Crawlers that know to try the .md path receive the canonical source directly. This is the cleanest pattern for documentation and developer-tools sites, where the content already originates as Markdown and the conversion question does not arise.
The llms.txt convention
The llms.txt proposal asks site owners to publish a structured index of important pages, with short summaries, at the site root, optionally alongside a fuller llms-full.txt. It is a community convention introduced in 2024, not a standard any major engine has committed to honoring, and there is no published metric showing it changes crawl behavior or citation rates. Its cost to add is low and it is harmless if ignored, which makes it a reasonable low-effort addition rather than a dependable signal.
What AI-ready HTML looks like
Because HTML is the format every engine consumes, improving it is the highest-leverage move available, and it is the work that pays off regardless of any Markdown decision. Clean HTML for AI is less about novel techniques than about disciplined application of established structure.
Semantic structure comes first: a single h1, logical heading nesting with no skipped levels, appropriate use of article, nav, aside, and footer, and lists for any enumeration of three or more parallel items. Structured data follows, with JSON-LD blocks for the relevant schema types such as Article, Product, Organization, FAQPage, and HowTo, validated against Google's Rich Results Test, because JSON-LD is explicitly read by Google to ground AI Overviews and AI Mode. The reasons schema markup helps AI citations are well documented, and its value holds whether or not a Markdown endpoint sits alongside the page.
Content placement matters as much as markup. The answer to a page's primary question belongs in the first 100 to 200 words, and each major section reads better for extraction when it opens with a short summary before the detail, a structure that also supports how content is chunked for retrieval. Performance is part of parseability: server-rendered rather than client-only HTML, a fast time to first byte, and no gating of primary body content behind client-side execution. Metadata and accessibility complete the picture, with accurate title, meta description, and canonical tags, a declared language attribute, alt text on images, and captions or transcripts on media. Clean HTML for AI is therefore a checklist of established practices applied consistently, not a new discipline. A technical SEO checklist for LLM retrieval consolidates these into a repeatable audit.
Which should you choose
The best format for AI crawlers in a given case depends on engineering bandwidth and content type more than on any citation promise, because the citation promise does not yet exist. The recommendations below sort by situation.
A small content site with no engineering bandwidth should concentrate entirely on HTML quality: semantic structure, JSON-LD, server rendering, and fast response times. If the site already sits behind Cloudflare, enabling Markdown for Agents is a one-click, zero-code addition that costs nothing during the beta and can be left on. Custom Markdown negotiation is not worth building at this scale.
A mid-market site with development resources can reasonably implement origin-level content negotiation, serving Markdown when the Accept header requests it, caching correctly with Vary: Accept, and keeping the HTML and Markdown representations semantically identical to avoid any cloaking exposure. Adding llms.txt for index discoverability is a low-cost complement.
A documentation or developer-tools site is the clearest case for Markdown delivery, because the content is usually authored in Markdown already. Exposing raw Markdown at /page.md routes, and optionally an llms-full.txt, is the natural pattern, whether implemented through the build or through an edge toggle.
An enterprise team treating this as a high-stakes decision should measure AI citations before and after any format change, run the comparison over a 60 to 90 day window rather than reading noise as signal, and revisit the question quarterly as engine behavior around Accept headers evolves. The discipline here is measurement, not the format switch itself.
Across all four situations, the sequence is the same. Clean, semantic, server-rendered HTML with valid structured data is the requirement. Markdown delivery is an optional layer whose value is highest for documentation-style content and lowest where the case rests on an unproven citation lift.
Measuring the outcome before changing the format
The reason to resist a format-first approach is that the markdown vs HTML AI decision is testable, and testing requires visibility into where a brand is actually cited across engines. Retrieval systems reward content quality and authority, so the productive question is not which serialization to serve but which pages earn citations, on which prompts, and on which engines, and what those cited pages have in common. Answering it calls for tracking mention and citation rates across ChatGPT, Perplexity, Gemini, Claude, and Google AI Mode rather than inferring impact from an infrastructure change.
This is the layer where an AI search analytics platform is useful before any format decision is made. DeepSmith tracks how those engines answer questions about a brand, reports mention rate, citation rate, and share of voice against competitors, and identifies which of a site's pages AI actually cites. Establishing that baseline first turns a format question into an evidence-based one: a team can see whether a Markdown rollout, or more likely an improvement to HTML structure and content quality, moves the citation metrics that matter, rather than adopting a format on the strength of a token-count headline. DeepSmith pairs that measurement with a content production pipeline that builds semantic structure, internal linking, schema, and metadata into articles during creation, which addresses the HTML quality that the evidence actually supports. Teams that want to establish that baseline can start a DeepSmith free trial and measure before they change anything.
