How to Create and Implement an llms.txt File for AI Search Visibility

You keep hearing that you need an llms.txt file, and every explainer stops right before the part where you actually build one. That is frustrating, and it is also normal. The good news: creating one is a short, concrete job you can finish in an afternoon.

This guide walks you through it end to end: how to create llms.txt from scratch and implement llms.txt on your own domain. By the end you will have authored, formatted, hosted, and validated a working file, plus an optional llms-full.txt companion, and you will know exactly where it lives and how to keep it current. No theory, no hand-waving. Just the build.

Let's take it one step at a time.

Step 1: Understand what an llms.txt file is (and what it is not)

Before you write a line, get the concept straight, because it saves you from two common wrong turns.

An llms.txt file is a small Markdown file placed at the root of your domain, served at a path like https://yourdomain.com/llms.txt. It gives large language models a curated, human-readable map of your most important pages. The idea was proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI, on the page at llmstxt.org. The reference implementation lives in the AnswerDotAI/llms-txt repository on GitHub under an Apache-2.0 license.

Here is the part people miss. It does not replace robots.txt, and it does not replace sitemap.xml. Your sitemap is an exhaustive machine list of every indexable URL, built for search crawlers. Your llms.txt is a short, curated, editorial pick of the pages most useful to an AI model, written in Markdown so a model can parse it directly. It complements both. It replaces neither.

How to tell this step is done: you can explain in one sentence why llms.txt is a curation file, not a crawl-control file.

Where people go wrong: treating it as a replacement for robots.txt or your sitemap, then wondering why nothing behaves the way they expected. It is a handoff, not a gate.

This is the whole point of llms.txt AI search readiness: you hand a model a clean, opinionated shortlist instead of making it guess. Get that mental model right and the rest of the build is mechanical.

Step 2: Inventory the URLs you want AI to find

Now the real work starts, and it is more editorial than technical. You are choosing what matters.

Pull your 30 to 80 most authoritative pages. Do not dump your whole site. Every entry you add dilutes the signal of the ones that count, so a tight list beats a long one every time. Pages that almost always belong:

• Your pricing page, with the canonical price facts written into the note
• Your main product or feature pages
• Your top five to ten documentation pages
• Your top five to ten blog posts or guides by traffic
• About, company, and contact pages
• Any page that answers a real buyer question directly

Not sure which pages carry the most weight? Look at what already earns attention. The pages AI engines already cite, and the ones where a competitor is winning instead of you, are the clearest signal of what belongs on the shortlist. This is where an AI search analytics platform like DeepSmith helps: its AEO Pages view shows which of your pages AI actually cites and where the citation gaps are, so your inventory is grounded in real data instead of a hunch. You are prioritizing pages by evidence, not by guesswork.

How to tell this step is done: you have a ranked list of 30 to 80 URLs, and you could defend why each one made the cut.

Where people go wrong: stuffing hundreds of URLs in to feel thorough. Curate hard. Everything that is not top-tier goes in an ## Optional section later, or gets left off.

Step 3: Publish a Markdown mirror of every page you link

Here is the detail that quietly makes or breaks the file: llms.txt points to Markdown, not to rendered HTML.

If your links point at normal HTML pages, a model has to wade through navigation, JavaScript, and styling to find your actual words. Point it at a clean .md version instead, and it reads your content directly. You have three ways to produce those mirrors:

1. Publish static .md files alongside your HTML. This is the Stripe pattern, where docs.stripe.com/webhooks has a matching docs.stripe.com/webhooks.md.
2. Generate them at build time with a static-site plugin.
3. Add a server route that streams the Markdown source with Content-Type: text/markdown; charset=utf-8.

Pick whichever fits your stack. The output is the same: every URL you plan to list has a Markdown twin that returns clean text.

If your site is JavaScript-heavy, this step matters even more, because a rendered page can be nearly invisible to a model that does not execute your scripts. A Markdown mirror sidesteps that problem entirely.

How to tell this step is done: every URL on your Step 2 list has a working .md version that loads plain Markdown.

Where people go wrong: linking to rendered HTML anyway. The index still gives the model your titles and notes, but it never gets your actual content. You have built half a file.

Step 4: Author the file, and how to create llms.txt that a model can read

This is the part everyone worries about, and it is the easiest. The format is short and strict, and once you see it, you have it.

Your file has a required order:

1. An H1 heading with your site or project name. This is the only mandatory line.
2. A blockquote summary: two to four factual sentences on what the site is, who it serves, and what problem it solves. This is the highest-signal prose in the whole file, so write it plainly, with no marketing fluff.
3. An optional info block for extra context like usage or licensing. No headings allowed inside it.
4. One or more ## Section lists, each an H2 followed by a Markdown list of links.
5. An optional final ## Optional section for archives, changelogs, and edge cases that tools can skip under a tight context budget.

Each list item follows one shape:

- [Descriptive Title](https://full.url/path.md): optional one-line note

Write the title as a descriptive phrase, because that text is the highest-signal string the model sees. "Webhook signature verification" beats "Webhooks." Add the one-line note whenever there is a key fact the model should catch: a price, a version, a limit, a date.

Here is a copy-ready template. Edit the bracketed fields and you have a first draft:

# [Your Brand Name]

> [One factual sentence about what your product does and who it serves.]
> [One sentence on the problem it solves.]
> [One sentence on what makes it different.]

## Documentation

- [Getting started guide](https://[yourdomain]/docs/getting-started.md): Step-by-step setup in under 10 minutes.
- [Authentication](https://[yourdomain]/docs/auth.md): API keys, OAuth, and session tokens.
- [Webhooks](https://[yourdomain]/docs/webhooks.md): Event types, signature verification, retry behavior.

## API Reference

- [Create resource](https://[yourdomain]/api/create.md): POST endpoint, parameters, response shape.
- [List resources](https://[yourdomain]/api/list.md): Pagination, filtering, rate limits.

## Guides

- [How to migrate from a competitor](https://[yourdomain]/guides/migrate.md): Side-by-side mapping and a script.
- [Pricing explained](https://[yourdomain]/guides/pricing.md): Plan tiers, overages, and how invoices work.

## About

- [Company](https://[yourdomain]/about.md): Who we are and what we build.
- [Contact](https://[yourdomain]/contact.md): How to reach sales and support.

## Optional

- [Full changelog](https://[yourdomain]/changelog.md)
- [Legacy v1 API](https://[yourdomain]/legacy/v1.md)

Keep your section names short and topical: Documentation, API Reference, Guides, Pricing, About. Skip your internal information-architecture jargon.

Want a real llms.txt example to model yours on? The live files from Anthropic and Stripe docs are the ones the community points to most, and studying either before you finalize yours will sharpen your section choices.

If you write for the brand full time, you can author this by hand for full control. If you would rather start from your existing content, the same brand context that already shapes your articles, your positioning, product facts, and priority pages, is exactly what feeds a strong blockquote and note set. A platform like DeepSmith stores that context once and reuses it, so the factual summary you need here is already written down somewhere you can pull from.

Pro tip: write your blockquote first and read it aloud. If it does not tell a stranger what you do and who you serve, the rest of the file cannot rescue it.

How to tell this step is done: your file has an H1, a factual blockquote, at least one section of Markdown links, and every link points at a .md URL.

Where people go wrong: skipping the blockquote, or putting a heading inside the info block. Both break the format.

Step 5: Generate an llms-full.txt companion (optional)

Once your index is written, you can ship a second file that bundles the actual content. This is optional, so do not stress about it.

Where llms.txt is the index, llms-full.txt is the full Markdown body of every page you listed, fetched and concatenated in order at the same root path, /llms-full.txt. It lets a model ingest your entire curated corpus in a single fetch when its context budget allows. You do not need it for the file to work. Ship it only when your linked Markdown fits comfortably in a modern model's context window and you want one-shot ingestion.

Two ways to build it:

1. Run the reference CLI from the AnswerDotAI/llms-txt repo: llms_txt2ctx https://example.com/llms.txt > llms-full.txt. It fetches each linked .md and concatenates them in order.
2. Add a build step in your static-site pipeline that walks the URLs in your llms.txt, fetches each .md, concatenates them with their headings, and writes the result to public/llms-full.txt.

How to tell this step is done: if you chose to build it, llms-full.txt exists at your root and contains the concatenated Markdown of your listed pages, in the same order as your index.

Where people go wrong: confusing the two files. llms.txt is the map. llms-full.txt is the full text. They serve different needs, and both live at the root.

Step 6: Host the file at your domain root

To implement llms.txt, you serve it where a model can reach it: the root of your domain, over HTTPS, with no login required, and not blocked in robots.txt.

Serve it as plain text. Content-Type: text/plain is universally safe; text/markdown is more accurate when your host supports it. Use UTF-8 encoding. Where you drop the file depends on your stack:

• Static hosts (Netlify, Vercel, Cloudflare Pages, GitHub Pages, S3 with CloudFront): put it in the project root or your public/ output directory. It is served at /llms.txt automatically, no config.
• Next.js, Astro, Nuxt, Hugo, Eleventy, Jekyll: place it in public/. The build copies it verbatim.
• WordPress: install the "Website LLMs.txt" plugin by Ryan Howard, first published January 4, 2025. It auto-generates the file, supports custom post types and exclusion lists, and pulls URLs from Yoast, Rank Math, SEOPress, or AIOSEO sitemaps.
• Nginx or Apache with a custom CMS: if the file lives in the webroot, you are done. If you use rewrites, add an exception so they do not strip the .txt extension or route the path away.
• Object storage (Cloudflare R2, S3, GCS): upload it as a public object at the path llms.txt and set Content-Type: text/plain.

How to tell this step is done: you can open https://yourdomain.com/llms.txt in a browser and see your file.

Where people go wrong: accidentally blocking /llms.txt in robots.txt, which guarantees no AI system reads it. Never add a disallow rule that covers the file. And remember you need one file per host: a file on docs.example.com does not cover blog.example.com.

Step 7: Verify and validate your llms.txt file

You are almost there. Now confirm the file is both reachable and correctly formatted, because a file with a silent formatting error helps no one.

First, check reachability from the command line:

curl -I https://yourdomain.com/llms.txt

You want an HTTP 200, a text Content-Type, and a non-zero body length. Repeat it for llms-full.txt if you shipped one. Then open the body and eyeball it: H1 present, blockquote present, sections in order, every .md URL valid.

Next, validate against the spec. Paste your content or submit the URL to a checker like llmstxtchecker.net or the Radarkit LLMs.txt Validator. These confirm the H1 is present, the blockquote is present, there are no headings inside the info block, every list item is a valid Markdown link, ## Optional is recognized if you used it, and there is no stray HTML or images inside list items. If you are on Mintlify, its built-in checker does the same job inside the product. Fix whatever gets flagged, re-upload, and re-verify.

It also helps to check your live file the way an AI search analytics tool would: does it point only at pages that genuinely earn or deserve citations? DeepSmith's AEO Pages view keeps showing which of your pages AI cites after the file goes live, so you can see whether your curated shortlist is actually the set winning attention, and adjust the file when it is not.

How to tell this step is done: curl -I returns 200, and a spec validator reports no errors.

Where people go wrong: shipping without validating, then discovering weeks later that a missing blockquote or an HTML tag quietly made the file non-compliant.

Step 8: Set a refresh cadence and an owner

A file you never touch again slowly turns into a liability. The last step keeps it honest.

A file that points at removed or 404'd pages is actively misleading, worse than having no file at all. So treat it like a living document:

• Cadence: review quarterly for most sites, and per release for fast-moving SaaS docs and product pages.
• Ownership: assign one named owner, usually the docs lead or the content marketing lead, and add the review to your release checklist.
• Drift control: generate llms.txt from your sitemap at build time so the index cannot drift from your real URL set.
• Link health: fail your CI build if any link in the file returns a non-200.
• Audit header: embed a generated comment at the top, like <!-- llms.txt generated 2026-04-15 from sitemap @ commit a1b2c3d -->, so anyone can see when it last changed.

Keeping the shortlist current is easier when something already tracks your published pages and keeps them classified as they change, which is the kind of always-current content map that makes a quarterly refresh a ten-minute job instead of an afternoon.

How to tell this step is done: the file has a named owner, a review date on the calendar, and ideally a CI check guarding its links.

Where people go wrong: shipping once and forgetting. Set the reminder now, while you are thinking about it.

What to do next

You now have a valid, hosted, validated llms.txt file, and a plan to keep it current. That is the whole build, start to finish.

Here is the honest next move: real llms.txt AI search visibility comes from the file and the pages it points to both earning their place, and a clean file is a handoff, not a guarantee. The pages you point to still have to be the ones worth citing. Before you polish the file further, it is worth auditing where you actually show up in AI answers today, so your shortlist reflects real visibility, not just the pages you wish were winning.

If you want that visibility picture and the content engine to close the gaps in one place, start a free DeepSmith trial and see where you show up in AI search before you optimize a single line.