llms.txt is a plain-text Markdown file that sits at yourdomain.com/llms.txt and gives AI models a clean, curated summary of your website: your name, what you do, and links to your most important pages. Think of it as a sitemap, but written for a language model instead of a search engine crawler. Instead of forcing an AI system to wade through your navigation bar, cookie banner, ad scripts, and CSS just to find the actual content, you hand it a short document that says, in effect, “here’s who I am, and here’s what actually matters on this site.”
If you’ve already added schema markup and cleaned up your structured data, this is a natural next step in a GEO strategy. It also happens to be one of the few AI-search tactics you can finish in an afternoon instead of a quarter.
Why this file needed to exist in the first place
Websites are built for people, not machines. A single product page might load a mega-menu, three tracking pixels, a chat widget, a newsletter popup, and a slider carousel before you even reach the paragraph you actually wanted to read. Humans filter all of that out without thinking. A language model can’t — it reads the raw content it’s given, and every irrelevant token it processes eats into a hard limit called the context window: the amount of text a model can hold in its “working memory” at once.
So when a model burns that budget parsing boilerplate, less of it is left for the article that’s supposed to answer someone’s question. llms.txt sidesteps the whole problem by handing over a pre-cleaned index instead of asking the model to fight through the HTML.
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed the standard in late 2024. Since then, companies including Stripe, Cloudflare, and Zapier have added the file to their developer documentation, and the format has become something close to an informal convention among AI-facing sites — not an official web standard yet, but a widely recognized pattern.
llms.txt vs. llms-full.txt — they’re not the same file
People mix these up constantly, so it’s worth separating them clearly before going further.
| llms.txt | llms-full.txt | |
|---|---|---|
| What it contains | A curated index — titles, links, and one-line descriptions | The complete text of your key pages, pasted into one long document |
| Typical size | A few kilobytes | Can run into several megabytes |
| Best suited for | Most content sites, blogs, service businesses | Documentation-heavy products, SDKs, sites built for AI coding agents |
| How a model uses it | As a table of contents, deciding what’s worth fetching | As a single source it can read end to end without additional requests |
| Do you need it? | Yes, almost always | Only if AI tools need deep, self-contained context on your content |
For a typical SEO blog, marketing site, or small business, the standard llms.txt file covers what you need. Skip llms-full.txt unless you’re running documentation that developers or AI agents need to consume in full.
The exact format, line by line
The spec stays deliberately minimal — plain Markdown, nothing exotic:
# Your Brand or Project Name
> A one-sentence summary in blockquote form. This line does
> more work than everything else in the file combined,
> because it's what a model reads first to understand who
> you are.
An optional paragraph of extra context. No headings needed here.
## Core Pages
- [Page Title](https://yoursite.com/page-url/): One honest line about what this page covers
- [Another Page](https://yoursite.com/another-page/): Same idea, no fluff
## Optional
- [Lower-priority page](https://yoursite.com/page/): A page worth including, but not essentialA quick breakdown of what each part is actually doing:
| Element | Required? | What it’s for |
|---|---|---|
# H1 |
Yes | Your brand or project name — exactly one line |
> Blockquote |
Yes | The single most important sentence in the file; a model reads this to understand your identity |
| Free paragraph | No | Extra context, written in plain prose, no formatting |
## Core Pages (or similar) |
Yes | Grouped links to your best content, each with a short description |
## Optional |
No | A place for lower-priority pages a model can skip if it’s short on space |
Keep the page count sane — somewhere between 10 and 30 links usually works well. Add a thin, outdated, or purely promotional page and you’re diluting the file, not strengthening it. The entire value of llms.txt comes from curation. A dump of every URL on your site defeats the purpose completely.
Does it actually move the needle on AI visibility?
This is the part most guides oversell, so let’s be straightforward about it instead.
For general-purpose answer engines — ChatGPT Search, Perplexity, Google AI Overviews — there’s no solid evidence right now that these systems routinely check a site’s llms.txt before generating an answer, the way Googlebot indexes a normal page. Several publishers who tested the file in isolation, including inside the SEO industry itself, haven’t measured a clean, attributable lift in citations just from adding it.
Where the file earns its keep is one layer over, in what’s increasingly called the agentic web — the layer where AI tools act on a person’s behalf rather than just answering a chat prompt. Coding assistants and IDE agents like Cursor, GitHub Copilot, Claude Code, and Windsurf actively look for /llms.txt and /llms-full.txt when they’re pointed at documentation, and they use it to decide which pages to pull into context before writing code. Google added an “Agentic Browsing” audit category to Chrome Lighthouse in 2026 that checks for llms.txt as a readiness signal for exactly this kind of interaction.
So the honest framing looks like this:
| If you’re optimizing for… | Does llms.txt help? |
|---|---|
| Google AI Overviews / AI Mode | Not directly — content quality, schema, and authority signals still do the heavy lifting |
| ChatGPT Search / Perplexity citations | Unconfirmed, likely minor on its own |
| AI coding agents (Cursor, Copilot, Claude Code) | Yes — this is where the standard already has real traction |
| Documentation and developer-facing content | Yes — genuinely useful for RAG pipelines and agent workflows |
Treat llms.txt as low-cost infrastructure hygiene and a small bet on where AI browsing is heading, not a shortcut around the fundamentals. The pages that actually get cited in AI Overviews are still the ones with clear direct-answer formatting, real structured data, and third-party authority behind them — the same ground covered in our guides on structured data for AI overviews and E-E-A-T signals that get you cited.
Building your own llms.txt, step by step
You don’t need a developer for the basic version. Here’s the whole process:
1. Pick your 10 to 30 best pages. Pull from your pillar content, cornerstone guides, and core service or product pages. Leave out thin pages, old announcements, and anything purely promotional — a model won’t get much value from them, and neither will you.
2. Open a plain-text editor. VS Code, Sublime Text, or Notepad set to plain-text mode all work fine. Skip Word or Google Docs for this — rich-text editors quietly insert formatting that breaks Markdown parsing, and you won’t notice until the file misbehaves.
3. Write the H1 and blockquote first, and take your time with them. This is the line a model reads before anything else, so write it the way you’d describe your business to a stranger in one breath — specific, not salesy.
4. Group your links under clear H2 headings, each with a short, factual description instead of marketing copy. “A guide to fixing Core Web Vitals on WordPress” beats “The ultimate resource you need!”
5. Save the file as llms.txt and upload it to your site’s root directory, so it loads at yourdomain.com/llms.txt — not inside a subfolder.
6. On WordPress, you can add it through a plugin built for this or by dropping the file into your root folder through FTP or your host’s file manager. Either way, no custom code is required for the standard version.
7. Revisit it whenever you publish or retire a major page. Treat it the way you treat your sitemap — a stale llms.txt pointing at dead links or outdated pages works against you rather than for you.
Mistakes worth avoiding
| Mistake | Why it backfires |
|---|---|
| Treating it as a ranking factor | It has no confirmed effect on classic Google search rankings |
| Listing every URL on the site | Removes the curation that makes the file useful in the first place |
| Writing the blockquote like an ad | Models respond better to plain, specific description than to hype |
| Setting it once and forgetting it | Content changes; an outdated index misleads more than it helps |
| Using a rich-text editor to write it | Hidden formatting can silently break the Markdown structure |
Who’s already doing this well
Stripe, Cloudflare, and Zapier all publish llms.txt files for their documentation, and the pattern across them is consistent: a tight, specific blockquote up top, then a small number of tightly grouped links rather than a full site index. That’s the model worth copying — narrow and curated beats broad and exhaustive every time with this particular file.
FAQ
Is llms.txt the same thing as robots.txt?
No. Robots.txt controls what crawlers are allowed to access on your site. llms.txt tells AI models what your site is about and which pages matter most — one is about permission, the other is about orientation.
Will adding llms.txt get me cited in Google AI Overviews or ChatGPT answers?
Not by itself. Current evidence points toward llms.txt mattering more for AI coding agents and documentation tools than for consumer-facing answer engines. Citations in AI Overviews still come down mainly to content quality, structured data, and third-party authority.
Do I need llms-full.txt as well?
Only if you’re running documentation-heavy content or a product that AI coding tools need deep context on. Most content sites and small businesses can stop at the standard file.
How long does this actually take to set up?
For a site with 10 to 30 core pages already identified, budget 15 to 30 minutes once you sit down to write it.
Kia has worked in SEO and digital marketing for over a decade, building and optimising websites across different industries. He founded Technexies to share what actually works in modern search written from direct professional experience rather than theory. All content on Technexies is researched, written, and reviewed by Kia personally.