What is llms.txt used for?

It's a Markdown file listing your most important pages so AI systems can find them without parsing your whole site. It was designed to help AI coding tools read developer documentation efficiently, not to improve marketing visibility.

Does llms.txt help SEO?

No. It isn't a ranking factor, and Google has stated you don't need it for AI Overviews or AI Mode. Crawlability, content quality and structure all matter considerably more.

Do ChatGPT, Claude or Perplexity read llms.txt?

None of them have committed to it. Anthropic publishes its own llms.txt but has never said its crawlers use the standard. In controlled testing, AI bots didn't request the file once over three months.

Is llms.txt an official standard?

No. It's a proposed convention with no governing body and no formal adoption by any major AI provider.

How do I check if my llms.txt file is working?

Load yoursite.com/llms.txt in a browser, or run curl -I https://yoursite.com/llms.txt and look for a 200 response. To find out whether anything reads it, search your server logs for requests to that path.

What's the difference between llms.txt and llms-full.txt?

llms.txt is an index — a list of links. llms-full.txt is a complete text export of your content in a single file. The second is used mainly by documentation platforms and can become very large.

Should I create llms.txt even if AI might not read it?

If you run developer docs, yes — that's the genuine use case. Otherwise it's optional. It costs little and risks little, but your real effort belongs in crawler access, extractable writing and third-party corroboration

What Is llms.txt? And Does AI Actually Read It?

llms.txt is a plain-text file you put at the root of your site to show AI systems where your best content lives. It’s a proposed convention, not an official standard – and so far, no major AI provider has committed to reading it.

That’s the honest headline. Whether you should still build one is a slightly longer conversation, so let’s have it.

Key takeaways

llms.txt is a Markdown file at yoursite.com/llms.txt listing your most important pages.
It was built for AI coding tools reading developer docs, not for marketing visibility.
In a controlled experiment, no AI bot requested the file in three months – on sites those same bots were actively crawling.
Ahrefs looked at 137,000 domains: 28% had one, and 97% of those files were never requested.
Google says you don’t need it. John Mueller compared it to the old keywords meta tag.
Cheap to build, low risk, no measured benefit. Build it if you like. Don’t call it a strategy.

So what is llms.txt?

Think about a new hire on day one. You can let them wander the office working out who sits where, or you can hand them a single sheet of paper: here’s the important stuff, here’s where to find it.

llms.txt is that sheet of paper, written for machines.

It’s a Markdown file at your root domain listing the pages you most want AI systems to find, with a line explaining what each one covers:

# AEORanks

## Guides

- [What is AEO](/what-is-answer-engine-optimisation/): How answer engine optimization works

- [AEO Glossary](/aeo-glossary/): 75+ AI search terms explained

## Services

- [AEO Services](/aeo-service-agency/): What we do and who we do it for

As an idea, it’s completely sensible. We’ve had robots.txt since 1994 and XML sitemaps since 2005, both for the same reason – crawlers needed a hand finding things. So when AI assistants started reading the web at scale, someone reasonably thought: let’s do that again, for them.

Good instinct. What happened next is where it gets interesting.

It was never built for marketers

This is the detail that reframes the whole debate, and most articles skip it.

llms.txt was proposed to fix a developer problem. When an AI coding assistant reads your API documentation, it has to chew through HTML – nav bars, sidebars, cookie notices, styling, the lot – spending tokens on page furniture instead of actual content. A clean Markdown index cuts that waste.

That was the entire use case. Token efficiency for machines parsing technical docs.

John Mueller at Google said it plainly: llms.txt is “not done for search.” He described it as a “temporary crutch, perhaps to save some tokens.”

Somewhere between that proposal and your LinkedIn feed, it became an AI visibility tactic. Nobody announced the change. It just drifted – which is what tends to happen when a whole industry is anxious about a new channel and desperate for something concrete to do about it.

Does AI actually read it?

On the evidence so far, no. Not in any way you’d notice.

Somebody finally ran this properly instead of arguing about it.

The UK agency Reboot built a genuinely clean test. They picked two established sites and first confirmed in server logs that ChatGPT, Gemini and Claude had all crawled them in the previous six months – so the bots were definitely active. Then they published four brand-new pages on each site with no internal links and no external links. Completely orphaned. The only route to those pages, anywhere on the internet, was the llms.txt file.

Then they published llms.txt on both domains, pointed it at those pages, and watched the logs.

After three months, neither site’s llms.txt had been requested once. Not by any AI bot. And during that same window, those bots were crawling other pages on both sites perfectly happily. They just never went looking for the file.

Ahrefs saw the same thing at scale. Across 137,000 domains, 28% had published an llms.txt – more than one in four, which tells you how hard this idea has spread. Of those, 97% received zero requests in May 2026. No bots, no humans, nothing at all.

And here’s the thing about that sample: it’s drawn from sites using Ahrefs’ analytics, meaning tech-savvy owners who follow this stuff closely. If adoption is 28% among the most AI-aware slice of the web, it’s lower everywhere else – and the “nobody reads it” finding probably holds even harder.

The part where Google argues with itself

This bit is almost funny.

In May 2026, Google published guidance on optimising for generative AI features. It included a section headed – and I’m not editorialising here, this is the actual heading – “mythbusting,” which stated that machine-readable files like llms.txt aren’t needed to appear in AI search.

Within days, the Chrome team shipped an experimental Lighthouse audit that checks whether your site has an llms.txt file.

So one part of Google tells you not to bother, and another part adds a tick-box for it. If this file has left you confused, that’s not you failing to keep up. The signals genuinely contradict each other.

Pressed on it, Mueller reached for an older comparison: llms.txt is like the keywords meta tag. Remember that one? A field where you declared what your page was about, which engines eventually ignored because a site’s self-description isn’t evidence of anything. His point was blunt – if you can just read the site, why trust what the site claims about itself?

That’s not a small analogy. The keywords meta tag survived for years after it stopped mattering, kept alive by habit and by tools that still checked for it.

So should you build one?

Yes if you publish developer docs. Optional for everyone else. Just don’t mistake it for strategy.

Three situations, honestly:

You run API docs or a developer reference. Build it. This is the real use case, the one it was designed for. AI coding assistants pull from docs constantly, and a clean Markdown index genuinely helps them land on your current quickstart rather than a changelog from 2023. Platforms like GitBook generate the file automatically, so it may already exist.

You run a normal business website. It’s twenty minutes of work with no measurable downside, and if having one settles an argument internally, fine. One thing worth knowing before you do: you’re also publishing a tidy map of your most valuable content, which makes life marginally easier for anyone scraping you.

Someone is selling you llms.txt as an AI visibility service. That’s the one to question. Ask what evidence they have that it works. The honest answer is none – and if it’s leading their proposal, it’s worth asking what else in there is built on vibes.

How to create an llms.txt file

Still want one? It’s genuinely quick.

Pick your pages. Your strongest guides, product and pricing pages, docs. Leave out thin pages, duplicates, old campaign landing pages and anything archived.

Write it in Markdown. H1 for your site name, H2s to group sections, then a link and a one-line description for each page. Keep it tight – a couple of hundred lines at most. A sprawling file defeats the point, which was always efficiency.

Put it at the root, so it resolves at yoursite.com/llms.txt.

On WordPress: Rank Math and Yoast both generate llms.txt now, which is the path of least resistance. Otherwise upload the file to your root folder via FTP or your host’s file manager.

On Shopify: you can’t write directly to the root, so you’ll need an app from the store or a redirect from a page you create.

To check it: load yoursite.com/llms.txt in a browser. If the text appears, it’s live. For a cleaner confirmation, run curl -I https://yoursite.com/llms.txt and look for a 200.

Then do the thing almost nobody does – check your logs in thirty days and see whether anything actually requested it. You’ll have your own answer instead of borrowing mine.

What actually moves the needle

Here’s where the same research keeps pointing, and it’s considerably less fun than uploading a file.

Let the AI crawlers in. Open your robots.txt and confirm you’re not blocking GPTBot, ClaudeBot, PerplexityBot or Google-Extended. We’ve found this blocking more sites than anyone expects, often from a rule somebody added years ago for a reason nobody remembers. It’s the one technical setting that genuinely decides whether you can appear in AI answers at all.

Write passages that survive being lifted out. AI systems quote paragraphs, not pages. If a section only makes sense once you’ve read the two before it, it can’t be used – no matter how good it is.

Give them a reason to pick you. Original data, real numbers, something you learned by doing the thing. If your page restates what twenty others already say, a model has no reason to cite yours over its own summary.

Get corroborated somewhere else. This is the big one and the one most teams skip entirely. AI systems check whether independent sources back up what you say about yourself. It’s the most common reason AI finds your content but never cites it – the page is perfectly discoverable, nothing external vouches for it.

None of that is as satisfying as ticking a box. It’s just what the evidence supports. Our AEO and GEO best practices guide walks through the full picture, and earning AI citations goes deep on the corroboration side. If some of the vocabulary here is new, the AEO glossary has the definitions.

Want to know where you actually stand in AI answers? A text file won’t tell you – testing will. Book a free AI visibility audit and we’ll show you where ChatGPT, Perplexity and Gemini cite you, where they cite competitors instead, and what’s causing the gap.

What is llms.txt used for?
It’s a Markdown file listing your most important pages so AI systems can find them without parsing your whole site. It was designed to help AI coding tools read developer documentation efficiently, not to improve marketing visibility.
Does llms.txt help SEO?
No. It isn’t a ranking factor, and Google has stated you don’t need it for AI Overviews or AI Mode. Crawlability, content quality and structure all matter considerably more.
Do ChatGPT, Claude or Perplexity read llms.txt?
None of them have committed to it. Anthropic publishes its own llms.txt but has never said its crawlers use the standard. In controlled testing, AI bots didn’t request the file once over three months.
Is llms.txt an official standard?
No. It’s a proposed convention with no governing body and no formal adoption by any major AI provider.
How do I check if my llms.txt file is working?
Load yoursite.com/llms.txt in a browser, or run curl -I https://yoursite.com/llms.txt and look for a 200 response. To find out whether anything reads it, search your server logs for requests to that path.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is an index — a list of links. llms-full.txt is a complete text export of your content in a single file. The second is used mainly by documentation platforms and can become very large.
Should I create llms.txt even if AI might not read it?
If you run developer docs, yes — that’s the genuine use case. Otherwise it’s optional. It costs little and risks little, but your real effort belongs in crawler access, extractable writing and third-party corroboration

Team AEORanks

We’re the team behind AEORanks, combining SEO, AEO, GEO, and AI citation strategy to help brands get discovered, trusted, and recommended everywhere people search — from Google to ChatGPT, Gemini, Claude, and Perplexity.

Schedule a Demo

What Is llms.txt, and Does AI Actually Care About It?