What is llms.txt? A short index for AI agents — and we checked 50 sites to see who has one

The short version

• llms.txt is a small markdown file at your site root holding a description of your site and a list of your important pages, written so software does not have to pick your HTML apart.
• It is a proposal, not a standard. Version 2 went up on 10 August 2026. Nothing obliges any AI company to read it.
• We fetched /llms.txt from 50 sites a UK business actually uses. 26 had one; 19 returned a 404.
• The split is almost perfectly along one line: 8 of 8 developer-facing tools had one. Every marketplace and every UK government site we checked had none.
• The AI labs publish one for their developer docs, not their main sites. anthropic.com/llms.txt is a 404. docs.anthropic.com/llms.txt is 67,859 bytes.
• Google’s Lighthouse does audit for it — and by Google’s own description the audit cannot fail you. A missing file is marked Not Applicable.
• ⚠️ Nobody can tell you whether anything reads yours. We tried two ways and both are dead ends without server logs.
• ⚠️ Ours is 181,656 bytes — bigger than OpenAI’s, Anthropic’s and Google’s put together. That is the mistake in this post, and it is ours — and it grew by 509 bytes while this was being written.

What the file actually is

The proposal lives at llmstxt.org. It is the work of Jeremy Howard, first published on 3 September 2024, and the version now on the site is v2, which the page records as modified on 10 August 2026. The argument is short enough to quote:

“But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise. Context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money.”

So the proposal is to add “a /llms.txt markdown file to websites to provide LLM-friendly content” — because “Agents are best served by concise, expert-level information gathered in a single, accessible location.”

The format is deliberately plain. An H1 with the name of the site — “This is the only required section” — then a blockquote summarising it, then optional H2 sections each containing a list of links. Each list item is “a required markdown hyperlink [name](url), then optionally a : and notes about the file.” There is also a convention worth knowing about: a section headed Optional, for “links an agent can skip when a shorter context is needed.”

If you already have a sitemap, the difference is this. A sitemap lists every URL you have and says nothing about any of them. An llms.txt names the handful that matter and says why. One is an inventory; the other is a contents page.

Who has one: 50 sites, fetched on 15 September 2026

Rather than repeat the claim that “thousands of sites” have one — which the proposal itself makes, and which we have no way to check — we picked 50 domains a UK small business genuinely touches and requested /llms.txt from each. A status code on its own is not enough, because plenty of sites answer any unknown path with a 200 and a page of HTML, so every response was also checked for whether the body was actually a markdown file.

Grouped by what the company sells, the pattern is much sharper than the headline count:

Read down that table and the file stops looking like an SEO tactic and starts looking like what it is: a documentation convention. The closer a company sits to selling things to developers, the more likely it has one. The places an ordinary person actually goes for an answer — a marketplace, a government site — have none at all.

⚠️ Two honest caveats. This is our list of 50 domains, chosen by hand because they are the tools we see UK businesses using; it is not a random sample and it is not a survey of the web, so the 26 is a fact about this list and nothing more. And four sites refused our request outright — GoDaddy, Sage, OpenAI and Perplexity all returned 403 — while one, QuickBooks, dropped the connection twice. A blocked request is not evidence of a missing file, so those five are counted as unknown rather than quietly dropped into the “no” column.

⚠️ And one bug of our own, since it changed a number: the first run reported Amazon as a script error rather than a result. The response was gzip-compressed and our checker tried to read it as plain text. Re-fetched properly it is a clean 404. A check that errors is not a check that passed, and it would have been very easy to leave that in the unknown column and never look again.

The AI labs put theirs somewhere you would not guess

This is the single most useful thing we found, and it is the reason the headline count above is a bit misleading. The proposal notes that the AI labs publish llms.txt files for their own developer docs, and names OpenAI, Anthropic and Gemini. That is true. But look at where:

All six fetched by us on 15 September 2026. The companies building the agents this file is for have put one on their API documentation and not on the site where they sell to the public. That is not hypocrisy; it is a signal about what the file is good at. It works where a machine needs to find the right page out of eight hundred technical ones. It does far less on a ten-page site about a plumbing business, because there was never much for an agent to get lost in.

If you want the question behind this one — whether AI tools can find and describe your business at all — we wrote that up separately in can AI actually find your business, and the answer there had far more to do with what other sites say about you than with any file you control.

Does Google use it? The honest answer, with Google’s own words

There is one place Google mentions llms.txt in its own documentation, and it is worth reading closely because it is routinely oversold. Chrome’s Lighthouse — the auditing tool built into Chrome’s developer tools — now has an Agentic Browsing category, and inside it an llms.txt audit. Google describes the file there as “an emerging convention used to provide a machine-readable summary of a website’s content, specifically designed for LLMs and AI agents”, and says that without it “agents may spend more time crawling the site to understand its high-level structure and primary content.”

Then comes the part that almost never travels with the headline. Here is what the audit does, verbatim:

“Lighthouse flags the pages if a server error occurs when attempting to retrieve the llms.txt file. If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.”

So the audit cannot fail you for not having the file. It only complains if your server breaks while being asked. And the category it sits in is hedged just as carefully: Google notes that “The Agentic Browsing category and WebMCP support are experimental and based on proposed standards”, and that unlike every other Lighthouse category it “does not have a weighted average score from 0 to 100”, because “the standards for the agentic web are still emerging, the current focus is to gather data and provide actionable signals rather than a definitive ranking.”

None of that is Google Search. It is a developer tool, in an experimental category, reporting an optional file. If someone tells you llms.txt will improve your Google ranking, ask them to show you where Google says so. We looked and could not find it.

The question we could not answer, and the test that fooled us

The obvious thing to want to know is whether anything actually reads the file. We thought we had a way to check. Google’s URL Inspection API will report, for any URL on a site you own, when Google last crawled it. We pointed it at our own /llms.txt and it came back: unknown to Google, never crawled. That reads like proof.

It is not. Before publishing it we ran the same check against our /robots.txt and got exactly the same answer. Google’s own crawler documentation states that “Google generally caches the contents of robots.txt file for up to 24 hours” — so ours is plainly being fetched, repeatedly, and the API still reports it as never crawled. The tool reports whether a URL is in the web search index, not whether a file has ever been fetched. The control killed the finding, and we would have published something confidently wrong without it.

So the honest position is this: from outside your own server, you cannot tell whether anything reads your llms.txt. The only real answer is in your access logs, and if your site is on a hosted platform you may not be able to see them. We would rather say that than publish a number we cannot stand behind. It is the same discipline we applied to the security side of this file in what happens when something does read your llms.txt — which is the other half of the question, and the less comfortable one.

Our own file is the cautionary tale

We have published an llms.txt since long before writing this, and reviewing it for this post was not a pleasant experience. Here it is against the three AI labs’ own files, all fetched the same afternoon:

Those three add up to 107,107 bytes. Ours is 69% bigger than all three combined, and roughly 31 times the size of OpenAI’s. It holds 108 links across 5 sections, and the longest single entry — the descriptive note attached to one blog post — runs to 6,221 characters. That is a note about a link that is longer than OpenAI’s entire file.

Nothing about that breaks the format. The spec explicitly allows notes after each link. But the whole argument for the file is that “every wasted token costs time and money” — and at a rough four characters to the token, ours costs something like 45,000 tokens to read. We built the thing that was supposed to save an agent from reading our whole site, and made it about as long as a good chunk of the site.

The fix is in the spec and we had not used it: one sentence per link, and push everything secondary into the Optional section that exists precisely for “links an agent can skip when a shorter context is needed.” We are publishing the number before fixing it, because a post that quietly tidied up first would be worth less to you.

We ran the whole thing again this morning

Everything above was measured on 15 September. Before publishing we re-ran the lot on 17 September — all fifty requests, all four spec and documentation pages, and every file size. Three things came out of it.

The adoption count did not move. Fifty domains, same classification: 26 with a real file, 19 returning a genuine 404, four blocking us, one dropping the connection. Amazon answered 404 again and QuickBooks failed again, so both stay where they were. A count taken once is a claim; taken twice on dated occasions it is a finding, and you can repeat it yourself in about a minute.

Two of the files had grown, ours most of all. OpenAI’s and Google’s were byte-for-byte identical. Anthropic’s had gained 177 bytes. And ours had gained 509 — because yesterday we published a blog post, and our publishing routine adds a paragraph about it to the file. We wrote 509 more bytes into the file whose length is the confession in this article, while writing the article. That is not irony so much as the actual mechanism: nobody decides to let a file get too long, a process just keeps adding to it and no one is watching the total.

And one of our own numbers nearly turned out to be nonsense. On the re-run every single file appeared to have shrunk by about two thirds — ours from 181,000 to 73,000, Anthropic’s from 68,000 to 8,500. Four simultaneous rewrites overnight would be an extraordinary coincidence, which is the tell. The cause was one flag: we had asked for compressed transfer this time, so the tool was reporting the compressed size on the wire rather than the size of the file. Measuring the saved files instead settled it in seconds, and the original figures were right.

We are leaving that in because it is the same mistake the whole article is about. A number is only as good as your knowledge of what it counted — and “bytes” turned out to mean two different things depending on how we asked.

So should you write one?

A straight answer, with no upsell attached.

• If you have a large documentation site, technical or otherwise: yes, and it is the clearest case. That is the job the file is demonstrably good at, and it is the reason every developer-facing company in our sample has one.
• If you have an ordinary small-business website: it is cheap and it is harmless, so write one if you like — but write it in an afternoon and do not pay anyone for it. There is no evidence it moves your search ranking, and an agent that can read your ten pages was never lost.
• If someone is selling you an llms.txt package: the file is a page of markdown with a title, a sentence and a list of links. Ask what the rest of the money is for.
• Whatever you write, keep it short. That is the one rule we broke ourselves, and the only one the proposal actually argues for at length.

What argues the other way

Three things, honestly.

Absence of evidence is not evidence of absence. We could not find Google saying llms.txt affects search, and we could not measure whether anything fetches ours. Neither of those proves nothing reads it. AI companies do not publish their retrieval mechanics, and a file being read by an assistant mid-conversation would leave no trace anywhere we can see.

The direction of travel is towards the file, not away from it. Two years ago this was one person’s proposal. Today the three big labs publish one for their docs, documentation platforms generate them automatically, and Google has put an audit for it in Chrome — hedged and toothless, but there. That is not the shape of an idea dying.

And the cost is close to zero. Everything above argues against paying for an llms.txt, not against having one. If it takes an hour and it might help, the sensible answer is to write it, keep it short, and stop thinking about it — which is roughly the opposite of how it is usually sold.

Sources, and what we checked

• The proposal: llmstxt.org, v2, published 3 September 2024, modified 10 August 2026. Fetched raw on 15 September 2026 — both the rendered page and its markdown source — and every quotation above matched against the text of the file rather than a summary of it.
• Google’s Lighthouse audit: the llms.txt and scoring pages under developer.chrome.com/docs/lighthouse/agentic-browsing, fetched raw the same day. The llms.txt page records a last update of 5 May 2026.
• The 50-site check and the six lab URLs: our own requests, run in full on 15 September and again on 17 September 2026, classified on the response body rather than the status code alone. File sizes were taken from the saved files, not from the transfer counter.
• Google on robots.txt caching: Google’s robots.txt specification under developers.google.com/crawling/docs/robots-txt, fetched raw on 15 September 2026 — the source of the “up to 24 hours” quotation used above as a control.
• Search volume: llms txt at about 1,600 UK searches a month, from Google’s own Keyword Planner via our advertising account, checked on 15 September 2026. Planner figures are rounded estimates, not counts.

The image at the top of this page was generated by AI.