llms.txt: what it does, and what it doesn’t

AI search 14 September 2026·9 min read

Every AI-visibility checklist written in the last year tells you to add an llms.txt file. The server logs say almost nothing reads it. Both things are true, and I publish one on this site anyway — so this is what the file is, what the evidence actually shows, and why I think it is worth ten minutes but not a strategy.

What llms.txt is

llms.txt is a plain markdown file served from the root of a site, at /llms.txt. Jeremy Howard proposed it on 3 September 2024 as a way to hand language models a short, curated map of a site, instead of making them wade through navigation, scripts and cookie banners.

The format is deliberately loose. Under the proposal, only the first of these is required:

  • An H1 with the name of the site or project. The only required part.
  • A blockquote with a short summary of what someone needs to know to understand the rest.
  • Any amount of prose or lists, without headings, for detail.
  • Sections under H2 headings holding lists of links, each with an optional note.
  • A section called “Optional”, by convention, for secondary material a tool can skip when space is short.

The proposal also suggests offering clean markdown versions of important pages at the same address with .md added. A minimal file:

# Example Consulting

> Independent consultancy for brands selling in several countries.

## Services

- [International SEO](https://example.com/seo/): hreflang, structure
- [Audit](https://example.com/audit/): five modules, fixed price

## Optional

- [Notes](https://example.com/notes/): long-form articles

Two things it is not. It is not robots.txt: it grants and blocks nothing, and a crawler that ignores it loses nothing. It is not a sitemap: a sitemap tries to list everything, while llms.txt is meant to be a short editorial selection.

What the evidence says

This is the part most articles about llms.txt leave out, so it is the part worth reading slowly.

Google, June 2025 and June 2026

Asked about the file, Google’s John Mueller wrote on Bluesky on 17 June 2025: “FWIW no AI system currently uses llms.txt.” He added that consumer chatbots do fetch pages, for training and for grounding answers, “but none of them fetch the llms.txt file” — and that anyone can see this in their own server logs.

Google’s documentation is now explicit. Its guide to optimising for generative AI features in Search lists llms.txt files first among the things site owners can ignore, because Google Search itself does not use them, and on 15 June 2026 Google added a note saying that creating one “will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Keeping one for other services that do use it, the note adds, is completely fine.

137,210 domains, May 2026

Ahrefs checked at scale. Across the server logs of 137,210 domains in May 2026, 28 per cent — 38,360 sites — published a valid llms.txt. Ninety-seven per cent of those files received no requests at all that month. Among the few that were requested, a coding agent, Claude Code, fetched the file more often than any AI search or assistant bot. And no AI bot ever requested llms.txt from a site that did not have one. Nothing goes looking for it.

83 sites, twelve weeks, 2026

The most granular test I have seen is EZY Research’s: 83 sites with an llms.txt, monitored for twelve weeks between 27 April and 19 July 2026. Meta’s crawler fetched the file 193 times, Googlebot 67, ClaudeBot 9, GPTBot 7 — and Perplexity not once. Another 1,001 fetches came from bots that could not be verified.

A horizontal bar chart of llms.txt requests across 83 sites over twelve weeks in 2026. Meta’s crawler made 193 requests, Googlebot 67, ClaudeBot 9, GPTBot 7 and Perplexity none. A note below adds 1,001 requests from unverified bots.
The most-read file in the study was read by Meta, not by any of the engines people usually mean by AI search. Data: EZY Research, April to July 2026.

What that adds up to

A file being fetched is not a file being used. Googlebot downloads almost everything it can reach, and Google’s guide makes the same point: crawling a file does not mean treating it in any special way. For Google Search, the question is now answered in writing. For everything else, I have not seen any published evidence connecting an llms.txt file to more citations, more AI traffic or better rankings — and the people selling it as a citation lever tend not to show any.

The honest summary

llms.txt is a reasonable proposal that the systems most people care about have, so far, largely not adopted. Treat it as a cheap, speculative bet — not as a tactic.

Why I publish one anyway

This site has an llms.txt, and you can read it here. I keep it for three reasons, and none of them is that it will get me cited.

  • It costs almost nothing. It is a text file that only changes when services or positioning do, which is rarely.
  • Some things do request it. Meta’s crawler asked for it more than anything else in the EZY data, and agents that a person points at a site request it too — in Ahrefs’ count, a coding agent out-fetched every AI search bot.
  • Writing it is useful on its own. Putting every fact about a business in one short document exposes contradictions between pages that nobody had noticed. That consistency matters far more to AI visibility than the file does.

The third reason is the real one. An engine that cannot work out who you are, consistently, will not assert things about you — the entity problem I cover in answer engine optimisation for multi-market brands. The file is a by-product of doing that work properly, not a substitute for it.

What does move AI visibility

If you have ten minutes for AI search this week, these are better uses of them:

  1. Check robots.txt — and your CDN

    This is where access is actually decided. OpenAI documents that sites which opt out of OAI-SearchBot will not be shown in ChatGPT search answers: one line that matters more than any llms.txt. Firewall and bot-management settings can block the same crawlers without anyone ever touching robots.txt. The robots.txt tester shows which AI crawlers your file lets in.

  2. Make sure the content exists without JavaScript

    Text that only appears after client-side rendering is text some systems never see.

  3. Get mentioned by other people

    In Ahrefs’ study of 75,000 brands in Google’s AI Overviews, branded web mentions correlated with visibility at 0.664 — against 0.218 for backlinks. That is one engine rather than all of them, but it is the best large-scale evidence available, and it points away from files on your own server.

If you write one, write it like this

  • Make it a fact sheet, not a copy of the site. Who you are, what you sell, what it costs, who it is for — and links to the pages that prove each one.
  • Every fact must match the page it summarises. A price in llms.txt that differs from the price on the service page is a contradiction you have published. I change both in the same commit.
  • Link, do not paste. The file should point at pages, not duplicate them.
  • Never say anything the pages do not. A file that tells machines something different from what people see follows the logic of cloaking, and it is the quickest way to make a curated source untrustworthy.
  • Decide how to handle a second language. Mine is one file with an English body and a Spanish section pointing at the Spanish pages, rather than two files. That is a judgement call: no crawler currently gives a reason to prefer either.

Sources

Questions

A markdown file at the root of a site, proposed by Jeremy Howard in September 2024, giving language models a short curated summary: a title, a one-paragraph summary, and lists of links to the pages that matter. It grants and blocks nothing — that is still robots.txt’s job.
Not for Search. Google’s guide to its generative AI features says Google Search does not use llms.txt files, and a note added on 15 June 2026 says one neither harms nor helps your visibility or rankings there. Googlebot may still request the file, but requesting is not using.
No evidence says so. Across 83 sites monitored for twelve weeks in 2026, GPTBot requested the file seven times in total. What decides whether ChatGPT search can show your pages is whether OAI-SearchBot may crawl them.
It takes minutes and does little harm, so there is no strong reason not to — provided every fact matches your pages exactly. Just do not expect it to move anything, and do not spend budget on it that belongs to crawlability, content or earning mentions.