llms.txt: what it is, what is known, and whether to bother

What is the llms.txt proposal?

The proposal was written by Jeremy Howard and first published on 3 September 2024; the spec page shows a modified date of 10 August 2026. It is not an IETF or W3C standard. It asks site owners to serve a Markdown file at /llms.txt (or at a sub-path, in which case it covers URLs beneath it) so that an AI agent reading a site at question time has a concise starting point instead of crawling navigation, scripts and ads.

The format is deliberately small. The only required element is an H1 with the site or project name. After that, in order: an optional blockquote summary, optional free-text sections, and optional H2 sections that each contain a list of links written as [name](url) with an optional colon and note. An H2 called Optional marks links an agent may skip when it needs a shorter context. The proposal also suggests clean Markdown copies of pages (the page URL with .md added) and discovery through alternate or describedby links.

The proposal describes itself as a complement to robots.txt and sitemap.xml, not a replacement, and says it is aimed at use during inference rather than training. It states that thousands of sites have adopted it. That adoption claim comes from the proposal itself and we did not verify it.

What have Google, OpenAI and Anthropic actually said?

Only Google has made an explicit statement we could verify in its own documentation. For OpenAI and Anthropic we can report what their crawler pages say and do not say; that is not evidence either way about what their products do internally. Note too that both companies publish llms.txt for their own developer documentation, which tells you they find it convenient for coding tools, not that their consumer search products consult it.

Is there published data on whether AI crawlers fetch llms.txt?

We could open one primary study. Ahrefs (an SEO software vendor, with its own commercial interest in the topic) published a log analysis on 15 June 2026 covering 137,210 domains that received traffic in May 2026 in Ahrefs Web Analytics. It identified domains serving a valid Markdown llms.txt with HTTP 200 (about 28%, roughly 38,000 domains) and then classified every request to those files by user agent using its Bot Analytics tool.

Its headline: about 97% of those files received no requests in the month. Of the requests that did arrive, about 96% were from bots, named AI bots were roughly 19.5% of requests (GPTBot first, Claude-Code second), and SEO audit tools were the largest single category at roughly 21.7%. The authors themselves caution that their customers skew technical, so 28% adoption is probably an upper bound; that they did not check whether files conformed to the spec; and that a fetch does not show a bot read or used the content.

What this does and does not show: it is a vendor study of requests, not of citations. It says little about whether a fetched file changes an answer. Other studies were mentioned in search results (for example an 83-site test by an AI-visibility vendor and a single-site test by another vendor) but we did not open them, so we do not report their figures.

When is llms.txt worth the ten minutes?

• You publish documentation, an API, a product reference or a long help centre and developers use coding assistants against it. This is where the proposal's own examples and the big vendors' own use sit.
• You already keep a clean list of your canonical pages and can generate the file from it, so there is no extra upkeep.
• An AI platform that matters to your customers tells you it wants the file. Mueller's reported advice follows this logic: if a platform that brings you customers says it needs the file, make one.
• It is not worth chasing if your goal is better placement in Google AI Overviews or AI Mode. Google says it does not use the file there.
• It is not a substitute for the basics that are documented: letting the right crawlers in via robots.txt, having indexable text pages, and accurate content. See the AEO checklist for those.

A minimal llms.txt you can copy

This is for a fictional company, Brightwell Analytics. Replace every name, URL and sentence. Keep the summary factual and short; list your most useful canonical pages, not every page.

# Brightwell Analytics

> Brightwell Analytics makes a web-analytics tool for small online shops. This file lists the pages that best explain what the product does, how it is priced and how to use it.

Brightwell is a hosted product. Pricing and plan limits change; always prefer the live pricing page over any cached copy.

## Product

- [What Brightwell does](https://www.example.com/product): scope, supported platforms and what it does not do
- [Pricing](https://www.example.com/pricing): current plans and limits

## Documentation

- [Quick start](https://www.example.com/docs/quick-start): install and first report in about 15 minutes
- [Data we collect](https://www.example.com/docs/data-collected): the full list of events and fields
- [Export API](https://www.example.com/docs/export-api): endpoints, rate limits, examples

## Company

- [About and contact](https://www.example.com/about): who runs it, where to write

## Optional

- [Changelog](https://www.example.com/changelog): release history
- [Blog archive](https://www.example.com/blog): older articles

How do I check it exists and is served as text?

Run these from a terminal. You want a 200 status, a text-like content type (text/plain or text/markdown; either is readable, but not text/html), and a body that starts with a Markdown H1 rather than an HTML page. A soft 404 that returns your homepage HTML with status 200 is the common failure.

On 2026-10-28 we ran the first check against https://citerlabs.com/llms.txt and got HTTP 200 with content-type text/plain; charset=utf-8. That confirms the file is served; it says nothing about whether any AI system uses it.

# 1. status and content type
curl -sI https://www.example.com/llms.txt | grep -iE "^(HTTP|content-type)"

# 2. first lines of the body: expect "# Your Name" and no "<html"
curl -s https://www.example.com/llms.txt | head -n 5

# 3. every link in the file should return 200
curl -s https://www.example.com/llms.txt \
  | grep -oE "\]\(https?://[^)]+\)" | sed -E 's/^\]\(//; s/\)$//' \
  | while read -r u; do printf "%s " "$(curl -s -o /dev/null -w "%{http_code}" "$u")"; echo "$u"; done

Does CiterLabs publish one, and has it helped?

Yes, citerlabs.com/llms.txt exists. We have not measured its effect on any AI answer, crawler visit or citation, and we are not claiming one. We publish it because it is cheap and a reader or tool might find it useful.