llms.txt: an overview with clear limits
An llms.txt links a site's key pages with short descriptions for AI systems. It helps them find their way around, but it does not guarantee any citations.
An llms.txt is a Markdown file in the root of a website (/llms.txt) that links the most important pages, with short descriptions, for language models and AI agents. Jeremy Howard of Answer.AI proposed the format in September 2024. It is a proposal, not a binding standard. Whether an AI system uses the file cannot be inferred from the fact that it exists.
Give an editorial reason for each link
The proposal describes a simple structure: a heading with the site's name, a short summary as a blockquote, then sections with lists of links, and at the end an optional section for less important material. A page belongs on the list only if it answers a question you can name, and that question goes next to the link as its description.
When pages or URLs change, the file has to change in the same step. Otherwise it points to outdated content. The proposal is still evolving, so its documentation remains the authoritative source for the format. An overview is worth as much as the upkeep behind it.
Name the limits
The llms.txt does not control access; that is what robots.txt is for. It does not replace a sitemap or navigation people can follow, and it does not replace search engine optimization. Google does not list an additional AI text file as a requirement for appearing in its AI search features.
Typical mistakes are outdated URLs, a list of every page instead of a selection, and promotional copy instead of plain descriptions. Anyone who sells guaranteed AI visibility is promising more than the file can do.
Separate observation from effect
When a system fetches the llms.txt, it shows up in the server log. That is a measurable technical signal. A fetch of the llms.txt does not show that the content will be cited later. Rising direct traffic cannot be traced to a specific source with any certainty either. So I evaluate these observations separately. Measuring AI visibility takes more than a log entry.
Sources: llms.txt proposal · Google on AI features in Search.
A fetch of the llms.txt does not show that the content will be cited later.
Judge the file by what it does
A documentation site lists its introduction, its API reference and a few selected guides in its llms.txt. Those links help people and compatible tools find their way. The linked pages still have to make sense on their own. If a guide lacks context, even the best entry in the overview cannot supply it.
Since 2023 I have been moving SUMAX as a whole company onto AI. One principle in that work is that process knowledge belongs in one place, not in individual heads and scattered spreadsheets. I see a content overview the same way. It lives in one place, it has been checked, and whoever publishes it answers for what it says.
My take: an llms.txt is worth having for websites with clear core content, because the effort is small. It is no guarantee of AI visibility.