--- title: "llms.txt: definition and example – Carl-Uwe Würfel" description: "llms.txt explained: how the Markdown file for AI agents is structured, how it differs from robots.txt and sitemap.xml, who reads it, and what it cannot do." url: https://carl-wuerfel.de/en/glossary/llms-txt author: Carl-Uwe Würfel --- 1. [Home](https://carl-wuerfel.de/en) 2. [Knowledge](https://carl-wuerfel.de/en/knowledge) 3. [Glossary](https://carl-wuerfel.de/en/glossary) 4. llms.txt # llms.txt llms.txt is an open proposal for a Markdown file that offers AI agents and language models a website's most important content as selected links with short explanations. ## On this page 1. How the file is structured 2. Where the file lives, and Markdown versions of pages 3. llms.txt, robots.txt and sitemap.xml compared 4. Who reads the file today 5. What belongs in it and what does not 6. Upkeep and measuring results 7. Where the term stops working 8. Frequently asked questions 9. Sources _Figure: A website’s public, well-linked pages remain the actual foundation; llms.txt adds orientation with selected links and short explanations, for example to an introduction, a reference and further guides. It is not access control, does not replace robots.txt, and its existence does not prove that any platform uses it._ **Example** A software company publishes an llms.txt on its domain with three sections: an introduction, a reference and further guides, each link with one explanatory sentence. Every page also has a Markdown version. Developers using an AI coding tool give it the address of the file, and the tool finds the right guide without crawling the entire website. llms.txt is a plain text file in Markdown format that usually sits at the root of a domain, at example.com/llms.txt. It works like a curated table of contents: the site owner lists the pages they consider important, each with a short explanation. [AI agents](https://carl-wuerfel.de/en/glossary/ki-agenten), developer tools and language models that need to understand a website get the shortest route to the content that matters. Jeremy Howard (Answer.AI) published the proposal on llmstxt.org on September 3, 2024; a revised second version is now posted there. The problem behind it is practical: web pages are built for people and consist largely of navigation, scripts and design. A language model can only process a limited amount of text at once. llms.txt helps point that limited capacity at what matters. It is a proposal, not a binding standard from a standards body. For executives, llms.txt is a small technical add-on, not a strategy. A website's public, well-linked pages remain the actual foundation; llms.txt adds orientation on top. The effort is small. There is no evidence that it improves visibility in AI answers, and Google has stated explicitly that its search does not use the file. Major budget decisions do not belong with this file. ## How the file is structured. The description on llmstxt.org sets a fixed order. It starts with a level-one heading containing the name of the website or project. That heading is the only required element. Next comes a blockquote with a short summary that states the essentials in one or two sentences: who is speaking, what it is about, who the content is for. Any number of text sections without headings may follow, such as notes on how to read the content. The main part consists of sections with level-two headings, each followed by a list of links. Every entry is a Markdown link with a name and an address, optionally followed by a colon and a short explanation. A documentation site might split its links into an introduction, a reference and further guides. A section headed “Optional” is commonly used for links an agent can skip when space is tight. In the first version it had a fixed technical meaning; in the second it is just a convention. ## Where the file lives, and Markdown versions of pages. The usual location is the root directory, at example.com/llms.txt. The second version of the proposal also allows files in subdirectories. Such a file covers all pages below its path, and the most specific file takes precedence. That helps organizations that only manage part of a domain. The proposal includes a second idea: websites should offer clean Markdown versions of their pages at the same address, either with “.md” appended or with “.md” replacing the existing extension. An agent then gets the plain text without navigation and scripts. The second version also describes how a page can point to its Markdown version and to the llms.txt file through link relations. A file called llms-full.txt is also widespread. Instead of links, it contains the full text of a documentation set in a single file, so the whole thing can be loaded into an AI tool with one link. This file is not part of the description on llmstxt.org; it is a convention that documentation platforms such as Mintlify generate automatically. For a typical company website, llms.txt as an overview is enough. ## llms.txt, robots.txt and sitemap.xml compared. All three files usually sit in the root directory, but they do different jobs. robots.txt contains rules on which parts of a website automated programs (crawlers) may fetch. Its format has been described since 2022 as the Robots Exclusion Protocol in RFC 9309. It is not protection either: the standard states explicitly that these rules are not a form of access authorization. Reputable crawlers follow them, but nothing is technically enforced. sitemap.xml lists a website's URLs in XML format so search engines can find every page. It is complete, but it carries no explanations and no weighting. llms.txt is a curated selection in Markdown, with short explanations, meant for language models and AI agents. It does not control access; it offers orientation. None of the three replaces another. Anyone who wants to stop AI crawlers from fetching content does that in robots.txt or on the server. llms.txt is a third, voluntary layer on top. It is not access control and does not replace robots.txt. ## Who reads the file today. Its existence does not prove that any platform uses it. Google has been clear. In its guide to optimizing for the AI features in Google Search, Google writes that Search does not use such files and that an llms.txt neither helps nor hurts visibility in Google Search. Anyone who wants to maintain the file for other services can do so without any downside. At the same time, Lighthouse, the audit tool built into Chrome, checks in its agentic browsing category whether an llms.txt exists and is properly structured. If the file is missing, the audit counts as not applicable; a failed fetch gets flagged. That shows the file is seen as a help for AI agents, not that it affects rankings. The developer documentation of OpenAI, Anthropic and Google Gemini each offer an llms.txt, and platforms such as Mintlify, GitBook and Wix, along with the WordPress plugins Yoast SEO and AIOSEO, can generate one automatically. The most tangible benefit today is for agents and coding tools pointed at a specific documentation set. Whether the major AI assistants use the file when choosing their sources is not publicly documented (as of November 2026). ## What belongs in it and what does not. The selection determines how useful the file is. It should include the pages someone ought to read when learning about the company, the product or the topic for the first time. For a company website, that typically means the company description, the core service or product pages, articles with real substance, frequently asked questions and the contact page. Internal areas, test pages, duplicates, thin pages and anything robots.txt blocks for crawlers stay out. Because llms.txt is publicly accessible, it only lists what may be public anyway. The explanations are short and factual. One sentence per link is enough, and it describes what is on the page, not how good the offering is. Marketing language does not help an agent find the right page. Length matters too. The file should shorten the route, not repeat the entire website. A few dozen well-chosen links are more useful than a complete list, which is what the sitemap is for. ## Upkeep and measuring results. llms.txt goes stale as soon as pages are moved, renamed or deleted. A link to a page that no longer exists is worse than no link, because it sends an agent down the wrong path. Anyone introducing the file should decide who maintains it and when it gets checked, for example at every relaunch and with any larger change to the site structure. Upkeep is easiest when the content management system generates the file automatically from existing pages. There are two simple ways to check results. Lighthouse shows whether the file is technically reachable. Server access logs show whether, and by which programs, it is actually fetched. A fetch does not prove, however, that a service uses the content in its answers. The realistic expectation is a website that is easier for agents to read. Anyone who wants more visibility in AI answers works on the fundamentals: clear, well-linked pages with real substance, clean structure and clear information about who stands behind the content. ## Where the term stops working The file is not access control and does not replace robots.txt. Its existence does not prove that any platform uses it, and there is no evidence that it improves visibility in AI answers. llms.txt is distinct from robots.txt, which gives crawlers rules, from sitemap.xml, which lists every URL, and from [GEO](https://carl-wuerfel.de/en/geo), the work on content and structure aimed at visibility in AI answers, to which the file contributes at most a small piece. ## Frequently asked questions What is the difference between llms.txt and robots.txt? robots.txt sets rules for which parts of a website crawlers may fetch, while llms.txt only offers language models orientation through selected links. llms.txt does not control or block anything. So it is not access control and does not replace robots.txt. Is llms.txt an official standard? No, llms.txt is an open proposal described on llmstxt.org, not a standard from a standards body. Adding it supplements a website but replaces neither SEO fundamentals nor structured data. Where does llms.txt go? Usually in the root directory of the domain, at example.com/llms.txt. Since the second version of the proposal, a file can also sit in a subdirectory and then covers the pages below it. Does Google use llms.txt? Google Search does not. In its guide to the AI features in Search, Google writes that such files neither hurt nor help because Search ignores them. Lighthouse, the audit tool in Chrome, does check the file in its agentic browsing category. Does an llms.txt improve visibility in ChatGPT or other AI assistants? There is no evidence that it does. The major [AI assistant](https://carl-wuerfel.de/en/glossary/ki-assistent) providers have not publicly stated that they use the file when choosing sources (as of November 2026). Based on what is known today, visibility in AI answers depends mainly on the content and structure of the pages themselves. Which pages belong in llms.txt? The few pages a language model should know first: an introduction or company description, core service or product pages, reference material and frequently asked questions. Internal areas, test pages and duplicate content stay out. Each link gets a factual one-sentence explanation. What is the difference between llms.txt and llms-full.txt? llms.txt contains selected links with explanations, while llms-full.txt contains the full text of a documentation set in one file. llms-full.txt is not part of the description on llmstxt.org; it is a convention used by documentation platforms. For company websites, llms.txt is usually enough. How much work is it to set up an llms.txt? Very little. Many content management systems and documentation platforms generate the file automatically, and written by hand it is a short text file. The real work lies in choosing the pages and keeping the file current when things change. ## Further reading * [**llms.txt: an overview with clear limits** Note](https://carl-wuerfel.de/en/notes/setting-up-llms-txt) * [**AI glossary**](https://carl-wuerfel.de/en/glossary) * [**GEO: Generative Engine Optimization**](https://carl-wuerfel.de/en/geo) * [**AEO: Answer Engine Optimization**](https://carl-wuerfel.de/en/geo) ## Sources 1. [Jeremy Howard (Answer.AI): The /llms.txt file, v2, 2024](https://llmstxt.org/) 2. [llmstxt.org: Changes (version 1 to version 2), 2026](https://llmstxt.org/changes.html) 3. [Google: Optimizing your website for generative AI features on Google Search, 2026](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) 4. [Mintlify: llms.txt (documentation)](https://www.mintlify.com/docs/ai/llmstxt) 5. [IETF: RFC 9309 – Robots Exclusion Protocol, 2022](https://www.rfc-editor.org/rfc/rfc9309) 6. [Chrome for Developers: Lighthouse llms.txt audit (documentation)](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt)