In short
Websites are built for people: menus, layouts, scripts and adverts. A language model reading a site has to dig the useful text out of all that, and often has limited room in its context window to do so. The llms.txt proposal suggests a simple fix: publish one concise markdown file that says what the site is and points to the pages that matter, ideally with clean markdown versions of those pages.
What goes in the file
| Title | A top-level heading with the name of the site or business. This is the only required part in the proposal. |
|---|---|
| Summary | A short quoted summary of what the site or business is, in a sentence or two. |
| Detail | Optional paragraphs or notes with key information a model should know. |
| Link sections | Headed lists of links to important pages, each with a short note on what the page covers. |
| Optional section | A section for secondary links that can be skipped when space is short. |
Where it came from
Jeremy Howard, co-founder of Answer.AI and of fast.ai, published the proposal in September 2024, with the specification at llmstxt.org. It was aimed first at software documentation, where developers increasingly ask AI tools questions about libraries and the tools struggle to read sprawling documentation sites.
The proposal also suggests offering markdown versions of individual pages at the same address with .md added. Separately, some sites publish a longer llms-full.txt containing the full text of their key pages in one file. That has grown up as a convention around the proposal and is not part of the core specification.
Do AI tools read it?
This is the important caveat. llms.txt is a community proposal that has not become a web standard, and the major AI providers have not, as far as we can confirm, publicly committed to reading it when they crawl or answer questions. Some documentation platforms and developer tools have adopted it, and some people point AI tools at a site’s llms.txt by hand. Beyond that, there is no reliable public evidence about which assistants use it or how much weight it carries.
It is also different from robots.txt. robots.txt tells crawlers what they may and may not fetch, and well-behaved crawlers follow it. llms.txt grants or blocks nothing. It is a summary offered in case something reads it.
Should a small business add one?
Probably yes, with modest expectations. A useful llms.txt takes an hour or two to write: a clear statement of who you are, what you do, where you work, and links to your main service pages, pricing and contact page. The cost is low and there is little downside.
Do not treat it as a substitute for the things known to matter for AI search visibility: clear and crawlable pages, consistent business details across the web, structured data and independent mentions. If your site is hard for people and search engines to understand, a summary file will not fix that.
If you do add one, keep it accurate. An out-of-date summary with old prices or services is worse than none. The simplest way is to generate the file from the same content that powers your website, so it changes when your pages change.
Check your robots.txt at the same time. If it blocks AI crawlers from your site, a summary file will make little difference to how those assistants describe you.
When this matters
Relevant to any business working on how AI assistants describe it. New Era AI publishes an llms.txt on this site and includes one in AI search work where it fits. RankOne measures how AI assistants mention your brand, which is the result that matters.
Questions
Is llms.txt an official standard?
No. It is a proposal published in September 2024. It has not been adopted by a standards body, and support among AI providers is limited and unconfirmed.
Does llms.txt control AI crawlers?
No. Access is controlled by robots.txt and your server settings. llms.txt only offers a summary.
Where does the file go?
At the root of your domain, so it is reachable at yoursite.com/llms.txt, served as plain text or markdown.
Will it help us appear in AI answers?
There is no published evidence that it does for the main assistants. It is cheap to add and may help tools that read it, but it should sit alongside the fundamentals.