What Is llms.txt and Does My Website Need One?
You’ve been told that your site needs a file named llms.txt by a plugin, newsletter or agency. The article explaining it is written by the person who sells the file. So, what is llms.txt? It is a proposed convention, not an adopted standard, and as of October 9, 2026 no major assistant operator has confirmed that it reads one.
This guide explains:
- What the file is and what it contains
- How it differs from robots.txt
- What the operators’ own documentation does and does not say
- Who might reasonably publish one anyway
- What to do with the hour instead
What is llms.txt?
Proposed for the root of a website, llms.txt is a plain text Markdown file containing a list of key pages with short descriptions, which a language model can read, rather than crawling everything. Jeremy Howard, Answer.It was suggested by AI in September 2024 and the specification can be found at llmstxt.org. A typical llms txt file exampleis a title, one line summary and a short list of key pages with a few words on each, and a model gets a map of your site structure that fits the number of tokens. The example below is for illustration only.
Most readers come here because they confuse this file with robots.txt which determines whether a crawler is allowed to read your site. The mix-up is common: A search for “llms txt vs robots txt” reveals that llms.txt only offers a summary and has no control. Our guide to the file that does control access helps you determine which crawlers can access your site and what you’ll lose by allowing each one. It includes training on crawlers and search crawlers.
Does any AI assistant actually read it?
No major assistant operator’s published documentation names llms.txt, and none has publicly confirmed consuming one, so as of October 9, 2026 there is no evidence that it is read.
The table below separates what people claim from what is documented.
| What people claim about llms.txt | What is actually documented | Status |
|---|---|---|
| It is a web standard | A proposed convention, published as a specification by its author, with no standards body behind it | Proposed, not ratified |
| AI assistants read it | No major operator’s documentation names it, and none has publicly confirmed consuming one | Unconfirmed |
| It improves your AI visibility | No operator has described any effect on retrieval, citation or ranking | No published evidence |
| It works like robots.txt | Robots.txt controls access and named crawlers honor it. llms.txt proposes a curated summary and controls nothing | Different thing entirely |
| Google supports it | Google’s AI features guidance names no additional file or markup as a requirement | Not referenced |
| Some sites publish one | True. Publishing a file and a crawler reading it are separate facts | Publishing is real, consumption is unconfirmed |
Beware of this proof. The operators’ documentation as of October 9, 2026, documents robots.txt based controls, and does not mention reading of llms.txt, nor have there been any confirmed uses. That’s a fact about the documentation, and it leaves open the possibility that some systems read the file. Google is more than just silence. It states that no new machine-readable files, AI text files or markup are required for AI features. According to Google’s documentation on AI features and your website, there are no extra technical requirements and that “normal search basics” determine if a page can be eligible for these features.
Each crawler has a specific task to perform: GPTBot, which trains models; OAI-SearchBot, which searches pages; and ChatGPT-User, which provides user pages. OpenAI’s GPTBot documentation doesn’t indicate that they read llms.txt, but rather that it can be controlled with robots.txt rules; also, there is no documentation that they read robots.txt.
Why does this keep coming up?
The file is easy to write, since it is a comparison to robots.txt and sitemaps, and many plugins and agencies sell something with it. Plenty of owners now ask, “Do I need llms txt?” because a plugin told them to. The proposal is reasonable, and its author was not selling anything. The trouble is the paid advice built on top of it. Vendors love to sell it as AI work, as it is new, easy to bill, and not many buyers are able to check it. Read our page on where AI genuinely fits to understand where AI supports a business, where it doesn’t, and what AI work is beyond just posting a text file.
The file also comes with new vocabulary; labels like AEO and GEO are sold as separate services and merge together. We’ve broken it down into the vocabulary being sold alongside it and defined each term and indicated the overlap in the work, so you can compare any proposal you receive in your inbox with the breakdown.
What is the honest cost and benefit?
The charge is an hour and an accurate file you’ll need to maintain. The upside is that it hasn’t been proven, and the actual risk is believing it was the visibility work of AI to publish it. It’s better to have no file than a stale llms.txt that conflicts with your live site. The only time the file is important is when someone reads it. It’s at that moment that the wrong summary is harmful. One of the most common questions that people ask is “is llms txt worth it” due to the low cost of the file. The reason why it is not being done is not because of harm, but opportunity cost and false confidence. The “llms txt seo” search term indicates a ranking improvement, and there is no documentation of an operator for this. This may change. A convention that might be adopted and become a standard is normal, as indicated by the sitemaps. Major operators that document support change the answer on this page that day.
What should I do instead?
Make the real pages readable without a script, give every section a self-contained answer, keep one accurate set of business details, publish a correct sitemap, and decide your crawler policy on purpose. The hour is better spent on documented basics that every assistant and search engine reads first, before anything else. Our view of search fundamentals done first is that they come before any file in the root folder, because every other method rests on them and none of them is speculative.
The table below pairs what llms.txt promises with the documented way that already does the job.
| What llms.txt promises | The documented mechanism that already does it |
|---|---|
| “Tell models what your site is about” | A clear homepage and service pages, plus Organization or LocalBusiness structured data |
| “Give models a clean version of your content” | Server-rendered HTML with real headings, so no curated duplicate is needed |
| “Point models at your most important pages” | An accurate XML sitemap and internal links from pages that already have authority |
| “Summarize each page for a model” | A self-contained answer passage under each heading, which also helps human readers |
| “Control what models use” | Robots.txt per user agent, which named crawlers do honor |
Most of the table is set up, and if you make a mistake in any row, it doesn’t show up, so it takes place without a sound. Our team works on the technical side of this, such as rendered HTML, sitemaps and crawler rules. Each item is checked on the live site, having the file open in front of us. Structure is the row that is easiest for most people to fix, since parsing is taking place by starting with a real heading and having an answer at the top of each section. As part of our on-page work, we cover the headings and structure a parser can follow, and each section can be lifted out of the page by a machine.
Google documents how its AI answers choose pages, the closest you can get to a real route. We have a documented route into the answers for you and explained what makes a page eligible and how a passage is selected. It expands on the brief mention of work in llms.txt. All of this is based on a standard older than the file, and it asks the question: is a page useful to a reader? The self-check questions are available in Google’s guidance on creating helpful content. They are applicable to each row of the table above, to whoever wrote the page, and they are the most accurate measure of an hour well spent.
Who might reasonably publish one anyway?
Documentation sites, API products and developer tools: the audience here is software agents and the file builds are built from the same source as the docs, and are almost free. There is a real case here, and ignoring it would make this page look motivated. Write one if it’s created from content you already have to maintain, and avoid creating one if it’s manual work.
Most small business websites are on the wrong side of that line, as the owner has little time to keep a second set of records of the first. Our work for small sites with limited developer time puts that time into pages and details that are documented, and a file can be added later in ten minutes if it ever matters.
How would I know if it started to matter?
Do not rely on marketing articles or operators’ own crawler documentation/changelogs the first reliable indicator is a named crawler documenting that it fetches the file. Operators like OpenAI, Google, Perplexity and Anthropic have crawler pages, so look for the presence of llms.txt on those twice a year.
So does my website need an llms.txt file?
If you are a newbie who is looking for what is llms.txt, you should take the hour to read the five lines above. The exception is a documentation site or a developer tool, where the file is self-built and is almost free. This is a statement about the evidence at hand, and if a key operator provides evidence the next day, the answer will be different the next day. Want a second opinion on your own site? Pink Socials will be able to tell you clearly what you should fix, what you don’t need to fix, and why, based on the same sources that this guide uses.
Leave a Reply