Ask around in any SEO or web development community lately and someone will tell you to add an llms.txt file to your site, as if it were the new robots.txt. It has a spec, a homepage at llmstxt.org, and adoption numbers that sound like a land grab. It also has almost no evidence that any AI model actually reads it.
Both of those things are true at once, and reconciling them is the point of this post. If you are wondering whether llms.txt deserves an afternoon of your time, the honest answer is that it depends heavily on what kind of site you run, and for most businesses it does not. Here is what the file actually is, what the data shows about who reads it, what Google has said on the record, and the narrow case where publishing one is still worth doing.
What llms.txt Actually Is
The idea comes from Jeremy Howard, the Answer.AI founder, who proposed the format on 3 September 2024 at llmstxt.org. The spec itself is short enough to read in five minutes.
An llms.txt file lives at your site’s root, at /llms.txt, and it is written in plain Markdown rather than XML. The only mandatory element is an H1 with your site or project’s actual name. After that comes an optional blockquote summary, then a series of H2-delimited sections, each listing curated links with a short note explaining what sits behind each one. A minimal valid file looks like this:
# Acme Analytics
> A usage analytics platform for product teams.
## Docs
- [Quickstart](https://acme.example/docs/quickstart): get a project running in under five minutes
- [API Reference](https://acme.example/docs/api): full endpoint documentationThere is also llms-full.txt, a separate and much larger variant that inlines full page content directly into the file instead of linking out to it. It was not part of Jeremy Howard’s original spec. It grew out of community practice as people wanted a single file a model could ingest without following links at all.
Why Everyone Started Talking About It
The appeal is easy to understand. robots.txt tells crawlers what not to touch. Sitemaps tell search engines what exists. llms.txt promises something similar for language models: a clean, hand-curated map of your most important pages, without the noise of navigation menus, cookie banners, and footer links that a model would otherwise have to wade through to find your actual content.
That pitch landed. Adoption data published by Originality.ai in June 2026, drawn from monitoring more than 3 million websites, found llms.txt use grew from 4,088 sites in June 2025 to 36,120 sites in May 2026, roughly 8.8 times over. llms-full.txt grew from 23 sites to 2,463 over the same period, and the newer ai.txt convention went from 4 sites to 397. Combined, close to 38,980 sites now publish at least one of the three.
Growth curves like that are exactly the kind of number that gets screenshotted into a blog post about a hot new SEO tactic. Taken on its own, it looks like early mover advantage waiting to be claimed. Taken alongside what actually happens to these files once they are published, the picture changes considerably.
What the Data Says About Whether Anyone Reads It
This is where the enthusiasm runs into a wall. Ahrefs published server log data in May 2026 drawn from 137,000 domains, and the headline number is blunt: 97% of llms.txt files received zero requests at all during the month studied.
Of the small remainder that did get a request, actual AI retrieval bots, the crawlers that models like ChatGPT and Claude use to fetch content, accounted for just 1.1% of those hits. The single largest category of requester, at 21.7%, was SEO audit tools: software checking whether a site has an llms.txt file, not software reading one to answer a question. Broken down by individual bot, GPTBot accounted for 4.51% of requests, ClaudeBot for 0.80%, and DeepseekBot for 0.02%.
Put plainly, the file that is supposedly written for AI models is mostly being requested by tools that exist to audit whether you have written one, not by the models themselves.
Key takeaway: Publishing an llms.txt file does not mean an AI model is reading it. In the month Ahrefs measured, real AI crawlers accounted for barely one in a hundred requests to these files, and most files received no requests whatsoever.
[!tip] Before spending time on a file with a 97% chance of going unread, it is worth confirming crawlers can even reach your important pages in the first place. Run our free AI Search Readiness Check to see where your site actually stands.
What Google Has Actually Said
Google’s public position has been consistent and unambiguous. John Mueller addressed it directly in a Reddit thread titled “LLM.txt, where are we at?” on 17 April 2025: “AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag.”
That comparison is deliberate. The keywords meta tag died because it let site owners describe their own pages in a field no user ever saw, which made it a magnet for cloaking and keyword stuffing rather than an honest signal. Mueller’s argument is that llms.txt carries the same structural weakness: AI crawlers already fetch your actual content, so a separate, self-reported summary adds an unverified layer on top of pages the model can already read directly. Gary Illyes gave a similar confirmation in July 2025.
Google’s own AI optimization guidance, published through Search Central on 15 June 2026, states it plainly: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search.” The more precise nuance, worth sitting with, is that publishing one will not negatively or positively impact your visibility or rankings. It is not a risk to have an llms.txt file. It is simply not a lever that moves anything in Google Search.
Key takeaway: Google has said outright that llms.txt does not affect rankings in either direction. The real cost of publishing one is not a penalty, it is the hours spent on a file instead of on work that does move AI visibility.
The Case For Publishing One Anyway
The strongest pushback on this comes from Carolyn Shelby, writing in Search Engine Land on 9 July 2025 in a piece titled “No, llms.txt is not the new meta keywords”. Her argument deserves a fair hearing, because it is not the same lazy hype driving most llms.txt coverage.
Shelby’s central point is that llms.txt is verifiable in a way the keywords meta tag never was. A keywords tag let you claim anything with no way to check it. An llms.txt file points to real URLs that either deliver on what the note promises or do not, and as she puts it, “you can’t fake clarity, authority, or substance once the model is on the page.” She also argues it can reduce server strain by routing crawlers toward lightweight Markdown rather than full rendered pages, that it helps models locate relevant content when working inside a limited context window, that it requires no content duplication the way AMP once did, and that both robots.txt and schema markup took years to become genuinely foundational before anyone trusted them either.
There is also a real, working use case that has nothing to do with search visibility. Stripe, Vercel, Cloudflare, Anthropic, OpenAI, and Mastercard all publish an llms.txt file, and the pattern among them is telling: these are documentation-heavy technical products. The actual consumers of these files today are coding agents and agentic browsers, tools like Cursor, Claude Code, and GitHub Copilot, which check for the file to locate developer documentation when someone asks for help integrating an API. That is a genuine use case. It just is not a search citation use case, and no major AI vendor, not OpenAI, not Anthropic, not Google, not Meta, not Mistral, has committed to using llms.txt as a citation signal in production. Neither OpenAI’s nor Anthropic’s own crawler documentation mentions the file. Both point site owners to robots.txt instead.
So Do You Actually Need One?
Here is the verdict. If you run developer documentation, an API, or a technical product where coding agents are a plausible category of reader, publishing an llms.txt file is a reasonable use of an hour, because there is a real, documented audience for it. If you run a local service business, an ecommerce store, or a content and publishing site, skip it. The opportunity cost is the actual argument here, not any risk of penalty: there simply is not an audience reading it yet.
That hour, or afternoon, is better spent on things that provably do influence what AI systems cite. Start with crawlability: if a search bot or AI crawler cannot render your page quickly and cleanly, no file at the root of your domain will fix that, and a proper technical SEO review is the higher-value place to start. From there, look at whether your pages actually answer questions in a form a model can lift cleanly, direct statements early in the page, not paragraphs it has to interpret, which is the kind of work a real SEO content strategy built around topical depth handles far better than a curated links file sitting at your root.
[!tip] Structured data does more of the real work llms.txt is often credited with, giving machines an explicit, verifiable description of what your business is and what your pages contain. Our free Schema Pro tool generates Article, FAQ, LocalBusiness, and Organization schema in minutes.
None of this means llms.txt is harmful to have. It means that for most sites, it is a rounding error compared to the fundamentals, and treating it as a shortcut past those fundamentals is the actual mistake.
If You Do Publish One, Do It Properly
If you have decided your site fits the narrow case above, do it properly rather than generating a token file to check a box. A common failure mode looks like this:
# Example Corp
## Everything on our site
- [Home](https://example.com/)
- [About](https://example.com/about)
- [Blog Post 1](https://example.com/blog/post-1)
- [Blog Post 2](https://example.com/blog/post-2)
- [Blog Post 3](https://example.com/blog/post-3)
(400 more lines)That is a sitemap wearing a Markdown costume, and it defeats the entire premise of a curated file. Do this instead:
- Put the file at the actual root path, /llms.txt, not buried in a subdirectory where nothing will find it.
- Open with an H1 that is your real site or project name, not a tagline or marketing slogan.
- List only the pages that genuinely matter: core docs, key API references, or your most authoritative content, not a dump of every URL on the domain.
- Write a short, honest note for each link describing what is actually there, since an inaccurate note is worse than no file at all.
- Keep the file a manageable size. If you need to include full page content rather than links, that is what llms-full.txt exists for, not the main file.
- Review it whenever your site structure changes, and remove or update links to pages that have moved or been retired.
The part that actually helps, if a bot or coding agent ever does arrive, is having clean Markdown versions of the pages you have linked to. The llms.txt file is just the index. The pages it points to are the part doing any real work.
FAQ: llms.txt
Does llms.txt help SEO or search rankings?
No. Google has stated directly that llms.txt will not negatively or positively affect visibility or rankings in Search. It is not parsed as a ranking signal, and the systems behind AI Overviews pull from the same crawled, indexed content they always have. If you are optimizing for AI citations, the value is in crawlability, structured data, and quotable content, not in a curated links file at your root.
Can llms.txt hurt my site?
Not directly, and Google has been explicit that publishing one carries no penalty. The realistic cost is not a technical risk, it is opportunity cost: time spent writing and maintaining a file with a documented 97% chance of receiving zero requests is time not spent on technical SEO, content structure, or schema markup, all of which have a measurable relationship with AI and search visibility.
What is the difference between llms.txt and robots.txt?
robots.txt is a long-established, universally respected protocol that tells crawlers which parts of your site they may or may not access, and every major search engine and most AI crawlers honor it. llms.txt is a newer, unofficial convention that suggests which pages are worth reading, but no major AI vendor has confirmed using it to guide crawling or citation decisions, which is the core distinction critics keep pointing to.
What is the difference between llms.txt and llms-full.txt?
llms.txt is meant to be a short, curated index: an H1, an optional summary, and links to your most important pages with brief notes. llms-full.txt is a separate, much larger file that inlines the full content of those pages directly, rather than linking out to them. It was not part of the original spec and exists mainly so a model could ingest everything in one file without following links.
Do ChatGPT and Claude actually read llms.txt?
The evidence says rarely, if at all. Ahrefs’ server log analysis found GPTBot responsible for only 4.51% of requests to llms.txt files and ClaudeBot for 0.80%, and John Mueller has said Google’s own logs show AI services largely do not check for the file. The coding agents built on top of these models, like Claude Code or GitHub Copilot, are the more consistent readers, and mainly for developer documentation.
What should I do instead if I decide to skip it?
Put the time into things with a documented connection to AI visibility: make sure crawlers can render your pages quickly and cleanly, restructure your key pages so the answer appears in the first sentence or two, add Article, FAQ, and LocalBusiness schema, and build genuine topical depth around the questions your customers actually ask. Those are the signals AI Overviews and chatbot citations are actually built on.
Where Five Minutes Are Better Spent
If you run a documentation-heavy technical product, publish the llms.txt file properly and move on. It will not hurt you, and it might genuinely help a coding agent find your docs. If you run anything else, the file was never the opportunity. Run the free AI Search Readiness Check higher up on this page before you spend another afternoon reading AI SEO hot takes about the next must-have file; it will tell you, in minutes, whether your crawlability, structured data, and content are actually in shape. If you would rather have someone look at all of it for you and build a plan, get in touch and we will.
