Somebody has probably told you your website needs an llms.txt file. Maybe it appeared as a line item on a quote. Maybe a tool generated one automatically and called it AI optimization. The pitch sounds reasonable. Artificial intelligence is reading the web now, so hand it a file that explains your business in language it understands.
There's a problem with that. Google has stated plainly that you don't need one. Ahrefs pulled server logs from 137,210 websites and found that 97 percent of these files were never fetched by anything at all during May 2026. And while that file sat there, the things that genuinely put a business in front of an AI answer went untouched.
This is not an argument that AI search doesn't matter. It matters a great deal. It's an argument that you're being pointed at the wrong lever, and here's where the real one is.
- llms.txt was proposed in September 2024 as a convenience for software documentation, not as a way to rank or get recommended.
- Google's own documentation says you don't need to create new AI text files to appear in AI Overviews or AI Mode.
- An Ahrefs study of 137,210 domains found 97 percent of llms.txt files received zero requests in May 2026.
- The file that actually controls AI access to your site is robots.txt, and you already have one.
- What correlates with getting mentioned by AI is being talked about on sites you don't own.
What llms.txt actually is, and who it was built for
Jeremy Howard, co-founder of Answer.AI, proposed the format on September 3, 2024. He was solving a real technical problem. Language models work inside a limited context window, and a modern web page is mostly not content. It's navigation, scripts, tracking pixels, cookie banners and advertising. A model burns through its budget reading page furniture before it reaches a single useful sentence.
His fix was a plain markdown file at yoursite.com/llms.txt that lists your important pages in clean, stripped-down text. The intended use case was software documentation. A coding assistant needs to look up how an API works, and a markdown map of the docs saves it from parsing a rendered documentation site.
Notice what isn't in that description. Nothing about rankings. Nothing about local search. Nothing about getting a roofing company recommended when somebody asks an assistant who to call. It was a file format for developer docs, and it was a sensible idea for developer docs.
So how did a documentation convenience end up on marketing invoices? Because it's easy to sell something you can point at. A file is visible. A file is deliverable. When a client asks what they paid for, "we added your llms.txt" is a much cleaner answer than explaining that visibility takes months and depends partly on things outside your control.
What Google says about it, in Google's own words
Google publishes guidance on how content shows up in AI Overviews and AI Mode. On the question of special files, its position is one sentence long: "You don't need to create new machine readable files, AI text files, or markup" to appear in those features. The same page states there are no additional requirements and no special optimizations necessary beyond ordinary search fundamentals.
Gary Illyes, speaking at Google Search Central Live, confirmed that Google does not support llms.txt and has no plans to. John Mueller went further and compared it to the old keywords meta tag, pointing out that no AI service has said it uses the file, and asking why a system would trust a site owner's self-description instead of just reading the site.
That last comparison is worth sitting with, because it explains the whole thing.
Ninety-seven percent of these files are never read
In June 2026, Ahrefs published a study by Louise Linehan built on server log data from 137,210 domains. Not survey responses. Not opinions. Actual records of what fetched what during May 2026.
Ninety-seven percent of the llms.txt files in that dataset received zero requests. Nothing pulled them. Of the three percent that did see any traffic, the single largest category was SEO audit tools at 21.7 percent, meaning the most common visitor to an llms.txt file is a piece of software checking whether an llms.txt file exists. All AI-related categories combined, including agents, training crawlers and retrieval bots, accounted for 19.5 percent of requests to those files.
Two caveats belong here, and Ahrefs states both. Their customer base skews technical and SEO-aware, so the adoption figures should be read as an upper bound rather than a picture of the average small business website. And a bot fetching a file only proves the file was downloaded. It says nothing about whether any model used a word of it.
The most frequent visitor to an llms.txt file is a tool checking whether an llms.txt file exists.
Why a file you write about yourself was never going to count
Mueller's keywords meta tag comparison isn't a throwaway insult. It's a structural point, and it's the reason this was predictable.
The keywords meta tag let a site owner list the terms they wanted to rank for. Search engines used it briefly, discovered that people simply wrote whatever they wanted, and stopped trusting it. The tag didn't fail because it was badly designed. It failed because it asked the interested party to grade their own work.
llms.txt has the same shape. You write it. You decide what it says your business is best at. There's no verification step, no cross-check, no cost to overstating. Worse, because the file is served to machines and rarely read by people, it creates a tidy path to showing one story to bots and another to customers. Search engines have spent twenty-five years closing doors like that one.
Any signal a business fully controls gets discounted eventually. That's not cynicism about AI. It's just how trust works when the stakes are commercial.
The file that does control AI access is one you already have
Here's the part worth your actual attention, and it costs nothing to check.
Every website has a robots.txt file at yoursite.com/robots.txt. Type that into a browser right now and you'll see yours. Unlike llms.txt, this one is a genuine standard, and the major AI companies document that their crawlers respect it. It's the real access control, and most business owners have never looked at it.
The important thing to understand is that AI crawlers come in two kinds, and they do completely different jobs.
- Training crawlers. GPTBot, ClaudeBot and Google-Extended collect content that becomes part of a model's long-term background knowledge. Blocking these is a reasonable choice some businesses make on principle. Google states that Google-Extended has no effect on your inclusion or ranking in ordinary Google Search.
- Retrieval agents. OAI-SearchBot, Claude-SearchBot, Claude-User and PerplexityBot fetch live pages to answer a question somebody is asking right now. These are the ones that produce a citation with your business name attached to it.
That distinction is the whole ballgame. Blocking training crawlers keeps your writing out of the next model. Blocking retrieval agents removes you from the answers people are getting today. Those are two entirely different decisions, and a blanket rule makes both of them at once without meaning to.
I've opened robots.txt on more than one Hunt County business site and found a sweeping block nobody remembered adding. It usually arrived with a plugin, or with a well-meaning setting labeled something like AI protection. The owner wanted to keep their photos out of a training set. What they actually did was make themselves invisible to the assistants their customers are already asking.
Structured data is the other thing that genuinely helps machines read your business correctly, and unlike llms.txt it's a standard every major platform uses. Our free generator builds it for you.
Build your schema markupSo what actually gets a business mentioned by AI?
Ahrefs studied 75,000 brands to see which signals line up with showing up in AI Overviews. The strongest correlations were all off-site. Brand web mentions came in at 0.664, brand anchors at 0.527 and brand search volume at 0.392. Traditional backlinks, the thing the SEO industry has chased for two decades, sat at 0.218. A follow-up found brand mentions on YouTube correlating at roughly 0.737, the highest single signal they measured.
Correlation is not proof of cause, and Ahrefs is careful to say so. But the pattern is consistent enough to plan around, and it points somewhere uncomfortable for anyone selling file-based fixes. What predicts being recommended is being discussed in places you don't own.
Ranking still counts. A separate Ahrefs analysis of four million AI Overview URLs across 863,000 search results found that 38 percent of cited pages also rank in the top ten organically. So classic search work is far from dead. It's just no longer the whole story, because these systems break one question into several related ones and pull sources for each.
Translated for a business owner, the list looks like this. The chamber of commerce writeup that names you. The supplier page that lists you as a dealer. The local news piece. The YouTube video where somebody walks through the work you did. The review that describes what you actually fix rather than saying you were great. None of that lives on your website, and all of it is what an AI system reads to decide you exist.
We wrote more about how this is reshaping local discovery in our piece on AI search and local businesses, and the mechanics of machine-readable business facts in the guide to schema markup on Squarespace.
When llms.txt is genuinely worth having
Fair is fair. There is a real use case, and it's the original one.
If you publish technical documentation, an API reference, or a large knowledge base that developers work against, an llms.txt file earns its keep. Coding assistants do read documentation, and giving them a clean markdown map is a genuine courtesy that saves everyone tokens. That was Howard's proposal and it still holds up.
Google has also never said the file causes harm. It's a text file. It won't hurt your rankings and it costs nothing to leave in place. If your platform generated one automatically, there's no emergency and nothing to undo.
The only mistake is counting it as marketing. If it appeared on an invoice as an AI visibility service, that's the part to question, not the file itself. Ask what else was done, and ask how it will be measured.
A plain plan for the next ninety days
None of this requires an agency. Here's the order I'd work in.
- Read your robots.txt. Visit yoursite.com/robots.txt and look for any Disallow rule aimed at everything. If you find one, you may be blocking the assistants your customers use.
- Decide training and retrieval separately. Keeping your content out of model training is a legitimate position. Removing yourself from live AI answers usually isn't what anyone intended. Set those two rules on purpose.
- Answer the question in the first two sentences. AI systems extract direct answers. A page that opens with three paragraphs of throat-clearing before saying what you charge or where you serve gives them nothing to pull.
- Make your business facts identical everywhere. Name, address, phone, hours and services should match across your site, your Google Business Profile and every directory listing. Contradictions make a machine hedge, and hedging looks like being left out.
- Add structured data. This is the machine-readable format that actually has adoption behind it, and it describes your business in a vocabulary every major platform already parses.
- Earn mentions you don't control. Get listed by suppliers and trade associations. Answer a reporter. Post a real project walkthrough to YouTube with your business name in the title and description. This is slow work and it's also the work that correlates best.
- Measure it. In Google Analytics, look at referral traffic from chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. Small numbers today, but the trend line is the point, and you cannot argue about a lever you never measured.
Step six is the one people skip, and it's the one the data keeps pointing at. Why does that happen? Because it's the only step you can't finish in an afternoon.
What chasing the wrong file actually costs
The hour spent generating an llms.txt file is not the real expense. These are.
- False confidence. A business that believes it handled AI visibility stops looking, and the gap widens quietly for a year.
- Opportunity cost. That same hour spent fixing a blanket crawler block, or getting one supplier to list you properly, produces something measurable.
- Budget spent on the visible instead of the useful. When a vendor is paid for deliverables you can see, the incentive runs toward files and reports rather than outcomes.
- Trust, eventually. When the promised AI visibility never shows up in the numbers, business owners conclude that all of it was noise. That's the worst outcome, because the underlying shift is real.
Search is genuinely changing. That's exactly why it deserves an honest answer rather than a convenient one.
Frequently asked questions
Should I delete my llms.txt file?
No. There's no benefit to removing it and no penalty for keeping it. Google has said the file has no effect on Search, not that it causes harm. Leave it alone. The only thing worth changing is the expectation attached to it. If it was sold to you as AI visibility work, ask your vendor what else was delivered and how the result will be measured.
Does llms.txt help with ChatGPT or Claude even if Google ignores it?
The evidence says no. Ahrefs measured actual server logs across 137,210 domains and found that AI-related bots accounted for a small fraction of requests to these files, with 97 percent of files never fetched at all. No major AI company has publicly committed to reading llms.txt in production. That could change, but you should not spend money today on a maybe.
Is AI optimization worth paying for at all?
Yes, when it means real work. Fixing crawler access, writing pages that answer questions directly, correcting your business facts across the web, adding structured data and earning mentions on sites you don't own all have evidence behind them. Be cautious when the deliverable is a file. Ask what will be measured, and ask to see the number before and after.
How do I know if AI is already sending me customers?
Check referral traffic in Google Analytics for chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. The volume will look small next to Google, and that's normal right now. What matters is the direction over several months. Also ask new customers how they found you, because plenty of AI-driven discovery ends in a direct visit that no report will ever attribute.
My web platform added llms.txt automatically. Is that a problem?
Not at all. Several platforms generate one by default now, and it's harmless. Just don't treat its presence as evidence that AI visibility is handled. Spend ten minutes on robots.txt instead, since that file genuinely controls whether AI assistants can read your pages, and a mistake there has real consequences you can fix the same day.
Not sure whether your site is open to AI assistants or quietly blocking them? We'll check your crawler rules, structured data and business listings, and tell you what we find at no charge.
Request a visibility checkGood guidance means telling you when something doesn't work, even when the something is easy to sell. If you want the full picture on how we approach search and AI visibility together, our SEO and AI optimization services page lays out the method.
Sources
- Google Search Central, AI features and your website, retrieved 2026-09-08. developers.google.com
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Louise Linehan, published 2026-06-15, retrieved 2026-09-08. ahrefs.com
- Ahrefs, An Analysis of AI Overview Brand Visibility Factors, 75,000 brands, retrieved 2026-09-08. ahrefs.com
- Ahrefs, Update: 38% of AI Overview Citations Pull From The Top 10, Louise Linehan, published 2026-03-02, retrieved 2026-09-08. ahrefs.com
- Answer.AI, /llms.txt, a proposal to provide information to help LLMs use websites, Jeremy Howard, published 2024-09-03, retrieved 2026-09-08. answer.ai
- Search Engine Journal, Google Says LLMs.txt Comparable To Keywords Meta Tag, retrieved 2026-09-08. searchenginejournal.com
- Anagram, AI Crawlers Explained: GPTBot, ClaudeBot, PerplexityBot and How to Let Them In, retrieved 2026-09-08. anagram.ai
- PPC Land, llms.txt adoption rises 8.8x but 97% of files get zero AI requests, retrieved 2026-09-08. ppc.land