llms.txt vs robots.txt vs sitemap.xml: what bots actually read
Two of these three files are load-bearing. The third is cheap insurance, and it is fine to treat it that way.
Three text files sit at the root of your site, and founders keep asking us which one is the AI visibility lever. Fair question, and the answer is boring in a useful way. Two of these files get fetched by real crawlers and change what happens to your pages. One of them is a proposal whose own author, Jeremy Howard, describes as an inference-time context aid designed to coexist with current web standards rather than replace robots.txt or sitemap.xml. That framing has gotten lost in the hype, so let's put all three side by side.
| Criterion | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Do bots actually fetch it? | Yes, routinely requested | Yes, used for URL discovery | Rarely. 97% of published files got zero requests in a month |
| Does Google Search use it? | Yes, to manage crawler traffic | Yes, to help discover URLs | No. Google says Search ignores it |
| On the critical path for AI answers? | Yes. A page must be crawlable, indexed, and snippet-eligible | Indirectly, by helping discovery | No AI system uses it today, per Google's John Mueller |
| Does it guarantee indexing? | No. Blocked URLs can still appear in results | No. Sitemaps guarantee nothing about crawling or indexing | No |
| Who needs it? | Effectively everyone with a site | Helpful for discovery, though sites of about 500 pages or fewer may not need one | Optional. Even Google's own Lighthouse audit marks it Not Applicable when missing |
| Cost of getting it wrong | High. A stray rule can cut off crawling | Low. Mostly a missed discovery opportunity | None measurable. Google says it neither harms nor helps |
| Verdict | robots.txt is the only one of the three where a mistake costs you real visibility. Sitemap.xml is helpful plumbing. llms.txt is a free lottery ticket, not a strategy. | ||
robots.txt: access control, and the most expensive typo on your site
Google's documentation is blunt about what robots.txt is for: it manages crawler traffic, and it is not a mechanism for keeping a web page out of Google. Blocked URLs can still show up in results. So treat it as an access valve, not a privacy setting or a visibility dial.
Here is why it still matters most for AI answers. Google states that eligibility for its generative AI features runs on ordinary Search mechanics: a page must be indexed and eligible to be shown in Google Search with a snippet. That puts robots.txt and your snippet directives directly on the critical path, and puts llms.txt nowhere near it. If you audit one file this quarter, audit this one.
sitemap.xml: discovery help that promises nothing
Sitemaps are useful and honest about their limits. Google says well-linked sites are usually discovered without a sitemap, and that small sites, meaning roughly 500 pages or fewer, may not need one at all. Sitemaps help discovery, but they guarantee nothing about crawling or indexing.
We still ship them, because the cost is near zero and the upside is real for new or deeply buried pages. Just don't expect a sitemap to fix a page nobody links to and nobody wants to read.
llms.txt: a reasonable idea that almost nothing reads yet
Now the elephant. In June 2025, Google's John Mueller said publicly that no AI system currently uses llms.txt, noting that consumer chatbots do fetch site pages but none fetch the file itself.
Then the data arrived. Ahrefs analyzed server logs and bot traffic across 137,210 domains and found 28% of them publish an llms.txt file, while 97% of those files were never fetched by anything at all in May 2026. Published, indexed by nobody, read by nothing.
The sharper finding is what happened on sites without the file: zero requests came from AI bots for llms.txt files that don't exist. As Ahrefs put it, they never go looking. AI crawlers don't probe for llms.txt the way they routinely request robots.txt. That is the whole difference between a standard and a proposal.
There is one flicker of institutional interest. Google's Chrome team does check for llms.txt in Lighthouse's experimental agentic-browsing audits, but the audit is marked Not Applicable when the file is missing, because providing it is optional at the moment. Even Google's own tooling files it under nice-to-have.
So should you ship llms.txt anyway?
Yes, if it takes you ten minutes. No, if you were about to make it a project.
Google's generative-AI optimization guide says outright in its mythbusting section that you don't need llms.txt or other special machine-readable files to appear in Search or its generative AI features, and that creating one will neither harm nor help your site's visibility or rankings in Google Search. Neither harm nor help is a decent definition of insurance. If the ecosystem turns and crawlers start reading the file, you are already covered. If it doesn't, you lost an afternoon once.
What you should not do is skip the indexable, snippet-eligible, well-linked version of your content because a machine-readable summary file felt more futuristic.
Ship the file, then forget it. Spend the afternoon you saved writing a page worth quoting.
The order of operations we actually recommend
- First, robots.txt. Make sure crawlers can reach the pages you want quoted, and remember that blocking a URL is not the same as hiding it from Google.
- Second, indexing and snippets. Google says a page must be indexed and snippet-eligible to be considered for its generative AI features. That is the gate.
- Third, sitemap.xml. Cheap discovery help, no guarantees, and possibly unnecessary if your site runs about 500 pages or fewer and is well linked.
- Last, llms.txt. Ship it as a hedge, log it as done, and move on.
Where Goblyn fits in all this
Honest of us to mention: we handle all four of those by default. Technical SEO and GEO work, including schema, sitemaps, internal linking, and llms.txt, is part of the service rather than an upsell, because the point is keeping your blog legible to search engines and AI answer engines at the same time.
Goblyn is a small team of AI goblyns that crawls your public site to build a fact file, drafts grounded posts with cited sources, publishes them to a white-labeled blog on your own domain, and tracks how they perform in Google Search Console and AI answer engines. Cadence lands around three posts a week, tuned by data rather than a quota. One plan, $99/mo, and a 30-day free trial at $0 with no payment method required while your blog sits at a private, unindexed preview URL.
We make no promises about rankings or traffic, and we are not going to start with a text file that 97% of the internet's crawlers ignore.
Two of these files earn their keep. One is a lottery ticket. Buy the ticket, but keep working on the pages.
- robots.txt is the file with real consequences: it manages crawler traffic, and Google says it is not a way to keep a page out of Search, since blocked URLs can still appear.
- Google ties eligibility for its generative AI features to ordinary Search mechanics, a page being indexed and snippet-eligible, which puts robots.txt and snippet directives on the critical path instead of llms.txt.
- Sitemaps help discovery but guarantee nothing, and Google says sites of roughly 500 pages or fewer may not need one.
- llms.txt is still a proposal: Google says Search ignores it, John Mueller said no AI system uses it, and 97% of published files got zero traffic in May 2026.
- Ship llms.txt as cheap insurance, since Google says it neither helps nor hurts, then spend the real effort on crawlable, quotable pages.
- The llms.txt spec author framed the file as an inference-time context aid designed to coexist with current web standards rather than replace robots.txt or sitemap.xml.
- Google's robots.txt docs say it manages crawler traffic and is not a mechanism for keeping a web page out of Google, since blocked URLs can still appear in results.
- Google states that eligibility for its generative AI features depends on the page being indexed and eligible to show a snippet in Google Search.
- Google says you don't need llms.txt or other special machine-readable files for Search or generative AI features, and creating one neither harms nor helps visibility or rankings because Search ignores it.
- John Mueller said on June 17, 2025 that no AI system currently uses llms.txt, and that consumer chatbots fetch site pages but none fetch the llms.txt file.
- Ahrefs analyzed 137,210 domains and found 28% publish an llms.txt file, but 97% of those files received zero traffic in May 2026.
- In the Ahrefs dataset, no AI bot ever requested an llms.txt file on sites that didn't have one, and AI crawlers don't probe for it the way they routinely request robots.txt.
- Google's Chrome team checks for llms.txt in Lighthouse's experimental agentic-browsing audits, but marks the audit Not Applicable when the file is missing because providing it is optional at the moment.
- Google says well-linked sites are usually discovered without a sitemap, that sites of about 500 pages or fewer may not need one, and that sitemaps help discovery but guarantee nothing about crawling or indexing.
Does llms.txt help my site appear in Google's AI features?
No. Google's own AI optimization guide says you don't need llms.txt or other special machine-readable files to appear in Google Search or its generative AI features, and that creating one will neither harm nor help your visibility or rankings because Search ignores it. Eligibility depends on the page being indexed and allowed to show a snippet.
Do AI crawlers ever look for an llms.txt file?
Mostly no. In Ahrefs' study of 137,210 domains, zero requests came from AI bots for llms.txt files that don't exist, meaning crawlers never probe for it the way they routinely request robots.txt. Google's John Mueller also said in June 2025 that no AI system currently uses the file.
Do I still need a sitemap.xml if my site is small?
Maybe not. Google says well-linked sites are usually discovered without a sitemap and that small sites, meaning roughly 500 pages or fewer, may not need one. Sitemaps help discovery but guarantee nothing about crawling or indexing, so they are worth shipping when cheap and not worth agonizing over.