Does llms.txt work? 1,227 requests, zero AI crawlers

I added an llms.txt to this site in July, in the Astro rebuild. It seemed obviously correct: a clean Markdown manifest telling AI assistants who I am and what I write about. Cheap to write, plausibly useful, everyone else was doing it.

Two months later I went looking for evidence that it did anything. Here is the honest audit of my own decision.

What the server logs show

The most useful study I found tracked roughly 900 domains from September 4, 2025 to April 13, 2026 — seven months — and logged every request for llms.txt-family files. It counted 1,227 requests. The breakdown is the whole story:

Requester	Requests	Share
A commercial data aggregator	794	64.7%
Chrome (actual humans, curious)	392	31.9%
Security and purpose-built scanners	33	2.7%
Impersonators and other	8	0.7%
GPTBot, ClaudeBot, PerplexityBot, Google-Extended	0	0.0%

Not “a few”. Zero. Across seven months and nine hundred sites, not one frontier AI crawler fetched the file. And 793 of the 900 sites (88%) received no request for it at all — from anyone.

A separate November 2025 study across 300,000 domains looked for a correlation between having llms.txt and being cited by AI engines. It found none. Removing the variable from the model actually improved accuracy — the file was functioning as statistical noise.

What Google says

Gary Illyes said at Search Central Live in July 2025 that Google does not support llms.txt and is not planning to. Google’s AI features guidance, updated June 15, 2026, states plainly that you do not need special machine-readable files to appear in Search or its AI features.

John Mueller put it more bluntly, and his version is the one worth keeping:

“AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag.”

The keywords meta tag comparison stings because it is structurally right. llms.txt is a self-declaration: the site owner’s claim about the site. Anything a site owner controls and search engines cannot verify gets discounted — that is the entire history of on-page signals. Worse, serving a curated file to bots while humans get something else is cloaking with better branding.

The honest counter-case

Two things stop this being a clean dunk, and skipping them would make this post the same kind of overconfident content I am complaining about.

First, Google’s own products disagree with each other. Search says the file is irrelevant, but Chrome shipped a Lighthouse audit for llms.txt — filed under agentic browsing audits, explicitly not SEO. That is a real signal about where the file might matter: not search citations, but agents operating inside your site, where a map of the place is genuinely useful.

Second, “no crawler fetches it today” is a statement about today. Standards sometimes arrive late. robots.txt was a convention before it was universal.

So the fair verdict is narrower than “delete it”: llms.txt does nothing for AI search citations, and may eventually do something for in-site agents. Price it accordingly — which means it should cost you nearly nothing.

The part that actually cost me: mine rotted

Here is the failure that changed my mind, and it has nothing to do with crawlers.

I hand-wrote that file on July 10 and never touched it again. By this week it advertised 10 of my 12 posts and 4 of my 6 case studies. It confidently described a version of this site that stopped existing in August.

That is the real cost of a manifest: not the twenty minutes to write it, but the fact that it silently drifts out of sync while claiming to be authoritative. A stale manifest is strictly worse than no manifest — if anything ever does read it, what it reads is wrong. And I am a technical SEO consultant who does this professionally. If mine rotted in eight weeks, yours will too.

So I stopped hand-maintaining it. This site now generates /llms.txt at build time from the same content collections that build the blog, in about forty lines:

export const GET: APIRoute = async () => {
  const posts = (await getCollection('blog', ({ data }) => data.lang === 'en' && !data.draft))
    .sort((a, b) => b.data.pubDate.valueOf() - a.data.pubDate.valueOf());

  const blog = posts
    .map((p) => `- [${p.data.title}](${SITE.url}/blog/${slugOf(p.id)}/): ${p.data.description}`)
    .join('\n');

  return new Response(`${INTRO}\n\n## Blog\n\n${blog}\n\n${OPTIONAL}\n`, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

It can never drift again, because nothing about it is manual. That is the only version of this file I would now recommend to a client — and note what it implies: if you cannot generate it, that is a strong argument for not having one.

What actually gets you cited

The uncomfortable answer is that there is no shortcut file. AI crawlers fetch your HTML, the same HTML people get. So the work is the work you already know:

• Semantic markup and real structured data, so the machine reading your page can tell a byline from a nav item.
• Pages that load, because crawlers have budgets too — the same crawl-budget logic that makes you block infinite search URLs, and the same Core Web Vitals work you do for humans.
• Content with something specific in it. Models cite what is quotable: numbers, named versions, dates, first-hand results. Generic summaries of other people’s posts get summarized right back into nothing.

That last one is why this post contains a table of server-log counts instead of the sentence “studies suggest adoption is mixed.”

The verdict

Generate it or delete it. Never hand-maintain it — and never count it as AI-search work you have done, because the logs say it is not work at all.

Want to know what is actually costing you AI and search visibility? That is what a technical audit is for — measured on your site, not on best-practice folklore.