Does llms.txt Actually Work? An Honest Assessment

As of 2026, no major AI engine has confirmed that llms.txt improves your visibility in AI answers, and server-log data shows AI crawlers rarely fetch it. It’s cheap to add and unlikely to hurt, but it is not a ranking lever, and you should not expect it to move your presence rate. That’s the honest verdict. This article explains what llms.txt is, what the evidence actually says, and the one genuinely useful thing you can do with it — checking whether AI bots fetch yours — which ties into the broader work of Generative Engine Optimization.

What llms.txt is (and isn’t)

llms.txt is a proposed standard — documented at llmstxt.org — for a single Markdown file at the root of your site (/llms.txt) that gives large language models a curated, human-and-machine-readable guide to your most important content. Think of it as a hand-picked table of contents: a short description of your site, followed by links to the pages you’d most want an LLM to read, optionally with notes on each.

The crucial distinction is what it is not:

• It is not robots.txt. robots.txt controls crawler access (allow/disallow). llms.txt is the opposite intent — it’s an invitation that says “here’s the good stuff,” not a gate. If you want to control what AI crawlers can fetch, that’s still robots.txt for AI crawlers, not this file.
• It is not a sitemap. A sitemap is an exhaustive machine list of every URL for discovery. llms.txt is a short, opinionated, curated set of your best pages, written for an LLM’s benefit at inference time.
• It is not a ranking signal. Adding it does not, on any current evidence, make engines mention or cite you more.

What the evidence says

Here’s where honesty matters, because the marketing hype around llms.txt has run well ahead of the facts.

Google has said it doesn’t use it. Google has stated that llms.txt files are not required for Google Search and carry no positive or negative effect on rankings, and Google’s John Mueller said in mid-2025 that no AI system was using llms.txt at that point — comparing the enthusiasm to the long-discredited keywords meta tag.

OpenAI and Anthropic point elsewhere. Both direct site owners to robots.txt for crawler management, and reference llms.txt only in their developer documentation rather than committing their production crawlers to it. (Amusingly, several of these companies publish their own llms.txt files while giving this guidance.) Anthropic’s Claude products and Perplexity have been reported to respect llms.txt in some retrieval workflows, so it’s not universally ignored — but that’s a narrow, unconfirmed slice, not broad engine support.

The crawlers mostly don’t fetch it. The clearest signal is behavioral. An analysis of Ahrefs server-log data reported that 97% of llms.txt files received zero requests in a recent month, and that among the files which did get requests, AI retrieval bots accounted for only about 1% of the traffic — trailing far behind SEO audit tools. The big AI crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, Google-Extended) overwhelmingly skip the file and fetch your HTML directly. Adoption of the file is rising fast; actual consumption of it by AI engines is not.

Put together: llms.txt is a well-intentioned proposal with real momentum among publishers and near-zero confirmed uptake among the engines that would need to honor it for it to affect your AI visibility.

So should you add one?

Probably yes — but for the right reason and with the right expectations.

flowchart TD
A["Considering llms.txt?"] --> B{"Costs more than<br/>an hour to add?"}
B -->|"Yes"| C["Skip it —<br/>spend the time on<br/>citable content"]
B -->|"No"| D["Add it —<br/>low cost, low risk"]
D --> E{"Expecting a<br/>ranking / presence lift?"}
E -->|"Yes"| F["Reset expectations —<br/>no engine confirms this"]
E -->|"No"| G["Good. Now verify<br/>if bots actually fetch it"]

Add it if it’s cheap. If you can generate a good llms.txt in an hour — many doc frameworks and static-site generators produce one automatically — do it. The downside is essentially zero, and there’s a real, non-visibility upside below.

Don’t build a program around it. If maintaining llms.txt would take meaningful ongoing effort, that time is far better spent on content AI engines actually cite and on earning presence in your cited-domain map — both of which have evidence behind them. llms.txt belongs in the same bucket as schema markup: cheap, tidy, unproven for AI visibility, fine to do, not where the leverage is.

Where it may genuinely help: agents, not answers. The most credible use for llms.txt isn’t AI search visibility at all — it’s a business-to-agent surface. AI coding assistants and autonomous agents that fetch documentation in real time (think a coding tool pulling your API docs into context) can use a curated llms.txt to grab the right pages with less token waste. If your product has developer docs and you care about how agents consume them, that’s the real argument for shipping one.

The one thing worth measuring: do bots fetch yours?

Whatever the aggregate statistics say, the question that matters for your site is specific: is anything actually requesting your /llms.txt, and who? You can’t answer that with JavaScript analytics — crawlers don’t run your JS, so tag-based tools report exactly zero AI crawler traffic (this is the whole blind spot). You need server-side data.

This is a natural fit for Kitbase’s bot & crawler detection. Once you’re forwarding server or edge requests, every AI crawler hit is stored with the path it requested — so /llms.txt shows up in your bot top-paths just like any other URL. You can see:

• Whether any AI bot requests /llms.txt at all — and if so, which ones (GPTBot, ClaudeBot, PerplexityBot, …) and how often.
• Whether the requests are verified or spoofed — Kitbase labels each crawler visit using vendors’ published IP ranges and cryptographic Web Bot Auth signatures, so a fake “GPTBot” hitting your file doesn’t fool you.
• How /llms.txt traffic compares to your real content — which is usually the reality check: the file gets a trickle while your docs and comparison pages get the actual crawl budget.

That turns the whole debate from opinion into a number for your site. If ClaudeBot fetches your /llms.txt weekly, keep it current. If nothing has touched it in a month, you have your answer and can stop thinking about it. This is the same evidence-over-assumption approach that underlies tracking which pages AI bots crawl and auditing AI crawlers in your server logs.

Bottom line

llms.txt is a reasonable thing to add and a bad thing to believe in. Ship one if it’s an hour of work, especially if you serve developer docs to agents. Don’t expect it to change what ChatGPT or Perplexity says about you — that comes from citable content and third-party presence, measured over time in AI Visibility. And if you’re curious whether it’s doing anything, don’t guess: watch your bot paths and let the crawlers tell you.

FAQ

Does llms.txt improve my ranking in AI search? There’s no evidence it does. Google has said it’s not a ranking factor, and no major AI engine has confirmed using it to decide mentions or citations. Treat it as neutral tidy-up, not a visibility lever.

Do AI crawlers actually read llms.txt? Rarely. Server-log analyses find the overwhelming majority of llms.txt files get zero AI-crawler requests, and the big crawlers fetch HTML directly instead. Some retrieval workflows (Claude products, Perplexity) have been reported to respect it, but broad engine support doesn’t exist.

Is llms.txt the same as robots.txt? No. robots.txt controls crawler access; llms.txt is a curated invitation listing your best content for LLMs. Use robots.txt for control, llms.txt (optionally) for curation.

Should I bother creating an llms.txt file? If it’s cheap — an hour or auto-generated — yes, especially for developer docs consumed by AI agents. If it would take ongoing effort, put that time into citable content and cited-domain presence instead.

How do I know if bots read my llms.txt? Watch your server-side crawler data. With crawler detection, requests to /llms.txt appear in your bot top-paths, labeled by crawler and verified-vs-spoofed — so you can see exactly who fetches it, if anyone.

Want to see whether AI bots actually fetch your llms.txt? Start your free trial — 7 days, no credit card required — and watch which crawlers request which paths on your site.