llms.txt: what it is, who reads it, and why it is not in your AI answer

Checked against the providers' own published documentation on 10 September 2026. Where nobody has published a position, this piece says so.

Someone your team trusts has told you to publish an llms.txt file. They are not wrong that it is cheap. It takes an afternoon, it breaks nothing, and there are real reasons to have one. What it will not do is decide whether an AI assistant names your brand, and the largest answer engine has written that down in its own documentation.

What reads the file and what decides your visibility are two different systems. There is also a file at your root that genuinely decides whether you can be cited at all, and it is not this one.

What the file is, and who it was built for

llms.txt is a plain markdown file at the root of a domain, alongside robots.txt. Inside it is a curated index: a short site description, then links to the pages that matter with a line of context on each. Some sites also publish llms-full.txt, the whole documentation set in one file.

Jeremy Howard proposed it in September 2024 to solve a context window problem. His own framing: "Large language models increasingly rely on website information, but face a critical limitation: context windows are too small to handle most websites in their entirety," and it is "particularly important for use cases like development environments, where LLMs need quick access to programming documentation and APIs". Search Engine Land's write-up of the proposal.

That is still where the file lives. OpenAI publishes one for its own developer documentation, an index routing a coding agent to the API guides, the Codex pages, the cookbook. Be clear about what that does and does not show. It shows a provider finding the format useful for its own manual. It says nothing about whether that provider's crawlers read yours.

Google has written down that Search ignores it

It is in Google's guidance on generative AI in Search, updated 10 July 2026, in the section on what you do not need to do.

You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.

The same page then names the file directly.

It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.

Its own summary is blunter: "you can ignore tactics like 'chunking' content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions". Read the guidance.

Google's Gary Illyes was reported as saying the same at a Search Central Deep Dive event in 2025: Google does not support the file and is not planning to. That one reached the trade press second hand, through an attendee's notes rather than a Google post, which is why the documentation above is the citation to lean on.

Google's page on AI features adds the rule that matters more. To be eligible to be shown as a supporting link in AI Overviews or AI Mode, it says, "a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. There are no additional technical requirements." Read the page. The unit is a page in the index, not a file at your root.

Nobody who writes the answers has committed to reading yours

Google is the clearest because it has published a position. The other providers have mostly published nothing either way, which is not the same as a refusal, and we are not going to write it up as one.

What is on the record is Google's John Mueller, quoted in January: "None of the AI services have said they're using llms.txt, and you can tell when you look at your server logs that they don't even check for it." That is one engineer reading his own logs rather than a policy statement from the other providers, and it is still the most specific thing anybody has published.

Ana Fernández, writing in Search Engine Land in January 2026, tracked ten sites across a ninety day window after each published one. Most showed no measurable change and one went backwards. The two that rose had both shipped something else in the same weeks: one had just landed national business coverage while fixing crawl errors, the other had just published a set of downloadable tools. Her reading of it is the useful line: sites that launched new, functional content saw gains, sites that documented existing content saw none, and the file "documented those efforts. It didn't drive them." Her verdict on the format is useful infrastructure rather than a growth lever, closer to an XML sitemap than to a campaign. Read the analysis.

Now the complication, before you conclude the file is dead. In May 2026 Google added an experimental Agentic Browsing category to Chrome's Lighthouse tool, which checks for a machine readable summary at the domain root, on the reasoning that "without llms.txt, agents may spend more time crawling the site to understand its high-level structure and primary content". Asked why Google's own browser checks for a file Google's Search documentation ignores, Mueller's answer was that "the short answer is that it's not done for search", and he called markdown files of this kind "more of a temporary crutch, perhaps to save some tokens". He also pointed out that most sites get very little agent traffic today. Read the report. Helping an agent work efficiently on a site it has already reached is one job. Deciding which pages get reached and quoted is another.

What the file is genuinely good for

• Documentation an agent has to read. If you sell to developers, or you have a help centre, an API or product specifications an assistant will be asked to work through, this is the file's home ground.
• Agents already on your site. Google's wording for its own browser check is about crawl efficiency, not ranking. If agent traffic matters to you, an index that saves an agent wandering your navigation is a small kindness that costs you nothing.
• The discipline of writing one. To build the index you have to decide which pages state your facts plainly and which are marketing wrapper. Most teams find something worth fixing there, and the fix has value whether or not any engine reads the file.

So publish it. Just book it on the site maintenance list next to the sitemap, rather than on the line where you are trying to change what a model says about your category.

What decides whether you are in the answer

Google describes its own mechanism. Its generative features use retrieval augmented generation, "relying on our core Search ranking systems to retrieve relevant, up-to-date web pages from our Search index", plus query fan-out, "a set of concurrent, related queries generated by the model" that fetches further results. An answer is assembled out of pages the ranking systems retrieve, and a file at your root is not in that path.

Which pages, then. Foundation Marketing and AirOps, May 2026, found that 90% of the sources AI cites when answering are ones the brand does not control. The market the sample covers is not stated. Read the study. So the paragraph a buyer reads about your category is written mostly from pages on other people's domains. Your own pages do real work on the questions that carry your name, and almost none on which third-party page gets picked for the ones that do not. How brands actually get cited works through that mechanism.

One caveat, from the same Google document, and it cuts at our end of this as much as anyone's. It says its generative features "can show what's being said about products and services across the web", and in the same breath that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem", because "our core ranking systems focus on high-quality content while other systems block spam". Read those together and they are one instruction rather than two. What is said about you off your own domain is in scope, and the machinery is built to sort the genuine from the manufactured. That is an argument for commissioning the kind of piece a publication's own editors would run on their own standard, and against buying mentions by the yard.

The root file that does matter, and the token most sites get wrong

llms.txt is optional. robots.txt is not, and it is where the real damage gets done, because the crawler that reads in order to answer is not the crawler that reads in order to train, and the two carry separate tokens.

OpenAI runs four, set out in its own crawler documentation at developers.openai.com/api/docs/bots. OAI-SearchBot is the one that puts sites into ChatGPT's search results. GPTBot is the training crawler. ChatGPT-User fetches a page when a user's question sends it there. OAI-AdsBot validates the safety of pages submitted as ads. Each setting is independent of the others, and that is the sentence to take away: a rule written to keep your content out of training does nothing to the crawler that decides whether you can be quoted, and a blanket block on everything takes you out of the answer.

Google splits the same way and has published the split. robots.txt for Googlebot is the control for Search, AI Overviews and AI Mode included. Google-Extended is a separate token that governs training and grounding in Gemini Apps and Vertex AI, and in Google's own words it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search". Read the crawler list. So a team that disallowed Google-Extended to stay out of Gemini's training has not removed itself from AI Overviews, and a team that disallowed it hoping to stay out of AI Overviews has not done that either.

One caution about where an access checklist usually comes from. OpenAI's published guidance for advertisers reads as exactly that sort of list: fix robots.txt, the firewall, the CDN, bot mitigation, authentication and rate limiting so the pages can be crawled. Read the guidance. It is worth reading, and worth reading for what it is. It is written about OAI-AdsBot and landing pages submitted as ads on ChatGPT, so it is a readiness step for a paid placement, not a route into an organic answer. The token that governs the organic answer is the search one, and nobody sells you access to that.

If you have an hour for root files this quarter, spend it on robots.txt. That file has a failure mode that costs you the answer. llms.txt does not have one.

Publish it, then keep the visibility question separate

Two questions for whoever recommended it. Which engine reads it, and where has that been published? And if it were read, what would change, and how would we see it? Both have good answers if the goal is agents reading your documentation.

If the goal is being named when a buyer asks an open question in your category, neither has a good answer, and the work moves off your own site. The site half of AI visibility is real and your search team should keep doing it. The half that decides the answer is bought the way media is bought, a different owner and a different line on the plan.

So publish the file, then find out where you actually stand. A free AI Visibility Report runs the questions your buyers ask across the engines and shows where you appear, what gets said, and which publications the answer is built from. That source list is the brief for the work llms.txt was never going to do, and one brief can reach many of those publishers at once.