What Is llms.txt? The AI-Readability Standard, Explained (2026)

llms.txt is a proposed web standard: a single Markdown file placed at the root of your site (at /llms.txt) that gives large language models a clean, curated map of your most important content. Think of it as a reading guide written for machines — not a wall, but a welcome mat that points an AI to the pages you most want it to understand and cite.

It was proposed by Jeremy Howard, co-founder of Answer.AI, in September 2024 as "a proposal to provide information to help LLMs use websites." The official specification at llmstxt.org keeps it deliberately simple: the only required element is an H1 with your project or brand name, followed by an optional one-line summary in a blockquote and H2-delimited lists of links to your key pages. It is designed for inference time — the moment an AI is answering a question — not for training.

How is llms.txt different from robots.txt?

This is the most common point of confusion, and the distinction matters.

robots.txt tells bots where they may go. llms.txt blocks nothing — it simply hands an AI a tidy index so it doesn't have to fight through your navigation, scripts, and boilerplate to find what matters. The two are meant to coexist alongside your sitemap.xml. There is also a companion file, llms-full.txt, which Mintlify says it developed in collaboration with Anthropic, containing your full plain-text content in one document for models that want everything at once.

Is anyone actually using it?

Adoption is climbing fast, but from a small base. As of June 2026, llms.txt was present on 5.61% of the top 10,000 websites, up from just 1.04% a year earlier — roughly 5.4x growth in twelve months. Much of that surge is platform-driven rather than organic: Shopify silently pushed llms.txt to every store by default in spring 2026, taking adoption across Shopify sites to over 78%.

The critical question is whether the major AI engines read it. Here the evidence is sobering. Google's John Mueller said no AI services have confirmed using llms.txt, comparing it to the long-ignored keywords meta tag — a self-declared signal anyone can write. Gary Illyes later confirmed Google "doesn't support llms.txt and isn't planning to," though server logs suggest OpenAI's crawler may occasionally check for the file.

Where llms.txt clearly earns its keep is developer documentation. The file is consumed mainly by coding and IDE agents — Cursor, Windsurf, Claude Code, GitHub Copilot — pointed at docs, which is why the developer-doc sites of Anthropic, Perplexity, Mistral, and Cohere host one even though their consumer front doors do not.

Should you add llms.txt in 2026?

Our honest take: yes if it's cheap, but with clear eyes about why. A minimal, accurate llms.txt takes about fifteen minutes to write, the downside risk is essentially zero, and adoption momentum is real. If you publish technical documentation or an API, the case is strong — that's exactly the content coding agents read today. For a general marketing site, treat it as a low-cost hedge, not a growth lever.

What it is not is a substitute for the work that demonstrably moves AI visibility: well-structured, crawlable, genuinely authoritative content. That foundation — covered in our guide to whether your site is even letting AI crawlers in and our primer on Answer Engine Optimization — is what actually gets you cited. llms.txt is a small, sensible addition on top of it, and it's the kind of detail our SEO & AI search team and web development team handle as part of making a site genuinely machine-readable.

Sources

• https://www.answer.ai/posts/2024-09-03-llmstxt.html
• https://llmstxt.org/
• https://www.mintlify.com/blog/the-value-of-llms-txt-hype-or-real
• https://caseyrb.com/blog/state-of-llms-txt-adoption/
• https://shopify.dev/changelog/customize-llmstxt-llms-fulltxt-and-agentsmd
• https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/
• https://www.seroundtable.com/openai-crawling-llms-txt-files-39811.html