What Is llms.txt?
Definition
An emerging web standard — a plain-text file at /llms.txt that provides AI crawlers with a curated, structured summary of a website's most important content.
Why It Matters
llms.txt gives AI systems a clean, navigable map of your site — similar to what robots.txt does for traditional search crawlers. Early adopters including Anthropic, Mintlify, Stripe, and Vercel publish llms.txt files. As adoption grows, having one is cheap insurance for AI discoverability.
How It Works
An llms.txt file is a markdown document at the root of your domain. It starts with an H1 (site name), a blockquote summary, and curated H2 sections listing key pages with short descriptions. A companion llms-full.txt file (optional) concatenates full page content for AI training and retrieval.
An Australian digital agency publishes /llms.txt listing its pillar services, lead magnets, comparison guides, and glossary. Within weeks, Perplexity and Claude begin citing pages from the site more frequently because the llms.txt gave the AI systems a clear, curated index of what's available.
Quick Facts
- llms.txt was proposed in September 2024 by Jeremy Howard of Answer.AI
- Anthropic, Mintlify, Stripe, and Vercel are early adopters
- llms.txt is markdown; robots.txt is a separate crawler-directives file
- The companion llms-full.txt provides full text for training and RAG systems
Need Help With llms.txt?
Our team of experts can help you implement effective strategies.
- Expert consultation
- Tailored strategy
- Measurable results
No long-term commitment. Cancel anytime. 100% satisfaction guaranteed.