LLM.txt
A proposed standard file (similar to robots.txt) that provides a Markdown-formatted map of a website for AI crawlers to consume efficiently.
What is LLM.txt?
The file is conventionally named llms.txt and placed at the site root (for example, https://example.com/llms.txt). It is a plain Markdown document that opens with the site or project name, a short blockquote summarising what it is, and then sections of links to the pages that matter most, each with a one-line description. An optional companion, llms-full.txt, can contain the full text of key documentation in one file so a model can ingest it in a single request.
The idea, put forward as an open proposal in 2024, is that HTML pages are noisy for language models: navigation, scripts, cookie banners and ads dilute the useful content and consume context. A curated Markdown index gives an AI system a clean starting point that says "here is what this site is, and here are the canonical pages for each topic". It is meant to complement robots.txt (which says what crawlers may fetch) and sitemap.xml (which lists every URL), not replace them.
Adoption is voluntary on both sides. Publishing the file is cheap and low-risk, and some AI tooling and documentation platforms read it, but the major answer engines have not publicly committed to using it for ranking or retrieval. Treat it as a hygiene item and a clear signal of intent rather than a guaranteed lever.
Why it matters for AI search
llms.txt is one of the few AI-specific technical signals a site can control directly. It costs little, it clarifies which pages are canonical for which topics, and it forces a useful editorial exercise: deciding what the brand most wants an AI system to read first. Combined with correct robots.txt access for retrieval crawlers and server-rendered pages, it removes avoidable friction between a site and the engines that might cite it.
Related terms
Synthetic Traffic
Web traffic where the primary visitor is an AI agent or crawler acting on behalf of a user.
AEO (Answer Engine Optimization)
The practice of optimizing content to be cited and recommended by AI-driven search engines and conversational assistants like ChatGPT and Perplexity.
RAG (Retrieval-Augmented Generation)
A technique that allows LLMs to pull in real-time information from external sources (like a website) before generating an answer.
Knowledge Graph
A network of entities and their relationships that AI models use to provide structured, factual answers.
Frequently asked questions
Is it llm.txt or llms.txt?+
The proposed standard uses llms.txt (plural), served at the site root. llms-full.txt is the optional expanded version containing full page content.
Do ChatGPT or Google use llms.txt?+
Neither has publicly confirmed using it for search or citations. Some developer tools and documentation hosts do read it. It is worth publishing because it is cheap and harmless, but it should not be relied on as the main GEO tactic.
What should go in llms.txt?+
A title, a one-paragraph summary, and grouped links to the most important pages: product overview, pricing, documentation, key guides, and an "optional" section for lower-priority pages. Keep descriptions factual and short.