What is an llms.txt file?

An llms.txt file is a markdown-formatted document located at the root of a domain (e.g., yoursite.com/llms.txt) designed to provide AI models and autonomous agents with a curated roadmap of clean, canonical, and high-value content.

How does llms.txt differ from robots.txt?

robots.txt is a strict access-control protocol that instructs crawlers which URLs they may or may not fetch. llms.txt is an informational curation standard that tells AI models where to find the highest-signal markdown and text content.

Can an llms.txt file block AI crawlers from scraping my website?

No. llms.txt cannot block crawlers or restrict data usage. To prevent crawling or ingestion, you must enforce rules in robots.txt or deploy server-level web application firewall (WAF) blocks.

How many links should be included in an llms.txt file?

An effective llms.txt file contains 10 to 30 curated, high-value links with brief factual descriptions. Overloading the file with hundreds of links creates noise and exhausts LLM context windows.

What is the difference between llms.txt and llms-full.txt?

llms.txt acts as an index containing titles, URLs, and summaries, whereas llms-full.txt concatenates the full markdown text of all core documentation and specifications into a single downloadable file for deep ingestion.

llms.txt: The New robots.txt for AI Search & LLM Ingestion

CORE_CONCEPT

llms.txt

A standardized, markdown-based document served at the root of a domain that indexes high-signal pages, API documentation, and core product facts specifically formatted for ingestion by LLMs and autonomous agents.

CORE_CONCEPT

llms-full.txt

An extended variant of the llms.txt convention that bundles the complete raw markdown text of key website documentation into a single file, allowing AI systems to ingest the entire knowledge base in one request.

lightbulb

STRATEGIC_PLAYBOOK

Architectural Distinction: `robots.txt` governs permission (who can crawl); `sitemap.xml` lists raw URLs (everything available); `llms.txt` curates semantic authority (what matters most for AI synthesis). Do not dump your entire XML sitemap into `llms.txt`—curate only your 10–30 highest-density pages.

Comparing Discovery & Control Protocols: robots.txt vs. sitemap.xml vs. llms.txt

Protocol	Format	Primary Consumer	Purpose / Role	Crawl Enforcement
`robots.txt`	Plaintext Directive Format	All Web Crawlers & Bots	Controls crawl access, blocks sensitive directories, declares sitemaps	Legally & technically honored by standard bots
`sitemap.xml`	XML Schema with timestamps	Search Engine Indexers	Comprehensive inventory of all indexable URLs and modification dates	Crawlers discover URL paths without guarantee of indexing
`llms.txt`	Clean Structured Markdown	LLMs, AI Agents, RAG Pipelines	Provides an uncluttered, token-efficient table of contents of core facts	Voluntary ingestion by AI models to minimize parsing latency
`llms-full.txt`	Raw Markdown Full Text	Deep Context LLM Agents	Single-payload documentation ingestion for zero-friction training & RAG	Allows instant context loading without scraping hundreds of pages

What to Include vs. What to Exclude in llms.txt

Category	Include in llms.txt	Exclude from llms.txt	Reasoning
Product & Features	Canonical product specifications, feature matrices, API docs	Marketing fluff, sales banners, seasonal promotions	LLMs require unambiguous technical capabilities, not promotional adjectives.
Pricing & Plans	Clean tabular pricing tiers with inclusions/exclusions	Dynamic calculator scripts, gated quote forms	Ensures ChatGPT and Claude accurately quote current pricing without hallucination.
Company & Entities	Founding team, mission, official contact, verified social URLs	Unformatted press releases, investor disclosures	Anchors brand entity attributes firmly in the model's knowledge graph.
Guides & Docs	Top technical tutorials, integration manuals, GEO benchmarks	Generic listicles, clickbait articles, thin blog archives	Focuses the AI's attention on your highest information-gain assets.

CORE_CONCEPT

Token Efficiency Optimization

The engineering practice of eliminating DOM boilerplate, CSS, tracking scripts, and navigation clutter from AI-facing payloads to maximize the density of actionable information per token processed.

Reference Structure of a Production-Ready llms.txt

Markdown Section	Syntax / Formatting Example	Target Function
H1 Header (Required)	`# Acme AI — Platform Overview`	Declares the entity name and canonical domain identity.
Blockquote Summary	`> Enterprise API for vector embeddings and search.`	Provides a concise, neutral 20-word entity definition for zero-shot retrieval.
Section Headers (H2)	`## Core Documentation` / `## Pricing`	Categorizes knowledge nodes into distinct semantic groups.
Curated Links	`- [REST API](https://acme.com/docs): Complete endpoints.`	Supplies direct URLs paired with 10–15 word descriptions.
Optional Full Export	`- [Full Documentation](https://acme.com/llms-full.txt)`	Points heavy AI models to the complete markdown text stream.

lightbulb

STRATEGIC_PLAYBOOK

Hosting Configuration: Place your `llms.txt` file in your root `/public` directory so it resolves at `https://yourdomain.com/llms.txt`. Ensure your web server serves it with `Content-Type: text/plain; charset=utf-8` and does not require authentication or JavaScript evaluation.

llms.txt Rollout & Maintenance Checklist

Deployment Step	Action Item	Validation Method	Priority
1. Asset Selection	Select 15–25 highest-value canonical URLs across product, pricing, and docs.	Verify all selected links have 200 HTTP status and clean HTML	Critical
2. Markdown Authoring	Write neutral, Wikipedia-style summaries for each link without hype words.	Check information density and ensure token count remains under 2,000 tokens	High
3. Server Placement	Deploy file to `/public/llms.txt` with CORS enabled (`Access-Control-Allow-Origin: *`).	Curl `https://yourdomain.com/llms.txt` via CLI to verify instant plaintext response	Critical
4. Sitemap Integration	Add `https://yourdomain.com/llms.txt` to your `sitemap.xml` for crawler discovery.	Verify inclusion in Google Search Console and Bing Webmaster Tools	Medium
5. Quarterly Review	Synchronize links whenever pricing tiers, product features, or docs change.	Automated CI/CD link-checker test to prevent broken links in llms.txt	High

llms.txt: The New robots.txt for AI Search & LLM Ingestion

Key Takeaways & Executive Summary

llms.txt

llms-full.txt

STRATEGIC_PLAYBOOK

Comparing Discovery & Control Protocols: robots.txt vs. sitemap.xml vs. llms.txt

What to Include vs. What to Exclude in llms.txt

Token Efficiency Optimization

Reference Structure of a Production-Ready llms.txt

STRATEGIC_PLAYBOOK

llms.txt Rollout & Maintenance Checklist

Is Your Brand Being Cited by ChatGPT & Claude?

Related Learning Guides

From SEO to AEO: Understanding the Shift

What is Answer Engine Optimization (AEO)?

Keywords vs. Entities: The Building Blocks