Skip to content

llms.txt: Making Your Project Discoverable to AI Agents

llms.txt is a plain-text file at /.well-known/llms.txt or /llms.txt that tells AI agents and language models what your project does and how to navigate its content efficiently.

The llms.txt specification (Jeremy Howard, answer.ai) defines a Markdown file published at {site-root}/llms.txt. LLM context windows are too small to read a full website, and HTML adds noise. So llms.txt gives agents a curated, structured index instead. One fetch replaces undirected crawling, and the agent spends its context budget on content rather than discovery.

A companion convention is /llms-full.txt: every linked page concatenated, so an agent gets the whole site in one fetch.

When this backfires

  • Static sites that change rarely benefit most. High-churn sites risk serving stale link lists that mislead agents.
  • No major LLM provider has published documentation confirming they read llms.txt at inference time. Google's John Mueller said in June 2025 that "no AI system currently uses llms.txt" and that server logs show AI bots are not fetching it (Search Engine Roundtable: Google on llms.txt). Treat adoption as forward-compatible infrastructure, not a guaranteed citation signal.
  • The 2026 data shows no citation uplift. An analysis of 300,000 domains found no measurable effect of llms.txt presence on AI citation likelihood, and adoption stays near 10% even among tech-forward publishers (Search Engine Journal: llms.txt shows no clear effect across 300k domains; ALLMO analysis of 94k cited URLs).
  • A poorly curated llms.txt that points to dead links is worse than none. Agents that follow broken links waste context and may return errors.

See llms.txt: Spec, Adoption, and Honest Limitations for the full adoption picture and known limitations.

Example

A minimal llms.txt for a documentation site:

# Acme Docs

> Developer documentation for the Acme platform — REST API, SDKs, and tutorials.

## Core Documentation

- [Quick Start](/docs/quickstart): First app in 5 minutes
- [API Reference](/docs/api): Full endpoint reference

## Optional

- [Changelog](/changelog): Release notes

The H1 is the only required element per the llms.txt spec. Sections and the ## Optional block are conventions that help agents budget context. Publish /llms-full.txt alongside it to give agents the full concatenated content in one fetch.

Key Takeaways

  • llms.txt is a curated Markdown index at /llms.txt that tells agents where your content lives without forcing them to crawl HTML pages.
  • The spec requires only an H1; blockquote summaries, H2 sections, and an ## Optional block are conventions that help agents budget context.
  • Publishing /llms-full.txt alongside it lets agents fetch all linked content in a single request, eliminating multi-step navigation.
  • Treat llms.txt as forward-compatible agent infrastructure — empirical 2026 data shows no measurable citation uplift, so do not invest in it as an SEO or ranking signal.
  • A stale or broken llms.txt is worse than none; only publish if you can keep the link list current.
Emerging — single recent source Last reviewed: 2026-05-27
Feedback