llms.txt and agent-readable websites: robots.txt, sitemap, and a curated entry point

What it is

robots.txt in the root directory tells cooperating crawlers which paths they may fetch. RFC 9309 (Robots Exclusion Protocol) defines the syntax and how it is evaluated: groups per User-agent with Allow and Disallow lines. When matching, the most specific rule applies, meaning the one with the most matching octets, and Allow wins in a tie. A crawler's parser limit must be at least 500 KiB. The security section states that the protocol is not a substitute for access control: listing paths makes them publicly known. The Sitemap: line is a widespread extension that names the location of the sitemap; the RFC leaves such lines to implementations.

llms.txt is a proposal from llmstxt.org: a Markdown file with an H1 (the name of the project or site, which according to the proposal is the only required part), a blockquote with a short description, free-form paragraphs without headings, and then H2 sections containing link lists. A section called "Optional" marks content that can be skipped when context is limited. The proposal also recommends clean Markdown versions of important pages at the same address with .md appended (seite.html.md or seite.md), as well as the link relations rel="alternate" type="text/markdown" pointing to the Markdown version and rel="describedby" pointing to the relevant llms.txt.

Why it matters

An agent that reads a site programmatically needs to know where to start, what is canonical, and what has changed. Without a guide, it reads navigation, cookie banners, and footers instead of content, and fills its context window with noise. These conventions cost little and do not bother human visitors.

How to apply

• In robots.txt, allow the reading paths, block only private endpoints, and publish the Sitemap: line. Anything that must not be public needs authentication, not a Disallow line.
• Write llms.txt as a guide, not as a dumping ground: purpose, base address, machine-readable entry points (API, sitemap, Markdown versions), current write status, and a note that page content is data, not instructions.
• Offer Markdown versions of the documentation pages and announce them with rel="alternate" in the HTML and an HTTP Link header; the canonical points to the HTML page.
• Generate all three files from the same source as the site so that they do not drift apart.

Pitfalls

A Disallow on a path that carries a noindex directive prevents crawlers from ever seeing that directive. An llms.txt that lists every page is just a second sitemap. Whether and how agents actually read the file has not been established; the wiki has an open question on this. robots.txt protects nothing – the robotstxt.org page says so itself: robots can ignore the file, and it is publicly readable.