llms.txt

llms.txt is a proposal from September 2024 by Jeremy Howard of Answer.AI at llmstxt.org. It is a plain Markdown file served at /llms.txt. It gives AI models and agents a curated map of your site.

It has one required element: an H1 with your site or project name. Then a blockquote with a one paragraph summary. Optional paragraphs may follow. After that, H2 sections each hold a list of links in the form [title](url): one line description. A section titled Optional marks links an agent can skip when its context is short. /llms-full.txt can optionally carry the full text of pages as Markdown.

It is not access control, an opt out or a licence file. robots.txt governs what a crawler may fetch. llms.txt says what the site is and where the useful pages are.

# YourProduct

> A tool for small teams to track bugs and ship fixes. Web app and API. Free tier. Docs, changelog and pricing below.

We prioritise stability and clear docs.

## Docs
- [Getting started](/docs/getting-started): Install, invite your team, first project in 10 minutes.
- [API reference](/docs/api): REST endpoints with examples.
- [Webhooks](/docs/webhooks): Events you can subscribe to.

## Product
- [Pricing](/pricing): Plans and limits.
- [Changelog](/changelog): What shipped this month.
- [Status](https://status.yourproduct.com): Uptime and incidents.

## Company
- [About](/about): Team and contact.
- [Security](/security): Data handling and audits.

## Optional
- [Blog](/blog): Commentary and tutorials.

A good file reads like a tidy menu, not a dump. Keep link titles human. Keep descriptions to one clean line each. Put long lists in their own H2.

1. 1

   Draft the outline

   Write the H1 as your site name: "YourProduct". Add a one paragraph summary in a blockquote. Add one or two plain paragraphs if they help.

2. 2

   Group key pages

   Create H2 sections like Docs, Product, Company. List 5 to 15 links that define the site. Format each as [Title](URL): one line description.

3. 3

   Mark optional links

   Add an H2 named Optional for nice to have links, like blog archives or long indexes.

4. 4

   Save and serve it

   Save as UTF 8 without BOM. Name it llms.txt. Place it at the web root so it resolves at https://yourproduct.com/llms.txt.

5. 5

   Consider /llms-full.txt

   If you want agents to ingest full content, publish a separate /llms-full.txt with Markdown copies of key pages.

On a static site, add llms.txt to the public folder. On a framework, map a static route to /llms.txt with text/plain or text/markdown. Avoid redirects.

• Open /llms.txt in a browser. Check the H1, blockquote, H2s and the link lines render as plain text.
• Click a few links. They must be absolute or correct relative URLs. No 404s. No redirects if you can avoid them.
• View the response headers. text/plain or text/markdown is fine. Status 200. Cache it for a day or a week.
• Validate by eye against llmstxt.org. Confirm the Optional section is only things an agent can skip.
• Set a quarterly reminder. Update when you add a major page like /pricing or /docs/api.

For /llms-full.txt, spot check the Markdown renders, headings are clear and code blocks are fenced. Keep it in sync with live pages after big edits.

• Treating llms.txt as a crawler gate. It is not. Use robots.txt for access control.
• Listing everything. Curate. Ten strong links beat fifty weak ones.
• Forgetting the blockquote summary. Many agents will read that first.
• Letting URLs rot. A single stale /docs link penalises trust by agents and users.
• Burying critical info under Optional. Keep docs, pricing and status in main sections.
• Serving HTML. Keep it plain Markdown at /llms.txt, not a webpage.