I was wrong about LLMs.txt

Lately I’ve been digging into the area of agentic discovery and how agents crawl, navigate and use websites & products. It’s a fascinating area that’s actively emerging now and will be the major source of growth of internet usage over the coming years. Today, agentic usage is small, but it’s bound to 10x (maybe someday 1000x) the usage of the web. Cloudflare says bot traffic finally outpaced humans.

Understanding how these tools navigate a website is critical for them using websites effectively and I now think that LLMs.txt is a critical component of doing this well. That said, it’s not in the traditional sense of how the spec was envisioned.

What is llms.txt

The core concept behind llms.txt is that it’s a simple text file that acts like a sitemap.xml. Sitemap.xml is a traditional SEO/crawler tool that tells bots, search engines, indexing platforms what pages exist on a website. LLMs.txt is a way of putting these URLs into an AI/LLM friendly format.

Similar to the way you might have a robots.txt, security.txt, etc.. file at the root of your domain. You can see an example here for my open tool, Agent Tools: https://agtls.dev/llms.txt

The traditional way of implementing it is to just put text of all the pages of your site with a short title/description of the content next to the link. This is usually in markdown format.

Why I was wrong

First off, and just to get it out of the way, the spec is really annoying from a technical perspective. It recommends implementing a markdown format, but is a text (.txt) file. It should be a .md file for markdown. Okay, that's out of the way. Let’s get to the real reasons.

The problem with new internet specs is that it requires adoption by both platforms (think AI chatbots, indexing tools, crawlers, etc..) and websites. There’s always a chicken-and-egg problem for adoption with any new protocol. Frequently, it’s painfully slow and knowing when the tipping point hits is hard to predict.

From my work at Duda, where we host a whole lot of SMB websites, I could always see that llms.txt never gets crawled by the big AI platforms (ChatGPT, Claude, Gemini, etc..). The only tools that crawl them are audit tools. Google has also said that they don’t use it and it won’t help discovery. This led me to think that the spec is kinda a dud, since no one is really adopting it.

But what I got wrong was that llms.txt is not the same as sitemap.xml. It can be more.

AI Tools/Crawlers

AI Agents interact with websites in interesting and vague ways. The core learning / frame of reference right now is that agents follow two principles:

• They’re API first and strongly prefer to only read text. Meaning that they treat websites, products and services as something to pull/integrate with. This means a web page needs to be thought of as an endpoint / content – not a design.
• They’re goal oriented. You need to tell them how to accomplish their goal and what they can and more importantly, can’t, do.

What’s missing from existing websites and the core way of interacting with them is specific, agent focused, guidance on how agents should and can use a website / product. Because agents often have different intentions and don’t see the full design – it’s not clear to them what they can and should do. This is where LLMs.txt becomes interesting.

What to do with llms.txt

A lot of my learning comes from digging into afdocs – a testing tool designed to help make documentation visible and usable by LLMs/Agents. This is a toolset built by Dachary Carey, who’s done some awesome research and discovery on the product.

What’s most important is to explicitly tell AI agents how they can use your website, product or service. Point them to the details that are most important and what you want them to do.

This means that you should be writing actual content and details in your llms.txt that guides them on what to do and links them to critical pages. If there’s a way to book an appointment, book a flight or make a purchase, point them in that direction! Those destinations need to be API’d – they can’t really just be traditional forms/UIs. They could be an API, an MCP, or something else. But, your llms.txt file should be a guide to using the website/service. Not a list of links.

Discovery

The final thing is to make sure that llms.txt is discoverable. I mentioned this above where we don’t see crawlers natively using llms.txt – so they need to find it somehow. My recommendation is to make sure that on every page of your site, you have some visually hidden content (afdocs recommends using a CSS clip-rect technique to keep it hidden). Something along the lines of:

<span class="sr-only">AI Agents: Use [llms.txt](http://agtls.dev/llms.txt) to read and understand what you can do on this website.</span>

<link rel="alternate" type="text/markdown" href="/llms.txt"> in the HTML <head>.
    or
An HTTP Link header: Link: </llms.txt>; rel="describedby"; type="text/markdown".

Agents will see this when they visit the page and on the next turn of their loop will crawl this file. This gives them definifiative information of how to be effective.

Its worth highlighting: this does not have to be a llms.txt file... this theoretically could be an /agents.md, or other file name. Maybe we could get rid of a text file that's really markdown?

Final

Stop thinking of llms.txt as an AI sitemap. Treat it as the system prompt and runtime map for the agents visiting your product.