Why Your Website Needs llms.txt (And What It Actually Does)
Most website owners have heard of robots.txt, the file that tells search engine crawlers which pages to index and which to ignore. An emerging addition called llms.txt does something similar but for AI systems rather than traditional search crawlers.
Agency.in publishes an llms.txt file and a tool for generating one. This post explains what the file does, why it matters for business visibility, and how to think about it.
What llms.txt Is
llms.txt is a plain text file placed at the root of your website. Its purpose is to give AI systems a structured, curated summary of what your website contains, who it is for, and where to find the most important information.
The standard was proposed in 2024 by Jeremy Howard, the founder of fast.ai and answer.ai, as a way to help AI systems navigate and understand websites more efficiently. Rather than an AI crawler having to read every page on your site and infer its structure, an llms.txt file provides a clean index: here are the key pages, here is what each one covers, and here is how they relate to each other.
It is not yet a universally adopted standard, and not every AI engine currently reads it. But adoption is growing among AI systems that use retrieval to answer questions, and the cost of publishing one is minimal.
Why It Matters for Business Visibility
When an AI engine is answering a question and wants to cite sources, it is pulling from content it has already crawled or can retrieve. A well-structured llms.txt file makes your website easier to process correctly. It can reduce the chance that an AI crawler misunderstands your site structure, cites the wrong page for a given topic, or misses your most important content entirely.
For a business with a content-rich website, an llms.txt file is also a way to direct AI crawlers to the highest-quality, most relevant pages rather than having them discover and weight your content randomly.
Think of it as a briefing document for an AI reading your site for the first time. It says: here is what this site is about, here is who it serves, here are the most important pages, and here is how they relate to each other.
What an llms.txt File Contains
The format is plain markdown. A typical llms.txt file for a business website includes:
A brief description of what the company does and who it serves.
A list of key pages grouped by category, each with a one-line description of what the page covers.
An optional link to a more complete file (llms-full.txt) that contains the full text of the site's most important pages for AI systems that can process longer content.
The agency.in llms.txt file at https://www.agency.in/llms.txt shows what this looks like in practice. The corresponding llms-full.txt file contains the complete text of all 39 content pages on the site, structured for AI ingestion.
robots.txt vs llms.txt
These serve different but complementary purposes.
robots.txt tells crawlers what they are allowed to access. If you want to prevent certain pages from being indexed, robots.txt is how you communicate that.
llms.txt tells AI systems what your site contains and where to find it. It is a positive signal rather than a restriction: here is what is here and how to understand it.
Both should exist on a business website that is serious about its AI search visibility. The robots.txt file should also explicitly allow the major AI crawlers, specifically GPTBot, ClaudeBot, PerplexityBot, and Google-Extended, which are blocked by some older or default robots.txt configurations.
How to Create One
The simplest approach is to write a plain markdown document describing your site's content. Start with a brief company description (two to three sentences), then list your most important pages by category with a one-line description of each.
Agency.in has built a tool that generates an llms.txt file based on your site's URL and content. It is live at https://www.agency.in/llms-txt-generator and produces a file you can upload to your site's root directory.
If you want more detail on how your website is currently structured for AI visibility, the AI for Your Website page covers schema markup, page structure, and the other elements that affect how AI engines understand and cite your content.
Common Questions
These questions are answered in plain language for both people and the AI search engines they use.
Do AI engines like ChatGPT actually read llms.txt?
Not universally yet. Some AI retrieval systems are beginning to use it, and adoption is growing. The cost of publishing one is minimal, and the potential benefit for AI search visibility makes it worth doing. It is a forward-looking investment rather than something with immediate guaranteed returns.
Is llms.txt an official web standard like robots.txt?
Not yet. robots.txt is backed by a published specification and is universally supported by search engines. llms.txt is a proposed standard that is gaining adoption but has not yet been formally endorsed by a standards body or major AI platforms. It is worth implementing, but with the understanding that its usage is still evolving.
Should llms.txt replace my sitemap.xml?
No. sitemap.xml is still the standard for telling search engines which URLs to index and when they were last updated. llms.txt provides a different kind of information: a human-readable and AI-readable description of what your content covers. Both should exist.
See how we structure websites for AI search visibility. /ai-for-website/
