llms.txt Implementation Guide: The New robots.txt for AI?

robots.txt tells automated crawlers where they are allowed to go.

sitemap.xml tells search engines which pages exist.

llms.txt is trying to solve a different problem:

If an AI agent needs to understand this website quickly, where should it start?

That distinction is important because llms.txt is often described as “robots.txt for AI.”

It is a useful analogy.

But technically, it is wrong.

robots.txt is an access-control convention used by crawlers. llms.txt, by contrast, is an emerging proposal for giving AI agents a concise, curated map of the information that matters most on a website.

The current llms.txt proposal was originally published in 2024 and updated to version 2 in August 2026. The specification describes a Markdown file placed at /llms.txt—or within a subpath such as /docs/llms.txt—that provides a short description of the site and links to important, preferably LLM-friendly resources.

That sounds promising.

But there is a second reality site owners need to understand.

Google explicitly says Google Search does not use llms.txt, and creating one will neither improve nor hurt your Google Search visibility or rankings, including Google’s generative AI search experiences.

OpenAI’s current crawler documentation also focuses on robots.txt controls for OAI-SearchBot and GPTBot rather than documenting llms.txt as a requirement for ChatGPT Search.

So the right question is not:

Do I need llms.txt to rank in AI search?

A better question is:

Could llms.txt make my site easier for compatible AI agents and tools to navigate and understand?

For some sites—especially documentation-heavy websites—the answer may be yes.

This guide explains what llms.txt actually does, how to implement it correctly, where it fits beside robots.txt and sitemap.xml, and when maintaining it is worth the effort.

What Is llms.txt?

llms.txt is a proposed standard for providing AI agents with a concise, Markdown-based overview of a website or section of a website.

The file typically lives at:

https://example.com/llms.txt

A documentation section could also have its own file:

https://example.com/docs/llms.txt

The current specification says the file should contain a site or project name, an optional short description, optional explanatory information, and grouped links to useful resources.

A simple example might look like this:

# Example Analytics

Example Analytics is a SaaS platform for ecommerce reporting and attribution.

This file points AI agents toward the most useful product and documentation resources.

## Product

- [Product Overview](https://example.com/product/)

- [Pricing](https://example.com/pricing/)

- [Integrations](https://example.com/integrations/)

## Documentation

- [Getting Started](https://example.com/docs/getting-started/)

- [API Documentation](https://example.com/docs/api/)

- [Authentication](https://example.com/docs/authentication/)

## Company

- [About](https://example.com/about/)

- [Security](https://example.com/security/)

An AI agent that understands the convention can read this small file first and then follow only the links relevant to the user’s task.

That is the core idea.

Why llms.txt Exists

Modern websites are designed primarily for browsers and humans.

A useful documentation page may be surrounded by navigation, JavaScript, menus, cookie notices, footers, widgets, related articles, ads, and other interface elements.

Humans can usually navigate that environment easily.

AI agents have a different problem.

They may need to extract a relatively small amount of information while staying within a context budget.

The llms.txt proposal is designed to make that process easier by creating a concise entry point.

Instead of forcing an agent to inspect hundreds or thousands of URLs before understanding the site, the publisher can say:

These are the pages that explain our product, API, policies, and documentation most clearly.

Version 2 of the proposal describes agents using llms.txt as a discovery layer: the agent reads or searches the file, identifies the relevant resource, and then follows that link for deeper information.

This makes llms.txt closer to a curated table of contents for AI agents than an AI equivalent of robots.txt.

llms.txt vs robots.txt

The easiest way to understand the difference is to think about the question each file answers.

robots.txt answers:

What may this crawler access?

llms.txt answers:

What information should an agent look at first?

Those are fundamentally different responsibilities.

OpenAI, for example, currently documents robots.txt as the mechanism publishers should use to manage OAI-SearchBot and GPTBot behavior. A site may allow OAI-SearchBot so it can appear in ChatGPT Search while independently blocking GPTBot from training-related crawling.

llms.txt does not override those permissions.

If a URL is blocked by a crawler’s access rules, listing it in llms.txt should not be treated as permission to crawl it.

The files should therefore coexist.

llms.txt vs sitemap.xml

A sitemap answers another different question:

Which URLs does this site want search engines to discover?

A sitemap may contain thousands or millions of pages.

That is useful for crawling.

It is not necessarily useful as an AI context document.

Imagine an ecommerce site with:

120,000 products
2,000 collections
8,000 blog posts

A sitemap can list them.

An llms.txt file should probably not.

Instead, llms.txt might point an agent toward:

• product documentation
• category explanations
• policies
• API docs
• company information
• selected guides
• important reference pages

The llms.txt specification explicitly positions the file as complementary to sitemaps rather than a replacement for them.

robots.txt vs sitemap.xml vs llms.txt

The files solve different problems.

A mature website may eventually use all three.

The GUIDE Framework for llms.txt Implementation

A practical way to implement llms.txt is the GUIDE Framework.

G — Give the Site a Clear Identity
Explain what the website, company, or project actually is.

U — Use Only High-Value Resources
Curate rather than dump every URL.

I — Include LLM-Friendly Destinations
Prefer pages or Markdown resources that are easy to extract and understand.

D — Divide Resources by Intent
Group product, documentation, company, policy, and other resources logically.

E — Evaluate and Maintain
Keep the file synchronized as important URLs and information change.

This is not a ranking framework.

It is an implementation model for making the file genuinely useful.

01Decide Whether You Actually Need llms.txt

Not every site needs one.

A five-page local-business website with a clear homepage, services page, contact page, and About page may gain very little from maintaining an additional AI-specific navigation file.

A technical documentation site with thousands of API pages has a much stronger use case.

Good candidates include:

software documentation sites, because agents frequently need exact API and implementation information;

developer platforms, because coding agents may need to find specific references quickly;

large SaaS websites, where product, pricing, security, integration, and documentation information is distributed across many sections;

and knowledge-heavy organizations, where important reference material is difficult to discover from navigation alone.

The value increases when the site’s information architecture is deep and agents would benefit from a curated entry point.

02Place the File in the Correct Location

The most common location is:

/llms.txt

which would resolve to:

https://example.com/llms.txt

Version 2 of the proposal also allows subpath-specific files.

For example:

/docs/llms.txt

can describe resources under:

/docs/

The specification says that when more than one file applies, the most specific applicable file should be used.

This is useful for large websites.

A company could maintain:

/llms.txt

/docs/llms.txt

/developers/llms.txt

with each file focused on a different information environment.

03Start With a Clear H1

The only required part of the current format is an H1 identifying the project or website.

For example:

# SEOBrain

or:

# Acme Developer Documentation

Keep it simple.

This is not the place for:

# The World’s Most Revolutionary Next-Generation AI-Powered Solution

Use the real entity or project name.

04Add a Concise Summary

The next useful element is a blockquote explaining what the project or website is.

For example:

SEOBrain is an AI-powered SEO and AEO platform that helps businesses research, create, optimize, and publish search-focused content.

The description should establish context.

It should help an agent understand what the rest of the links relate to.

Think of it as the answer to:

What is this site?

not:

What marketing slogan do we want to promote?

05Add Important Context Before the Links

The format allows explanatory paragraphs before the main link sections.

This can be useful when the agent needs interpretive guidance.

For example:

Product capabilities and feature availability should be verified against the official product and documentation pages below.

Pricing information should be taken only from the official pricing page.

Developer implementation guidance is located in the Documentation section.

This is particularly useful for businesses whose website contains older blog posts, announcements, or historical information that should not be treated as current product documentation.

But keep this section short.

If your llms.txt file becomes a 15,000-word company handbook, it has stopped serving its purpose.

06Organize Links Into Useful Sections

The specification uses H2 headings to divide link groups.

For a SaaS company, the file might look like:

## Product

– [Product Overview](https://example.com/product/)

– [Features](https://example.com/features/)

– [Pricing](https://example.com/pricing/)

## Documentation

– [Getting Started](https://example.com/docs/getting-started/)

– [API Reference](https://example.com/docs/api/)

– [Authentication](https://example.com/docs/authentication/)

## Integrations

– [WordPress Integration](https://example.com/integrations/wordpress/)

– [Webflow Integration](https://example.com/integrations/webflow/)

## Company

– [About](https://example.com/about/)

– [Security](https://example.com/security/)

The exact sections should reflect the site.

A university, ecommerce store, personal portfolio, and API platform should not use the same template.

07Add Descriptions to Important Links

A useful feature of the format is the ability to add a short note after a link.

For example:

– [API Reference](https://example.com/docs/api/): Complete endpoint and parameter reference.

or:

– [Pricing](https://example.com/pricing/): Current official plans and pricing.

That annotation helps the agent decide whether it needs to fetch the page.

This is especially valuable when several resources have similar titles.

08Curate Instead of Listing Everything

One of the easiest ways to ruin an llms.txt file is to turn it into a sitemap written in Markdown.

Suppose your site contains 5,000 blog posts.

Do not automatically insert all 5,000.

The purpose is to reduce search cost.

A better approach might link to:

AI Search Guide

Technical SEO Hub

Product Documentation

API Reference

Research Library

Then the agent can navigate deeper only when necessary.

The file should represent your site’s information hierarchy, not its entire URL inventory.

09Consider Markdown Versions of Important Pages

The v2 llms.txt proposal goes beyond the root file.

It also recommends providing clean Markdown versions of pages where practical.

The proposal allows patterns such as:

page.html.md

or:

page.md

and introduces standard link relationships that can tell agents where the Markdown version lives.

For example, an HTML page might expose:

<link

rel=”alternate”

type=”text/markdown”

href=”/docs/api.md”

This gives an agent the option to retrieve a cleaner representation of the same information.

That can be particularly useful for technical documentation where page chrome adds little value.

10Use rel=”describedby” for Discoverability

One of the main changes in version 2 is improved discoverability.

The proposal recommends using:

<link rel=”describedby” href=”/llms.txt”>

to indicate which llms.txt file describes a page.

A documentation page could therefore include:

<link

rel=”describedby”

href=”/docs/llms.txt”

The same information can also be provided through an HTTP Link response header.

This matters because otherwise an agent has to guess whether an llms.txt file exists.

The v2 proposal attempts to solve that discoverability problem explicitly.

11Keep robots.txt Separate

Do not use llms.txt to express access rules.

If you want to control whether a crawler can access content, use the mechanism documented by that crawler.

For OpenAI, for example, OAI-SearchBot and GPTBot can be controlled independently through robots.txt.

A site might choose:

User-agent: OAI-SearchBot

Allow: /

User-agent: GPTBot

Disallow: /

That configuration expresses:

The site may appear in ChatGPT Search, but the publisher does not want GPTBot crawling the site for model-training purposes.

llms.txt does not replace that control.

12Do Not Use llms.txt as a Ranking Hack

This is probably the most important SEO section of this guide.

As of 2026, Google explicitly says llms.txt is not needed for Google Search, including its generative AI features.

Google says the file does not positively or negatively affect Search visibility or rankings and that Google Search ignores it.

That means you should not expect:

Add llms.txt

↓

AI Overview rankings increase

or:

Add llms.txt

↓

Gemini starts citing the site

There is no Google basis for either claim.

Google recommends continuing to focus on traditional search fundamentals and high-quality, non-commodity content rather than AI-specific text-file hacks.

So if your technical SEO is broken, fix that before implementing llms.txt.

Does ChatGPT Use llms.txt?

OpenAI’s current public crawler documentation describes OAI-SearchBot as the crawler used to surface websites in ChatGPT Search and GPTBot as a crawler associated with improving generative AI models.

The documented publisher controls are based on robots.txt.

OpenAI does not currently document llms.txt as a requirement for appearing in ChatGPT Search.

That distinction matters.

It is possible for tools, agents, integrations, or future systems to make use of the format without it being a formal ChatGPT Search ranking signal.

So the safe statement is:

llms.txt can make a site easier for compatible agents to navigate, but it is not a documented ChatGPT Search ranking requirement.

Does Google Gemini Use llms.txt?

Google’s answer is clearer.

No special AI text file is needed for Google Search or its generative AI experiences.

Google specifically names llms.txt in its current AI optimization guidance and says Google Search ignores it.

So if your reason for implementing it is:

I want higher rankings in Gemini-powered Google Search

that is not supported by Google’s documentation.

Use llms.txt only if it serves another agent, documentation, or interoperability purpose.

Does llms.txt Control Model Training?

No.

This is another reason the “new robots.txt” description can be misleading.

If you want to communicate crawler permissions, use the controls documented by the crawler or platform.

For OpenAI, GPTBot behavior is managed through robots.txt.

The llms.txt proposal describes itself primarily as an information-discovery mechanism for agents rather than a training-consent protocol.

Do not write:

AI companies may not train on this content.

inside llms.txt and assume that creates a crawler policy.

That is not what the specification is designed for.

A Good llms.txt Architecture for a SaaS Website

Imagine a SaaS company with:

• marketing pages
• documentation
• integrations
• API references
• security documentation
• research
• blog content

A practical root file could look like:

# Example SaaS

Example SaaS is a platform for automating technical SEO workflows.

Use the product and documentation resources below for current capability information.

## Product

– [Product Overview](https://example.com/product/): Official overview of the platform.

– [Pricing](https://example.com/pricing/): Current plans and pricing.

– [Integrations](https://example.com/integrations/): Supported integrations.

## Documentation

– [Getting Started](https://example.com/docs/getting-started/): Initial setup guide.

– [API](https://example.com/docs/api/): API documentation.

– [Authentication](https://example.com/docs/authentication/): Authentication methods.

## Trust and Security

– [Security](https://example.com/security/): Security architecture and policies.

– [Privacy](https://example.com/privacy/): Official privacy policy.

## Company

– [About](https://example.com/about/): Company information.

Then the documentation section could maintain its own:

/docs/llms.txt

with deeper links.

That keeps the root file concise.

What Should Not Go in llms.txt?

You usually do not need to include every low-value resource.

For example, an agent probably does not need direct links to:

• every tag archive
• every author archive
• every thin category page
• every press release
• every historic product announcement
• every pagination URL

unless those resources are genuinely important to understanding the site.

Think of the file as editorial curation.

Ask:

If an intelligent assistant had only one minute to understand our website, which pages would I give it?

Those are strong candidates.

Keep Current Facts on Canonical Pages

Do not use llms.txt as a second product database.

Suppose your pricing changes every month.

Avoid manually writing:

Pro Plan costs $79/month.

inside llms.txt while also maintaining a pricing page.

Now you have two places that can become inconsistent.

A better pattern is:

– [Pricing](https://example.com/pricing/): Current official pricing and plan information.

Let the canonical pricing page remain the source of truth.

The llms.txt file should point to authoritative facts rather than duplicating every fact.

Automate llms.txt Where the Site Changes Frequently

Manual maintenance works for small sites.

Large documentation environments may be better served by generation.

The current llms.txt ecosystem includes automatic generation in several documentation and CMS platforms. The specification’s project site currently lists integrations including Mintlify, GitBook, Yoast SEO, AIOSEO, Wix, and various framework plugins.

Automation becomes particularly useful when:

New documentation page

↓

Documentation index updates

↓

llms.txt regenerated

↓

Agent navigation stays current

But generation still needs editorial rules.

Automatically dumping every URL recreates the exact problem the file is supposed to solve.

WordPress llms.txt Implementation

A WordPress site can implement llms.txt in several ways.

For a small site, the simplest option is a static text file at the site root.

For a frequently changing content site, generating it dynamically may be more practical.

The current llms.txt project lists WordPress SEO plugins such as Yoast SEO and AIOSEO among tools that can generate or maintain the file.

Regardless of implementation, the editorial question remains the same:

Which pages should an AI agent treat as the authoritative entry points?

A plugin can generate syntax.

It cannot automatically determine your information strategy perfectly.

Shopify llms.txt Implementation

For Shopify, the implementation challenge is different because merchants have less direct control over server-root files than on a self-hosted website.

If an app, theme, proxy route, or platform capability exposes /llms.txt, the content should focus on high-value resources rather than every product URL.

For a store, useful sections might include:

Store information

Major collections

Buying guides

Shipping

Returns

Product-support documentation

Company information

A 100,000-SKU store should generally not turn llms.txt into a 100,000-link product feed.

Product discovery already has better mechanisms.

Documentation Sites Are the Strongest Use Case

The most compelling current use case for llms.txt is technical documentation.

Imagine a coding agent trying to answer:

How do I authenticate with Example API?

Without a curated guide, it may inspect:

• homepage
• marketing site
• blog
• old changelog
• community posts
• random documentation pages

With a good llms.txt:

Agent

↓

/docs/llms.txt

↓

Authentication

↓

Official auth guide

That reduces unnecessary retrieval.

The proposal’s own documentation says the format has seen particularly strong use in software documentation and coding-agent workflows.

llms.txt and AI Search SEO Are Not the Same Thing

This distinction is worth keeping clear.

AI search SEO is about making content discoverable, relevant, credible, extractable, and useful to AI search systems.

llms.txt is one optional technical layer that some compatible agents may use.

A strong AI-search strategy still needs:

crawlability, good information architecture, clear entities, high-quality content, original evidence, accurate sourcing, and authority.

Google explicitly recommends focusing on those traditional and people-first fundamentals instead of special AI text files.

So think of the stack as:

Technical SEO

↓

High-quality content

↓

Strong entities and authority

↓

AI-search optimization

↓

Optional agent-friendly layers

↓

llms.txt

Do not reverse that hierarchy.

Common llms.txt Mistakes

Treating It as a robots.txt Replacement

It is not.

Use robots.txt for crawler access policies.

Use llms.txt to help compatible agents discover important information.

Expecting a Google Ranking Boost

Google explicitly says llms.txt does not improve or hurt Google Search visibility.

Implement it for interoperability, not as a Google ranking tactic.

Listing Every URL

That turns the file into an inferior sitemap.

Curate.

Duplicating Volatile Facts

Point agents to the canonical pricing, product, policy, or documentation page rather than manually maintaining the same facts in multiple locations.

Linking to Outdated Documentation

A curated file containing stale references can be worse than no file at all.

Maintain it.

Assuming Every AI Platform Uses It

llms.txt is an evolving proposal, not a universal AI-web standard.

Platform behavior should be verified individually.

How to Validate Your llms.txt File

After publishing, open:

https://yourdomain.com/llms.txt

in a browser.

It should return successfully as readable text.

Then inspect:

• Markdown formatting
• URLs
• redirects
• broken links
• outdated pages
• duplicate resources

The file should remain useful without requiring special client-side JavaScript.

For larger deployments, validation can be incorporated into CI/CD.

For example:

Site build

↓

Generate llms.txt

↓

Validate syntax

↓

Check URLs

↓

Deploy

That prevents documentation drift.

How Often Should llms.txt Be Updated?

Update it when your information architecture changes.

That might happen when:

• a documentation section moves
• an API version changes
• a major product is launched
• a pricing URL changes
• an integration is retired
• authoritative pages are replaced

You do not need to change the file simply to make it look fresh.

The file’s value comes from accuracy.

Not timestamps.

A Practical llms.txt Checklist

Before publishing the file, answer a few questions.

Does it clearly identify the website?

Does the summary explain what the organization or project actually does?

Are the listed pages authoritative and current?

Are the most important links grouped logically?

Does the file remain reasonably concise?

Are volatile facts kept on canonical pages rather than copied into the file?

And most importantly:

If an AI agent followed these links, would it gain a more accurate understanding of the site?

If not, revise the file.

Frequently Asked Questions

Where SEOBrain Fits

llms.txt is a good example of why AI-search optimization needs a system rather than a collection of hacks.

A business may need to manage:

robots.txt

sitemap.xml

llms.txt

structured data

brand entities

author entities

AI search visibility

content freshness

internal linking

But those components should all reflect the same underlying website and content strategy.

For an SEO automation platform such as SEOBrain, the useful role is not simply:

Generate an llms.txt file.

The stronger opportunity is to keep AI-facing information synchronized with the rest of the SEO system.

For example:

Website inventory

↓

Identify authoritative pages

↓

Entity + content classification

↓

Generate/update llms.txt

↓

Validate links

↓

Monitor website changes

↓

Regenerate when needed

That turns llms.txt from a manually maintained experiment into part of a controlled technical workflow.

More importantly, the same system should continue focusing on the fundamentals that matter regardless of whether a particular AI platform adopts the proposal:

good content, clear entities, crawlability, factual accuracy, strong information architecture, and measurable search visibility.

Final Takeaway

llms.txt is worth understanding.

But it is not the new robots.txt.

robots.txt tells crawlers what access is permitted.

sitemap.xml tells search engines which URLs exist.

llms.txt attempts to tell compatible AI agents:

Here is the most useful path through this website.

That can be genuinely valuable—especially for documentation-heavy websites, developer platforms, and information-rich SaaS products.

The format has also evolved. Version 2 of the proposal now supports scoped files, Markdown page alternatives, and explicit discoverability through standard link relationships.

But expectations need to remain realistic.

Google explicitly ignores llms.txt for Search visibility.

OpenAI currently documents robots.txt, not llms.txt, as the publisher control for its search and training crawlers.

So do not deploy the file because someone promises an AI ranking boost.

Deploy it because you want to make your site’s authoritative information easier for compatible agents to navigate.

And if you do implement it, treat it like any other technical interface:

keep it concise, authoritative, current, and connected to the real source of truth.