Robots.txt for AI Crawlers: GPTBot, Claude & Perplexity Configuration

For years, robots.txt was mainly an SEO infrastructure file.

You used it to tell search-engine crawlers which parts of your website they could or could not crawl.

AI search has made that decision more complicated.

A modern website may now receive requests from several different types of AI crawler:

one may crawl content for model development, another may build a search index, and another may fetch a page because a user explicitly asked an AI assistant to retrieve it.

Those uses are not interchangeable.

For example, OpenAI currently separates GPTBot, which may crawl content used to improve its generative AI models, from OAI-SearchBot, which is used to surface websites in ChatGPT Search. A site can block one while allowing the other.

Anthropic follows a similar model. Its current documentation lists ClaudeBot for potential model-training data, Claude-SearchBot for search indexing and search quality, and Claude-User for user-directed retrieval.

Perplexity operates PerplexityBot as its search crawler and says that crawler follows explicit robots.txt limits.

So the question is no longer simply:

Should we block AI bots?

A better question is:

Which AI use cases do we want to allow?

What robots.txt Actually Controls

A robots.txt file sits at the root of a domain:

https://example.com/robots.txt

It contains rules associated with crawler user-agent names.

A simple rule might look like:

User-agent: ExampleBot
Disallow: /

That tells a crawler identifying itself as ExampleBot that it should not crawl the site.

An empty Disallow means crawling is allowed:

User-agent: ExampleBot
Disallow:

The important point is that robots.txt controls crawler access behavior.

It does not directly mean:

never mention this website anywhere.

For some AI systems, search indexing, user-triggered fetching, training crawls, and navigational links are handled separately.

That is why bot-specific configuration matters.

OpenAI: GPTBot vs OAI-SearchBot

OpenAI currently documents several different agents.

The two most important for website owners are GPTBot and OAI-SearchBot.

GPTBot

GPTBot is associated with crawling content that may be used to improve OpenAI’s generative AI foundation models.

OpenAI says disallowing GPTBot indicates that future crawled content from your site should not be used for that training purpose.

To block GPTBot:

User-agent: GPTBot
Disallow: /

OAI-SearchBot

OAI-SearchBot serves a different purpose.

OpenAI says it is used to surface websites in ChatGPT’s search features. Sites that opt out of OAI-SearchBot will generally not appear as sources in ChatGPT Search answers, although they may still appear in navigational contexts.

To allow ChatGPT Search crawling:

User-agent: OAI-SearchBot
Disallow:

This means a publisher can make an intentional distinction:

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow:

In plain English:

Do not crawl my site for model-development purposes, but allow it to be discovered for ChatGPT Search.

OpenAI explicitly documents these controls as independent.

What About ChatGPT-User?

OpenAI also documents ChatGPT-User.

This agent may retrieve pages because a user has explicitly asked ChatGPT or a Custom GPT to access something.

That is different from systematic crawling.

OpenAI says ChatGPT-User is not used to determine whether a website appears in ChatGPT Search, and user-initiated requests may not always behave like normal automated crawling under robots.txt.

So if your goal is controlling ChatGPT Search visibility, focus primarily on:

OAI-SearchBot

not ChatGPT-User.

Anthropic: ClaudeBot, Claude-SearchBot, and Claude-User

Anthropic currently documents three distinct robots.

This is important because older AI-crawler lists may use outdated names such as Claude-Web.

Anthropic’s current search crawler is Claude-SearchBot.

ClaudeBot

ClaudeBot is used to collect public web content that may contribute to model development.

To block it:

User-agent: ClaudeBot
Disallow: /

Anthropic says restricting ClaudeBot signals that future material from the site should be excluded from its AI model-training datasets.

Claude-SearchBot

Claude-SearchBot is used for web search.

Anthropic says blocking it prevents its system from indexing your content for search optimization and may reduce your site’s visibility and accuracy in Claude’s search results.

To allow it:

User-agent: Claude-SearchBot
Disallow:

Claude-User

Claude-User handles user-directed web retrieval.

When someone asks Claude a question that requires a specific website, Claude may access the page using this agent.

Anthropic says blocking Claude-User can prevent its system from retrieving the site’s content in response to user queries, which may reduce visibility for user-directed web search.

To allow it:

User-agent: Claude-User
Disallow:

A publisher that wants Claude search visibility without model-development crawling could therefore use:

User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow:
User-agent: Claude-User
Disallow:

This is conceptually similar to separating GPTBot from OAI-SearchBot.

Anthropic Also Supports Crawl-delay

Anthropic says its robots support the non-standard Crawl-delay extension.

For example:

User-agent: ClaudeBot
Crawl-delay: 2

This can be useful if crawler frequency is causing server-load concerns but you do not want to block the crawler completely.

Remember that Crawl-delay is not part of the core robots.txt standard and support varies by crawler.

Use it only where the crawler explicitly documents support.

PerplexityBot

Perplexity operates PerplexityBot as a search crawler.

Perplexity says the crawler complies with explicit restrictions in robots.txt and adjusts crawling behavior based on limits and site availability.

To allow it:

User-agent: PerplexityBot
Disallow:

To block it:

User-agent: PerplexityBot
Disallow: /

Blocking the search crawler can naturally reduce Perplexity’s ability to index and retrieve your content.

That matters if you are actively trying to earn Perplexity citations.

PerplexityBot vs Perplexity-User

Perplexity also distinguishes between automated search crawling and user-driven fetching.

Its current technical materials reference both PerplexityBot and Perplexity-User.

This follows the same broader pattern emerging across AI platforms:

systematic search crawler

≠

training crawler

≠

user-triggered agent

Website owners should therefore avoid treating every request containing an AI user-agent as the same type of traffic.

Recommended Configuration for AI Search Visibility

For many publishers, the practical goal is:

Let AI search engines discover public content, but limit model-training crawlers where separate controls exist.

A configuration could look like this:

# OpenAI model-development crawler
User-agent: GPTBot
Disallow: /
# ChatGPT Search
User-agent: OAI-SearchBot
Disallow:
# Anthropic model-development crawler
User-agent: ClaudeBot
Disallow: /
# Claude Search
User-agent: Claude-SearchBot
Disallow:
# User-directed Claude retrieval
User-agent: Claude-User
Disallow:
# Perplexity Search
User-agent: PerplexityBot
Disallow:

This is only an example.

Your actual configuration should reflect your publishing policy.

Some businesses may allow everything.

Others may block all AI crawling.

Others may distinguish search visibility from training access.

The important thing is to make that choice intentionally.

Allow Public Content, Block Private Sections

You do not necessarily need site-wide rules.

Suppose you want AI search crawlers to access your public articles but not internal account areas.

You could use:

User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /private/
User-agent: Claude-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /private/
User-agent: PerplexityBot
Disallow: /account/
Disallow: /checkout/
Disallow: /private/

Public content remains discoverable.

Low-value or private areas are excluded.

This is usually more useful than automatically blocking the entire site.

Do Not Block CSS or JavaScript Without a Reason

AI crawlers increasingly need to parse modern websites.

Overly aggressive blocking can make content difficult to interpret.

A rule such as:

User-agent: \*
Disallow: /assets/

might unintentionally block resources required to understand the page properly.

As with traditional technical SEO, block content because there is a specific reason—not because the directory happens to look technical.

robots.txt Is Not a Security System

This is essential.

robots.txt is a crawler directive.

It does not protect confidential information.

Anyone can visit:

example.com/robots.txt

and see which directories you listed.

Sensitive information should be protected with actual access controls such as authentication and authorization.

Do not put private documents online and assume:

Disallow: /secret/

makes them secure.

It does not.

Does Blocking GPTBot Remove You From ChatGPT Search?

Not necessarily.

OpenAI separates GPTBot from OAI-SearchBot.

GPTBot is associated with model-development crawling, while OAI-SearchBot is used for ChatGPT Search visibility.

So this is valid:

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow:

Blocking GPTBot alone does not mean you have also opted out of ChatGPT Search.

That separation is one of the most important things publishers should understand.

Does Blocking ClaudeBot Remove You From Claude Search?

Not necessarily.

Anthropic also separates model-development crawling from search.

ClaudeBot is associated with potential model-training data, while Claude-SearchBot serves search indexing and Claude-User supports user-directed retrieval.

So blocking only:

ClaudeBot

is different from blocking:

Claude-SearchBot

and:

Claude-User

Configure them according to the outcome you actually want.

Should SEO Sites Allow AI Search Crawlers?

If AI-search visibility matters to your acquisition strategy, blocking search-specific crawlers can work against that objective.

For example, OpenAI explicitly states that sites opted out of OAI-SearchBot will not normally be surfaced in ChatGPT Search answers.

Anthropic similarly says blocking Claude-SearchBot can reduce a site’s search visibility.

Perplexity says its search crawler follows robots.txt, meaning a block can prevent that crawler from accessing the content.

So an SEO/AEO strategy should include crawler configuration as part of technical visibility.

Publishing AI-friendly content while blocking the bots that need to retrieve it is self-defeating.

A Simple AI Crawler Policy

For most marketing sites, the decision can be simplified into three questions.

Do we want to appear in AI search?

If yes, allow the relevant search crawlers.

Do we want our public content available for model-development crawling?

Configure training-oriented crawlers separately where the provider offers that distinction.

Which areas should no automated crawler access?

Block private, low-value, duplicate, or operational sections as appropriate.

That is usually enough to build a sensible policy.

Common AI robots.txt Mistakes

Blocking Every AI Bot With a Copy-Pasted List

Large bot lists become outdated quickly.

Crawler names change and new agents appear.

Maintain only rules you understand.

Confusing Search With Training

GPTBot is not OAI-SearchBot.

ClaudeBot is not Claude-SearchBot.

Treating them as identical can accidentally remove your site from AI search.

Using Old Crawler Names

Anthropic’s current search crawler is Claude-SearchBot.

Do not rely on outdated configuration articles that still use older names without checking Anthropic’s current documentation.

Assuming robots.txt Creates Security

It does not.

Protect sensitive resources at the server or application level.

Never Reviewing the File

AI crawler policies are evolving rapidly.

Review the file periodically against official provider documentation.

Frequently Asked Questions

Where SEOBrain Fits

AI crawler configuration is a small technical task, but it sits directly underneath AI-search visibility.

A broader SEO/AEO system should know whether the site is technically eligible to be retrieved before spending time optimizing content for ChatGPT, Claude, or Perplexity.

A useful workflow looks like:

AI visibility audit

↓

robots.txt review

↓

Search crawlers allowed?

↓

Pages crawlable?

↓

Content + entity optimization

↓

Citation monitoring

For SEOBrain, this type of technical check can become part of the site’s AI-readiness audit rather than a standalone configuration task.

The important principle is simple:

crawler policy should match business strategy.

If you want AI search visibility, allow the search crawlers you want to serve.

If you do not want model-training crawling, use the separate controls offered by providers that support that distinction.

Do not accidentally block both because their names sound similar.

Final Takeaway

AI crawlers are no longer one category.

OpenAI separates GPTBot from OAI-SearchBot.

Anthropic separates ClaudeBot from Claude-SearchBot and Claude-User.

Perplexity operates PerplexityBot for search crawling.

Those distinctions matter because training, search indexing, and user-directed retrieval are different use cases.

So do not copy a giant “block all AI” robots.txt file without understanding what each rule does.

For sites that want AI-search visibility, the practical strategy is usually:

allow search-specific crawlers, protect sensitive areas, make deliberate decisions about training crawlers, and review the configuration as provider policies evolve.

A two-line robots.txt rule can decide whether an otherwise excellent page is accessible to an AI search engine.

That makes crawler configuration a small technical task with potentially large visibility consequences.