robots.txt for AI crawlers: who to allow and who to block
In short
Each AI company runs several crawlers with different jobs. Here are the names to know and how to decide which ones to let in.
Your robots.txt file decides which crawlers may read your site. AI companies run several crawlers each, with different jobs, so one rule does not fit all.
The names to know
- OpenAI: GPTBot (model training), OAI-SearchBot (search results in ChatGPT), ChatGPT-User (fetches a page when a person asks).
- Anthropic: ClaudeBot (model training), Claude-SearchBot (search), Claude-User (fetches a page when a person asks).
- Perplexity: PerplexityBot (search index), Perplexity-User (fetches a page when a person asks).
- Google: Google-Extended is a robots.txt token that controls use of your content for Google’s AI models. It does not change how you rank in Google Search.
How to decide
Split the question in two. Do you want to appear in AI answers? Then allow the search and user-request crawlers. Do you want your content used to train future models? That is a separate choice, made with the training crawlers.
A common mistake
Blocking everything with “AI” in the name. That keeps you out of training data, and also out of the answers people see today.
Check yours
Open yoursite.com/robots.txt and read it. A crawler with no rule is allowed by default. Also check your firewall or CDN: some block AI crawlers before robots.txt is ever read.