robots.txt for AI crawlers: who to allow and who to block

In short

Each AI company runs several crawlers with different jobs. Here are the names to know and how to decide which ones to let in.

Your robots.txt file decides which crawlers may read your site. AI companies run several crawlers each, with different jobs, so one rule does not fit all.

The names to know

  • OpenAI: GPTBot (model training), OAI-SearchBot (search results in ChatGPT), ChatGPT-User (fetches a page when a person asks).
  • Anthropic: ClaudeBot (model training), Claude-SearchBot (search), Claude-User (fetches a page when a person asks).
  • Perplexity: PerplexityBot (search index), Perplexity-User (fetches a page when a person asks).
  • Google: Google-Extended is a robots.txt token that controls use of your content for Google’s AI models. It does not change how you rank in Google Search.

How to decide

Split the question in two. Do you want to appear in AI answers? Then allow the search and user-request crawlers. Do you want your content used to train future models? That is a separate choice, made with the training crawlers.

A common mistake

Blocking everything with “AI” in the name. That keeps you out of training data, and also out of the answers people see today.

Check yours

Open yoursite.com/robots.txt and read it. A crawler with no rule is allowed by default. Also check your firewall or CDN: some block AI crawlers before robots.txt is ever read.

Tarek Riman
Tarek Riman is the founder of Riman Agency and AEOMCP. He works on SEO, AEO and AI search, and teaches digital marketing and AI at Canadian universities. About

All guides

Can AI read your site? Check it for free.

Enter your domain. In a few seconds you see what AI crawlers can read, and what to fix first.