The main AI crawlers
- GPTBot (OpenAI): training data.
- OAI-SearchBot (OpenAI): ChatGPT search index.
- ChatGPT-User (OpenAI): fetches pages when a user asks.
- ClaudeBot (Anthropic): training data. Claude-SearchBot: search results.
- PerplexityBot: Perplexity’s answer index.
- Google-Extended: controls Gemini and AI training use. It does not affect Google Search.
- Applebot-Extended: Apple Intelligence training.
- CCBot: Common Crawl, an open dataset used by many models.
- Meta-ExternalAgent: Meta AI.
Allow AI search, block training (example)
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
This keeps your content out of training sets while ChatGPT search, Perplexity and Google AI Overviews can still read and cite you.
Check your own site
Run free auditSee how your site scores on this in about two minutes.
Common accidental blocks
- Security plugins or CDN “bot fight” modes that block unknown user agents.
- A copied robots.txt with
User-agent: * Disallow: /left over from a staging site. - Firewall rules that block data-centre IP ranges.
robots.txt is only part of the picture. If your firewall returns errors to AI crawlers, they still cannot read you. Check your CDN logs.
Frequently asked questions
Will blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search, which uses OAI-SearchBot and ChatGPT-User. It stops your content being used for future model training.
Do AI crawlers respect robots.txt?
The major companies state that their crawlers follow robots.txt. Some smaller scrapers do not.