The short answer
OpenAI, Anthropic and Perplexity each run separate crawlers for search, for model training and for pages a user asks about. You can allow search and block training. Blocking GPTBot or ClaudeBot opts you out of training; it does not remove you from ChatGPT or Claude search. Blocking OAI-SearchBot or Claude-SearchBot does.
Three jobs, three kinds of bot
Most AI companies now split their crawlers by purpose:
- Search bots index pages so the assistant can find and cite them when a customer asks a question.
- Training bots collect pages that may be used to train future models.
- User-fetch agents visit a page because a person asked the assistant about it, or pasted its link.
These are separate decisions. Many businesses want to be found in AI search but would rather not contribute to training. The controls allow that, as long as you name the right bots.
The bots, company by company
| Company | User agent | Purpose | Follows robots.txt? |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Yes |
| OpenAI | GPTBot | Content that may be used to train models | Yes |
| OpenAI | ChatGPT-User | Visits pages for user actions in ChatGPT | OpenAI says robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | Indexes pages to improve Claude's search results | Yes |
| Anthropic | ClaudeBot | Collects content for model training | Yes (also honours Crawl-delay) |
| Anthropic | Claude-User | Fetches pages when a user asks Claude a question | Yes |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity search; not used for training | Yes |
| Perplexity | Perplexity-User | Visits pages to answer a user's question | Generally ignores robots.txt, per Perplexity |
Google-Extended | A token controlling Gemini training and grounding; not a separate crawler | Yes; does not affect Google Search |
Sources: OpenAI's crawler overview, Anthropic's crawler article (updated 7 April 2026), Perplexity's crawler documentation and Google's common crawlers.
Example: allow AI search, opt out of training
This robots.txt lets the search and user-fetch agents in and asks the training crawlers to stay out. Adapt the paths to your site. If you already have a User-agent: * group, keep it for every other crawler, including Googlebot and Bingbot. But a bot with its own named group ignores the * group completely, so repeat any path you block for everyone (an admin area, say) inside the named group too.
# AI search and user requests: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /account/
Disallow: /checkout/
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /We checked this file with two independent robots.txt parsers on 26 September 2026: each search and user-fetch agent above may fetch product pages but not /account/ or /checkout/, and GPTBot, ClaudeBot and Google-Extended are kept out of the whole site. Googlebot and Bingbot aren't named, so they follow your * group or, without one, may crawl everything. Other training crawlers, such as Common Crawl's CCBot, need their own line if you want them out. Test the file on your own deployment before you rely on it, and check the live file at yourdomain.com/robots.txt after you publish.
robots.txt is only half of it
Many sites allow the right bots in robots.txt and still block them further down the stack:
- CDN and bot protection. Services such as Cloudflare can challenge or block AI crawlers by default. Check the dashboard, not just the file.
- Firewalls. If you allowlist bots, match both the user agent and the company's published IP ranges. OpenAI and Perplexity publish theirs; a user-agent string on its own is easy to fake.
- Page-level tags. A
noindexornosnippetin the HTML or theX-Robots-Tagheader can keep a page out even when the crawler is allowed. - Server logs. The honest test is whether the bot actually got a 200 response. Search your logs for the user agents above.
Seeing ChatGPT visitors in your analytics
OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to links in search answers. Filter your analytics for it to see visits that came from ChatGPT answers. It counts clicks, not how often you were mentioned without a click.
Check yours
Scope's free check reads your robots.txt and reports search access for ChatGPT, Claude, Perplexity, Google and Bing separately from training preferences. Opting out of training is shown as information, not as a problem, because it doesn't affect AI search.
What this can't tell you
- Crawler names and behaviour change. The table reflects each company's documentation on 25 September 2026.
- Allowing a search crawler makes a page eligible to be found. It does not mean an assistant will cite or recommend it.
- Whether a model already trained on your content before you opted out is outside what robots.txt controls.
Sources
Primary sources first. Research papers are cited with their limits. Every source was re-opened on the date shown.
- 1Overview of OpenAI crawlers
OpenAI · checked 26 September 2026
OAI-SearchBot surfaces sites in ChatGPT search; GPTBot collects content that may be used for training; ChatGPT-User acts on user requests and robots.txt rules may not apply to it.
- 2Publishers and developers FAQ
OpenAI Help Center · checked 25 September 2026
Public sites can appear in ChatGPT search if OAI-SearchBot is not blocked; ChatGPT adds utm_source=chatgpt.com to referral links.
- 3Does Anthropic crawl data from the web, and how can site owners block the crawler?
Anthropic (Claude Help Center) · 7 April 2026 · checked 25 September 2026
ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (user-directed retrieval) are separate; Anthropic honours robots.txt and Crawl-delay.
- 4Perplexity crawlers
Perplexity · checked 25 September 2026
PerplexityBot surfaces and links sites in Perplexity search and is not used for foundation-model training; Perplexity-User fetches pages for user questions and generally ignores robots.txt; published IP ranges for both.
- 5Google's common crawlers (Google-Extended)
Google Search Central · checked 25 September 2026
Google-Extended controls use for Gemini model training and grounding; it does not affect inclusion or ranking in Google Search.
- 6Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
Google Search Central · checked 25 September 2026
How noindex, nosnippet, data-nosnippet and max-snippet work, including in AI features.
Get the next guide by email
Short emails on how ChatGPT, Gemini and Claude recommend businesses: the new piece, one real answer we saw, and one fix you can make. Unsubscribe any time.