ChatGPT, Claude and Perplexity crawl controls: search, training and user fetches

Most AI companies now split their crawlers by purpose:

• Search bots index pages so the assistant can find and cite them when a customer asks a question.
• Training bots collect pages that may be used to train future models.
• User-fetch agents visit a page because a person asked the assistant about it, or pasted its link.

These are separate decisions. Many businesses want to be found in AI search but would rather not contribute to training. The controls allow that, as long as you name the right bots.

Sources: OpenAI's crawler overview, Anthropic's crawler article (updated 7 April 2026), Perplexity's crawler documentation and Google's common crawlers.

This robots.txt lets the search and user-fetch agents in and asks the training crawlers to stay out. Adapt the paths to your site. If you already have a User-agent: * group, keep it for every other crawler, including Googlebot and Bingbot. But a bot with its own named group ignores the * group completely, so repeat any path you block for everyone (an admin area, say) inside the named group too.

# AI search and user requests: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /account/
Disallow: /checkout/

# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

We checked this file with two independent robots.txt parsers on 26 September 2026: each search and user-fetch agent above may fetch product pages but not /account/ or /checkout/, and GPTBot, ClaudeBot and Google-Extended are kept out of the whole site. Googlebot and Bingbot aren't named, so they follow your * group or, without one, may crawl everything. Other training crawlers, such as Common Crawl's CCBot, need their own line if you want them out. Test the file on your own deployment before you rely on it, and check the live file at yourdomain.com/robots.txt after you publish.

Many sites allow the right bots in robots.txt and still block them further down the stack:

• CDN and bot protection. Services such as Cloudflare can challenge or block AI crawlers by default. Check the dashboard, not just the file.
• Firewalls. If you allowlist bots, match both the user agent and the company's published IP ranges. OpenAI and Perplexity publish theirs; a user-agent string on its own is easy to fake.
• Page-level tags. A noindex or nosnippet in the HTML or the X-Robots-Tag header can keep a page out even when the crawler is allowed.
• Server logs. The honest test is whether the bot actually got a 200 response. Search your logs for the user agents above.

OpenAI's publisher FAQ says ChatGPT adds utm_source=chatgpt.com to links in search answers. Filter your analytics for it to see visits that came from ChatGPT answers. It counts clicks, not how often you were mentioned without a click.

Scope's free check reads your robots.txt and reports search access for ChatGPT, Claude, Perplexity, Google and Bing separately from training preferences. Opting out of training is shown as information, not as a problem, because it doesn't affect AI search.