bisicaDocs
FREE CHATGPT CHECK

AI crawlers and robots.txt: which to allow

To appear in AI answers, let the crawlers that read pages for answers into your site. For OpenAI that is OAI-SearchBot and ChatGPT-User, for Anthropic Claude-SearchBot and Claude-User, for Perplexity PerplexityBot and Perplexity-User. Keeping training crawlers out is a separate choice.

Updated · 3 min read

View as markdown Ask ChatGPT Ask Claude
On this page

Each AI company runs more than one crawler. OpenAI’s OAI-SearchBot reads pages for ChatGPT’s search, while its GPTBot collects training data. Your robots.txt file can let one in and keep the other out.

The crawlers

We read each company’s own documentation on October 7, 2026 (Microsoft’s on October 11).

Company Crawler What it does Group
OpenAI OAI-SearchBot Reads pages for ChatGPT’s search Answers
OpenAI ChatGPT-User Fetches a page when a person in ChatGPT asks for it Answers
OpenAI GPTBot Collects training data Training
Anthropic Claude-SearchBot Reads pages for Claude’s search Answers
Anthropic Claude-User Fetches a page when a person in Claude asks for it Answers
Anthropic ClaudeBot Collects training data Training
Perplexity PerplexityBot Reads pages for Perplexity’s search Answers
Perplexity Perplexity-User Fetches a page when a person in Perplexity asks for it Answers
Google Googlebot Reads pages for Google Search, AI Overviews included Answers
Google Google-Extended A robots.txt name with no crawler of its own. It controls whether Google may use your pages to train Gemini models and to ground answers in Gemini apps. Google says it does not affect Google Search Control
Microsoft Bingbot Reads pages for Bing Answers

Sources: OpenAI, Anthropic, Perplexity, Google, Microsoft.

What the companies say about robots.txt

  • OpenAI: “we recommend allowing OAI-SearchBot in your site’s robots.txt file”.
  • Perplexity: “we recommend allowing PerplexityBot in your site’s robots.txt file”.
  • Anthropic says its bots honor robots.txt.
  • OpenAI on ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.”
  • Perplexity on Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

Let the answer crawlers in

If your robots.txt has no rule for these names and no rule for all crawlers (User-agent: *) that blocks them, they are already allowed. To be explicit, add a group like this one:

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

A crawler follows the most specific group that names it. If your file has a group for GPTBot only, OAI-SearchBot still follows User-agent: *. Check that group too.

Keep training crawlers out, if you choose

Blocking a training crawler is your company’s choice. It does not stop the answer crawlers. Weigh Google-Extended first. Google says it also covers grounding in Gemini apps. Blocking it can keep your pages out of Gemini’s answers. To keep all three out:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

Firewalls can block crawlers too

A firewall or a bot rule at your hosting provider can turn a crawler away even when robots.txt lets it in. Cloudflare, for one, has a setting that blocks AI crawlers across a whole site. Ask your web or security team to check those settings against the names above.

Check your site

Can AI read your site reads your robots.txt and shows what each of these crawlers is told. It does not test your firewall. The free snapshot requests your pages under the names of six AI crawlers and reports any refusal it can confirm.

Was this page helpful?

Related articles