> The full list of bisica's docs pages is at https://docs.bisica.com/llms.txt

# AI crawlers and robots.txt: which to allow

To appear in AI answers, let the crawlers that read pages for answers into your site. For OpenAI that is OAI-SearchBot and ChatGPT-User, for Anthropic Claude-SearchBot and Claude-User, for Perplexity PerplexityBot and Perplexity-User. Keeping training crawlers out is a separate choice.

Each AI company runs more than one crawler. OpenAI’s OAI-SearchBot reads pages for ChatGPT’s search, while its GPTBot collects training data. Your robots.txt file can let one in and keep the other out.

## The crawlers

We read each company’s own documentation on October 7, 2026 (Microsoft’s on October 11).

| Company | Crawler | What it does | Group |
| --- | --- | --- | --- |
| OpenAI | `OAI-SearchBot` | Reads pages for ChatGPT’s search | Answers |
| OpenAI | `ChatGPT-User` | Fetches a page when a person in ChatGPT asks for it | Answers |
| OpenAI | `GPTBot` | Collects training data | Training |
| Anthropic | `Claude-SearchBot` | Reads pages for Claude’s search | Answers |
| Anthropic | `Claude-User` | Fetches a page when a person in Claude asks for it | Answers |
| Anthropic | `ClaudeBot` | Collects training data | Training |
| Perplexity | `PerplexityBot` | Reads pages for Perplexity’s search | Answers |
| Perplexity | `Perplexity-User` | Fetches a page when a person in Perplexity asks for it | Answers |
| Google | `Googlebot` | Reads pages for Google Search, AI Overviews included | Answers |
| Google | `Google-Extended` | A robots.txt name with no crawler of its own. It controls whether Google may use your pages to train Gemini models and to ground answers in Gemini apps. Google says it does not affect Google Search | Control |
| Microsoft | `Bingbot` | Reads pages for Bing | Answers |

Sources: [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), [Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), [Microsoft](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0).

## What the companies say about robots.txt

- OpenAI: “we recommend allowing OAI-SearchBot in your site’s robots.txt file”. 
- Perplexity: “we recommend allowing PerplexityBot in your site’s robots.txt file”. 
- Anthropic says its bots honor robots.txt.
- OpenAI on ChatGPT-User: “Because these actions are initiated by a user, robots.txt rules may not apply.” 
- Perplexity on Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” 

## Let the answer crawlers in

If your robots.txt has no rule for these names and no rule for all crawlers (`User-agent: *`) that blocks them, they are already allowed. To be explicit, add a group like this one:

```
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
```

A crawler follows the most specific group that names it. If your file has a group for `GPTBot` only, OAI-SearchBot still follows `User-agent: *`. Check that group too.

## Keep training crawlers out, if you choose

Blocking a training crawler is your company’s choice. It does not stop the answer crawlers. Weigh Google-Extended first. Google says it also covers grounding in Gemini apps. Blocking it can keep your pages out of Gemini’s answers. To keep all three out:

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
```

## Firewalls can block crawlers too

A firewall or a bot rule at your hosting provider can turn a crawler away even when robots.txt lets it in. Cloudflare, for one, has a setting that blocks AI crawlers across a whole site. Ask your web or security team to check those settings against the names above.

## Check your site

[Can AI read your site](https://bisica.com/can-ai-read-your-site/) reads your robots.txt and shows what each of these crawlers is told. It does not test your firewall. The free snapshot requests your pages under the names of six AI crawlers and reports any refusal it can confirm.

---

Source: https://docs.bisica.com/make-your-site-readable/ai-crawlers-and-robots-txt/ · Updated November 2, 2026