Make your site readable by AI

Guides for your web team. Let AI crawlers in and give them pages they can read.

1. AI crawlers and robots.txt: which to allowTo appear in AI answers, let the crawlers that read pages for answers into your site. For OpenAI that is OAI-SearchBot and ChatGPT-User, for Anthropic Claude-SearchBot and Claude-User, for Perplexity PerplexityBot and Perplexity-User. Keeping training crawlers out is a separate choice.3 min read
2. Make your pages readable before any script runsPut your page’s words in the HTML your server sends. A crawl study by Vercel and MERJ found that none of the major AI crawlers ran JavaScript, OpenAI’s and Anthropic’s included. Words that appear only after a script runs can be invisible to them.2 min read
3. llms.txt and plain-text copies of your pagesllms.txt is a proposed file that lists a website’s pages in plain text for AI tools. A plain-text copy is a page’s words in markdown with the layout taken out. Both are cheap to add on top of pages AI crawlers can already read.2 min read