Docs AI readiness

AI crawlers allowed in robots.txt

Checks that your robots.txt does not block GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended from the whole site.

AI readinessWeight 3In reports as ai-robots

What it checks

We read your robots.txt and check six crawler names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. For each, we find the group of rules that names it, or the User-agent: * group if none does.

A crawler counts as blocked when that group has Disallow: / and no Allow: /. Rules that block only some folders are not counted. A site with no robots.txt passes.

Why it matters

AI tools such as ChatGPT, Claude and Perplexity read pages through these crawlers. If you block them, they cannot read your pages and are less able to mention or cite you in their answers.

Not all of them do the same job. Some collect pages for training models, others fetch pages for search results or when a user asks. You may decide to block some on purpose; this check flags any of the six, so a fail there may be your choice rather than a mistake.

How to fix it

Remove the whole-site block for the crawlers you want to reach you. Check where the rule came from first: it may have been added by a plugin, a CDN setting or a template rather than by you.

Platform notes

Cloudflare

Cloudflare can manage your robots.txt and add rules for AI crawlers. Check that setting if the rules are not in your own file.

Squarespace

Squarespace has a site setting to block known AI crawlers, which adds rules to robots.txt. Turn it off if you want to be readable by them.

WordPress

Some SEO and security plugins add AI crawler blocks. Check their settings and the robots.txt editor.

Shopify

Rules come from the robots.txt.liquid template, if your theme has one.

Webflow and Wix

Edit the rules in the robots.txt editor in the SEO settings.

Your report shows what this check found on your site. Pro shows the exact fix for your site. See pricing.

Want to test just this? Use the free AI crawler checker.

Questions

Can I allow AI search crawlers but block training crawlers?

Yes. The AI companies publish separate crawler names for training and for search, and robots.txt can have rules for each one. This check flags any of the six it tests when blocked, so a deliberate block will still show as a fail.

See what this check finds on your site

Free, about 30 seconds, with every other check included.