Docs AI readiness

AI crawlers not blocked by firewall

Requests your homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and checks your firewall or CDN lets them through.

AI readinessWeight 3In reports as ai-firewall

What it checks

We request your homepage four times, each time with the user agent of one crawler: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Each request waits up to 10 seconds, with one retry if there is no answer.

A crawler counts as blocked when there is no answer, or the answer is 403, 429 or a 5xx error.

The requests come from scorebadge's servers, not from the AI companies' own addresses. A firewall that checks crawlers by their IP address can block this test while still letting the real crawler in. If you have set that up on purpose, a fail here can be expected.

Why it matters

A robots.txt can welcome AI crawlers while the CDN or firewall in front of the site turns them away. The site looks open from the inside, but the crawlers only ever get an error page.

How to fix it

Look at the bot settings of your CDN, firewall and host, and at any custom rules that match on user agent or limit request rates. Allow the AI crawlers you want, and check the firewall's logs to confirm the real crawlers get through.

Platform notes

Cloudflare

Check the bot settings, including the option that blocks AI bots, Bot Fight Mode, and the per-crawler AI controls, as well as any custom WAF rules.

WordPress

Security plugins and host firewalls can block unknown bots or rate-limit them. Check their blocked request logs for the crawler names.

Next.js, Vercel and Netlify

Check any firewall or bot protection rules on the project, including ones you added for scrapers.

Your report shows what this check found on your site. Pro shows the exact fix for your site. See pricing.

Want to test just this? Use the free AI crawler checker.

Questions

My robots.txt allows AI crawlers. Why does this fail?

robots.txt is read by the crawler after it gets through. The firewall or CDN decides first whether the request reaches your site at all, and many have bot blocking settings that are separate from robots.txt.

See what this check finds on your site

Free, about 30 seconds, with every other check included.