Docs AI readiness
AI crawlers not blocked by firewall
Requests your homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and checks your firewall or CDN lets them through.
What it checks
We request your homepage four times, each time with the user agent of one crawler: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Each request waits up to 10 seconds, with one retry if there is no answer.
A crawler counts as blocked when there is no answer, or the answer is 403, 429 or a 5xx error.
The requests come from scorebadge's servers, not from the AI companies' own addresses. A firewall that checks crawlers by their IP address can block this test while still letting the real crawler in. If you have set that up on purpose, a fail here can be expected.
- PassThe homepage loads for all four.
- FailAt least one is blocked.
Why it matters
A robots.txt can welcome AI crawlers while the CDN or firewall in front of the site turns them away. The site looks open from the inside, but the crawlers only ever get an error page.
How to fix it
Look at the bot settings of your CDN, firewall and host, and at any custom rules that match on user agent or limit request rates. Allow the AI crawlers you want, and check the firewall's logs to confirm the real crawlers get through.
Platform notes
Check the bot settings, including the option that blocks AI bots, Bot Fight Mode, and the per-crawler AI controls, as well as any custom WAF rules.
Security plugins and host firewalls can block unknown bots or rate-limit them. Check their blocked request logs for the crawler names.
Check any firewall or bot protection rules on the project, including ones you added for scrapers.
Your report shows what this check found on your site. Pro shows the exact fix for your site. See pricing.
Want to test just this? Use the free AI crawler checker.
Questions
My robots.txt allows AI crawlers. Why does this fail?
robots.txt is read by the crawler after it gets through. The firewall or CDN decides first whether the request reaches your site at all, and many have bot blocking settings that are separate from robots.txt.
Free, about 30 seconds, with every other check included.