Free AI crawler access checker
Can ChatGPT, Claude and Perplexity read your site? Enter your domain to check your robots.txt rules and whether your firewall turns their crawlers away. No login.
AI crawler questions
What does this checker test?
Two things. First, your /robots.txt: whether it blocks the whole site for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot or Google-Extended, either by name or through a User-agent: * group. Second, your firewall: it loads your homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot and fails if any gets a 403, 429, server error or no answer.
Why would my firewall block AI crawlers?
Many CDNs and security plugins have bot settings, such as Cloudflare's Block AI bots or Bot Fight Mode, that turn these crawlers away even when robots.txt allows them. The site works for people, so nobody notices.
Is the firewall result exact?
Close, not exact. The check sends the same user agent the real crawlers send, but from Cloudflare's network, not from OpenAI, Anthropic or Perplexity addresses. A firewall that checks crawler IP addresses can answer the real crawlers differently.
Should I allow every AI crawler?
Only the ones you want. Allowing search crawlers like OAI-SearchBot and PerplexityBot lets those tools find and cite your pages. Blocking training crawlers like GPTBot or Google-Extended is a fair choice if you do not want your content used to train models.
Does Google-Extended affect Google Search?
No. Google says Google-Extended only controls whether your content is used for its Gemini models. Blocking it does not change how your site shows in Google Search.