AI Bot Access Checker
Sends real requests with GPTBot, ClaudeBot, PerplexityBot, and 15 more AI crawler User-Agents, see what your server actually returns
Check URL
Test sent to the homepage. 18 parallel requests. Takes 5-15 seconds.
What this tool does
Sends real HTTP requests to your site with the actual User-Agent strings of 18 major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, etc.) and reports what your server actually returns. Detects Cloudflare/WAF blocks, robots.txt rules, redirects, and X-Robots-Tag headers.
Who it's for: Webmasters and SEO pros who need to verify that their intended AI bot policy is actually enforced - not just declared.
When to use
- •Confirm a new robots.txt rule is respected by your stack end-to-end.
- •Check whether Cloudflare's AI bot blocking is actually triggering (HTTP 403).
- •Spot cases where robots.txt says 'blocked' but the server still serves 200.
- •Verify Google-Extended and Applebot-Extended opt-outs at the robots.txt layer.
How it works
- 1Enter your site URL.
- 2We send 18 parallel HTTP requests, each with a different AI bot User-Agent.
- 3robots.txt is fetched and parsed for per-bot rules.
- 4Results compare robots.txt declaration with actual server response.
Tips
- •If a bot shows 'robots-blocks-but-server-serves', the rule is declarative only - upgrade to WAF blocking.
- •HTTP 403 + 'cf-mitigated: block' confirms Cloudflare is enforcing the rule.
- •Retest after changing WAF rules or robots.txt to confirm propagation.