AI crawler access checker

One robots.txt line can remove your site from an entire AI engine. This checker tests your robots.txt, meta tags and HTTP behavior against every documented AI bot, by name.

Check your siteHow scoring works

12 checks · 5 critical gates · 15% of the Slurp Score

Why it matters for AI search

AI companies run three kinds of bots, and each needs its own decision. Search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot) build the indexes that answers come from. Block one and you disappear from that engine. Training bots (GPTBot, ClaudeBot, Google-Extended) only feed model training, so blocking them is a rights decision that costs you no visibility. Live-fetch bots (ChatGPT-User, Claude-User, Perplexity-User) open pages when a user pastes your URL into a chat.

Most robots.txt files were written before these bots existed, and many block the wrong tier. Some sites block the search bots and leave the training bots open, the exact opposite of what they wanted. GetSlurp tests each bot by name, flags noindex and snippet rules that quietly pull you out of AI Overviews, and reports training-bot blocks without penalty.

What GetSlurp checks

This list comes straight from the audit engine, so it always matches what the extension and MCP server actually run.

crawler.oai-searchbot-allowedGate

OAI-SearchBot builds the index behind ChatGPT Search; blocking it removes this site from ChatGPT Search answers entirely.

Fix: Remove the robots.txt rule that disallows OAI-SearchBot (or add "User-agent: OAI-SearchBot\nAllow: /").

crawler.perplexitybot-allowedGate

PerplexityBot builds the index behind Perplexity; blocking it removes this site from Perplexity answers entirely.

Fix: Remove the robots.txt rule that disallows PerplexityBot (or add "User-agent: PerplexityBot\nAllow: /").

crawler.claude-searchbot-allowedGate

Claude-SearchBot builds the index behind Claude web search; blocking it removes this site from Claude web search answers entirely.

Fix: Remove the robots.txt rule that disallows Claude-SearchBot (or add "User-agent: Claude-SearchBot\nAllow: /").

crawler.googlebot-allowedGate

Googlebot builds the index behind Google AI Overviews and AI Mode; blocking it removes this site from Google AI Overviews and AI Mode answers entirely.

Fix: Remove the robots.txt rule that disallows Googlebot (or add "User-agent: Googlebot\nAllow: /").

crawler.meta-noindexGate

A noindex directive removes the page from search indexes, which is where AI answer engines retrieve from.

Fix: Remove the noindex value from the robots meta tag or X-Robots-Tag header on pages that should appear in AI answers.

crawler.snippet-controlsGraded

nosnippet and max-snippet:0 are the directives Google actually honors for AI Overviews; they silently remove the page from AI answers while normal rankings look fine.

Fix: Remove nosnippet / max-snippet:0 from the robots meta tag, or set max-snippet to -1 or a generous value.

crawler.live-fetch-botsGraded

ChatGPT-User, Claude-User and Perplexity-User fetch pages when a user directly asks about a URL; blocking them breaks those on-demand answers.

Fix: Allow the user-triggered fetch agents (ChatGPT-User, Claude-User, Perplexity-User) in robots.txt.

crawler.robots-txt-validGraded

A robots.txt that errors or blocks everything takes the whole site out of every crawler-fed index.

Fix: Serve a valid robots.txt with HTTP 200 (an allow-all file is fine) and make sure it does not contain "Disallow: /" for all agents.

crawler.sitemap-presentGraded

Sitemaps are how search-index crawlers (including OAI-SearchBot and PerplexityBot) discover pages beyond the ones already linked.

Fix: Publish a sitemap.xml and declare it in robots.txt with a "Sitemap:" line.

crawler.http-cleanGraded

Crawlers give up on error statuses and long redirect chains; every hop also dilutes which URL gets cited.

Fix: Serve the page with HTTP 200 and at most one redirect hop from the canonical URL.

crawler.training-botsInformational

Blocking training bots (GPTBot, ClaudeBot, Google-Extended...) is a content-rights choice; it does not remove the site from AI answers, so GetSlurp reports it without scoring it.

Fix: No action needed. Review whether the current allow/block set matches your content policy.

crawler.search-bot-coverageInformational

Summary of which answer-engine index crawlers can currently reach this site.

Fix: No action needed.

Try GetSlurp free

Audit any page with 60+ checks across 12 categories. No sign-up, runs on your machine.

Get the extension

Related checkers

All 12 categories