AI crawler access checker

AI crawlers need permission before they can read your page. One robots.txt line can remove your site from an entire AI search engine. This checker tests your rules against every documented AI bot, by name.

Get the extension How the score works

12 checks · 5 critical gates · 15% of the Slurp Score · Updated July 2026

Why AI crawler access matters for AI search

AI companies run three kinds of bots, and each one needs a different decision. Search bots build the indexes that answers come from. Block OAI-SearchBot, PerplexityBot, Claude-SearchBot or Googlebot and you disappear from that engine. Training bots like GPTBot only feed model training, so blocking them costs you no visibility. Live-fetch bots open your page when someone pastes the URL into a chat.

Most robots.txt files were written before these bots existed. Many block the wrong tier by accident. Some sites block the search bots and leave the training bots wide open, which is the opposite of what they wanted. GetSlurp tests each bot by name and tells you which engine every rule affects.

Studies and sources: OpenAI bot documentation · Perplexity crawler documentation

What GetSlurp checks

Search bot access Whether OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot can reach the page.
Robots.txt rules Valid syntax, no accidental site-wide block, sitemap declared.
Noindex and snippet tags Meta rules that quietly pull you out of AI Overviews.
Live-fetch agents ChatGPT-User, Claude-User and Perplexity-User, which open URLs on request.
Training bot status Reported for your records, never counted against your score.

Every check in this category

This list comes straight from the audit engine, so it always matches what the extension and MCP server actually run.

crawler.oai-searchbot-allowedGate

OAI-SearchBot builds the index behind ChatGPT Search; blocking it removes this site from ChatGPT Search answers entirely.

Fix: Remove the robots.txt rule that disallows OAI-SearchBot (or add "User-agent: OAI-SearchBot\nAllow: /").

crawler.perplexitybot-allowedGate

PerplexityBot builds the index behind Perplexity; blocking it removes this site from Perplexity answers entirely.

Fix: Remove the robots.txt rule that disallows PerplexityBot (or add "User-agent: PerplexityBot\nAllow: /").

crawler.claude-searchbot-allowedGate

Claude-SearchBot builds the index behind Claude web search; blocking it removes this site from Claude web search answers entirely.

Fix: Remove the robots.txt rule that disallows Claude-SearchBot (or add "User-agent: Claude-SearchBot\nAllow: /").

crawler.googlebot-allowedGate

Googlebot builds the index behind Google AI Overviews and AI Mode; blocking it removes this site from Google AI Overviews and AI Mode answers entirely.

Fix: Remove the robots.txt rule that disallows Googlebot (or add "User-agent: Googlebot\nAllow: /").

crawler.meta-noindexGate

A noindex directive removes the page from search indexes, which is where AI search engines retrieve from.

Fix: Remove the noindex value from the robots meta tag or X-Robots-Tag header on pages that should appear in AI answers.

crawler.snippet-controlsGraded

nosnippet and max-snippet:0 are the directives Google actually honors for AI Overviews; they silently remove the page from AI answers while normal rankings look fine.

Fix: Remove nosnippet / max-snippet:0 from the robots meta tag, or set max-snippet to -1 or a generous value.

crawler.live-fetch-botsGraded

ChatGPT-User, Claude-User and Perplexity-User fetch pages when a user directly asks about a URL; blocking them breaks those on-demand answers.

Fix: Allow the user-triggered fetch agents (ChatGPT-User, Claude-User, Perplexity-User) in robots.txt.

crawler.robots-txt-validGraded

A robots.txt that errors or blocks everything takes the whole site out of every crawler-fed index.

Fix: Serve a valid robots.txt with HTTP 200 (an allow-all file is fine) and make sure it does not contain "Disallow: /" for all agents.

crawler.sitemap-presentGraded

Sitemaps are how search-index crawlers (including OAI-SearchBot and PerplexityBot) discover pages beyond the ones already linked.

Fix: Publish a sitemap.xml and declare it in robots.txt with a "Sitemap:" line.

crawler.http-cleanGraded

Crawlers give up on error statuses and long redirect chains; every hop also dilutes which URL gets cited.

Fix: Serve the page with HTTP 200 and at most one redirect hop from the canonical URL.

crawler.training-botsInformational

Blocking training bots (GPTBot, ClaudeBot, Google-Extended...) is a content-rights choice; it does not remove the site from AI answers, so GetSlurp reports it without scoring it.

Fix: No action needed. Review whether the current allow/block set matches your content policy.

crawler.search-bot-coverageInformational

Summary of which answer-engine index crawlers can currently reach this site.

Fix: No action needed.

AI crawler access checker: frequently asked questions

How do I block AI bots in robots.txt?

Add a User-agent line naming the bot, then a Disallow rule below it. The catch is picking the right tier. Blocking a search bot like OAI-SearchBot or PerplexityBot removes you from that engine's answers. Blocking a training bot like GPTBot only opts you out of model training.

Does blocking GPTBot remove me from ChatGPT search?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses a separate bot called OAI-SearchBot. Block GPTBot and you opt out of training while staying visible in ChatGPT answers. Block OAI-SearchBot and you vanish from ChatGPT search.

Which AI crawlers should I allow?

Allow the search bots if you want AI visibility: OAI-SearchBot for ChatGPT, plus PerplexityBot, Claude-SearchBot and Googlebot. Training bots like GPTBot, ClaudeBot and Google-Extended are a separate choice with no visibility cost either way.

How do I check if my site blocks AI crawlers?

Run the page through GetSlurp. It tests your robots.txt and meta tags against every documented AI bot by name, tells you which engine each rule affects, and gives you the exact line to change.

Check this before you publish, not after

GetSlurp runs 60+ checks across 12 categories on any page, with a ranked fix for every issue it finds. No sign-up, and it runs on your machine.

Get the extension

Related checkers

All 12 categories