Check if AI bots can crawl your site
Enter your website (or paste your robots.txt) and a page path, and this checker shows which of 12 AI bots from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl are allowed or blocked, and which rule decides it. It splits training bots from AI search bots, so you can see whether you are blocking reuse or blocking visibility.
| Bot (robots.txt token) | Type | Result | Why |
|---|
Matching follows the Robots Exclusion Protocol (RFC 9309): a bot uses the group(s) that name it, or the * group if none do; the longest matching rule wins; on a tie, Allow wins; * matches any characters and $ marks the end. Google applies the same group and longest-path logic (Google robots.txt spec, updated 23 Oct 2026). User-agent tokens and purposes checked on each vendor's documentation, 7 Nov 2026.
Three kinds of AI bot
Training bots vs search bots vs user-triggered fetchers
AI companies now run separate bots for separate jobs. Training bots collect pages that may be used to build future models. Search bots index pages so AI search can show and link them. User-triggered fetchers open a page only when a person asks the assistant about it. You can block one kind and allow another.
| Token | Company | Job, in the vendor's words (summarised) | Type |
|---|---|---|---|
GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | Training |
OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. | Search |
ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply to these user-initiated visits. | User-triggered |
ClaudeBot | Anthropic | Collects web content that could contribute to model training. | Training |
Claude-SearchBot | Anthropic | Navigates the web to improve search result quality for users. | Search |
Claude-User | Anthropic | Fetches pages when people ask Claude questions. Anthropic says blocking it stops Claude retrieving your content for those questions. | User-triggered |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search. Perplexity says it is not used to crawl content for foundation models. | Search |
Perplexity-User | Perplexity | Visits a page to help answer a user's question. Perplexity says it generally ignores robots.txt. | User-triggered |
Google-Extended | Not a separate crawler. A token that controls use of Google-crawled content for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search. | Training control | |
Applebot-Extended | Apple | Does not crawl. Controls whether Applebot's data is used to train Apple's foundation models. Pages that block it can still appear in Apple search results. | Training control |
CCBot | Common Crawl | Builds Common Crawl's free, public web archive. It checks robots.txt before fetching. | Dataset |
Meta-ExternalAgent | Meta | Crawls the web for uses such as training AI models and indexing content. | Training |
Sources, all checked 7 Nov 2026: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google common crawlers (updated 4 Nov 2026), About Applebot, Common Crawl FAQ, Meta web crawlers. We left out ByteDance's Bytespider because we could not find an official ByteDance page that documents it.
What we set up for clients
Which AI bots should you allow?
If you want to be found and cited in AI answers, allow the search bots and the user-triggered fetchers. Blocking training bots is a separate choice about reuse; the vendors say it does not remove you from their search. For a B2B company that wants leads from AI search, our default advice is to allow all three.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent Disallow: / User-agent: * Disallow: /admin/
The block above is for teams that decide to opt out of training but stay visible in AI search. Several User-agent lines in a row share one set of rules. Every other bot, including the search bots, falls through to the * group.
- The specific-group trap. Once a bot has its own group, it ignores the
*group completely. If*blocks/admin/and you add a GPTBot group that only blocks/drafts/, GPTBot may now crawl/admin/. Repeat any shared rules inside each named group. Load the second example above to see it. - The staging leftover.
User-agent: *plusDisallow: /copied from a staging site blocks every bot that follows robots.txt, Google included. It is a common mistake after a site relaunch. - robots.txt is not the only gate. A firewall, CDN bot setting or login wall can block a bot that robots.txt allows. If the tool says "Allowed" but you never appear in AI answers, check your server logs for the bot's visits. Our free AI visibility snapshot checks what AI answers say about you.
- Changes are not instant. OpenAI says it can take about 24 hours for its systems to pick up a robots.txt change for search.
Common questions
AI Crawler Checker: FAQs
Related