Free tool · AI crawler checker

Check if AI bots can crawl your site

Enter your website (or paste your robots.txt) and a page path, and this checker shows which of 12 AI bots from OpenAI, Anthropic, Perplexity, Google, Apple, Meta and Common Crawl are allowed or blocked, and which rule decides it. It splits training bots from AI search bots, so you can see whether you are blocking reuse or blocking visibility.

Check your site's robots.txt

We fetch your public robots.txt file and check it here. Nothing is stored.

Or paste it yourself: open https://yoursite.com/robots.txt, copy everything and paste it below. Use this if the check above cannot reach your site.

Fills in automatically when you check your site above. Rules are checked in your browser.
A path such as /blog/my-post or a full URL. Paths are case sensitive.

Result for /

AI search bots allowed
–
User-triggered fetchers allowed
–
Training and dataset bots allowed
–
User-agent groups found

Paste a robots.txt file to see which AI bots it lets in.

File checks

    Robots.txt result for each AI crawler
    Bot (robots.txt token)TypeResultWhy

    Matching follows the Robots Exclusion Protocol (RFC 9309): a bot uses the group(s) that name it, or the * group if none do; the longest matching rule wins; on a tie, Allow wins; * matches any characters and $ marks the end. Google applies the same group and longest-path logic (Google robots.txt spec, updated 23 Oct 2026). User-agent tokens and purposes checked on each vendor's documentation, 7 Nov 2026.

    Three kinds of AI bot

    Training bots vs search bots vs user-triggered fetchers

    AI companies now run separate bots for separate jobs. Training bots collect pages that may be used to build future models. Search bots index pages so AI search can show and link them. User-triggered fetchers open a page only when a person asks the assistant about it. You can block one kind and allow another.

    TokenCompanyJob, in the vendor's words (summarised)Type
    GPTBotOpenAICrawls content that may be used to train OpenAI's foundation models.Training
    OAI-SearchBotOpenAISurfaces websites in ChatGPT's search features.Search
    ChatGPT-UserOpenAIVisits a page when a ChatGPT user asks. OpenAI says robots.txt rules may not apply to these user-initiated visits.User-triggered
    ClaudeBotAnthropicCollects web content that could contribute to model training.Training
    Claude-SearchBotAnthropicNavigates the web to improve search result quality for users.Search
    Claude-UserAnthropicFetches pages when people ask Claude questions. Anthropic says blocking it stops Claude retrieving your content for those questions.User-triggered
    PerplexityBotPerplexitySurfaces and links websites in Perplexity search. Perplexity says it is not used to crawl content for foundation models.Search
    Perplexity-UserPerplexityVisits a page to help answer a user's question. Perplexity says it generally ignores robots.txt.User-triggered
    Google-ExtendedGoogleNot a separate crawler. A token that controls use of Google-crawled content for Gemini training and grounding. Google says it does not affect inclusion or ranking in Google Search.Training control
    Applebot-ExtendedAppleDoes not crawl. Controls whether Applebot's data is used to train Apple's foundation models. Pages that block it can still appear in Apple search results.Training control
    CCBotCommon CrawlBuilds Common Crawl's free, public web archive. It checks robots.txt before fetching.Dataset
    Meta-ExternalAgentMetaCrawls the web for uses such as training AI models and indexing content.Training

    Sources, all checked 7 Nov 2026: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google common crawlers (updated 4 Nov 2026), About Applebot, Common Crawl FAQ, Meta web crawlers. We left out ByteDance's Bytespider because we could not find an official ByteDance page that documents it.

    What we set up for clients

    Which AI bots should you allow?

    If you want to be found and cited in AI answers, allow the search bots and the user-triggered fetchers. Blocking training bots is a separate choice about reuse; the vendors say it does not remove you from their search. For a B2B company that wants leads from AI search, our default advice is to allow all three.

    User-agent: GPTBot
    User-agent: ClaudeBot
    User-agent: Google-Extended
    User-agent: Applebot-Extended
    User-agent: CCBot
    User-agent: Meta-ExternalAgent
    Disallow: /
    
    User-agent: *
    Disallow: /admin/

    The block above is for teams that decide to opt out of training but stay visible in AI search. Several User-agent lines in a row share one set of rules. Every other bot, including the search bots, falls through to the * group.

    • The specific-group trap. Once a bot has its own group, it ignores the * group completely. If * blocks /admin/ and you add a GPTBot group that only blocks /drafts/, GPTBot may now crawl /admin/. Repeat any shared rules inside each named group. Load the second example above to see it.
    • The staging leftover. User-agent: * plus Disallow: / copied from a staging site blocks every bot that follows robots.txt, Google included. It is a common mistake after a site relaunch.
    • robots.txt is not the only gate. A firewall, CDN bot setting or login wall can block a bot that robots.txt allows. If the tool says "Allowed" but you never appear in AI answers, check your server logs for the bot's visits. Our free AI visibility snapshot checks what AI answers say about you.
    • Changes are not instant. OpenAI says it can take about 24 hours for its systems to pick up a robots.txt change for search.

    Common questions

    AI Crawler Checker: FAQs

    Open yoursite.com/robots.txt in your browser, copy everything, and paste it into the checker above with the page you care about. It shows each AI bot as allowed or blocked and the exact rule behind it. If your site has no robots.txt at all, every bot is allowed.
    Not according to OpenAI. GPTBot is its training crawler. ChatGPT search uses OAI-SearchBot, which you control separately. OpenAI says disallowing GPTBot signals your content should not be used to train its foundation models, and that changes for search can take about 24 hours to apply.
    The crawlers listed here are documented by their vendors as following robots.txt. User-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page. A firewall or login is the only hard block.
    Google-Extended is a robots.txt token, not a separate crawler. Disallowing it tells Google not to use content it crawls from your site to train future Gemini models or for grounding in Gemini apps and Vertex AI. Google says it does not affect your inclusion or ranking in Google Search.
    Yes. Enter your website and we fetch its public robots.txt file, then check the rules in your browser. We request only that one file and store nothing. If your site blocks automated requests, open the file yourself and paste it in.

    Related

    Related tools and guides

    AI search visibility

    Bots allowed, but still not cited?

    Access is only the first gate. We find out why AI answers skip you and fix the pages and signals they rely on.

    We reply within one business day.

    We use cookies to understand how visitors use this site. Analytics cookies stay off unless you say yes — see our Cookie Policy for details.

    Cookie Preferences

    Strictly Necessary
    Required for the site to function. Always on, cannot be disabled.
    Analytics
    Helps us understand how visitors use the site (Google Analytics). Off by default.