Free AI Crawler Checker
Check if your website blocks AI crawlers like GPTBot, ClaudeBot, and Google-Extended. We'll analyse your robots.txt, meta tags, and headers instantly.
What are AI Crawlers?
AI crawlers are web bots used by companies like OpenAI, Anthropic, and Google to index content for their AI models. When these crawlers are blocked, your brand becomes invisible to AI assistants — they can't cite what they can't read.
Why robots.txt Matters
- Controls which bots can access your site content
- Blocking AI crawlers is the #1 cause of AI invisibility
- Many sites unknowingly block AI bots with wildcard rules
What We Check
- robots.txt rules for 11 AI crawler user agents
- Meta robots tags (noindex, noai, noimageai)
- X-Robots-Tag HTTP headers
Frequently Asked Questions
AI crawlers are the bots AI companies use to fetch web pages — some to train models, some to build the index behind an assistant’s answers, some to open a link the moment a user asks about it. If your robots.txt blocks the answering ones, your content cannot appear in responses from ChatGPT, Claude, Gemini, or Perplexity, no matter how good it is. That is the most common reason a brand is absent from AI answers.
Your robots.txt file controls which bots can access your website. Many sites unknowingly block AI crawlers like GPTBot, ClaudeBot, and Google-Extended, either through specific Disallow rules or a broad wildcard block. This means AI assistants cannot read your content and will never cite or recommend your brand.
Eleven user agents in total. OpenAI: GPTBot (training), OAI-SearchBot (the ChatGPT search index), and ChatGPT-User (fetches triggered by a person). Anthropic: ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (user-directed fetches), plus the legacy anthropic-ai token still found in older files. Google-Extended (Gemini grounding and training), PerplexityBot, Bytespider (ByteDance), and CCBot (Common Crawl). They are controlled independently, so you can allow the ones that cite you and disallow the ones that train on you.
Beyond robots.txt, pages can restrict crawler access through <meta name="robots"> tags in the HTML and X-Robots-Tag HTTP headers. Directives like noindex, nofollow, noai, and noimageai tell crawlers not to index or use your content. Our tool checks all three layers of crawler access control.
Yes. GeoVector's AI crawler checker is completely free with no sign-up required. You can scan any URL and get instant results showing your robots.txt AI crawler status, meta robots tags, and X-Robots-Tag headers. For deeper site-wide audits and ongoing monitoring, GeoVector offers premium plans.
AI crawlers this tool checks
Eleven user agents, grouped by operator. The important thing is that they are independent: allowing an answer engine and blocking a training crawler is a supported configuration, not a contradiction.
| Crawler | Operator | Used for | If you disallow it |
|---|---|---|---|
GPTBot | OpenAI | Crawling content that may be used to train OpenAI foundation models. | Your pages are excluded from that training data. |
OAI-SearchBot | OpenAI | Building the index behind ChatGPT's search answers. | You drop out of ChatGPT search answers, though you can still appear as a navigational link. |
ChatGPT-User | OpenAI | Fetching a page when a ChatGPT user or tool opens a link. | ChatGPT can't open your pages on request during a conversation. |
ClaudeBot | Anthropic | Collecting web content that may contribute to model training. | Future material is signalled as out of scope for training. |
Claude-SearchBot | Anthropic | Indexing pages to improve Claude's search results. | Your content stops being indexed for Claude search. |
Claude-User | Anthropic | Retrieving a page when a Claude user asks about it. | Claude can't pull your content into a user-directed answer. |
anthropic-ai | Anthropic | A legacy token that still appears in older robots.txt files; Anthropic no longer lists it among its active agents. | No effect on the three current agents. Harmless to leave in place. |
Google-Extended | Controlling whether your content is used for Gemini grounding and training. | You are removed from Gemini grounding. Google Search crawling, indexing, and ranking are unaffected. | |
PerplexityBot | Perplexity | Indexing pages so Perplexity can surface and cite them. | Perplexity stops citing your pages. |
Bytespider | ByteDance | Crawling for ByteDance's models and products. | Your content is excluded from ByteDance's collection. |
CCBot | Common Crawl | Building the open Common Crawl corpus that many model builders train on. | You are excluded from a dataset used well beyond any single vendor. |
Three robots.txt patterns
Copy whichever matches your policy. robots.txt lives at the root of each host and each subdomain needs its own.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xml# Answer questions about us
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# But do not train on us
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /This split is what most teams actually want when legal asks for a training opt-out and marketing asks for citations. It only works where the operator separates the two roles: OpenAI, Anthropic, and Google do. PerplexityBot and CCBot do not, so those are a single yes-or-no decision.
User-agent: *
Disallow: /
User-agent: GPTBot
Allow: /Under RFC 9309 a crawler obeys the most specific group that matches its own name and ignores every other group. So GPTBot here reads only its own group and is allowed, while every crawler without a named group — including OAI-SearchBot and Claude-SearchBot — falls into the wildcard and is blocked. A staging robots.txt that survives launch fails exactly this way.
Page-level controls: meta robots and X-Robots-Tag
robots.txt governs whether a page may be fetched. These two govern what a crawler may do once it has fetched it, which is why a page can be perfectly crawlable and still be withheld from results.
<meta name="robots" content="noindex, nofollow">X-Robots-Tag: noindexYou will also see noai and noimageai. They came out of image-hosting platforms and are not part of any specification; none of the crawlers listed above documents support for them. We report them because they record your intent, not because they block anything.
Sources and specifications
Everything on this page is checked against the primary documentation below. If a vendor changes a rule, that is where it shows up first.
- RFC 9309: Robots Exclusion Protocol — IETF
- Overview of OpenAI crawlers — OpenAI
- Does Anthropic crawl data from the web? — Anthropic
- Google crawlers, fetchers, and user agents — Google Search Central
- PerplexityBot — Perplexity
- How Google interprets the robots.txt specification — Google Search Central
Last reviewed . We re-check this page against the source specifications whenever a vendor updates one.