AI CRAWLER
ACCESS CHECKER
Instantly check if GPTBot, ClaudeBot, and Google-Extended can access your site. Our AI automatically fetches and parses your robots.txt.
FREQUENTLY ASKED
What is GPTBot and should I block it?
GPTBot is OpenAI's web crawler used to train ChatGPT. If you block GPTBot in your robots.txt file, your content will not be included in future ChatGPT training datasets, which may reduce the likelihood of ChatGPT citing or recommending your brand. For most businesses, allowing GPTBot is beneficial for Generative Engine Optimization (GEO). You should only block it if you have proprietary or sensitive content you do not want in AI training data.
What is Google-Extended and why does it matter?
Google-Extended is a user-agent token introduced by Google specifically for controlling access by Bard and Google's AI products. It is separate from Googlebot (the standard search crawler). Blocking Google-Extended prevents your content from informing Google AI Overviews and Gemini responses, even if your site still ranks normally in Google Search.
What AI crawlers should I allow in my robots.txt?
To maximize AI search visibility, you should allow: GPTBot (OpenAI/ChatGPT), ClaudeBot (Anthropic/Claude), Google-Extended (Google AI/Gemini)Bot (ChatGPT AI), and CCBot (Common Crawl, used by many LLMs for training). Blocking these crawlers creates a significant GEO disadvantage as AI search continues to grow.
How do I add or allow AI crawlers in my robots.txt?
In your robots.txt file (located at yourdomain.com/robots.txt), you can allow specific bots by not disallowing them, or by explicitly allowing them. Example to allow all AI crawlers: User-agent: GPTBot / Allow: / User-agent: ClaudeBot / Allow: / User-agent: Google-Extended / Allow: / If you currently have 'User-agent: * / Disallow: /' make sure specific AI bots are listed above this catch-all rule.