AI crawler checker
AI assistants can only recommend what their crawlers are allowed to read. Enter your domain to see, for each AI crawler, whether your robots.txt lets it in and which exact line decides.
What does the checker test?
It fetches your robots.txt once and applies the standard matching rules to ten AI crawlers, grouped by purpose:
- AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) decide whether assistants can cite your pages in their answers.
- User-triggered fetchers (ChatGPT-User) visit a page when a person asks about it.
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider) decide whether your content may be used to train models. Blocking them is a legitimate choice.
How do you read the result?
Each row says allowed or blocked, which group of rules applied (the crawler's own or the general * group) and the exact line that decided it. If a search crawler is blocked, copy the fixed robots.txt below the table: it lets the chosen crawlers in and keeps your own private paths disallowed.
What can the checker not tell you?
robots.txt states your policy, but it is not a lock. A firewall or bot-protection service can block a crawler even when robots.txt allows it, and some user-triggered fetchers may not follow robots.txt at all. If the checker shows access allowed and assistants still cannot read your site, check those settings. The AI crawlers guide lists what each vendor documents.
See if ChatGPT actually recommends you
These tools check whether AI can read your site. The free AI Visibility Check asks ChatGPT and Perplexity the questions your customers ask and shows who they recommend.
Run the free check →How to use it
- Enter your website’s domain.
- Read the verdict for each AI crawler and the robots.txt line that decides it.
- If search crawlers are blocked, copy the fixed robots.txt and upload it to your site’s root.
Frequently asked questions
What is the difference between GPTBot and OAI-SearchBot?
OAI-SearchBot fetches pages for ChatGPT’s search answers, so blocking it keeps you out of those answers. GPTBot collects data to train OpenAI’s models; blocking it does not remove you from ChatGPT search.
Should I block AI training crawlers?
It’s your choice. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) keeps your content out of future model training, but don’t block the search crawlers if you want AI assistants to recommend you.
My robots.txt allows everything. Why doesn’t ChatGPT mention me?
Access is only the first step. AI assistants recommend businesses they find described clearly and cited by other sites. Run the free AI Visibility Check to see who they recommend instead.