Can AI bots read your site?
Enter a page to see whether robots.txt lets 22 AI and search bots in, which line decides, whether a firewall challenge answers instead, and whether the content is in the server HTML.
Free tool
What it checks
robots.txt, per bot
AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot), user-requested fetches (ChatGPT-User, Claude-User) and training crawlers (GPTBot, ClaudeBot, Google-Extended), each with the rule that applies.
Page-level rules
Meta robots and X-Robots-Tag directives, including ones addressed to one bot by name, and a firewall or bot challenge answering instead of the page.
Content in the HTML
Whether the main text is in the server HTML that AI crawlers read, or only appears after JavaScript runs.
Questions
How do I know if GPTBot or ClaudeBot can crawl my site?
Read the site's robots.txt the way the bot does: find the group that names the bot (or the * group when none does), then the longest matching Allow or Disallow rule for the page's path. This checker does exactly that for every AI bot in our open list and shows the line that decides.
If I block GPTBot, does my site disappear from ChatGPT?
No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks about it. You can block training and still allow search, which is the most common setup.
Why does the checker warn about a firewall?
When a bot-protection service (Cloudflare, DataDome, Imperva…) answers with a challenge instead of the page, AI crawlers can't solve it either, so they may be blocked even when robots.txt allows them. We can only see what our own crawler, optmzBot, was served.
Why does it matter whether the content is in the server HTML?
AI crawlers fetch the HTML and don't run JavaScript. A page that renders its text in the browser looks empty to them, whatever robots.txt says.