Skip to content
optmz
Start free

Every AI crawler, and what it's for

22 AI crawlers and user agents: who runs each, whether it trains models, powers AI search or fetches pages for a user, and its robots.txt token. Updated November 2026.

Open data

Download JSONDownload CSVrobots.txt generator

AI bots do three different jobs. AI search crawlers index pages so assistants can cite them; user-requested fetchers read a page when someone asks about it; training crawlers collect content for future models. You can allow search and block training. Check what a page allows today.

AI search

Index pages so AI assistants can find and cite them. Block them and the site drops out of those answers.

  • OAI-SearchBot

    OpenAI

    Finds and indexes pages for ChatGPT search results and citations.

    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

    robots.txt: User-agent: OAI-SearchBotOperator docsIP ranges

  • Claude-SearchBot

    Anthropic

    Indexes pages to improve Claude's search results.

    robots.txt: User-agent: Claude-SearchBotOperator docsIP ranges

  • PerplexityBot

    Perplexity

    Indexes pages for Perplexity answers. Perplexity says it is not used to train foundation models.

    Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

    robots.txt: User-agent: PerplexityBotOperator docsIP ranges

  • Amzn-SearchBot

    Amazon

    Indexes pages for search experiences in Amazon products such as Alexa and Rufus.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36

    robots.txt: User-agent: Amzn-SearchBotOperator docs

  • DuckAssistBot

    DuckDuckGo

    Fetches pages to source DuckDuckGo's AI-assisted answers (DuckAssist). DuckDuckGo says the data isn't used to train AI models.

    DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

    robots.txt: User-agent: DuckAssistBotOperator docs

  • MistralAI-Index

    Mistral AI

    Indexes pages for Mistral's search, which helps answer user questions.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)

    robots.txt: User-agent: MistralAI-IndexOperator docs

User-requested fetches

Fetch a page live when a person asks an assistant about it.

  • ChatGPT-User

    OpenAImay not follow robots.txt

    Fetches a page when a ChatGPT user asks about it. OpenAI says robots.txt rules may not apply to these user-initiated visits.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

    robots.txt: User-agent: ChatGPT-UserOperator docsIP ranges

  • Claude-User

    Anthropic

    Fetches a page when a Claude user asks about it.

    robots.txt: User-agent: Claude-UserOperator docsIP ranges

  • Perplexity-User

    Perplexitymay not follow robots.txt

    Fetches a page a Perplexity user asked about. Perplexity says it generally ignores robots.txt for these requests.

    Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

    robots.txt: User-agent: Perplexity-UserOperator docsIP ranges

  • Meta-ExternalFetcher

    Metamay not follow robots.txt

    Fetches a page a person asked a Meta AI product about. Meta says that because a person requested the fetch, it may not follow robots.txt.

    meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

    robots.txt: User-agent: Meta-ExternalFetcherOperator docs

  • Amzn-User

    Amazon

    Fetches a page live when an Amazon assistant user asks for current information.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36

    robots.txt: User-agent: Amzn-UserOperator docs

  • MistralAI-User

    Mistral AI

    Fetches a page when a Mistral assistant user asks about it.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)

    robots.txt: User-agent: MistralAI-UserOperator docs

AI training

Collect content for model training. Blocking them doesn't remove the site from AI search.

  • GPTBot

    OpenAI

    Crawls content that may be used to train OpenAI models. Blocking it does not remove the site from ChatGPT search.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

    robots.txt: User-agent: GPTBotOperator docsIP ranges

  • ClaudeBot

    Anthropic

    Crawls content that may be used to train Anthropic models.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

    robots.txt: User-agent: ClaudeBotOperator docsIP ranges

  • Google-Extended

    Googlerobots.txt token only

    A robots.txt token, not a crawler: it controls whether Gemini models may be trained on the site and use it to ground answers. Blocking it removes the site from Gemini app and Vertex AI grounding, but not from Google Search, AI Overviews or AI Mode, and doesn't affect ranking.

    robots.txt: User-agent: Google-ExtendedOperator docs

  • Applebot-Extended

    Applerobots.txt token only

    A robots.txt token, not a crawler: blocking it stops Apple from using pages Applebot crawled to train its foundation models. It doesn't remove the site from Siri or Spotlight.

    robots.txt: User-agent: Applebot-ExtendedOperator docs

  • Meta-ExternalAgent

    Meta

    Crawls the web for use cases such as training AI models or improving products by indexing content directly.

    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

    robots.txt: User-agent: Meta-ExternalAgentOperator docs

  • CCBot

    Common Crawl

    Builds Common Crawl's open web archive, a common source of AI training data.

    CCBot/2.0 (https://commoncrawl.org/faq/)

    robots.txt: User-agent: CCBotOperator docsIP ranges

  • Amazonbot

    Amazon

    Crawls to improve Amazon's products and services; Amazon says the content may be used to train its AI models.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

    robots.txt: User-agent: AmazonbotOperator docs

Search engine

Classic search crawlers. Google AI Overviews and AI Mode, and Copilot's web answers, build on their indexes.

  • Googlebot

    Google

    Google Search's crawler. Pages must be crawlable by it to appear in Google Search, AI Overviews and AI Mode.

    Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

    robots.txt: User-agent: GooglebotOperator docsIP ranges

  • Bingbot

    Microsoft

    Bing's crawler. Pages must be crawlable by it to appear in Bing and in Microsoft Copilot answers that cite the web.

    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

    robots.txt: User-agent: BingbotOperator docsIP ranges

  • Applebot

    Apple

    Apple's crawler for Siri, Spotlight and Safari search suggestions. Apple says its data may also be used for Apple Intelligence unless Applebot-Extended is blocked.

    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

    robots.txt: User-agent: ApplebotOperator docsIP ranges

robots.txt generator for AI bots

Pick a preset, adjust single bots, and paste the rules above your existing robots.txt.

Start from
Block these bots

robots.txt rules

# AI crawler rules, generated with optmz (2026-11-05)
# Paste above your existing rules. Bots not listed here follow your User-agent: * group.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Amazonbot
Disallow: /

Only blocked bots get a group. A bot's own group replaces your User-agent: * rules for it, so writing Allow: / for allowed bots would reopen paths you block for everyone, like /admin. Bots you allow keep following your existing rules. Some user-requested fetchers may not follow robots.txt at all.

Questions

Which AI bots should I allow?

To appear in AI answers, allow the AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user-requested fetchers (ChatGPT-User, Claude-User, Perplexity-User), and keep Googlebot and Bingbot allowed. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) are a business choice: blocking them doesn't remove the site from ChatGPT search, Google Search or AI Overviews.

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended is a robots.txt token, not a crawler. It controls Gemini model training and grounding in the Gemini app and Vertex AI. AI Overviews and AI Mode use Googlebot's index.

How do I verify that a visitor really is GPTBot?

A user agent can be faked. Check the request's IP address against the operator's published IP list (linked for each bot below), or do a reverse DNS lookup where the operator documents one.

Can I use this list in my own tools?

Yes. /ai-bots.json and /ai-bots.csv are free to use under CC BY 4.0 with a link back to this page. The list is checked against each operator's documentation and dated.

See what AI says about your clients.

Start free