AI Bot User Agents List 2026 — Complete Reference

Quick-copy bot list

Lowercase strings for case-insensitive substring matching. Use this array directly in your middleware. See the full table below for details on each bot.

// AI bot user agents — 2026 reference list
// Use case-insensitive substring matching (see detection section below)

const AI_BOTS = [
  // OpenAI
  'gptbot',           // Training crawler
  'chatgpt-user',     // ChatGPT browsing feature
  'oai-searchbot',    // SearchGPT indexing

  // Anthropic
  'claudebot',        // Training crawler
  'anthropic-ai',     // General Anthropic crawl
  'claude-web',       // Claude web tool

  // Google
  'google-extended',  // Gemini/Vertex AI training (NOT regular Googlebot)

  // Common Crawl (used by most LLMs)
  'ccbot',

  // Other major AI companies
  'bytespider',       // ByteDance / TikTok
  'applebot-extended',// Apple Intelligence training
  'perplexitybot',    // Perplexity AI search
  'diffbot',          // Knowledge graph
  'cohere-ai',        // Cohere
  'facebookbot',      // Meta AI / Llama
  'amazonbot',        // Amazon / Alexa AI

  // Data aggregators
  'omgili',           // Webz.io (also omgilibot)
  'omgilibot',

  // AI search engines
  'iaskspider',       // iAsk.ai
  'youbot',           // You.com

  // Image dataset builders
  'img2dataset',
];

Full bot reference table

robots.txt compliance is self-reported by operators. "Variable" means documented cases of non-compliance or inconsistent behaviour across deployments.

How bot detection works

Always use case-insensitive substring matching on the User-Agent header. Never use exact equality — user agent strings include version numbers, platform tokens, and additional metadata that change between versions.

JavaScript / TypeScript

const AI_BOTS = ['gptbot', 'claudebot', 'anthropic-ai', 'google-extended',
  'ccbot', 'bytespider', 'applebot-extended', 'perplexitybot',
  'diffbot', 'cohere-ai', 'facebookbot', 'amazonbot', 'omgili',
  'omgilibot', 'iaskspider', 'youbot', 'img2dataset',
  'chatgpt-user', 'oai-searchbot', 'claude-web'];

function isAIBot(userAgent: string): boolean {
  const ua = userAgent.toLowerCase();
  return AI_BOTS.some(bot => ua.includes(bot));
}

Python

AI_BOTS = [
    'gptbot', 'chatgpt-user', 'oai-searchbot',
    'claudebot', 'anthropic-ai', 'claude-web',
    'google-extended', 'ccbot', 'bytespider',
    'applebot-extended', 'perplexitybot', 'diffbot',
    'cohere-ai', 'facebookbot', 'amazonbot',
    'omgili', 'omgilibot', 'iaskspider', 'youbot', 'img2dataset',
]

def is_ai_bot(user_agent: str) -> bool:
    ua = user_agent.lower()
    return any(bot in ua for bot in AI_BOTS)

PHP

$AI_BOTS = [
    'gptbot', 'chatgpt-user', 'oai-searchbot',
    'claudebot', 'anthropic-ai', 'claude-web',
    'google-extended', 'ccbot', 'bytespider',
    'applebot-extended', 'perplexitybot', 'diffbot',
    'cohere-ai', 'facebookbot', 'amazonbot',
    'omgili', 'omgilibot', 'iaskspider', 'youbot', 'img2dataset',
];

function isAIBot(string $userAgent): bool {
    $ua = strtolower($userAgent);
    foreach ($AI_BOTS as $bot) {
        if (str_contains($ua, $bot)) return true;
    }
    return false;
}

Go

var aiBots = []string{
    "gptbot", "chatgpt-user", "oai-searchbot",
    "claudebot", "anthropic-ai", "claude-web",
    "google-extended", "ccbot", "bytespider",
    "applebot-extended", "perplexitybot", "diffbot",
    "cohere-ai", "facebookbot", "amazonbot",
    "omgili", "omgilibot", "iaskspider", "youbot", "img2dataset",
}

func isAIBot(userAgent string) bool {
    ua := strings.ToLower(userAgent)
    for _, bot := range aiBots {
        if strings.Contains(ua, bot) {
            return true
        }
    }
    return false
}

robots.txt directives for all bots

Each AI bot that respects robots.txt has its own User-agent token. List them individually to be explicit. A single User-agent: * disallow would also block Googlebot — don't do that.

# robots.txt — AI training crawler block

User-agent: *
Allow: /

# OpenAI
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
Disallow: /

# Anthropic
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
Disallow: /

# Google AI (NOT regular Googlebot)
User-agent: Google-Extended
Disallow: /

# Common Crawl
User-agent: CCBot
Disallow: /

# ByteDance
User-agent: Bytespider
Disallow: /

# Apple AI
User-agent: Applebot-Extended
Disallow: /

# AI search engines
User-agent: PerplexityBot
User-agent: iaskspider
User-agent: YouBot
Disallow: /

# Data aggregators
User-agent: Diffbot
User-agent: cohere-ai
User-agent: FacebookBot
User-agent: Amazonbot
User-agent: omgili
User-agent: omgilibot
Disallow: /

How often does this list change?

Major new AI bot user agents appear 2–4 times per year, typically when a large AI company launches a new product or changes their crawling strategy. The core list has been stable since 2023.

Block these bots on your platform

Now that you have the bot list, implement the block in your specific framework or platform:

Don't see your platform? Browse all guides →

FAQ