robots.txt Generatorfor AI Bots
Choose which AI crawlers can access your site. Get a copy-paste robots.txt snippet in seconds. Covers 49+ AI bots including GPTBot, ClaudeBot, PerplexityBot, and Bingbot.
Quick presets
Filter by type
AI model training data collection
Real-time web browsing for ChatGPT users
AI-powered search indexing
AI model training data collection
AI search engine indexing and real-time retrieval
AI model training (Gemini, Bard)
Real-time web browsing for Gemini users
AI model training and content indexing
Open web dataset for AI training and research
AI assistant answers and product indexing
AI model training for Apple Intelligence
Link previews and AI model training
AI model training for Llama and Meta AI
Enterprise AI model training
AI search engine indexing
Structured data extraction and knowledge graph building
Search engine indexing for Petal Search
SEO analysis and backlink mapping
Decentralized search indexing
AI-scored search and content evaluation
Content indexing and data analytics
Web indexing and AI training
Forum and discussion content crawling for AI datasets
Multilingual AI research and translation training
Large-scale data collection for AI and analytics
General-purpose web scraping framework
AI model training for Grok — real-time knowledge and news indexing
Real-time page fetching for DuckAssist AI summaries
Search indexing for Bing + real-time grounding for Microsoft Copilot
Open research dataset collection for AI model training
AI model training data collection
Search indexing and Google AI Overviews data sourcing
Siri answers, Spotlight search, and Safari web suggestions
Backlink analysis and SEO intelligence platform data collection
SEO competitive intelligence and site audit data collection
Yandex search indexing and YaGPT AI training
Baidu search indexing and ERNIE Bot AI training
Independent search indexing for Brave Search
Naver search indexing and HyperCLOVA X AI training
AI dataset collection for Hugging Face Hub
Search indexing and DuckAssist AI answer support
Link preview metadata and LinkedIn AI features
Web preservation and Wayback Machine archiving
Academic plagiarism detection and AI content indexing
AI-powered search engine indexing
Sogou search indexing and Tencent Hunyuan AI training
SEO data collection and AI-powered marketing research
Link intelligence database and AI SEO tool support
SEO metrics and AI content tool support
AI model training data collection
Web scraping for AI agent applications
LLM-ready content extraction and AI embedding
Neural search indexing for AI applications
Privacy-focused independent search index
Web scraping and AI training data collection
AI model training data collection
AI search summaries and LLM training
Bulk image dataset collection for AI training
News and media dataset collection for NLP/LLM training
Web indexing for Yahoo Search and partner properties
Link card previews, Open Graph metadata fetching, and AI training data collection for Grok
Your robots.txt snippet
Configure rules above to generate output
# No AI bot rules configured yet. # Use the controls above to allow or block bots, then copy your robots.txt snippet.
How to use: Add this snippet to your robots.txt file at the root of your domain (e.g. yourdomain.com/robots.txt). Note: bots that ignore robots.txt will crawl regardless — blocking those requires server-level firewall rules.
Why configure AI bots?
AI crawlers from OpenAI, Anthropic, Google, and dozens more are indexing your content right now — for training datasets, search results, and live AI assistant queries. robots.txt lets you control exactly who gets in.
Does it always work?
Most reputable AI companies (OpenAI, Anthropic, Google, Meta) respect robots.txt. Some crawlers — notably Bytespider (ByteDance) and Scrapy deployments — do not. For those, you'll need server-level IP blocking or firewall rules.
Block training, keep search
The most popular configuration: block AI Training bots (GPTBot, ClaudeBot, CCBot) to prevent your content from feeding LLM datasets, while keeping AI Search bots (PerplexityBot, Bingbot, Googlebot) so you still appear in AI-generated answers.
Check your current setup
Not sure what your robots.txt looks like right now? Run a free scan to see which AI bots you're allowing or blocking, your AI readiness score, and how your brand appears in AI search engines.
Scan my site →