Robots.txt Generator
Create, optimize, and validate your robots.txt directives to ensure search engine crawlers index your site perfectly while blocking AI scrapers.
⚡ Quick Presets
User-Agent Directives (6)
Sitemap & Host Directives
Known AI Crawler Reference List
How It Works
Select Preset or Custom
Choose Allow All, Block All, or Block AI Bots preset, or add custom user-agent rules.
Configure Directives
Specify Allow, Disallow, Crawl-delay, and Sitemap directives for search crawlers.
Test & Download
Use the live URL tester to verify access permissions, then copy or download robots.txt.
Frequently Asked Questions
Where should I place the robots.txt file?
How do I block AI scrapers?
Does robots.txt stop pages from indexing?
Understanding Robots Exclusion Protocol & AI Bot Control
Common AI Crawlers & Scraper Directives
Modern websites require specific rules to govern search engines and AI model training scrapers:
| Crawler Bot Name | User-agent String | Organization / Purpose |
|---|---|---|
| GPTBot | GPTBot | OpenAI web crawler for model training |
| ChatGPT User | ChatGPT-User | Real-time web browser for ChatGPT |
| Claude Web | anthropic-ai | Anthropic Claude AI training scraper |
| Common Crawl | CCBot | Open repository web crawler |
| Google AI | Google-Extended | Google Gemini AI model training |
Robots.txt Best Practices & Optimization
1. Include XML Sitemaps: Always append Sitemap: https://yourdomain.com/sitemap.xml. Use our Sitemap Generator to build valid XML feeds.
2. Audit Web Page Size: Inspect page resources with our Page Size Checker to ensure fast crawl budgets.