Generate robots.txt
Generate robots.txt crawler rules and sitemap directives.
Input & settings
How to use this tool
Enter your own site data. Process and inspect the generated result before adding it to your website.
Limits and behavior
Rules control crawling, not access authorization or index removal. Verify rules on your own server. Maximum total input 1 MB, output 3 MB, JSON nesting 64 levels. Processing stays in your browser worker. Input is limited to 1 MB and output to 3 MB. No input content is saved by CraftAI.
Robots.txt generator for search engine crawlers
Create robots.txt rules for a website, including crawler directives and sitemap references. This robots.txt creator helps website owners express which URL paths crawlers may request.
Does robots.txt prevent pages from appearing in Google?
Not reliably. Robots.txt controls crawling, not guaranteed search-result removal. Use appropriate index directives or access controls when content must not appear in search.