Analyze robots.txt rules
Evaluate pasted robots.txt rules for a crawler and request path.
Input & settings
How to use this tool
Enter your own site data. Process and inspect the generated result before adding it to your website.
Limits and behavior
Offline longest-path matching with allow winning equal specificity. Most-specific product-token groups merge. Does not fetch robots.txt or account for status/caching policies. Maximum total input 1 MB, output 3 MB, JSON nesting 64 levels. Processing stays in your browser worker. Input is limited to 1 MB and output to 3 MB. No input content is saved by CraftAI.
Test robots.txt crawl permissions
Paste robots.txt rules, choose a crawler name and path, and inspect the supported allowance calculation.
Does allowing crawling guarantee Google indexing?
No. Crawl permission alone does not guarantee discovery, indexing, or search rankings.
Related tools: inspect webpage meta robots rules · validate xml sitemap