Technical SEO · Practical tutorial
How to test robots.txt Allow and Disallow rules
A robots.txt file tells cooperating crawlers where they are allowed to crawl. Use this worked example to check an exception rule and avoid confusing crawl controls with search indexing.
What is a robots.txt file?
A robots.txt file is normally available at the root of a website, for example https://example.com/robots.txt. It contains groups of instructions for automated crawlers. A User-agent line identifies the crawler group; Disallow asks that matching paths not be fetched, while Allow can make an exception. Crawlers do not treat robots.txt as a user-login mechanism.
Example: block private paths but allow a public subfolder
User-agent: *
Disallow: /private/
Allow: /private/public/Test these two paths separately:
/private/customer → Disallowed, because it matches Disallow: /private/.
/private/public/receipt → Allowed, because the longer, more specific Allow: /private/public/ rule applies.
Use CraftAI's offline robots.txt tester
- Open the Robots.txt Tester.
- Click Test allowed exception or Test blocked private URL to load the sample rules and run them automatically.
- Review the readable crawl decision and the matched rule. You can also inspect the original JSON output.
- Replace the sample with your own robots.txt content. Enter the crawler product token (such as
Googlebot) and the path, including the leading/. - Test several paths before deploying any changes. A single test path cannot establish that the whole website is crawlable.
Why does the longest matching rule matter?
For a crawler group, the more specific matching path determines the decision. If equally specific Allow and Disallow rules conflict, Google uses the less restrictive rule. Other directives or syntax may have different support between crawlers, and your real robots.txt file may differ from the text you pasted.
Can robots.txt prevent a page from appearing in Google?
Not reliably. Robots.txt controls crawling, not all ways an address can appear in search results. Google may discover a blocked URL through other links without fetching the page contents. If your goal is to exclude a public HTML page from indexing, a noindex directive on a crawlable page can be more appropriate. Don't block the crawler from reading the page while expecting it to see a noindex tag.
Does Allow guarantee indexing or a higher rank?
No. A page can be crawlable but still not indexed or ranked prominently. Search engines also consider duplicate pages, canonical URLs, usefulness and other signals. Use Google Search Console URL Inspection to check a particular page's status; generating or testing a robots rule is not an indexing request.
Privacy and security: use access controls
Disallow: /private/ is not password protection. People can still open accessible URLs, and the robots.txt file itself publicly lists the paths you specify. Use proper authentication and authorization for genuinely private documents or account areas.
Limitations of an offline tester
CraftAI evaluates the text you provide in your browser. It does not fetch a website's live robots.txt, simulate cache behavior, check HTTP status codes, or guarantee exactly how all search engines interpret every extension or invalid file. For significant changes, inspect the deployed file and verify critical behavior using the real crawler's documentation.
Read Google's technical specification
For the latest documented Googlebot behavior, consult Google's robots.txt interpretation and the Robots Exclusion Protocol (RFC 9309). CraftAI is an independent tool and is not affiliated with Google.