Robots.txt Allow and Disallow Explained

The History of Allow vs. Disallow

When the original Robots Exclusion Protocol was drafted in 1994, it only contained a single operative directive: Disallow. If you wanted to block a directory, you disallowed it; everything else was presumed allowed. However, as web applications grew more sophisticated, webmasters needed a way to block an entire directory while keeping a specific subfolder or landing page open for crawling.

Early search engines implemented proprietary solutions for Allow directives, leading to subtle cross-engine bugs. With the publication of RFC 9309, the interaction between Allow and Disallow has been formally codified for all compliant crawlers.

The Principle of Longest Octet Match

The core rule of modern robots.txt evaluation is simple yet frequently misunderstood:

In practice, this means order in the file does not matter. Unlike firewall rules that stop evaluating at the first match, a robots.txt parser reads all applicable directives and selects the rule that has the longest matching character length.

Concrete Specificity Examples

Examine how the following rules interact with various URLs:

User-agent: *
Disallow: /catalog/
Allow: /catalog/featured/
Disallow: /catalog/featured/archive/

The Universal Tie-Breaker: Allow Wins

What happens if both an Allow directive and a Disallow directive match the identical path with the exact same length? Consider:

User-agent: *
Disallow: /temporary
Allow: /temporary

Under RFC 9309, if multiple rules match with the same octet length, the Allow rule takes precedence. The path /temporary is considered ALLOWED. This design principle ensures that access is prioritized over blocking when ambiguous conflicting rules exist.

Empty Directives and Their Effects

• Empty Disallow (Disallow:): An empty Disallow line signifies that no paths are disallowed. It effectively acts as a reset or full permit for the crawler group.
• Empty Allow (Allow:): An empty Allow line has no defined effect and is ignored by compliant parsers.

Before launching major site changes, verify your path specificity using our Bulk Robots.txt Tester to ensure no accidental blocks occur.

Frequently Asked Questions

Does rule order matter in robots.txt?

No. Under RFC 9309, rule order is completely irrelevant for Allow and Disallow. The rule with the greatest matching character length always wins.

Can I use Allow without Disallow?

Technically yes, but everything is allowed by default anyway. Allow is primarily used to carve out exceptions within broader Disallow rules.

How do query strings affect matching length?

Query parameters are treated as part of the path octet string. A rule matching query parameters (e.g., Allow: /page?ref=) will have a longer octet count than Disallow: /page, taking precedence for that specific query.