Pattern Matching in Modern Robots Exclusion
In the early days of search engines, robots.txt rules were limited to simple prefix matching. A rule like Disallow: /private/ matched any URL beginning with /private/, but there was no standard way to block specific file types (like .pdf or .xls) or target dynamic URL query strings across the entire website.
With RFC 9309 Section 2.2.2, two essential pattern-matching characters have been officially standardized across all major search engines: the wildcard asterisk (*) and the end anchor dollar sign ($).
The Wildcard Character (*)
The asterisk designates zero or more arbitrary characters. It can appear at the beginning, middle, or end of a path pattern.
Common Wildcard Use Cases
- Blocking Dynamic Parameters:
Disallow: /*?*sort=blocks any URL containing a sorting parameter anywhere on the site, regardless of the folder path. - Blocking Subdirectories Globally:
Disallow: /*/temp/blocks/blog/temp/,/shop/products/temp/, and/temp/simultaneously. - Targeting File Extensions:
Disallow: /*.jsonblocks any URL containing.json.
The End Anchor Character ($)
By default, robots.txt rules are prefix matches. A rule like Disallow: /*.pdf will block:
https://example.com/brochure.pdfhttps://example.com/brochure.pdf?view=inlinehttps://example.com/brochure.pdf-archive/details
The dollar sign ($) forces the match to anchor strictly to the exact end of the URL path or query string. When you write:
Disallow: /*.pdf$The crawler will block /brochure.pdf, but it will ALLOW /brochure.pdf?version=2 because the trailing query parameter prevents the URL from ending with .pdf.
Interactive Pattern Matching Matrix
| Directive Pattern | URL Tested | Status | Reasoning |
|---|---|---|---|
Disallow: /*.doc$ |
/files/contract.doc |
BLOCKED | Ends exactly with .doc |
Disallow: /*.doc$ |
/files/contract.docx |
ALLOWED | Does not end with .doc (ends with .docx) |
Disallow: /search?*q= |
/search?lang=en&q=shoes |
BLOCKED | Wildcard * matches intermediate characters lang=en& |
Disallow: /*? |
/products/item?color=red |
BLOCKED | Matches any URL containing a question mark (query string) |
Testing Complex Wildcards Before Publishing
Because wildcards can match unintended strings, a small typo can inadvertently block wide sections of a domain. Always validate your patterns using our Bulk Robots.txt Tester. Paste a batch of sample URLs—including edge cases with trailing slashes, parameter variations, and uppercase characters—to confirm that only the intended paths match.
Frequently Asked Questions
Does robots.txt support regular expressions (Regex)?
No. Full regular expressions (such as character classes [a-z], quantifiers +, or alternation |) are NOT supported in robots.txt. Only * and $ are standard under RFC 9309.
Can I combine multiple wildcards in one line?
Yes. Patterns like Disallow: /*/cache/*/*.tmp$ are valid and will be evaluated properly by compliant parsers.
How do wildcards affect rule length specificity?
The wildcard characters themselves count toward the total octet length of the directive pattern when calculating which rule is longest and most specific.
