The Challenge of Dynamic URL Parameters
Modern web applications rely heavily on query parameters (everything following the ? in a URL) for sorting, filtering, tracking, session management, and pagination. However, search engine crawlers treat every unique parameter string as a distinct page, leading to severe duplicate content and wasted crawl budget.
Under RFC 9309, query parameters are fully supported in robots.txt pattern matching. In this guide, we examine how to construct robust, bulletproof directives that target dynamic query parameters without breaking clean canonical pages.
The Question Mark (?) in Robots.txt
A question mark in a robots.txt directive matches a literal question mark in the URL path. It does not act as a regex quantifier.
Pattern 1: Blocking All Query Strings Globally
User-agent: *
Disallow: /*?This directive blocks any URL that contains a question mark. It allows clean paths like /category/shoes, but blocks /category/shoes?color=blue and /page?ref=home.
Disallow: /*?. If your site uses URL parameters for canonical pagination (e.g., ?page=2), this directive will block search engines from crawling your paginated archives!
Pattern 2: Targeting Specific Parameter Keys
Because query parameters can appear in any order (e.g., ?sort=asc&color=red vs ?color=red&sort=asc), you must structure wildcards to catch the parameter regardless of its position in the query string:
# Targets the "sort" parameter anywhere in the query string
User-agent: *
Disallow: /*?*sort=
Disallow: /*&*sort=Why two lines?
/*?*sort=matches whensortis the first parameter (e.g.,/shop?sort=price) or preceded by other parameters after the question mark./*&*sort=matches whensortis preceded by an ampersand delimiter (e.g.,/shop?cat=1&sort=price).
Pattern 3: Allowing Specific Parameters While Blocking Others
Under RFC 9309 longest-match specificity, you can permit specific canonical parameters while disallowing all other query strings:
User-agent: *
# First, block all parameters
Disallow: /*?
# Carve out specific allowed parameters (longer matching rules take precedence)
Allow: /*?page=*
Allow: /*?p=*Testing Parameter Combinations Safely
| Directive | Tested URL | Status | Reasoning |
|---|---|---|---|
Disallow: /*?*filter= |
/catalog/shoes?filter=boots |
BLOCKED | Matches parameter key directly after question mark |
Disallow: /*?*filter= |
/catalog/shoes?cat=footwear&filter=boots |
BLOCKED | Wildcard matches intermediate string cat=footwear& |
Disallow: /*?*filter= |
/catalog/shoes |
ALLOWED | No question mark or filter parameter present |
Test your complex parameter rules using our Bulk Robots.txt Tester to ensure canonical pagination remains open while filtering traps are blocked.
Frequently Asked Questions
Are parameter names case-sensitive in robots.txt?
Yes! Disallow: /*?*Sort= will not match ?sort=asc. Ensure your directives match the exact casing generated by your web server.
Does URL parameter handling in Google Search Console still exist?
No. Google officially retired the Search Console URL Parameters tool in early 2022. Robots.txt and canonical tags are now the primary mechanisms for managing parameter crawling.
How does robots.txt handle URL-encoded characters (like %20 or %26)?
RFC 9309 dictates that crawlers normalize percent-encoded octets for unreserved characters, but reserved characters (such as &, =, and ?) should be matched carefully.
