Robots.txt Across Subdomains, Protocols, and Ports

The Origin Boundary Principle in RFC 9309

One of the most persistent architectural misunderstandings in technical SEO is assuming that a robots.txt file located on a root domain cascades down to its subdomains. Webmasters frequently add directives to https://example.com/robots.txt expecting them to protect https://staging.example.com or https://shop.example.com.

Under RFC 9309 Section 2.1, the scope of a robots.txt file is strictly bounded by the Web Origin concept defined in RFC 6454:

Subdomain Independence: No Inheritance

Consider the following distinct hostnames:

• https://example.com/robots.txt → Governs ONLY the apex domain.
• https://www.example.com/robots.txt → Governs ONLY the www host.
• https://blog.example.com/robots.txt → Governs ONLY the blog subdomain.
• https://api.example.com/robots.txt → Governs ONLY the api subdomain.

If you disallow /private/ on example.com, Googlebot is completely free to crawl https://blog.example.com/private/ unless blog.example.com explicitly hosts its own robots.txt file with an identical restriction.

Protocol Boundaries: HTTP vs. HTTPS

The URI scheme represents a distinct origin boundary. http://example.com and https://example.com are technically separate origins:

• If an old HTTP version of your site is still crawlable and returns a 200 status code with a legacy robots.txt, search crawlers may continue reading directives from the unencrypted endpoint.
• Best Practice: Enforce a site-wide HTTP 301 Permanent Redirect from HTTP to HTTPS for all requests, including /robots.txt.

Port Scoping

Similarly, non-standard network ports define separate origins:

• https://example.com:443/robots.txt (Default HTTPS port)
• https://example.com:8080/robots.txt (Custom alternate port)

Directives defined on port 443 do not govern resources accessed through port 8080.

Multi-Subdomain Management Strategies

For organizations managing dozens of subdomains (e.g., regional portals or multi-tenant SaaS applications):

1. Centralized Dynamic Routing: Configure your web server (Nginx or Apache) to route all requests for /*/robots.txt to a central backend script that serves appropriate directives based on the Host: header.
2. Staging Protection: Never rely on apex robots.txt to protect staging subdomains. Protect staging environments with HTTP Basic Authentication or IP whitelisting rather than robots.txt.
3. Audit Each Subdomain Separately: Use our Bulk Robots.txt Tester to test URLs under each specific origin, verifying that the appropriate subdomain host is selected.

Frequently Asked Questions

Can I use a wildcard for subdomains in robots.txt?

No. You cannot write Disallow: *.example.com/admin/ inside robots.txt. The file can only control paths within its own host.

Does www.example.com require its own robots.txt?

If you redirect example.com to www.example.com (or vice versa) via HTTP 301, crawlers follow the redirect to the canonical host’s robots.txt. If both hosts respond with 200, each must have its own file.

How do reverse proxies (e.g., Cloudflare) affect subdomain robots.txt?

Reverse proxies can be configured to serve custom robots.txt files per subdomain using edge workers or URL rewrite rules without changing origin server files.