Why Website Migrations Fail at the Robots.txt Layer
Website migrations—whether migrating from WordPress to Shopify, replatforming to a headless Next.js stack, or consolidating international domains—are high-risk technical operations. Months of design, content optimization, and 301 redirect mapping can be wiped out in seconds if the robots.txt file is misconfigured at launch.
The classic migration horror story is familiar: developers keep staging environments locked down with Disallow: /, and upon launch day, the staging configuration is pushed directly to the production domain. Search engines hit the live site, encounter the sitewide block, and begin deindexing the company’s organic catalog.
Pre-Launch Migration Checklist (Staging Phase)
1. Audit the Staging vs. Production Rule Differential
Never copy a staging robots.txt directly to production. Review line-by-line differences between your current legacy robots.txt and your proposed new file.
2. Map New URL Taxonomy and Directory Changes
If your URL structure changes (e.g., from /catalog/products/ to /shop/items/):
- Ensure legacy Disallow rules are updated to reflect the new paths.
- Verify that your 301 redirects are crawlable. Crucial: If legacy URLs are blocked in robots.txt, Googlebot cannot crawl them to discover your 301 redirects!
3. Test 100 Critical Paths in Sandbox Mode
Use the Bulk Robots.txt Tester in Paste Mode:
- Paste your proposed production robots.txt file.
- Input a batch of 100 representative URLs from your new architecture (core pages, new category paths, new faceted parameters, and critical CSS/JS bundles).
- Verify that all essential landing pages show ALLOWED.
4. Verify XML Sitemap Declarations
Ensure the Sitemap: declaration points to the new canonical sitemap index URL on the live domain, not the staging domain.
Launch Day Checklist (Go-Live Hour)
1. Immediate Live Fetch Verification
Within minutes of DNS propagation or deployment completion, visit https://yourdomain.com/robots.txt in an incognito window or fetch it via our Live Fetch Mode to confirm:
- HTTP status code is 200 OK.
- The staging rule
Disallow: /is NOT present. - The correct XML sitemap index is declared.
2. Check HTTP to HTTPS and WWW Canonicalization
Confirm that requesting http://yourdomain.com/robots.txt returns a clean 301 redirect to https://yourdomain.com/robots.txt.
Post-Launch Checklist (Days 1 to 7)
1. Request Expedited Re-crawl in Google Search Console
Navigate to Google Search Console → Settings → Crawl stats → Robots.txt. Verify that Googlebot has successfully fetched the new version. Submit your new sitemap in the Sitemaps tab.
2. Monitor Search Console Indexing Trends
Check the Page Indexing report daily for unexpected spikes in "Blocked by robots.txt." If valid canonical pages appear in this report, identify the decisive directive immediately using our bulk tester and deploy a fix.
Frequently Asked Questions
Should I disallow old URLs during a migration?
NO! Old URLs must remain ALLOWED in robots.txt so that search crawlers can fetch them, register the 301 redirect headers, and pass PageRank equity to the new destination.
How quickly does Google detect a robots.txt change after launch?
Google typically fetches robots.txt within a few hours to 24 hours of launch. You can check the exact fetch timestamp in Search Console.
What is the fastest way to fix an accidental staging block in production?
Immediately revert your live robots.txt file to allow all crawling (Disallow:), verify the 200 OK response with our Live Fetch tool, and use Search Console’s URL Inspection Tool on your homepage to force a re-fetch.
