How WordPress Handles Robots.txt by Default
When you install WordPress, you may look in your server’s root directory (via FTP or cPanel File Manager) and wonder why there is no physical robots.txt file. WordPress utilizes a built-in rewriting mechanism to generate a virtual robots.txt file on the fly whenever a request hits /robots.txt.
By default, WordPress serves a minimal template similar to this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlNotice the critical Allow: /wp-admin/admin-ajax.php rule: many WordPress themes and plugins require AJAX endpoints to render dynamic frontend content.
Method 1: Editing via SEO Plugins (Recommended)
The safest and most convenient way to customize your WordPress robots.txt is through a reputable SEO plugin.
Using Yoast SEO:
- Navigate to Yoast SEO → Tools in your WordPress dashboard.
- Click on File editor.
- If no physical file exists, click "Create robots.txt file".
- Add your custom directives into the editor and click Save changes to robots.txt.
Using Rank Math:
- Navigate to Rank Math → General Settings → Edit robots.txt.
- Edit the directives directly in the syntax-highlighted code box.
- Click Save Changes.
Method 2: Creating a Physical Robots.txt File
If you prefer complete control without plugin dependencies, you can create a physical plain-text file named robots.txt and upload it to the root of your WordPress installation (the folder containing wp-config.php).
robots.txt file takes complete precedence over the WordPress virtual file. Once uploaded, WordPress will stop generating the virtual file, and your physical file will be served directly by your web server (Apache or Nginx).
Essential WordPress Directives to Include (and Avoid)
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# Block search query parameters if using separate faceted navigation
Disallow: /?s=
Disallow: /search/
# Ensure media and uploads remain fully accessible
Allow: /wp-content/uploads/
Sitemap: https://example.com/sitemap_index.xmlWhat NEVER to Block in WordPress:
- Do NOT block
/wp-content/plugins/or/wp-content/themes/: These directories contain the CSS and JS files required for Googlebot to render your pages properly. - Do NOT block
/wp-includes/indiscriminately: Core scripts (such as jQuery) reside here and are needed for frontend rendering.
Always test your custom WordPress rules using our Bulk Robots.txt Tester to ensure plugins and category archives remain accessible.
Frequently Asked Questions
Why does my WordPress robots.txt say "Disallow: /"?
Check Settings → Reading in your WordPress dashboard. If "Discourage search engines from indexing this site" is checked, WordPress injects a site-wide block. Uncheck the box and save.
Does a physical robots.txt overwrite plugin settings?
Yes. If a physical file exists in your server root, plugins like Yoast and Rank Math cannot override it.
Why can’t I edit robots.txt in WordPress multisite?
In WordPress multisite networks, sub-sites share the network root virtual file. Custom rules require specialized multisite plugins or server-level rewrites.
