Crawl errors in WordPress are one of the most common and most ignored technical SEO problems on the web. If Google cannot crawl a page on your site, that page cannot rank. It does not matter how well written the content is or how many backlinks it has. A page Google cannot access is effectively invisible.
The good news is that most crawl errors follow predictable patterns, have clear causes, and can be fixed without any coding knowledge. This guide walks you through every type of crawl error you will find in Google Search Console, what each one means for your WordPress site, and exactly how to fix it.
What Crawl Errors Actually Are
A crawl error occurs when Google’s bot called Googlebot. It attempts to access a URL on your WordPress site and cannot retrieve it successfully. Googlebot discovers URLs through your sitemap, through internal links from other pages, and through external backlinks pointing to your domain. When it follows a link and finds an error instead of a working page, it logs that error in Google Search Console.
Crawl errors matter because Google can only index pages it can successfully crawl. According to Google’s own crawling documentation, Googlebot discovers and crawls billions of pages daily and even minor technical issues can prevent your content from reaching users. On WordPress specifically, crawl errors often appear after plugin updates that change URL structures, after permalink changes, or after deleting content without setting up redirects.
Not every page flagged in Google Search Console as “not indexed” is an error you need to fix. Some pages are intentionally excluded author archives you have set to noindex, tag pages, date archives. The goal is to identify and fix errors on pages you actually want Google to index.
How to Find Crawl Errors in Google Search Console
Go to Google Search Console → Indexing → Pages. This is the main report for crawl and indexing data in 2026. The report shows all URLs Google has discovered on your site, split into two groups: Indexed and Not Indexed.
Click on the “Not Indexed” tab. Below the chart you will see a table listing every reason Google has not indexed certain pages. Each reason is a separate error type. Click on any reason to see the specific URLs affected.
The most common error reasons you will see on a WordPress site are listed in order of urgency:
| Error Type | What It Means | Priority |
|---|---|---|
| 404 Not Found | Page deleted or URL changed without a redirect | High |
| Server Error (5xx) | Server failed to respond to Googlebot | High |
| Blocked by robots.txt | robots.txt disallow rule is preventing crawling | High |
| Submitted URL has noindex tag | Page is in sitemap but has a noindex directive | High |
| Soft 404 | Page returns 200 status but contains no real content | Medium |
| Crawled, currently not indexed | Google visited but decided not to index — thin or duplicate content | Medium |
| Discovered, currently not indexed | Google knows about URL but has not crawled it yet | Low |
Before fixing any crawl errors, go to WordPress → Settings → Reading and verify that “Discourage search engines from indexing this site” is unchecked. If this box is checked, every page on your site is blocked from crawling. This single setting causes more widespread crawl errors than any other WordPress configuration mistake.
How to Fix 404 Errors in WordPress
A 404 error means Googlebot followed a link to a URL that no longer exists on your site. The most common causes on WordPress are: deleting a post or page without redirecting it, changing a post’s slug after it had been indexed, or changing your permalink structure globally under Settings → Permalinks.
The fix depends on whether you want to restore the content or redirect the old URL to something else.
In the Pages report, click on “Not Found (404)” to see all affected URLs. Export this list using the download icon at the top right of the table. This gives you a full list to work from rather than fixing one at a time.
Install and activate the free Redirection plugin from the WordPress plugin directory. This lets you create and manage all your 301 redirects from inside WordPress without editing .htaccess directly.
Go to Tools → Redirection. Click “Add New.” In the “Source URL” field, enter the old broken URL. In the “Target URL” field, enter the most relevant existing page on your site. Set the redirect type to 301 (Permanent). Click Add Redirect.
Go back to the Pages report → “Not Found (404)” → click “Validate Fix.” Google will re-crawl the affected URLs and confirm the redirect is working. This process takes a few days.
Never redirect 404 pages to your homepage as a catch-all. Google detects these as “soft 404s” pages returning a 200 status code with no relevant content. Redirecting every broken URL to your homepage creates dozens of new soft 404 errors. Only redirect to genuinely relevant content. If there is no relevant page to redirect to, let the URL return a clean 404 and remove it from your sitemap.
How to Fix Soft 404 Errors in WordPress
A soft 404 happens when a page returns a 200 HTTP status code meaning “page found, everything is fine” but the page contains no real content. Google detects this mismatch and logs it as a soft 404. The most common WordPress cause is empty tag archive pages.
When you create a tag in WordPress, a tag archive URL is automatically generated. For example, technexies.com/tag/seo/. If that tag only has one post assigned to it, or the post has been unpublished, the archive page exists but is empty. Google reaches it, finds nothing useful, and flags it as a soft 404.
Fix for tag soft 404s: Go to Rank Math → Titles and Metas → Tags. Set “Robots Meta” to “noindex, follow.” This tells Google not to index tag archive pages, which eliminates the soft 404 problem for empty tags. Alternatively, if you have high-value tags with multiple posts, add a proper description to each tag under Posts → Tags → Edit to give the archive page unique content worth indexing.
Fix for homepage soft 404s after redirects: Check whether you have created redirect rules pointing multiple unrelated 404 URLs to your homepage. Remove these and replace them with proper redirects to relevant content, or simply remove the redirect and allow the old URL to return a genuine 404.
How to Fix noindex Errors in WordPress
This error appears in Search Console as “Submitted URL marked noindex.” It means you have a page in your XML sitemap that also has a noindex tag telling Google not to index it is a direct contradiction. Pepperland Marketing’s analysis of common Search Console errors identifies this as one of the most frequent mixed-signal issues on WordPress sites, often caused by Rank Math settings being changed after a sitemap was already submitted.
There are two possible fixes depending on what you actually want:
If you want the page indexed: Find the page in WordPress, open Rank Math in the editor sidebar, go to the Advanced tab, and make sure “Robots Meta” is set to “index.” Save and update the page. Then go to Google Search Console → URL Inspection → Request Indexing.
If you do not want the page indexed: Keep the noindex setting and remove the URL from your sitemap. In Rank Math → Sitemap Settings, check which post types and taxonomies are included. If author archives, tag archives, or date archives are included in the sitemap but noindexed, toggle them off in the sitemap settings. Rank Math respects noindex settings in its sitemap if configured correctly but it requires you to go through Titles and Metas and verify each content type’s robots settings match its sitemap inclusion status.
How to Fix robots.txt Blocked Pages
If you see “Blocked by robots.txt” in Search Console, a Disallow rule in your robots.txt file is preventing Googlebot from crawling those URLs. The most serious version of this error is when important content directories or your entire site are accidentally blocked.
Go to technexies.com/robots.txt in your browser to see your current file. Compare it against this correct WordPress configuration:
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Allow: /wp-admin/admin-ajax.php Sitemap: https://technexies.com/sitemap_index.xml
If your robots.txt has any Disallow rules beyond those two lines — particularly anything starting with Disallow: /wp-content/ or Disallow: /. Those rules may be blocking legitimate content from being crawled. Edit your robots.txt via Rank Math → General Settings → Edit robots.txt. Remove any incorrect Disallow rules, save, and then go to Search Console to validate the fix.
How to Fix Server Errors (5xx) in WordPress
Server errors occur when Googlebot reaches your site but your server fails to deliver a response. SEO Testing’s comprehensive error guide identifies three types you will see on WordPress: 500 (internal server error – usually a PHP or plugin conflict), 502 (bad gateway – typically a hosting or WordPress CMS issue), and 503 (service unavailable – server overloaded or in maintenance mode).
For WordPress-specific server errors, work through these steps in order:
Go to Plugins → Installed Plugins → select all → Deactivate. Then visit the affected URL. If the server error disappears, reactivate plugins one by one until you identify the conflict. Plugin conflicts are the most common cause of 500 errors on WordPress.
Go to Appearance → Themes and activate a default WordPress theme like Twenty Twenty-Four. If the error resolves, the problem is in your active theme’s code. Contact the theme developer or check for updates.
Log into your hosting control panel (cPanel or similar) → Error Logs. The log will show the exact PHP error causing the 500. Copy the error message and search for it — most WordPress PHP errors have documented solutions.
Persistent 503 errors usually indicate a server resource problem — your hosting plan cannot handle your site’s traffic or database queries. A reputable managed WordPress host (SiteGround, Kinsta, WP Engine) includes server monitoring and can diagnose infrastructure-level issues your control panel cannot show you.
Crawl Budget: What It Is and Why It Matters for WordPress
Crawl budget is the number of pages Googlebot will crawl on your site within a given time frame. For small sites under 100 pages, crawl budget is rarely a concern, Google will crawl your entire site on every visit. But as your WordPress site grows, crawl budget becomes important because you want Google spending its crawl allowance on your valuable content pages, not on thin archive pages, tag pages, or duplicate URLs.
On WordPress, the biggest crawl budget wasters are: unnoindexed tag and date archives (hundreds of thin pages), media attachment pages (one URL per uploaded image), paginated archive pages, and URL parameters from plugins that create variations like ?replytocom= or ?share=twitter.
Fix crawl budget waste in Rank Math by noindexing author archives, date archives, and tag archives (if low-value), and enabling “Redirect Attachments” under General Settings → Links to point media attachment URLs back to the parent post. For URL parameters, go to Google Search Console → Legacy Tools → URL Parameters and mark any tracking or session parameters as “doesn’t affect page content.”
Understanding how to configure Rank Math correctly is essential for managing crawl errors and noindex settings. Read our complete Rank Math SEO setup guide to see every setting that affects indexing and crawling.
This article is part of our Technical SEO for WordPress series. For a full overview of every technical issue WordPress sites face, read our complete guide: Technical SEO for WordPress — The Complete 2026 Fix-It Guide.
Frequently Asked Questions
Q. How do I fix crawl errors in WordPress without a developer?
Most WordPress crawl errors can be fixed using three free tools: Google Search Console (to identify errors), the Redirection plugin (to fix 404s with 301 redirects), and Rank Math (to fix noindex and robots.txt issues). No coding is needed for any of these fixes — all settings are managed through plugin interfaces inside your WordPress dashboard.
Q. How long does it take for Google to fix crawl errors after I fix them?
After you fix a crawl error and click “Validate Fix” in Google Search Console, Google will re-crawl the affected URLs within a few days to a week. The Search Console report updates with a 2 to 4 day lag, so allow up to two weeks before checking whether the error count has dropped. For urgent pages, use URL Inspection → Request Indexing to trigger a faster re-crawl.
Q. Should I worry about every URL in the “Not Indexed” report?
No. The Not Indexed report includes pages you intentionally excluded noindexed author archives, tag pages, date archives, alongside genuine errors. Focus only on pages you want indexed like your posts, core pages, and category pages. Ignore noindex statuses on pages you deliberately excluded.
Q. What causes the most crawl errors on WordPress sites?
The most common causes are: deleting or renaming posts without setting up 301 redirects, changing the permalink structure under Settings → Permalinks without redirecting old URLs, plugin updates that alter URL structures, and the “Discourage search engines” checkbox being left on after site development. Regular audits in Search Console catch these before they compound.
Kia has worked in SEO and digital marketing for over a decade, building and optimising websites across different industries. He founded Technexies to share what actually works in modern search written from direct professional experience rather than theory. All content on Technexies is researched, written, and reviewed by Kia personally.
