A fall in traffic does not prove that a site has been deindexed. Start by checking whether important URLs are indexed and accessible, then trace what changed.
This diagnostic guidance has been updated for current robots.txt and noindex practice.
Separate three different problems
- A crawling block: robots.txt can stop a crawler fetching a page. That is not the same instruction as noindex, and a blocked URL can still appear in search without its content being crawled.
- An indexing block: a noindex meta tag or HTTP header tells supporting search engines not to index the page. The crawler must be able to fetch the response to see that rule.
- A missing or redirected page: the response may be an error, a redirect to the wrong destination, or a page that no longer contains the expected content.
Use Search Console’s URL Inspection tool for representative affected URLs. Inspect the live response, headers, robots rules and rendered page. Do not use the order of a site: search as proof of a fault.
Trace the release or configuration change
Check recent theme, plugin, hosting and deployment changes. The original 2016 article described crawl blocks across both www and non-www versions. Checking only one hostname would have missed part of the problem.
Keep intentional privacy and staging restrictions intact. Remove only the unintended production block, then test the actual public response again. A configuration file that looks correct is not enough.
Request and monitor recovery
After the fix, use URL Inspection where appropriate and keep an accurate sitemap. Monitor indexing and important business journeys. Recrawling and indexing are not instant, and removing a block does not guarantee a particular ranking or recovery date.
See the historical agency-exit incident and hosting ownership checklist.