A Redesign That Blocked Google Without Anyone Noticing

A new site nobody can find

The redesign launched in the fall. It looked better than the old site on every screen, the owner approved every page, and everyone went back to their usual work.

A few weeks later, searching for the business's main service no longer brought up the website. Searching for the business name did, but with no description beneath it, only a note that none was available.

Nothing looked broken to a person. The site was turning search engines away, exactly as it had been told to during the build.

It happens often enough that every launch deserves a check for it.

Why sites are blocked during a build

While a site is being built it usually lives at a temporary address, often called a staging site. Nobody wants half-finished pages and placeholder text in Google, so developers tell search engines to stay away.

• A robots.txt rule that disallows the whole site.
• WordPress's visibility setting ticked to discourage search engines.
• A noindex tag in the theme or set by an SEO plugin.
• A server header that tells search engines not to index anything.
• Canonical tags that still point at the staging address.

The second causes the most trouble on small business sites. It is one tickbox, and it travels with the database when the finished site is copied to its real address.

What the WordPress setting does

In WordPress the setting lives under Settings, then Reading. It is a single box labelled discourage search engines from indexing this site.

When it is ticked, WordPress adds a noindex instruction to every page. Unless the site has its own robots.txt file, WordPress also publishes one asking search engines to stay out entirely.

Developers tick it on staging for good reasons. The trouble comes at launch, when the database is copied to the live server with every setting intact and nobody looks at that screen again.

How Google treats a blocked site

A robots.txt rule stops Google from reading pages. Pages it already knew can linger in the results for a while, usually with a note that no description is available because of the site's robots.txt.

A noindex tag works differently. Google has to read the page to see the tag, and then it drops the page from its results.

Either way, the pages that used to bring visitors fade over days and weeks. That slow fade is easy to mistake for a quiet month, so a block can sit there unnoticed for a long time.

A worked example

A two-person drafting service draws house plans, additions and permit drawings for homeowners and small builders in a growing suburb.

Its new WordPress site was built on the developer's staging server over the summer and launched in October, with a page for each type of drawing and a portfolio of finished plans.

By December the phone was quieter than the season explained, and a builder they had worked with mentioned that a search had turned up only an old directory listing for them.

A search for the business name showed the site with no description. A search for drafting services in their town did not show it at all.

The visibility box was still ticked from staging, carried across with the database, and the live robots.txt was asking every search engine to stay out.

The developer cleared the box, checked the robots.txt and page source, submitted the sitemap in Search Console and asked Google to fetch the main pages again. Over the following weeks the pages returned to the results.

Check it the morning after launch

The checks take ten minutes and need no special tools.

Type the site's address followed by /robots.txt into a browser. Under User-agent: *, a line reading Disallow: followed by a single slash blocks the whole site.

View the source of the home page and search it for noindex. If the word appears in a robots meta tag, the page is asking to be left out of search results.

In WordPress, open Settings, then Reading, and confirm the visibility box is clear. Then type site: and your domain name into Google, which lists the pages it holds, and note the number.

What Search Console shows

If the site is verified in Search Console, several reports show the problem before the phone does.

The robots.txt tester shows the live file and which rule blocks a given page. Fetch as Google shows whether Google may read a page and what it sees when it does.

Index Status shows how many pages Google holds over time, and a line that falls after launch day is the signature of a block. Search Analytics shows clicks and impressions dropping from the same date.

Other launch-day blocks

SEO plugins can set noindex on whole sections, such as every page or every post, from a settings screen few people open after the build.

Canonical tags, which tell Google which address is the main version of a page, sometimes still point at the staging site after launch.

And the reverse happens too: a staging copy left open to the world gets indexed itself, so Google finds two versions of every page. Staging sites belong behind a password.

Getting back into the results

Removing the block does not bring the pages back overnight. Google has to visit each page again, and a site that was blocked for weeks may take weeks to settle.

Submit an up-to-date sitemap, run the key pages through Fetch as Google and submit them to the index, and fix any missing redirects from the old site's addresses at the same time.

Then leave the site alone for a while. Rewriting titles and moving pages during the recovery makes it hard to tell what is working.

The counter-case

A business that launches its site before opening its doors may reasonably keep it out of search until opening day, as long as someone owns the date for switching it on.

Some pages should stay out of Google permanently: thank-you pages, login pages and internal documents. Blocking those is correct, provided the rule names them and nothing else.

For everything else the rule is the same. Whatever told search engines to stay away during the build comes off on launch day and gets checked again the morning after.

The morning after launch

1. Open robots.txt on the live address.
2. Search the page source for noindex.
3. Clear the WordPress visibility box under Reading.
4. Check canonical tags point to the live site.
5. Send Search Console a fresh sitemap.
6. Run a site: search and note the count.
7. Repeat the search a week later.

Step three is the one most often missed. Nobody sees that box on the live site, and it rides along with every copy of the database.

A blocked site loses enquiries as quietly as the leaks in the enquiries your business lost last year.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.