Robots and the Page You Accidentally Blocked

The file and the setting

Two separate things can hide a site, and they are frequently confused.

The robots file is a small text file at the root of your site, giving instructions to crawlers about what to request.

The setting is a checkbox in most platforms, intended for use during development, that tells search engines not to index the site.

Either alone will cause a site to disappear, and the second is the more common cause because it is designed to be temporary and is easy to forget.

Both are checkable in about a minute, which is worth doing on any site that is not appearing in search.

The launch-day mistake

The single most common way a small business site fails to appear at all.

A site is built with search visibility disabled, which is correct while it is unfinished.

Then it launches, everybody looks at the design and the content, and nobody unticks the box.

Weeks later the owner asks why they cannot find themselves, and the answer has been sitting in the settings the whole time.

Check it after every launch, every rebuild, and every migration, since a copy of a development site frequently brings the setting with it.

What the file usually contains

• Instructions about which paths not to request.
• Rules for specific crawlers, or for all of them.
• The address of your sitemap.

On most small business sites it should be nearly empty, since there is rarely anything worth blocking, and the shorter it is the fewer ways it can be wrong.

Blocking is not hiding

The misunderstanding that causes real problems, and it is worth stating precisely.

The robots file asks crawlers not to request a page. It does not remove a page from search results, and it does not keep the page private.

A blocked page that is linked from elsewhere can still appear in results, listed without a description, because the crawler was told not to look at it but knows it exists.

And anyone can read your robots file, so listing a sensitive path in it advertises exactly what you were trying to hide.

To keep something out of results, allow it to be crawled and mark it as not for indexing. To keep something private, put it behind a password.

Those are three different tools and using the wrong one is the usual error.

A worked example

A business relaunched their site and after two months had not appeared for their own business name.

They had assumed it was taking time, which is a reasonable assumption.

The search visibility setting had been enabled during the build and never disabled, and the platform had accordingly been asking search engines not to index anything.

It was a single checkbox, in the settings, described in one sentence.

They unticked it and requested indexing for the main pages, and the site appeared over the following fortnight.

Two months of a new site producing nothing had been one unticked box, and nobody had checked it because nobody had thought to.

The overly broad rule

The other failure mode, and it is usually inherited rather than authored.

A rule blocking a path can be broader than intended: blocking a folder blocks everything inside it, and a rule intended for one section can cover pages added later.

Rules copied from a template or from another site are the common source, since they were written for a different structure.

Blocking a folder that holds images or stylesheets is a particular problem, because a crawler that cannot fetch those cannot see how the page renders, which affects how it is assessed.

If your file blocks anything beyond an administrative area, understand why before leaving it there.

How to check

Three steps, all free and quick.

Open your domain followed by robots.txt in a browser and read it. It is a short text file and it will be obvious if it blocks everything.

Check your platform's search visibility setting, wherever it lives, and confirm it is not enabled.

Then use Search Console's inspection tool on two or three important pages, which reports whether each is indexed and, if not, why, naming the blocking rule specifically.

That last one is definitive and removes the guesswork, since it tells you what the crawler actually experienced rather than what you think the file says.

What is worth blocking

Very little, for a small business.

An administrative login area, internal search result pages, and anything generating endless variations such as filtered or sorted views.

That is close to the complete list, and many platforms handle it by default.

Staging environments should be password protected rather than blocked in this file, since a request not to crawl is not a guarantee and a staging copy in search is a real problem.

Everything else on a small site should be crawlable, including your images and stylesheets, which are needed to assess the page properly.

The counter-case

Not every page missing from search is blocked.

New pages take time, thin pages may be crawled and not indexed by choice, and a page can be excluded for being a near duplicate of another, none of which is a technical fault.

There is also a risk in editing this file without understanding it, since a syntax error can have broader effects than intended, and it is one of the few files where a small mistake is site-wide.

Read it, understand what each line does, and change it carefully or ask somebody.

Check the setting and the file. If both are clear, the problem is elsewhere and this is not the thing to keep adjusting.

The check

1. Open your domain followed by robots.txt.
2. Check the platform's search visibility setting.
3. Do both after every launch and migration.
4. Inspect two or three pages in Search Console.
5. Confirm images and stylesheets are not blocked.
6. Password-protect staging rather than blocking it.
7. Use not-for-indexing to keep something out of results.

Step two is the one that explains most cases of a new site not appearing, and it is a checkbox rather than a technical problem.

Keeping a test copy out of results is covered in the staging site nobody blocked from Google.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.