Check it in Search Console's URL inspection tool, which states plainly whether the page was crawled, whether it was indexed, and what stopped it if not.

How to tell

Search for the exact address using the site prefix. If the page appears, it is indexed. If nothing appears, it is not.

That test takes ten seconds and is more reliable than searching for the content, which depends on ranking as well as indexing.

For the reason why, the URL inspection tool in Search Console reports whether the page was crawled, when, whether it was indexed, and if not, what prevented it.

It states this in plain language, which makes the diagnosis considerably easier than it sounds.

The six reasons

The second and third look identical from outside and are entirely different problems with different fixes, which is why guessing is unproductive.

The no-index instruction

The most common cause and the most frustrating, because it is invisible in a browser.

A line in the page's code telling search engines not to index it. The page looks completely normal to a visitor.

It gets there from a development setting that was never turned off, a plugin default, a template option, or somebody ticking a box labelled discourage search engines.

Most platforms have a single global setting for this, intended for use during a build, which is routinely left on after launch.

Checking it is the first thing to do on any site that appears for nothing, because it accounts for a large share of cases and takes thirty seconds.

The robots file

A small text file that tells crawlers what they may fetch.

Visit your address followed by the standard filename to read yours. A line disallowing everything blocks the entire site.

That usually arrives by being copied from a staging server, where blocking was correct.

Worth noting: blocking in the robots file prevents crawling but does not reliably prevent a page appearing in results, since a blocked page linked from elsewhere can still be listed without content.

Which means for genuinely keeping something out of results, a no-index instruction is the right tool and the robots file is not.

A worked example

A business whose main service page had never appeared, while the rest of the site was indexed normally.

The URL inspection tool reported that the page was crawled, and that indexing was blocked by a no-index tag.

The page had been duplicated from a template during the build, and the template carried the tag.

Removing it and requesting indexing produced results within about ten days.

The business had spent four months assuming it was a competitive ranking problem for that service, and had been considering paying for advertising to compensate.

Ten seconds of checking would have identified it at any point during those four months.

Crawled and not indexed

A specific status worth understanding, because it is common and it is not an error.

It means the page was fetched and a decision was made not to store it.

Usual causes: the page is very thin, it substantially duplicates another page, or it is one of many near-identical pages such as location pages with the town swapped.

Sometimes it is temporary and resolves on its own, particularly on a new site.

The fix is making the page substantively different and genuinely useful, which is a content problem rather than a technical one, and no amount of resubmission changes it.

The canonical tag

A less common cause and a confusing one.

A canonical tag tells a search engine which version of similar content is the authoritative one.

Where it is set incorrectly, every page can point at the homepage, which tells search engines that none of the other pages matters.

That happens through a misconfigured plugin or a template error, and it produces a site where only the homepage is indexed.

The inspection tool reports the canonical it found, which makes this diagnosable in the same thirty seconds as the rest.

The counter-case

Where a page not being indexed is correct.

Thank you pages, checkout steps, search result pages on your own site, and anything with thin or duplicated content that exists for a functional reason.

Tag and category archives on many sites, which frequently duplicate content that exists elsewhere.

And any page you do not want people arriving on cold.

A site where every page is indexed is not the goal. A site where the pages you want found are found is.

How long to wait before worrying

Worth knowing, because impatience produces a great deal of unnecessary diagnosis.

A new page on an established, well-linked site is usually found within days.

A new page on a small or new site can take a few weeks, and that is normal rather than a fault.

Which means a page missing after four days needs no investigation at all, and the same page missing after six weeks does.

The inspection tool distinguishes the two clearly: a page never crawled is a different situation from one crawled and rejected, and it will tell you which you have.

If it reports the page has simply not been discovered yet, on a site otherwise indexed normally, waiting is the correct response.

After you fix it

The step people miss, having found and removed the cause.

Removing a no-index tag does not re-index the page by itself. It becomes eligible, and a search engine gets to it when it next crawls, which may be days or weeks.

Requesting indexing through the inspection tool shortens that considerably, and is worth doing once for each page you have fixed.

Then check back in a fortnight rather than daily, using the same site-prefix search that identified the problem.

And if a page you fixed a month ago is still absent, inspect it again, because the tool will now report whatever the remaining obstacle is rather than the one you already removed.

The diagnostic sequence

  1. Search for the exact address with the site prefix.
  2. If absent, open URL inspection in Search Console.
  3. Read what it says about crawling and indexing.
  4. Check the global discourage-search-engines setting.
  5. Visit your robots file directly.
  6. Check what canonical the tool reports.

Those six steps take under ten minutes and identify the cause in nearly every case, which is why guessing at this is never worth the time it appears to save.

What has to happen before any of this matters is covered in what Google has to do before it can rank you.

Frequently asked questions

How do I tell if a page is indexed?

Search for the exact address using the site prefix. If it appears, it is indexed. That is more reliable than searching for the content.

What are the six causes?

Nothing links to it, a no-index instruction, the robots file blocks it, a canonical pointing elsewhere, judged not worth keeping, or simply too new.

What is the most common cause?

A no-index instruction, which is invisible in a browser. Usually a development setting or a discourage-search-engines box that was never turned off.

Does the robots file keep pages out of results?

Not reliably. It prevents crawling, but a blocked page linked from elsewhere can still be listed without content. A no-index instruction is the right tool.

What does crawled but not indexed mean?

The page was fetched and a decision was made not to store it, usually because it is thin or duplicates another. That is a content problem, not a technical one.

Should every page be indexed?

No. Thank you pages, checkout steps and internal search results should not be. The goal is that the pages you want found are found.

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.

One page never appearing while the rest are fine?

Check it in URL inspection first. A no-index tag left over from a template is the usual answer.

Start a Conversation