Search engines find pages by following links. A crawler arrives at a page it already knows, reads the links on it, and adds those to a list of places to visit next. That single mechanism explains most discovery problems, because a page nothing links to is a page there is no route to.
The mechanism
There is no submission process, no directory to register with, and nothing to fill in. What happens is simpler and less reassuring.
A program visits a page it already knows about, reads the code, and notes every link on it. Those links go onto a queue. It works through the queue, repeating the process, indefinitely. Your site enters that queue the first time something already in it links to you.
Three consequences follow, and they explain nearly every discovery problem a small business has.
- Links are the roads. No link, no route, no discovery.
- Discovery is not instant. The queue is enormous and your pages sit in it alongside everything else.
- Being found is not the same as being ranked. A page can be discovered, read, and judged not worth showing.
How a brand new site gets found
This is the awkward case, because a new site has nothing pointing at it.
In practice it happens through a few routes. Your domain registration and hosting generate signals. A business listing that links to your site provides a route. Any directory, association, or supplier page that mentions you does the same. And a sitemap submitted through the search console tells the engine directly that these pages exist.
That last one is the closest thing to submission that still exists, and it is worth doing on a new site precisely because the natural routes are thin. It does not guarantee anything and it removes the discovery problem, leaving only the judgement problem.
Why internal links matter more than people expect
Most pages on a small business site are discovered through other pages on the same site rather than from outside.
That makes your own navigation and in-page links the primary discovery mechanism for your own content. A blog post published and linked from nowhere except a chronological listing is reachable in theory and treated as unimportant in practice, because nothing on your site suggests it matters.
Adding two or three links to a new page from relevant existing pages is ten minutes of work and it is regularly the difference between a page that gets crawled promptly and one that sits unnoticed for months.
What stops a page being found
| Cause | What it looks like |
|---|---|
| Nothing links to it | The most common. The page exists and there is no route to it |
| Blocked by instruction | A file on your site tells crawlers not to visit. Frequently left over from a build |
| Marked not to be indexed | A tag telling engines to exclude the page. Also usually a leftover |
| Only reachable by search or form | Pages behind a site search or a filter have no link to follow |
| Requires a login | Crawlers do not have accounts |
| Only appears after scripts run | Content assembled entirely by JavaScript is handled less reliably |
The first three account for the overwhelming majority, and all three are discoverable in a few minutes using the inspection tool in the search console, which will tell you directly whether a page is indexed and why not.
The leftover that catches new sites
Worth calling out on its own because it happens constantly.
During a build, the site is usually blocked from search engines so an unfinished version does not appear. That block is meant to be removed at launch. When it is not, the site goes live and is completely invisible, and because everything looks correct to a human visitor, nobody notices for weeks.
If a site has been live for a month with no search traffic at all, this is the first thing to check, before anything else.
How often pages are revisited
Not on a schedule you control. Engines revisit based on how often a page changes, how important it appears, and how much crawling the site can absorb.
A homepage updated regularly may be visited daily. A service page unchanged for a year may be visited every few weeks. That is normal and it explains why an edit does not appear in results immediately, and why requesting a recrawl for an important change is worth doing rather than waiting.
For a small site none of this needs managing. Crawl capacity becomes a consideration on sites with tens of thousands of pages, not on one with fifteen.
What is worth doing
- Have a sitemap and submit it once, particularly on a new site.
- Link new pages from existing ones, which is the fastest and cheapest discovery aid available.
- Check nothing is blocking the site, especially after a launch or a redesign.
- Make sure your listing links to your site, which provides an early external route.
- Use the inspection tool when a page seems missing, rather than assuming it is a ranking problem.
Discovery is the part of search that is genuinely mechanical. Once a page is found, whether it ranks is a different and much harder question, but nothing can rank that was never reachable in the first place.
Frequently asked questions
How do search engines find a new website?
By following links. A crawler reads pages it already knows, notes the links, and visits those next. A new site is usually found through its business listing, directories, or a sitemap submitted through the search console.
Do I need to submit my site to search engines?
There is no general submission process any more. Submitting a sitemap through the search console is the closest equivalent and it is worth doing on a new site, since the natural routes in are thin.
Why is my page not being found?
Most often nothing links to it. After that, a leftover instruction blocking crawlers or a tag telling engines not to index it. All three are visible in the search console inspection tool within a few minutes.
How long does it take for a new page to appear?
Days to a few weeks, depending on how established the site is and whether anything links to the page. Adding internal links from existing pages is the fastest way to speed it up.
What is the most common reason a new site is invisible?
A block left in place from the build. Sites are usually hidden from search engines during construction, and when that is not removed at launch the site is completely invisible while looking perfectly normal to visitors.
Do I need to worry about crawl budget?
Not on a small site. It becomes a consideration on sites with tens of thousands of pages, not on one with fifteen or fifty.
West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.
Site live and getting no search traffic at all?
We check whether it is a discovery problem or a ranking one, which are different faults with different fixes.
Start a Conversation