The robots file tells crawlers which parts of a site to leave alone. It is a small file capable of removing an entire website from search results, and it does so silently, which makes it one of the few technical items worth checking directly rather than assuming is fine.
What it is and is not
A plain text file at the root of a domain giving instructions to crawlers. It is a request that reputable crawlers honour voluntarily, not an access control.
The consequence people miss: it is not a security measure. Listing a private folder in your robots file tells every reader of that public file exactly where the folder is. Anything genuinely sensitive needs authentication, not a polite request.
A second consequence: blocking a page prevents crawling, not indexing. A blocked page linked from elsewhere can still appear in results, without a description, because the crawler was not permitted to read what it contained. If you want a page kept out of results, a noindex directive on the page itself is the tool, and that requires the page to be crawlable.
The mistake that matters most
A development or staging site is blocked from crawling so unfinished work stays out of search. When the site goes live, that block is supposed to be removed.
Frequently it is not, and the result is total: the site disappears from search results entirely. Nothing looks wrong on the site itself, so it is commonly discovered weeks later when someone asks why enquiries stopped.
This is the single most expensive avoidable technical error in web work, and it appears in redesigns that hurt SEO for the same reason. Check the live file on launch day, not the staging one.
The other recurring errors
Blocking resources the page needs
Disallowing folders containing stylesheets, scripts, or images prevents search engines rendering the page as visitors see it. This was common practice years ago and is now actively harmful, since a page that cannot be rendered cannot be assessed properly.
Using it to hide pages from results
As above, blocking prevents reading rather than listing. Pages you want excluded need to be crawlable and carry a noindex directive.
Listing sensitive paths
The file is public. A disallow rule for an admin area or a private directory is an index of where to look.
Syntax that does not do what was intended
A disallow rule with no path blocks nothing. A disallow rule with a single slash blocks everything. The difference is one character and the outcomes are opposite.
Rules under the wrong user agent
Directives apply to the group they sit under. A rule intended for all crawlers placed under a specific one applies only to that crawler, which is a common cause of rules that appear correct and do nothing.
Forgetting the sitemap line
Not an error exactly, and a missed opportunity. A sitemap reference in the robots file is a standard way to point crawlers at it.
What belongs in a typical small business file
Very little, which surprises people expecting something elaborate.
Most small sites want everything crawled. A reasonable file permits all crawlers, disallows any genuinely non-public area that is already protected by other means, and points at the sitemap. Anything longer usually reflects copied advice rather than a considered decision.
Decisions about AI crawlers also live here, and those are worth making deliberately rather than by default, as discussed in the website audit.
How to check yours
- Visit it directly by adding /robots.txt to your domain. It is public, so this takes seconds.
- Read it line by line. Most files are short, and most problems are visible on inspection.
- Use Search Console's testing tools to confirm specific pages are crawlable.
- Check it after every launch, migration, or hosting change. These are the moments the file gets replaced with a default or a leftover.
Two minutes, occasionally, against the possibility of vanishing entirely. Few technical checks have that ratio.
Frequently asked questions
What does robots.txt do?
It gives crawlers instructions about which parts of a site to leave alone. It is a voluntary request that reputable crawlers honour rather than an access control, so it should never be relied upon to protect anything genuinely sensitive.
Can robots.txt hide a page from search results?
Not reliably. Blocking prevents crawling rather than indexing, so a blocked page linked from elsewhere can still appear in results without a description. Keeping a page out of results requires a noindex directive, which means the page must remain crawlable.
What is the most damaging robots.txt mistake?
Launching a site with the crawler block that was applied to the staging version still in place. The site disappears from search results entirely while nothing appears wrong on the site itself, so it is frequently discovered only weeks later.
Is it safe to list private folders in robots.txt?
No. The file is publicly readable, so a disallow rule for an admin area or private directory tells anyone who looks exactly where to find it. Genuine protection requires authentication rather than a request not to look.
Should I block CSS and JavaScript files?
No. Blocking stylesheets, scripts, or images prevents search engines rendering the page as visitors see it, and a page that cannot be rendered cannot be assessed properly. This was once common practice and is now actively harmful.
What should a small business robots.txt contain?
Usually very little: permission for crawlers, any genuinely non-public area that is already protected by other means, and a line pointing at the sitemap. Anything longer typically reflects copied advice rather than a considered decision.
West Coast Media Solutions Inc. builds and maintains websites for businesses across Canada. Technical behaviour described here reflects how search engines and browsers currently work; specifics change, and your own configuration is what determines the outcome on your site.
Something technical not behaving?
We build and maintain websites for businesses across Canada, and we sort out this kind of thing without the drama.
Start a Conversation