Folders on the Website That List Every File

A search result nobody expected

A lawyer searches for her own office's name to see how the listing looks. Fourth on the page is a plain result titled Index of /uploads.

It opens a bare page listing file names, dates and sizes: letters, draft agreements and scanned forms, each put on the website at some point to share with a client by link.

Nobody linked to the list itself. The server made it, and Google found it.

Why a folder lists its files

When a browser asks for a page, it usually names a file. When it asks for a folder, the server looks inside for a home page, usually a file called index.

If there is none, many servers are set to build a page listing everything in the folder, with a link to each file. That page is titled Index of, followed by the folder's name.

Search engines treat it like any other page. Once one link leads to the folder, or to a file inside it, the crawler can follow the list to every document.

How private files get there

Private files usually reach the public website for practical reasons:

• Sharing by link, because the file was too large to email.
• The media library, which stores every upload in public folders.
• Old project folders left behind after a redesign or a move.
• Backups saved on the server, sometimes with the whole database inside.
• Upload forms that save what visitors send where anyone could reach it.

In WordPress, for example, every upload goes into a dated folder under uploads, and those folders have no home page of their own. On a server that lists folders, each month's files are on show.

Finding the listings

Search Google for your domain with the site: operator and the words index of. Any folder listing Google has found appears with that title.

Then type the address of each folder you know of directly: uploads, images, files, documents, downloads. A listing appears at once if the server allows it.

Your developer or host can check the server setting directly and scan the site for folders that hold private files.

Switching listings off

Directory listing is a server setting. On most shared hosting it can be switched off for the whole site with one line in a configuration file, or an option in the control panel.

Once it is off, a folder without a home page returns a forbidden message instead of a list. Test it by visiting the same folder addresses again.

An empty index file in each folder hides the list too, but only folder by folder, and only until someone creates a new folder without one.

Hiding the list does not hide the files

Switching off the listing removes the list, not the files. Every document stays where it was, and anyone who has its address, or finds it in a search result, can still open it.

So the private documents have to move. Take them off the public website entirely, then ask Google to drop their addresses with the Remove URLs tool in Webmaster Tools.

Blocking the folder in robots.txt is not a fix. It asks search engines not to crawl the folder, does not remove what they already found, and itself names the folder for anyone to read.

A worked example

A two-person laser cutting service in a small business park cuts signs, brackets and gift items from wood, acrylic and sheet metal, mostly from files its customers send in.

To share proofs and quotes, the owner uploaded PDFs to a folder on the website and emailed each customer the link. Over three years the folder filled with hundreds of files.

A customer mentioned that a search for the business had shown a list of other customers' drawings. The folder had no home page, and the server listed everything in it.

The developer switched off directory listing for the whole site, moved every proof off the website, and used the Remove URLs tool for the addresses Google had found.

Proofs now go out through a document-sharing service that asks each customer to sign in, and links are withdrawn once a job is finished.

The owner also wrote to the customers whose drawings had been listed, explaining what had happened and what had changed.

Sharing private files safely

A file shared by an unguessable link is only as private as the link. Links are forwarded, saved in email archives and sometimes found by search engines.

For documents that matter, use a system built for it: one where each person signs in, files can be removed or set to expire, and you can see who opened what.

For a professional office, that is often part of its practice software or a client portal your developer sets up. Either way, the login is checked on the server.

If personal information was listed

If the folder held other people's personal information, ask your lawyer what you need to do next and whether anyone must be told. This is general information, not legal advice.

Lawyers, accountants and other regulated professionals should also check what their professional body expects.

Your host can usually provide the server's access logs, which show which files were downloaded, when, and roughly from where. That tells you how far the files travelled.

The counter-case

Some folders are meant to be public. A folder of brochures, manuals or price sheets that anyone may download does no harm if it is listed.

A page written for the purpose still serves visitors better than a raw list: titles in words, a line about each file, and its size before they tap.

And a public folder still should not sit beside private ones. Keep the public files in their own place, with a page, and switch listings off everywhere else.

Closing the folders

1. Search Google for index of pages on your domain.
2. Visit each folder address you know of directly.
3. Switch off directory listing for the whole site.
4. Move private files off the public website.
5. Ask Google to remove the addresses it already has.
6. Share documents through a system with sign-in.
7. Check the access logs to see what was downloaded.

Step four protects the people named in the files. The other steps hide the list and clear the results, but a file still on the server opens for anyone with its address.

Files that sit behind a login page yet still open for anyone with the address were the problem in a dealer price list anyone can open.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.