Backups You Have Never Restored

Two different questions

Whether a backup is valid is one question, and a good many businesses have now checked it.

How long a full recovery would take, and whether anybody except one person could perform it, is a different question and almost nobody has answered it.

That second answer is what determines the cost of an incident, because the loss is measured in hours of downtime rather than in whether a file was readable.

The way to find out is to rehearse it once, deliberately, at a quiet time, with a clock running.

The number nobody knows

Ask any small business how long their site would take to bring back and the answer is a guess, usually optimistic.

The real figure includes finding the backup, working out which one to use, obtaining access to wherever it is going, the restore itself, reconnecting the domain, reinstating certificates, and testing that forms and payments still work.

For a typical small site that is frequently most of a day, and occasionally several, and the guess is usually an hour.

Knowing the real number changes decisions: whether the current arrangement is adequate, whether it is worth paying somebody to hold a warm copy, and what to tell customers when it happens.

It is also the only way to answer the question your insurer or a client will eventually ask.

Restore somewhere else

The mechanical requirement, and the reason this is safe to do in December.

Never restore over the live site as a test, since a failed restore then produces the outage you were rehearsing for.

Restore to a separate location: a staging area, a temporary subdomain, a local environment, or a new hosting account for the afternoon.

That lets you break things freely, take as long as you need, and abandon it if something goes wrong.

Most hosts provide somewhere to do this, and where they do not, a cheap account for a month is a reasonable cost for the information.

What the drill reveals

• How long it actually takes, end to end.
• Which credentials nobody could find.
• What was not in the backup.
• How old the most recent usable copy is.
• Whether anybody else could have done it.
• Which steps depend on somebody answering a phone.

The last is the one that surprises people. A recovery requiring a support ticket, a domain transfer, or a certificate reissue depends on somebody else's working hours, which is why an incident on a Friday evening is a Monday problem.

What is usually not in the backup

The finding that makes the drill worth an afternoon.

Backups typically cover the site files and the database, and stop there.

Not included: domain configuration and where it points, email, certificates, third-party services the site depends on, licence keys for paid components, and any settings held in a hosting control panel rather than in the site itself.

A restored site with the wrong domain settings and no working email is not a recovered business.

Write those down as part of the drill, in one document, because they are the parts that will be reconstructed from memory at the worst moment otherwise.

A worked example

A firm with nightly backups did a restore for the first time on a quiet afternoon in December.

The backup restored cleanly, which they had expected.

What took the time was everything around it: nobody knew the registrar login, the certificate had to be reissued, two paid components would not activate because the licences were on a former supplier's account, and the contact form pointed at a mailbox that no longer existed.

The elapsed time was around five hours against an expectation of one.

They wrote it down as they went, which produced a two-page procedure, and they fixed the licence and mailbox problems the same week.

The following year's drill took ninety minutes, which was the point of doing the first one.

Write the runbook as you go

The output that matters more than the timing.

Record each step as you perform it: where the backup is, how to reach it, which credentials are needed, what order things happen in, and what to check afterwards.

Written during the drill it takes no extra time. Written afterwards from memory it is wrong.

Keep it somewhere that does not depend on the systems it describes, which rules out the site itself and any document only reachable through a login that might be unavailable.

A printed copy in a drawer is not old-fashioned, it is the version that works when nothing else does.

Have somebody else do it

The variation that answers the question the drill exists for.

If the person who normally handles this performs the restore, you learn how long it takes them, which is not the situation you are worried about.

The useful drill has somebody else follow the written procedure, with the usual person available but not intervening.

Every point where they get stuck is a gap in the document, and those gaps are exactly what would stop a recovery happening while the usual person is unreachable.

For a sole trader, the equivalent is giving the document to somebody who could act on your behalf and asking whether they could follow it.

The counter-case

A full drill is disproportionate for some businesses.

A simple site on managed hosting, where the host restores from their own snapshots in minutes, does not need an annual rehearsal, and the useful check there is confirming the snapshots exist and knowing how to request one.

There is also a risk in over-preparing for the wrong failure. Most small businesses will never lose a site entirely, and are considerably more likely to lose access to an account, a mailbox, or a domain.

Those deserve the same attention and get less of it.

Scale the exercise to what the site actually does. If it takes orders, rehearse properly. If it is a brochure, know how to ask your host and where your content is.

The afternoon

1. Pick a quiet day, not a busy one.
2. Restore somewhere else, never over the live site.
3. Start a clock.
4. Write each step down as you do it.
5. Note what was missing from the backup.
6. Fix the gaps the same week.
7. Have somebody else follow the document next time.

Step three produces the number that everything else depends on, and it is almost always several times the guess.

Whether the backup itself works is covered in backups nobody has tested.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.