The Report That Made a Bad Decision Look Good

How it happens without dishonesty

Almost nobody fabricates figures.

What happens is more ordinary. A change was made, somebody wants to know whether it worked, and there are a dozen ways to look at it.

Several of those ways show an improvement and several show a decline, because that is what noisy data does.

The ones showing improvement feel like the correct way to look at it, so they are the ones presented, and everybody involved is being sincere.

The error is not deception. It is choosing the method after seeing the results.

The window

The single most common source of a misleading comparison.

Compare last month against the previous month and you have compared two periods that differ in working days, holidays, weather and season, none of which had anything to do with your change.

A business whose trade rises every spring will show an improvement for any change made in February, and the same change made in October will look like a failure.

The correction is to compare the same period against the previous year wherever you have the history, since that holds seasonality roughly constant.

Where you do not have a year of history, say so, and treat the result as an indication rather than a finding.

The baseline

Choosing what to compare against is where the second layer of selection happens.

An unusually poor month makes anything after it look like recovery. An exceptional month makes everything after it look like decline.

Comparing against a single prior period, rather than an average of several, gives whoever picks the period substantial control over the conclusion without doing anything obviously wrong.

Use a longer baseline: the same period last year, or an average of the preceding three months, chosen and written down before the change was made.

The metric chosen afterwards

• Visits rose, but enquiries fell.
• Bounce rate improved, because tracking changed.
• Time on page rose, because the page got harder to use.
• Impressions grew, while clicks stayed flat.
• Followers increased, while nothing else moved.

Each of these is a real number and each has been used to declare success. The tell is that the metric was not named before the change.

Say what would count as failure, first

The discipline that prevents nearly all of this, and it takes a sentence.

Before making a change, write down the measure, the period, and what result would mean it had not worked.

We are changing the enquiry form, we will compare enquiries over the following eight weeks against the same eight weeks last year, and fewer enquiries means it failed.

Having written that, no amount of subsequent searching for a favourable angle can be mistaken for analysis.

The step is uncomfortable precisely because it removes the escape route, which is why it works.

A worked example

An agency reported that a homepage redesign had increased engagement by a substantial margin.

The comparison was the six weeks after launch against the six weeks before, and the metrics were sessions and pages per session, both up considerably.

Two problems.

The before period included the Christmas fortnight, which for that business was its quietest of the year.

And pages per session had risen because the new navigation required an extra click to reach the service pages, so people were visiting more pages to do the same thing.

Enquiries, which nobody had put in the report, had fallen by about a fifth.

Nothing in the report was false. The window flattered it, and the metric that mattered had been left out, apparently without anybody deciding to leave it out.

Look for what is missing

The most reliable way to read a report critically, and it works even when you do not know the subject.

Ask what is not shown. If a report about a website change does not mention enquiries or sales, ask why.

Ask what the same numbers looked like in the two preceding periods, which turns a single comparison into a trend and usually deflates a dramatic one.

And ask what the person would have shown if the change had failed. A report with no possible failing version is advocacy rather than measurement.

Small numbers move a lot

A specific trap for small businesses, and it produces both false triumphs and false disasters.

Going from four enquiries to six is a fifty percent increase and is also two enquiries, which is entirely explicable by one busy week.

Reporting it as a percentage makes ordinary variation look like a result.

At low volumes, quote absolute numbers alongside any percentage, and be suspicious of any percentage where the underlying count is in single figures.

The trend that was already there

The subtlest version of this, and it survives even a careful year-on-year comparison.

If enquiries had been rising steadily for eighteen months before you changed anything, they will very likely keep rising afterwards, and the change will appear to have caused it.

The same works in reverse and is more damaging: a change made during a gentle decline gets blamed for a fall that was already underway, and something worth keeping gets reversed.

The remedy is to look at the shape before the change rather than a single prior period. Plot the preceding six or twelve points and ask whether the line after the change actually departs from the line before it.

Frequently it does not, and the honest conclusion is that the change made no detectable difference, which is a legitimate finding and the one most reports are structured to avoid reaching.

The counter-case

Rigour can be taken to a point where nothing is ever concluded.

Small businesses cannot run controlled experiments, will rarely have clean year-on-year comparisons, and cannot wait for statistical certainty before making decisions.

Demanding proof at that standard means never acting, which is worse than acting on imperfect evidence.

The workable position is to decide the measure in advance, accept the answer, and describe the confidence honestly. It looks better, on a comparison that has a seasonal problem, is a legitimate and useful sentence.

What is not acceptable is presenting an uncertain result as settled because it happened to support what somebody wanted to do.

Reading any report

1. Check the comparison period and what is in it.
2. Ask whether the metric was chosen before or after.
3. Look for the outcome number: enquiries, sales, jobs.
4. Ask for the two preceding periods as well.
5. Check absolute counts behind every percentage.
6. Write down the measure before the next change.

Step six is the only one that prevents the problem rather than detecting it.

Reading a report somebody else built is covered in reading a report you did not set up.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.