Numbers That Move Without Meaning Anything

The arithmetic of small numbers

Six enquiries becoming nine is a fifty percent increase.

It is also three enquiries, which is one customer changing their mind and two people who happened to phone this month rather than last.

The percentage is arithmetically correct and it describes something that would have happened anyway.

Which is the central problem with small business measurement: the numbers are small enough that ordinary variation produces changes that look dramatic.

What a normal range is

The single most useful thing to establish about any figure you watch.

Take twelve months of it and note the highest and lowest values.

That spread is what the number does on its own, without anybody doing anything.

Anything landing inside it is not news. Anything outside it might be.

Which converts every future reading from a question into an answer, and it takes ten minutes to work out once.

Where this misleads people

• Judging a change too early, on one month of data.
• Testing two versions on traffic too small to distinguish them.
• Reacting to a bad month that was within normal variation.
• Attributing a good month to whatever was done recently.
• Comparing percentages without stating the underlying counts.

The fifth is worth watching for in anything a supplier presents. A conversion rate improvement of thirty percent means something quite different on four hundred visits than on forty thousand.

A worked example

A business that changed its homepage and saw enquiries rise from seven to eleven the following month.

The change was declared successful and the same approach applied to two other pages.

Looking at twelve months of the figure afterwards, the monthly count had ranged between five and twelve throughout the year, with no changes made to the site at all.

Eleven was inside the normal range. The change may have helped, and nothing about the following month demonstrated it.

What would have demonstrated it: a sustained level above the previous range over three or four months, which is what they eventually had and could have waited for.

The homepage change turned out to be genuinely good, and the reasoning used to conclude that had been wrong.

Why this matters more than being pedantic

Because acting on noise costs real money.

A business that concludes a change worked, on one good month, will apply the same approach elsewhere, and some of those applications will make things worse.

One that concludes a change failed, on one bad month, will reverse something that was working.

And a supplier judged on a single quarter may be dismissed or retained for reasons unconnected to their work.

The cost is not a misunderstanding. It is a series of decisions made on evidence that was never there.

Why bigger businesses can do things you cannot

Worth understanding, because a good deal of marketing advice is written for a scale small businesses do not have.

A company with forty thousand conversions a month can detect a two percent difference reliably, and can test several things at once.

A business with eight enquiries a month cannot detect a fifty percent difference in under a year.

Which means the techniques that work at scale are not slower at small scale. They do not work at all.

The substitute is judgement: making changes there is good reason to believe in, based on what visitors evidently need, and evaluating them over quarters.

That is less satisfying than a test result and it is the honest method available at this size.

What to do instead

Wait longer and look at more of it.

Three or four months of a figure sitting above its previous range is evidence. One month above it is not.

Compare against the same period last year rather than against last month, which removes the seasonal component.

And look at the series rather than the latest value, since a chart of twelve months shows immediately whether a change is a step or a wobble.

None of that requires any statistics beyond looking at a column of numbers.

Testing two versions

Worth addressing specifically, since it is frequently recommended to businesses far too small for it.

Distinguishing a genuine difference between two versions of a page requires a substantial number of conversions on each, not a substantial number of visits.

A business with eight enquiries a month cannot detect anything short of an enormous difference, and would need many months to do so.

Which means split testing is not a small business tool, whatever the software suggests.

The alternative is making changes you have good reason to believe in, one at a time, and judging them over quarters rather than weeks.

Percentages without counts

A presentation habit worth insisting on, both for your own reports and for anybody else's.

Every percentage should appear beside the numbers it came from.

Conversions up thirty percent is unreadable. Conversions up thirty percent, from ten to thirteen, is immediately interpretable and immediately less impressive.

Which is precisely why percentages get quoted alone, and why a supplier presenting only percentages is worth asking about.

The habit costs nothing and it changes how a report is read, because the underlying counts tell the reader how much weight the percentage deserves.

Anybody unwilling to show the counts alongside is telling you something about the counts.

The counter-case

Where a single month does tell you something.

A sharp drop to near zero, which is breakage rather than variation and should be investigated immediately.

A change far outside anything the figure has ever done, in either direction.

Anything coinciding exactly with a known event: a site going down, a listing being suspended, a campaign starting.

The distinction is between a cliff and a wobble. Cliffs mean something and happen on a specific day; wobbles are the figure doing what it always does.

Establishing your ranges

1. List twelve months of enquiries.
2. Note the highest and lowest.
3. Do the same for traffic.
4. Write both ranges down.
5. Compare every future month against them.
6. Only investigate what falls outside.

The sixth saves the most time, because it removes the monthly exercise of explaining a movement that required no explanation.

The comparison that avoids this is covered in comparing this month to last month badly.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.