Deciding What Not to Measure

Measurement is not free

Adding a metric feels costless. Nothing is spent, the tool supports it, and more information sounds better than less.

It is not costless, and the costs are real enough to change the decision.

Attention is finite, and every figure in a report competes with the ones that matter. Configuration has to be maintained and breaks silently. Collecting more about visitors is a privacy cost you incur on their behalf.

And a number that cannot be acted on will eventually be acted on anyway, which is the largest cost of all.

The test

One question, asked before anything is configured.

What decision would this change, and at what value.

If you cannot name a decision, you are collecting for interest rather than for use, which is a legitimate hobby and not a reason to add a line to a report.

If you can name a decision but not a threshold, the metric will be watched and never acted on, because there will never be a moment that clearly requires action.

Both halves have to be answerable. Scroll depth below forty percent means we shorten the page is a complete answer. Scroll depth tells us how engaged people are is not.

What usually fails it

• Time on page, which nobody has a threshold for.
• Scroll depth on a site with twenty pages.
• Demographic breakdowns you cannot act on.
• Social followers, counted monthly.
• Search positions for terms nobody uses.
• Any rate calculated from a handful of events.

Each of those is genuinely useful somewhere, at a scale where the numbers are stable and somebody is employed to act on them. At forty enquiries a month they are decoration.

The false precision problem

The cost that does the most damage, because it looks like rigour.

A figure presented to one decimal place invites confidence it has not earned, particularly when it is derived from a small number of events.

A conversion rate of 3.4 percent, from eleven enquiries, is a number that will be compared against 4.1 percent next month and a conclusion will be drawn.

Nothing about that comparison is meaningful, and the presentation is what makes it look as though it is.

Round aggressively, show counts alongside rates, and refuse to report a rate where the underlying count is small.

Where somebody insists on the decimal place, that is a good moment to ask what they would do differently at each value.

A worked example

A business had a monthly report with fourteen figures, produced by a supplier.

Asked which had ever changed a decision, the owner could name two: enquiries, and where they came from.

The other twelve had been read, occasionally discussed, and never acted on, including four that nobody could define.

They cut the report to four figures and added two sentences of interpretation.

The immediate effect was that the report started being read, having previously been skimmed.

The second effect took longer to notice: discussions about the site became discussions about enquiries rather than about whichever figure had moved most, which changed what got worked on.

Refusing at the point of request

The practical skill, since most metrics arrive as a request rather than a decision.

Somebody asks whether you can track something, and the answer is almost always technically yes, which is how reports grow.

The useful reply is a question rather than a refusal: what would you do differently if it were high, and what if it were low.

That is not obstructive and it frequently produces a better version of the request, or a realisation that the interesting question is a different one.

Where the answer is that they just want to see it, agreeing to look at it once rather than adding it permanently is a reasonable compromise.

The one-in-one-out rule

The mechanism that keeps a report stable over years.

When something new deserves a place, remove something to make room.

That forces the comparison rather than deferring it, and it prevents the slow accumulation that turns four figures into fourteen over three years.

It also produces a useful conversation, since arguing about which existing figure to drop reveals which ones nobody was using.

Set the limit at whatever fits on one screen and treat it as fixed.

What changes the answer

Because this is a judgement about scale rather than a principle.

Volume changes it: at ten thousand visits a month, rates are stable and worth watching, and scroll depth on a landing page is a real signal.

Money changes it: anything with spending attached earns closer measurement, because the cost of not noticing is immediate.

Obligation changes it: a figure required by a client, a funder, or a franchise gets tracked regardless of whether you would choose to.

And a specific question changes it temporarily: measure something for three months to answer a question, then stop, which is different from adding it permanently.

The counter-case

Measuring too little has its own failure.

A business tracking only enquiries cannot tell whether a quiet month was fewer visitors, a broken form, or a worse conversion, and will spend a fortnight guessing.

Some measurement also earns its place by being cheap and occasionally decisive: a form submission test, a certificate expiry check, an uptime monitor.

There is also a version of this argument used to avoid accountability, where somebody declines to measure anything and therefore cannot be shown to have failed.

Track the few things that would change a decision, measure anything with money attached properly, and refuse the rest.

The decision

1. Name the decision a metric would change.
2. Name the value at which you would act.
3. Refuse it if either is unanswerable.
4. Round aggressively and show counts with rates.
5. Ask the requester what they would do at each value.
6. Remove one whenever you add one.
7. Measure temporarily to answer a specific question.

Step two is the one that removes most candidates, since a metric with no threshold will be watched indefinitely and acted on never.

Figures that fill a report without informing it are covered in vanity numbers in a monthly report.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.