Comparing Two Versions Properly

When a split test is not available

The proper method shows two versions simultaneously to comparable audiences, which removes almost everything that could otherwise explain a difference.

It also needs enough traffic for the difference to be distinguishable from noise, which most small sites do not have.

So the practical alternative is to change the page and compare the period after against the period before.

That is a legitimate method with a specific weakness: everything else also changed, and the discipline is in accounting for it.

What else moved

• The season, which is the usual answer.
• The weather, for anything outdoor or urgent.
• A holiday in one period and not the other.
• Your own advertising spend or targeting.
• A competitor starting or stopping.
• Anything else you changed that week.

The last is the most controllable and the most commonly ignored. Businesses change three things in a fortnight and then attribute the result to whichever one they were most interested in.

Change one thing

The rule that makes the comparison mean anything at all.

Where two changes go live together, the result is about both, and no amount of analysis afterwards separates them.

That is frustrating when both changes seem obvious, and it is the price of being able to learn anything.

Where you must do several, do them in sequence with a gap, and accept that the whole exercise takes months rather than weeks.

Or make one large change and accept that you will learn whether the package worked rather than which part did, which is a legitimate choice for a small site.

What does not work is several changes at once followed by a confident conclusion about one of them.

Compare whole weeks

Since days are not comparable to each other.

Traffic and enquiries vary enormously by day of the week, so a Tuesday to Tuesday comparison is meaningless.

Use four complete weeks before and four complete weeks after, at minimum, and prefer longer where the volume is low.

Avoid periods containing a public holiday on one side only, or note it explicitly if you cannot.

Start the after period a day or two following the change, so the day of the change itself is in neither.

Whole weeks is the single easiest improvement to this method and is skipped constantly.

Use last year as a control

The technique that addresses seasonality and almost nobody uses.

Look at the same two periods in the previous year, when the page had not changed.

Where enquiries also rose by a similar proportion last year, the season explains it and your change may have done nothing.

Where last year was flat and this year rose, that is considerably stronger evidence.

This requires having the figures from last year, which is the argument for keeping analytics running even when nobody looks at it.

It is not proof, and it is the difference between a guess and an informed one.

A worked example

A business rewrote a service page and saw enquiries rise by about forty per cent over the following month.

Before concluding, they checked three things.

They had also increased advertising spend that fortnight, which they had forgotten.

The same period the previous year had risen by about twenty per cent without any change, which is the seasonal pattern for that trade.

And one of the four weeks contained a holiday that had been in the before period the previous year.

Their honest conclusion was that the page probably helped somewhat and that the forty per cent was mostly other things, which is a more useful conclusion than the one they nearly reached.

Measure the right number

Since the wrong one produces confident errors.

Measure enquiries, or sales, rather than time on page, scroll depth, or clicks on a button.

Those intermediate numbers move for reasons unrelated to whether anybody bought, and a change that increases clicks and reduces enquiries is a common outcome.

Also watch quality, since more enquiries of worse fit is not an improvement, and that shows up in the conversion rate from enquiry to job rather than in the site figures at all.

Where volume is too low to compare enquiries, that is a signal that the method cannot resolve this change rather than a reason to measure something else.

Write down the prediction first

A small discipline that changes how honestly the result is read.

Before making the change, write what you expect: which number, in which direction, by roughly how much.

Then compare against that rather than against whatever the figures did.

Without it, any movement gets explained as a success, since a rise confirms the change and a fall is attributed to the season.

Writing the prediction takes a minute and is the difference between testing and confirming.

Some things can be split tested cheaply

Worth knowing, since the traffic problem does not apply everywhere.

Paid advertising platforms will run two versions of an advertisement against each other automatically, and that comparison is genuinely simultaneous.

The same applies to email subject lines on any list of reasonable size, where most tools will split the send without any setup.

Neither tells you about your pages, and both let you learn something about wording and offers with a proper method rather than a compromise.

Where a question is really about the message rather than the page, test it in the place where testing is easy and apply the answer to the page afterwards.

The counter-case

This method is genuinely weak.

Sequential comparison cannot separate a change from everything else that happened, and no amount of care fully fixes that.

At low volume the noise frequently exceeds any effect, and businesses have made and reverted good changes on the strength of six enquiries.

Where a change is obviously right, such as a broken form or a missing price, make it without measuring, since the measurement adds nothing.

Change one thing, compare whole weeks, check last year, and hold the conclusion loosely.

The method

1. Write the prediction before changing anything.
2. Change one thing.
3. Note everything else happening that month.
4. Compare four whole weeks each side.
5. Check the same periods last year.
6. Measure enquiries, not clicks.
7. Hold the conclusion loosely.

Step five is what separates a seasonal rise from an improvement, and it costs one report that most businesses already have and never open.

How much traffic a proper test needs is covered in how much traffic before a test means anything.

Frequently asked questions

West Coast Media Solutions Inc. provides web design, web development, hosting, digital marketing, and business consulting to organisations across Canada, drawing on more than twenty-five years in the field.