The Chatterbox

Conversion Testing 101 - Chapter 3

Continued from Chapter 2

In this entry we'll look at how we can make sure we have a large enough sampling of data to insure our results are something in which we can have confidence. This is where a lot of people seem to get lost, and it's easy to see why. When you're doing it by hand, this is the point where you have to start getting into some pretty heavy duty mathematical equations, formulas and functions. Things like computing Standard Deviation and the like.

This is where I'm going make a concerted effort to do everything I can to take out of the process for you. Starting with, I'm not even going to point you to any references for what the Standard Deviation formula looks like or how to incorporate it into your tests. Bottom line, unless you're a math geek or incredibly detail oriented you won't gain anything by either seeing or trying to understand the formula.

It'll just clutter up your mind and distract you from more important things.

So instead I'm going to be building all of these sorts of things into my tool so that you don't have to worry about them in the least.

There are a few numbers you will need to provide that will be plugged into the various formulas. But they're easy to come up with, so don't let it scare you in the least. Here's all you'll need to feed the tool to get the ball rolling:

The Minimum Conversion Difference You Wish To Detect
Let's say your site/page is converting at a 1% clip at the moment. You want to test to improve this, however you need some sort of minimum threshold that would be worth changing your site over. So you decide there needs to be a Minimum gain of .5% in conversions in order to make mass changes.

Note: The smaller the minimum difference is, the larger your required sample size in order to obtain valid and conclusive results. So a .25% difference is going to require more data in order to gain more specificity. Conversely, a .75% minimum difference would require a smaller sample size, but your end results will not be quite as specific. I find .5% to be a very good place to start in most cases. Though I will sometimes go lower when I get to the final stages of re-testing certain elements of a page.

The Average Success Rate of Your Control Page
This one is easy, but sometimes not so easy, depending upon your current stats program and how it reports things. It's your current conversion ratio. Basically, it's the number of sales per 100 unique visitors to your site/page as things stand before you begin testing. So if you're getting 1,000 new unique visitors per day and averaging 10 sales per day, your current conversion ratio is 1%. Or if you need to figure it out for your own site, it would be the number of sales per day (for example 10) divided by the amount of new, unique visitors (1,000) converted into a percentage.

The Degree of Confidence
This one is a personal choice you'll need to make. In a nutshell, how much Confidence do you want to be able to have in your test results? Generally speaking you'll want somewhere between a 90% and 95% Degree of Confidence. Anything less, and you won't be able to be sure the changes you're making are supported by valid and conclusive data. Anything more and your test will take longer and longer to complete.

Think of it like you might all of the political polls you see on TV. Most of them use a standard ~5% margin of error. This means the pollsters have 95% confidence in their results.

Note: The lower your Degree of Confidence, the smaller your sample size can be. Conversely, the higher your required Degree of confidence, the larger sampling of data required. I personally find 95% to be a pretty good choice, though when I'm testing radical design changes I'll sometimes take it down to as low as 90%. I never, ever go below 90% confidence. The tool will allow a Degree of Confidence from 80% all the way up to 99%, not that I would recommend either extreme.

New Arrivals Per Day
This is simply the number of new, unique visitors who typically hit your page/site each day. This one is only used to give you an estimate of how many days it will take to get a large enough sampling of data, so doesn't have to be 100% accurate. At the end of the day the tool will be looking at the Actual number of impressions, but by providing this number in a reasonably accurate form will give you a rough idea of how long your test will need to run in order to produce valid results.

Number of Treatments, Including Control
Simply the number of treatments you'll be testing. In our example this number would be 4. Or our Control page, plus our three new headlines.

Okay, let's roll it all together now to see just how easy it is.

Here's a link for the beginnings of the testing tool. (Will open in a new browser window.) For your reference, I'm also including a screen capture of what the tool will look like after we've entered our basic testing information.

1. We want to be able to detect a minimum conversion difference of half of one percent, or 0.5%. So we plug 0.5 into the first line.

2. Our current page is converting at 1%, or 1 out of every 100 new, unique visitors. This goes in line 2.

3. We want a 95% Confidence level in our results. So we plug 95 into line 3.

4. According to our stats program our current landing page is averaging 1,000 new, unique visitors per day. This number goes into the 4th line.

5. When we set up our example test (in the last chapter) we decided that we're going to test three new headlines against our control. So a total of 4 in the Number of Treatments section.

From there hit the Proceed button and the tool will start doing its magic. Not that it's really important for you to know, but it applies the Standard Deviation formula to the numbers we've entered, performs some addition, subtraction, multiplication and division, then spits out some preliminary information for us.

For our example test, it looks like we'll need a total of 3,043 impressions from 3,043 different people per headline treatment in order to have a sufficient sample size to make some determinations. This equates to 12,172 total impressions across all four of our headline versions, with the assumption that each will get an equal number.

If all of the numbers we entered are right, or at least close, the tool tells us that we'll need to let this particular conversion test running for 12 days in order to collect enough data to make an accurate assessment.

The tool also puts a variable I've called Current Control Successes Per Day into the page to give you an easy check of your current closure rate. Most people don't know exactly what their current conversion rate is running at, but most do know how many sales per day they average. So this quick check can let them know if the Average Success Rate needs to be tweaked.

Got it? See how it works?

Play with the tool a bit, changing different numbers a little bit at a time. For instance, see what happens when you reduce your Confidence Level to say 90%. Or increase it to 98%. Or change your Minimum Conversion Difference detection to 0.25%.

This will give you a sense of how all of the numbers work in conjunction with one another. They all depend upon each other to some degree.

Throw in the necessity of applying some fairly sophisticated mathematical formulas and it's easy to see why so many people get lost at this stage of the game if they have to do it by hand. It's simply too much, and everything depends upon everything else. My goal in designing a new tool is to keep this confusion and fall out from happening.

Next up: We'll skip over the nuts and bolts of the data collection side of things for the time being since it'll all be built into the final tool, and start looking at what happens after we've started collecting data. In short, how to determine the winners and losers, and making sure we have valid, conclusive results.