Conversion Confidence Tool

This tool is designed to help you determine the level of Confidence you should have in your conversion test data. It is basically the flip side of the coin from the Conversion Testing Tool in that the CCT tool gives you a pre-test estimate of the amount of traffic and time it will take to reach a conclusive result, while this tool works with your real testing and conversion data to provide you with the Confidence level you should have in the real results.

This tool is designed to show you the Confidence you should have in the results from a typical two-element A/B Split Test where one of the factors is the Control and the other is you Test Treatment. If you had run a multiple page A/B Split Test you'll want to use your original page as the Control and your Winning page as the Test page.

The same is basically true for full factorial (Multivariate) and fractional factorial (Taguchi) tests where you are testing a number of elements of a page or pages. These multi-element tests should give you a single Best Case result, combining several divergent elements. You'll want to enter the conversion data from this Winning page as the test page, with your original page being the Control.

When you conduct a Multivariate or Taguchi test you should always conduct a final two-element A/B split test between the Winning elements and your Control. There are a myriad of reasons why, but the main one is to make sure you're getting true, actionable data. This Final Test data is what you'll enter into this tool

- Usage -

Simply enter the Conversions and Impression of your Control page as recorded by your stats program, along with the Conversions and Impressions of your Test page. The tool will highlight your leader, show you the conversion rate for each treatment and provide you with the mathematical confidence you should have in the results.

Conversions of Control:
Impressions of Control:
Conversions of Test:
Impressions of Test:

- Intrepretation -

This is all about the amount of Confidence you can have in the results of your conversion testing. Plain and simple.

As a for instance, let's say you ran a test and the tool tells you there's a 98% Confidence rating for your Test Treatment. What this means is there is a 98% chance your new Test Treatment will outperform your original Control. Those are pretty darned good odds, so you'd want to run with it.

Could you continue running your test until the confidence number reaches 100%? You could, but to achieve such absolute certainty could require millions or billions of impressions, and take months or years to complete. All the while you'd be losing conversions on the half of the traffic that goes to your Control Treatment. Obviously this isn't a good idea.

Now that said, what exactly should you consider a Safe Confidence rating before you flip the switch and make your Test treatment the clear winner. As a general rule, I like a 95% Confidence rating. This is the mathematical standard used throughout the testing industry. Not just with web site conversion testing, but with all sorts of tests, polls, etc.

You may ask why everyone keys to 95%. Or to put it another way, if there's an 85% chance the Test Treatment will convert better than the control, why not go ahead and flip the switch? After all, 72% does look to be a pretty big number.

This next part is often misunderstood and is really, really important so make sure you read the next couple of paragraphs a time or two, or until you've committed it to memory!

What you need to remember with Confidence numbers however is that every test starts out with an assumption of a baseline 50% Confidence rating for each of the treatments. Or put another way, at the beginning of the process before any traffic hits either of yout test pages they each have an equal chance of proving to be the winner. So it's a 50/50 situation with each of your two treatments having a 50% baseline Confidence rating.

Viewed in this light, a 72% Confidence rating that looked so impressive before is not even halfway to the absolute 100% Confidence rating we'd like it to have. It's not all that much higher than its baseline 50% Confidence rating. That's shaky ground indeed !

So you're best off to stick with a higher Confidence rating before you start declaring a test page a clear winner.

If you run into a situation where your test doesn't look like it's ever going to reach 95% no matter how long you let it run (hey, it happens to all of us!) simply end the test and start another. Try to pick up on the elements that did appear to perform a little better as a starting point for your new test. And also try to get more divergence between your test elements. Many times the inability to attain an acceptable Confidence rating comes down to the test elements being too similar. You really need to concentrate in making them as different as possible.