What is A/B Testing? and Why Do Website Owners Care?

Back before there was an Internet, there was direct mail.  Many of the principles that we know about the Internet, and especially about testing websites, come from mail. Before we learn about how cool online A/B testing is (and even better, online multivariate testing), let's pause to understand the basics.

Direct mailers usually have a mail piece that works well for them.Then, with each mailing, they siphon off a portion of their mail audience from this "control" mail piece, and these other people instead get a test mail piece. One test “cell” might get a piece with a different headline. Another might get a wedding envelope instead of a #10 business envelope. Each cell might get assigned a letter (A, B, C) or for that matter a name -- whatever helps the marketing team do their job (and that's why we call it A/B testing, although A/B...N testing is a better name - you aren't limited to two cells.) The overriding principle is one that we all learned in eighth grade: you can only change one variable at a time if you are going to learn anything useful.

Enter the Internet, the most amazing of all direct response vehicles. Like direct mailers, Internet marketers want to know what works best -- which picture pulls, which copy matters? It's true that you can test over time, first putting up one page and then putting up another. But what accounts for the change in conversion rate and other results: the different content or the different time period? No matter how you do it, there will always be seasonal, economic, political and other factors affecting your tests. Your eighth grade science teacher will be disappointed because you've changed more than one variable.

A number of companies offer scientific A/B testing. The basic idea is, you create the two versions you want to test and then the A/B testing company's software splits your audience. If you are only doing two tests (perhaps the current page and a new page), the software splits your audience in half when the page that you are testing is requested, sending half the visitors to the current page and half to the test page. Usually, it puts a cookie on the visitor's computer so that the visitor always sees the same pages until the test is over. The company also installs (or lets you install) code on your "Thank You" page so that you can measure success -- "Oh, 8% of the customers who saw the new page made a purchase for $299, but only 7% of the customers who saw the old page made the purchase."

The problem that websites run into is audience size. Great, the conversion rate was 8% for the new page and only 7% for the old page, but did that represent an audience size of 100,000 unique visitors or only 100? When the sample size is too small, you can't have any statistical security in the response.

In the future, we will explore how many visitors you need to feel secure with your response, and the next level up from A/B...N testing: multivariate testing.

Oh, we almost forgot: LunaMetrics is an Offermatica agency - we use Offermatica's software when doing A/B and multvariate testing.

Read our other articles