A/B and Multivariate Testing, Not Scary at All


The final part of the Infinite Conversion Loop is A/B Testing. A/B testing is like exercising, everyone knows they should do it, it seems harder to do then it really is, and once you do it you wonder what took so long to get started.

First, an introduction. The concept of A/B testing is as old as marketing itself. An A/B test is simply showing one variation of a web page to 50% of your visitors, and another variation to the other 50%. You then measure what is more successful, and that one is “the winner”. Slightly more complicated is a multi-variate test. In this case, various specific parts of the page are changed, maybe a different picture shown, or the text justified differently, and every possible combination of the various parts are alternated to find the absolute best combination.

It’s been proven time and time again that slight variations in the way content is presented can have a big impact on conversions, and you can read all of the best practices in the world, A/B testing will always prove that what you think you know is wrong. Through A/B testing, I’ve found several counter-intuitive things improved conversion: Having a worse design, not listing prices, etc… You could test these same things and get different results. That’s the beauty of A/B testing.

Ok, so this sounds hard. But it’s not! Luckily Google has came up with a great, free tool called Website Optimizer that takes the pain out of A/B testing. It’s so easy, I’ve actually setup this blog post as a multi-variate test. So if you view this on several different browsers you should see completely different blog posts.

The key with A/B testing is to be clear what you want your goal to be. I’ve made my goal Tweets, so let’s see what variation (if any) has the best affect on that. That’s one last point to make. Often times there won’t be a statistically significant difference between your variations, that’s ok be prepared to give it time, the more traffic you have, the quicker your results will come in.

Update- The A/B Test that was originally being used for this post has ended, so you won’t see different variations on this page anymore. The winning combination is being displayed.

Phil is a former LunaMetrician and contributor to our blog.

  • Sarah

    Thanks for the reminder… keep meaning to do this…
    w.r.t your blog, I know which picture made me feel more warmly towards your page 🙂 Although just to mess with you I commented in the other one!!

Contact Us.


24 S. 18th Street, Suite 100,
Pittsburgh, PA 15203

Follow Us




We'll get back to you
in ONE business day.