Website Optimizer: 5 non-conversions required | LunaMetrics

Website Optimizer: 5 non-conversions required

/

Did you know that you needed five non-conversions for each of your combinations in Google’s Website Optimizer? Yeah, me neither.

When I was at Google last week, I was showing the results of a 72 way test to Eric, one of the engineers who is behind Google’s Website Optimizer product. (In case you are not familiar with WO, it is Google’s free multivariate software. It enables website owners to test multiple pieces of a page and even a funnel or series of pages to see which combinations of the variables convert the best. Read why multivariate testing is like a game of Clue.)

Well, anyway, I showed it to Eric, and he pointed out that even though we had a few combinations with greater than a 97% chance to beat the original, one variation was’t even showing up in the Combinations tab. “And that’s your best one so far,” he said. It converted 14 out of 17 times, the dashboard proclaimed. The reason that the combination data didn’t show, he explained, was that there needed to be enough data in general for that test, and there needed to be at least five conversions and (here is the part I didn’t expect) five non-conversions. In the case of this particular combination, there were only three non-conversions. (17-14, right?)

So why the need for non-conversions? After all, the more conversions, the merrier, right? Well in fact, no. “It does best when the conversion rate is 50%” Eric emailed to me.

In order to begin to wrap your head around this (assuming you aren’t a statistician), you have to stop thinking that conversions are good. Instead, there are two states here, a or b. Conversion or non-conversion. Heads or Tails.

So let’s say that we take a finite, maybe only 20, visitors and estimate the conversion rate based on the fraction of the 20 that converted. Is that a good estimate of the true conversion rate? The holy “Law of large numbers” in statistics says that the average conversion of a finite set of visitors becomes a good estimate of the true value as the number of trials becomes large. But, the fine print in this law states that the number must be really large when the true mean (the true conversion rate) is very small (very few convert) or large (nearly all convert). In fact, for the estimate based on finite visitors to be good you need to have enough counter examples. “Counter examples” are non-conversions when the conversion rate is high, or conversions when the conversion rate is low.

I know, you want to know where they got the number five from, and why it’s an absolute number and not a percent. Me too. I’m thinking that the issue is, it can’t be a percent, because if you have only 20 visits, you need a high percent, and if you have 100 visits to that combination, you need a low combination. By fixing a specific amount, you make sure you get something, for both conversions and non-conversions. For both heads and tails. But that last part is speculation. Now if you want to learn something really cool about WO, go over to ROI Revolutions’s blog and read Shawn Purtell’s magnificent piece on the marriage of GA and Website Optimizer.

Robbin Steif

Our owner and CEO, Robbin Steif, started LunaMetrics twelve years ago. She is a graduate of Harvard College and the Harvard Business School, and has served on the Board of Directors for the Digital Analytics Association. Robbin is a winner of a BusinessWomen First award, as well as a recent Diamond Award for business leadership. You should read her letter before you decide to work with us.

Contact Us.

LunaMetrics

24 S. 18th Street, Suite 100,
Pittsburgh, PA 15203

Follow Us

1.877.220.LUNA

1.412.381.5500

getinfo@lunametrics.com

Questions?
We'll get back to you
in ONE business day.