But we can hack it. So look — 3.3333% of 75% is 2.5 points. So instead of increasing the 75%, let’s take it down by 2.5 points to 72.5. To get it back up to 75%, we need to do an increase of 2.5 divided by 72.5 = 3.4483

Now, let’s go back to the calculator. Put in 72.5% as the current conversion rate. put in 3.4483 as the intended increase. It spits back 11.64 days. The same as a 25% control rate and a 10% expected increase.

Perfect mirrors.

The calculator is taking the new conversion rate into account as well. I can’t tell the specific formula they’re using, but it probably includes old variance + new variance (variance = conversions * non-conversions). So for the 25% example this would be .25 * .75 + .275 * .725 = .389 and for 75% it would be .75 * .25 + .775 * .225 = .362.

That’s why the sample size estimation is close but different for these opposites.

“Having used Google as well as xOs from Memetrics, I was very satisfied with the xOs Sample Size calculator. It has easy to use terms where a number of days to run the test is outputted. It’s five easy questions that you plug in and press calculate.

“The biggest difference is that Google optimizes one landing page for everyone while with a premium solution like xOs, you can optimize for different visitor segments, observed or pre-defined.”

Thanks for pointing this calc out, its a neat
little gizmo.

Upon ripping it apart it seems like it is based
upon the formula:

Duration (in days) = i * (test combinations) /[ (%visitors in experiment) * (pagviews per day)]

In the examples you showed, test combinations, page views per day
and % of visitors in experiment were all fixed.

That leaves the parameter (i) which they are deriving from
a function called impressions_to_divergence(p1, impr1, p2, impr2, conf).

It seems like an iterative algorithm which has as some of
its inputs the current conversion rate and the expected improvement
percentage (along with a parameter “conf”).

What I’m questioning is the parameter “conf” in the
impressions_to_divergence function. It seems that they’ve hard-coded
it at .28?

Any thoughts what it might be?

Using your example of 12 test combinations, 100 page views per day,
100 % Visitor participation, 25% current conversion, and 10% expected
improvement; I modified the code for the value of “conf” and obtained
these results for the duration (in days).

conf = .1 -> days = 1.44
conf = .2 -> days = 5.88
conf = .28 -> days = 11.64
conf = .3 -> days = 13.32
conf = .4 -> days = 23.76
…

Interesting…I’m wondering why they’ve chosen to fix “conf”
at .28? I see how it is passed down to the function that computes
the variance and is used to close in on the value…

Guess I’m off to dig out a stat book. Or maybe I should take it for
granted that the car runs, and not worry about the inner-workings
of the internal combustion engine?

Fun stuff…

Thanks,
Jim