I actually took the time and wrote a blog post about it ( In qualitative analysis 5-8 users are enough); as I find usability testing quite fascinating.

Cheers ..
Dennis

Dennis R. Mortensen, COO at IndexTools
My Web Analytics Blog

You were so politic here. It is really ME you are disagreeing with, not my customer. Thank you for teaching me something new. And of course I remember you. I am just waiting for the day when a customer comes to me and says, Robbin, we want your help with WA, and we use INDEXTOOLS!

First. Very nice to meet you in San Francisco (you might remember me sitting with Avinash). As you know, I am an avid reader of your blog and this is another great post. However; I would like to take the liberty to disagree with your client in his saying:

“I highly doubt that I am going to learn anything with 5-8 users”

I honestly think that this is incorrect and biased. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of user have a “decent” impact in a user test (whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability)

Jacob Nielsen introduced a model saying

N(1-(1-L) to the power n)

N is the total number of usability problems in the given design
L is the proportion of problems discovered testing 1 user

Having this is mind; you can do a couple of test calculations and will typically see that with 5 – 8 users you would get (assuming the first user finds about one third of the problems) between 80% and 90%. Thus concluding that one WILL discover (learn) most problems by using only 5-8 users.

Cheers..

Dennis R. Mortensen, COO at IndexTools
My Web Analytics Blog - …And learn how to Use Web Analytics to determine the width of your Internal Search Query box