Conversion on a limited budget: Users vs experts


Yesterday, I was talking to a potential client. Like most, he wanted to improve his conversion rate, and like even the biggest companies, he had to live within a budget. We discussed the pros and cons of doing user testing with 5-8 users vs. my doing an “expert” conversion analysis. The costs weren’t different enough to be a consideration.

“Don’t take this the wrong way,” he said, “But I highly doubt that I am going to learn anything with 5-8 users.”

Now, at some level he was absolutely right. Five to eight users would not be statistically significant. Nor, for that matter, would fifty or eighty.

So would one conversion analyst be more statistically significant? No. I can say, “Studies show that making the button red tends to work better,” but it might not work better for your site. Plus, it is incredibly hard for a best practices person to discard the “professional blindness” that knowing websites brings with it. (This is the reason that I don’t allow web designers to sign up for our user tests – same problem.) Web analytics are statistically significant, but we only know that everyone bails out on a page, we don’t know why. Multivariate testing is the best, but you still need to know what to test.

This is why I love user testing. It’s not that expensive, especially if you do it yourself. Sure, you won’t learn everything you always wanted to know, but when you hear four people out of five tell you that the deal isn’t good enough for them to be interested, you sure do know what to test.

Speaking of which, I think too many people think user testing is just usability. Usability does go hand in hand with conversion, but you learn about pricing, about trust, about what people notice on the page. And a great idea (I owe this to a LunaMetrics customer) is to do user testing on your competitors’ sites. That’s when you learn where they have elements that are worth testing on your own site.


Our founder, Robbin Steif, started LunaMetrics in 2004. She is a graduate of Harvard College and the Harvard Business School, and has served on the Board of Directors for the Digital Analytics Association. Robbin is a winner of a BusinessWomen First award, as well as a Diamond Award for business leadership. In 2017, Robbin sold her company to HS2 Solutions and has since retired from LunaMetrics.

  • Hi Robbin,

    First. Very nice to meet you in San Francisco (you might remember me sitting with Avinash). As you know, I am an avid reader of your blog and this is another great post. However; I would like to take the liberty to disagree with your client in his saying:

    “I highly doubt that I am going to learn anything with 5-8 users”

    I honestly think that this is incorrect and biased. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of user have a “decent” impact in a user test (whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability)

    Jacob Nielsen introduced a model saying

    N(1-(1-L) to the power n)

    N is the total number of usability problems in the given design
    L is the proportion of problems discovered testing 1 user

    Having this is mind; you can do a couple of test calculations and will typically see that with 5 – 8 users you would get (assuming the first user finds about one third of the problems) between 80% and 90%. Thus concluding that one WILL discover (learn) most problems by using only 5-8 users.


    Dennis R. Mortensen, COO at IndexTools
    My Web Analytics Blog – …And learn how to Use Web Analytics to determine the width of your Internal Search Query box

  • Dennis – I am so sorry that this took so long to post. It is my fault, it went into auto-moderation and I didn’t notice it.

    You were so politic here. It is really ME you are disagreeing with, not my customer. Thank you for teaching me something new. And of course I remember you. I am just waiting for the day when a customer comes to me and says, Robbin, we want your help with WA, and we use INDEXTOOLS!

  • No worries Robbin 🙂

    I actually took the time and wrote a blog post about it ( In qualitative analysis 5-8 users are enough); as I find usability testing quite fascinating.

    Cheers ..

    Dennis R. Mortensen, COO at IndexTools
    My Web Analytics Blog

  • Pingback: VisualRevenue | In qualitative analysis 5-8 users are enough()

  • Pingback: Online Marketing Blog » Blog Archive » Have you been putting off that new site or site tweaks?()

Contact Us.

Follow Us



We'll get back to you
in ONE business day.
Our Locations
THE FOUNDRY [map] LunaMetrics

24 S. 18th Street
Suite 100

Pittsburgh, PA 15203


4115 N. Ravenswood
Suite 101
Chicago, IL 60613


2100 Manchester Rd.
Building C, Suite 1750
Wheaton, IL 60187