Conversion on a limited budget: Users vs experts | LunaMetrics

Conversion on a limited budget: Users vs experts

/

Yesterday, I was talking to a potential client. Like most, he wanted to improve his conversion rate, and like even the biggest companies, he had to live within a budget. We discussed the pros and cons of doing user testing with 5-8 users vs. my doing an “expert” conversion analysis. The costs weren’t different enough to be a consideration.

“Don’t take this the wrong way,” he said, “But I highly doubt that I am going to learn anything with 5-8 users.”

Now, at some level he was absolutely right. Five to eight users would not be statistically significant. Nor, for that matter, would fifty or eighty.

So would one conversion analyst be more statistically significant? No. I can say, “Studies show that making the button red tends to work better,” but it might not work better for your site. Plus, it is incredibly hard for a best practices person to discard the “professional blindness” that knowing websites brings with it. (This is the reason that I don’t allow web designers to sign up for our user tests – same problem.) Web analytics are statistically significant, but we only know that everyone bails out on a page, we don’t know why. Multivariate testing is the best, but you still need to know what to test.

This is why I love user testing. It’s not that expensive, especially if you do it yourself. Sure, you won’t learn everything you always wanted to know, but when you hear four people out of five tell you that the deal isn’t good enough for them to be interested, you sure do know what to test.

Speaking of which, I think too many people think user testing is just usability. Usability does go hand in hand with conversion, but you learn about pricing, about trust, about what people notice on the page. And a great idea (I owe this to a LunaMetrics customer) is to do user testing on your competitors’ sites. That’s when you learn where they have elements that are worth testing on your own site.

Robbin

Our owner and CEO, Robbin Steif, started LunaMetrics twelve years ago. She is a graduate of Harvard College and the Harvard Business School, and has served on the Board of Directors for the Digital Analytics Association. Robbin is a winner of a BusinessWomen First award, as well as a recent Diamond Award for business leadership. You should read her letter before you decide to work with us.

  • http://visualrevenue.com/blog Dennis R. Mortensen

    Hi Robbin,

    First. Very nice to meet you in San Francisco (you might remember me sitting with Avinash). As you know, I am an avid reader of your blog and this is another great post. However; I would like to take the liberty to disagree with your client in his saying:

    “I highly doubt that I am going to learn anything with 5-8 users”

    I honestly think that this is incorrect and biased. If one read studies like: “A mathematical model of the finding of usability problems” and other similar literature like “Stochastic Processes” – one will see that a very limited number of user have a “decent” impact in a user test (whether that be for better usability and or specifically for increased conversion; which by the way typically is the result of increased usability)

    Jacob Nielsen introduced a model saying

    N(1-(1-L) to the power n)

    N is the total number of usability problems in the given design
    L is the proportion of problems discovered testing 1 user

    Having this is mind; you can do a couple of test calculations and will typically see that with 5 – 8 users you would get (assuming the first user finds about one third of the problems) between 80% and 90%. Thus concluding that one WILL discover (learn) most problems by using only 5-8 users.

    Cheers..

    Dennis R. Mortensen, COO at IndexTools
    My Web Analytics Blog – …And learn how to Use Web Analytics to determine the width of your Internal Search Query box

  • http://www.lunametrics.com Robbin

    Dennis – I am so sorry that this took so long to post. It is my fault, it went into auto-moderation and I didn’t notice it.

    You were so politic here. It is really ME you are disagreeing with, not my customer. Thank you for teaching me something new. And of course I remember you. I am just waiting for the day when a customer comes to me and says, Robbin, we want your help with WA, and we use INDEXTOOLS!

  • http://visualrevenue.com/blog Dennis R. Mortensen

    No worries Robbin 🙂

    I actually took the time and wrote a blog post about it ( In qualitative analysis 5-8 users are enough); as I find usability testing quite fascinating.

    Cheers ..
    Dennis

    Dennis R. Mortensen, COO at IndexTools
    My Web Analytics Blog

  • Pingback: VisualRevenue | In qualitative analysis 5-8 users are enough()

  • Pingback: Online Marketing Blog » Blog Archive » Have you been putting off that new site or site tweaks?()

Contact Us.

LunaMetrics

24 S. 18th Street, Suite 100,
Pittsburgh, PA 15203

Follow Us

1.877.220.LUNA

1.412.381.5500

getinfo@lunametrics.com

Questions?
We'll get back to you
in ONE business day.