Upcoming LunaMetrics Seminars
Washington DC, Sep 22-26 Boston, Oct 6-10 Chicago, Oct 20-24 Seattle, Nov 3-7

Greg Niland on testing: Not so excellent

Since it’s the weekend, I thought I could rant a little about Greg Niland’s recent podcast on testing your website. But first, a few nice things about Greg:

  • He has the world’s most adorable laugh. And it is even more fun to hear it over the phone after you have heard him do it on fifty podcasts
  • Unlike his SEO peers, he does a podcast for newbies. Most advanced web marketing professionals, be they in SEO or web analytics, want to show off their advanced techniques.
  • He devotes a lot of his time to charitable causes.

But on to the main attraction. I only this week had a chance to hear his podcast with Shimon Sandler. The exact topic was “Making money the Unfun way.” Hmm, I thought as I started to listen. What will be so unfun here? Rewriting URIs? Playing with your robots file? Canonical issues?

Oh, I was wrong. The major unfun issue they wanted to talk about was testing.

Testing? Unfun? Well, I guess what is fun to one person might be boring to another.

But then guest Shimon Sandler, who is an SEO, put testing into two categories, A/B and Mutivariate, “also known as Taguchi.” Really? Greg asked, I’ve never heard of multivariate testing called Taguchi. Maybe multivariate testing was designed by someone named Taguchi, Shimon answered.

(For the record, very simplified: the Taguchi method is one way of decreasing the number of page views and conversions your test needs, as opposed to a full-factorial analysis, which uses every variation. Offermatica uses Taguchi; Google Website Optimizer uses a full-factorial analysis. The Taguchi Method was designed by someone named Taguchi, but as a way of applying statistics to manufacturing.)

Sandler rattled off a few names in the MVT field, but forgot (or didn’t know) most of the important ones. He knew Offermatica, Vertster and at least one other, but missed Optimost, Sitespect and Website Optimizer. And then, the two of them talked about what you could test. But being SEOs instead of conversion “scientists” or MVT jocks, they could only come up with SEO-type things to test. Your PPC ads. Your urls.

Well, everyone who works in testing knows that to make a difference, you should start by testing BIG things. Your headlines. Your buttons. Your shopping cart. Your call to action.

But what was I expecting? Greg needed to have someone from the testing world on that show. How about Sean Purtell from ROI Revolution? How about Ophir Prusack? How about Bryan Eisenberg?

Well, in closing my rant, I do want to give credit to Shimon Sandler for addressing the web analytics issue. When I called Greg and pitched him on doing a show on web analytics (back when our company wasn’t a Google Analytics Authorized Consultant, life was sane and I didn’t spend every weekend working), he told me that his listeners “just aren’t interested in logfiles.” (I tried to explain that not all WA is server side, but I don’t think he cared.)

Robbin

Robbin Steif

About Robbin Steif

Our owner and CEO, Robbin Steif, started LunaMetrics ten years ago. She is a graduate of Harvard College and the Harvard Business School, and has served on the Board of Directors for the Digital Analytics Association. Robbin is a recent winner of a BusinessWomen First award, as well as a Diamond Award for business leadership.

http://www.lunametrics.com/blog/2007/06/02/greg-niland-on-testing-not-so-excellent/

One Response to “Greg Niland on testing: Not so excellent”

Matt Gershoff says:

I think what he meant was fractional factorial design. Taguchi is just one way of setting up the experiment that takes advantage of prior information (assumptions) that the designer possess. The prior in this case is the belief that the factors under consideration do not all interact with one another. This means that the size of the space that needs to be explored can be reduced and you find your answer quicker. Of course if you are wrong about your assumptions then your results will be biased – something to consider when using MVT.