Greg Niland on testing: Not so excellent


Since it’s the weekend, I thought I could rant a little about Greg Niland’s recent podcast on testing your website. But first, a few nice things about Greg:

  • He has the world’s most adorable laugh. And it is even more fun to hear it over the phone after you have heard him do it on fifty podcasts
  • Unlike his SEO peers, he does a podcast for newbies. Most advanced web marketing professionals, be they in SEO or web analytics, want to show off their advanced techniques.
  • He devotes a lot of his time to charitable causes.

But on to the main attraction. I only this week had a chance to hear his podcast with Shimon Sandler. The exact topic was “Making money the Unfun way.” Hmm, I thought as I started to listen. What will be so unfun here? Rewriting URIs? Playing with your robots file? Canonical issues?

Oh, I was wrong. The major unfun issue they wanted to talk about was testing.

Testing? Unfun? Well, I guess what is fun to one person might be boring to another.

But then guest Shimon Sandler, who is an SEO, put testing into two categories, A/B and Mutivariate, “also known as Taguchi.” Really? Greg asked, I’ve never heard of multivariate testing called Taguchi. Maybe multivariate testing was designed by someone named Taguchi, Shimon answered.

(For the record, very simplified: the Taguchi method is one way of decreasing the number of page views and conversions your test needs, as opposed to a full-factorial analysis, which uses every variation. Offermatica uses Taguchi; Google Website Optimizer uses a full-factorial analysis. The Taguchi Method was designed by someone named Taguchi, but as a way of applying statistics to manufacturing.)

Sandler rattled off a few names in the MVT field, but forgot (or didn’t know) most of the important ones. He knew Offermatica, Vertster and at least one other, but missed Optimost, Sitespect and Website Optimizer. And then, the two of them talked about what you could test. But being SEOs instead of conversion “scientists” or MVT jocks, they could only come up with SEO-type things to test. Your PPC ads. Your urls.

Well, everyone who works in testing knows that to make a difference, you should start by testing BIG things. Your headlines. Your buttons. Your shopping cart. Your call to action.

But what was I expecting? Greg needed to have someone from the testing world on that show. How about Sean Purtell from ROI Revolution? How about Ophir Prusack? How about Bryan Eisenberg?

Well, in closing my rant, I do want to give credit to Shimon Sandler for addressing the web analytics issue. When I called Greg and pitched him on doing a show on web analytics (back when our company wasn’t a Google Analytics Authorized Consultant, life was sane and I didn’t spend every weekend working), he told me that his listeners “just aren’t interested in logfiles.” (I tried to explain that not all WA is server side, but I don’t think he cared.)


Our founder, Robbin Steif, started LunaMetrics in 2004. She is a graduate of Harvard College and the Harvard Business School, and has served on the Board of Directors for the Digital Analytics Association. Robbin is a winner of a BusinessWomen First award, as well as a Diamond Award for business leadership. In 2017, Robbin sold her company to HS2 Solutions and has since retired from LunaMetrics.

  • Matt Gershoff

    I think what he meant was fractional factorial design. Taguchi is just one way of setting up the experiment that takes advantage of prior information (assumptions) that the designer possess. The prior in this case is the belief that the factors under consideration do not all interact with one another. This means that the size of the space that needs to be explored can be reduced and you find your answer quicker. Of course if you are wrong about your assumptions then your results will be biased – something to consider when using MVT.

Contact Us.

Follow Us



We'll get back to you
in ONE business day.
Our Locations
THE FOUNDRY [map] LunaMetrics

24 S. 18th Street
Suite 100

Pittsburgh, PA 15203


4115 N. Ravenswood
Suite 101
Chicago, IL 60613


2100 Manchester Rd.
Building C, Suite 1750
Wheaton, IL 60187