Archive for the ‘A/B and MVT’ Category
Posted on August 19, 2006 by Robbin Steif
It’s true. I cheated on Google AdWords so that I could measure A/B Google tests in my Google Analytics.
First, the background. Lots of small websites and advertisers aren’t ready to take the multivariate testing plunge, so start by using Google AdWords to A/B test. This is pretty tried and true: You create two ads for the same AdGroup which are absolutely identical, down to the URL that shows on the Google page (SERP). However, when the customer clicks, each ad has a different landing page. Then the advertiser compares conversion rate (or revenue, or average order size) for all the customers who start with Ad1 vs Ad2.
Admittedly, it has its limitations. Search engines are demographically skewed, and just because it works for Google customers doesn’t mean that it will work on Yahoo. But it’s way better than saying, “I know what will work. I just know.”
Google has recently made measuring different ad versions easier in the AdWords interface, but with Google Analytics, you still have to know how pull down the right menus and segment to see what you need. And even then, if the ads have the same name, you can’t tell them apart.
Step 1: Pulling down the right menus. (I can’t remember whether I learned how to do this from Justin’s GA blog or from ROI’s GA blog, and I can’t find the reference.)
Choose Marketing Optimization > Marketing Campaign Results > Campaign Conversion. Left click on the Analysis Options next to one of your Google AdWord campaigns (which you get with the little red circle to the left of the campaigns – follow the top red arrow in my picture); choose Cross Segment performance (that’s the middle red arrow I’ve drawn); finally, choose Content. When you choose Content, you’ll get a list of the different ads that are running for that campaign, by goal.
Step 2: This is where you cheat: Retitle your ads, ever so slightly. When you are using your Google AdWords to do A/B testing, as described above, the ads are identically worded. Google Analytics lists them out by title, which means, it can’t tell you that Ad1, titled, “Increase your Conversion Rate,” and which lands on www.lunametrics.com , is doing terribly, and that Ad2, titled, “Increase your Conversion Rate,” which lands on www.lunamerics.com/conversionrate, is doing great. It only sees one ad, called “Increase your Conversion Rate.” You can cheat on Google AdWords by changing the titles very very slightly. In this case, I would change one of the ads to have a capital Y in Your, so that it reads, “Increase Your Conversion Rate.” The difference is slight enough that it shouldn’t matter, and will enable you to read the results in GA.
Robbin Steif
LunaMetrics
View Comments (4 Responses) | Categories: A/B and MVT, Paid Search
Posted on April 19, 2006 by Robbin Steif
I’m at the eMetrics Summit in Santa Barbara, and there are lots of cool lessons to be heard, some of which I’ll write about in the following days. But today, I’m writing about lunch.
I didn’t really notice what I *ate*, but I sat with Matt Roche from Offermatica (whom I referenced a few days ago but only met today) and Bill Bruno from Stratigent. During the course of lunch, I asked them both what they thought about ABA testing, which companies sometimes use when they have too few visitors (too little data) to do standard split-path testing.
With ABA, the company does an A/B test with three groups instead of two. They randomly split visitors into three groups. One of them sees the test page (the B group.) The other two groups both see the same control page (Those are the two A groups.) When the first A group has the same conversion rate as the second A group, the company who is running the test decides that it has collected enough data, and then feels that it is in a position to declare either the control or the new test page a winner.
Matt and Bill both dismissed ABA testing as a not-too-great idea. Matt drew a picture on the back of Bill’s business card, showing how data usually presents itself:
It is never very clean, he said, and it doesn’t usually go in a beautiful curve, but it bounces all over the place. So in the above picture, the two “A” tests are in black and red, and the B test is in blue. At which arrow should we say that the two A tests are the same — the first one, where the blue is a winner, the second one, where they are all tied, or the third one, where the blue line is the loser?
Robbin Steif
LunaMetrics
View Comments (No Responses) | Categories: A/B and MVT
Posted on March 18, 2006 by Robbin Steif
Today, a colleague paid me (and my designer) the ultimate website compliment. He asked if he could copy my website structure. “I figured you had it all worked out,” he wrote.
There was a lot of truth in this — enough that I could feel good about telling him to go ahead. I really did try to work in best practices, because I had to. How can I tell customers with lead generation sites to put a contact form on every page if I don’t do it? How can I tell them to link to their privacy policies right by the email field if I don’t do the same?
On the other hand, I didn’t win every battle with my designers and still have a lot of work to do on my own. For example, I don’t have a great 404 error page, so that’s still on my list. I don’t have on-site search that can handle spelling errors and stemming — if you type in “emarketing” instead of “e-marketing,” you get a No Results page. (It’s a nice No Results page, but plain old Results would be better.) I didn’t have enough negotiating capital to get my navigation below the banner, where people would actually see it.
Which brings me to my point. Big companies like Amazon or a small web conversion company like mine aren’t always worth emulating. You assume they’ve tested everything — but maybe they haven’t. Sometimes the politics of an organization force upon companies sub-optimal solutions that everyone can live with.
Because my colleague wrote to me first, I was able to point out one change that would improve his site based on the experience I had with my own. But if you’re copying Amazon — well, the employees at Amazon are better about keeping secrets than the CIA is. Even if you do have a friend at Amazon, she won’t tell you anything. In that case, you have to measure and test, test and measure.
Robbin
LunaMetrics
View Comments (2 Responses) | Categories: A/B and MVT, Conversion Science
Posted on March 13, 2006 by Robbin Steif
So what is A/B testing, anyway?
A/B testing springs from old fashioned direct marketing. Mailers would have a mailpiece that worked well, but wanted to see if they couldn’t improve on it a little bit. So they changed just one variable — maybe it was the snipe (“Your free gift inside!”), or the offer – and tested the new piece against the control. Because they changed just one variable, they could scientifically measure not only whether the change made the mailpiece perform better than the control, but also what about the new mailpiece made the difference.
Let me take one last minute to distinguish between A/B, multivariate and Split Path, since I’ve mentioned them all. A/B is when you test one kind of creative against the current creative. Maybe you just change the headline on a test page. When doing A/B testing, never forget what your 8th grade science teacher taught: if you change more than one variable, you don’t know which one was responsible. Multivariate is when you change a bunch of elements on the test page, and then use statistical technology to figure out which elements mattered. (I will do another post on this if anyone is clamoring for more…) Finally, Split Path is just the technology to send some people to one page when they click on a link and everyone else to a different test page when they click on the same link. You can do 3-way or 4-way or nth-way tests, but you generally need more traffic so that your sample size is large enough to mean something.
(MarketingSherpa did a very interesting piece on testing multiple variables with a small audience size. I will dig it out when I get back from the UIE Usability conference.)
Robbin
LunaMetrics
View Comments (2 Responses) | Categories: A/B and MVT
Subscribe to this blog's feed
