I just wanted to exchange my theater tickets today.

So I called the Pittsburgh Public Theater, and had that standard box office conversation (”When can you get me good alternate tickets, how about this date, try that one.”) And oh, by the way, I said to him, where is the seating chart on your new site?

The box office guy pointed it out to me over the phone, and proudly indicated that I could see any seat’s view on the website. (In fact, you can play along at home.) “Put in your row and seat number,” he instructed, and that one was easy, I could see the boxes right there, begging me to fill them in. Immediately, the seat I was going to get lit up. “But wait,” I complained, “I thought I would be able to see the view from that seat!” Well, in fact, the box office guy explained, you can see the view. Just roll your mouse over your seat.

Now that I look at it again, I do see the little type with the instructions. But — where you sit in a theater is incredibly important. My best friend, a theater addict, taught me that seating is everything. For the person who really cares where she sits (and that is me, and a lot of people like me), this is a great opportunity to make the sale. A fabulous feature. Not one to hide with little type.

When you buy shoes online, you think that there might be an opportunity to see them from various angles. So, it might be somewhat intuitive to click on the shoe. But when you buy tickets online, do you expect to be able to see the view? No. The feature is so cool and so new - this site needs to make a much bigger deal of it.

OK, go ahead, tell me about the theaters around the world that are already doing this.

I would add, maybe they have tested it and found that I am wrong. But given that I can’t find any WA, I pretty much doubt that they have tested anything. Pretty website, though.

What’s the right price to charge for consulting? How much should I pay an experienced web analyst? How do I put those two things together?

I attended the Money, Jobs and Education WAA webinar last Thursday (9.20.07) for the sole purpose of learning the answer to that problem. Here’s the question I asked the panel who presented at the webinar:

Let’s assume that an experienced web analyst gets paid $100,000. Common wisdom says that you, the business owner, need to get 3-4 times an individual’s salary back in order to make a profit. Let’s just say, 3x. That means, s/he needs to bill $300,000.

But if the company bills at $150/hour, the analyst needs to bill out 2000 hours, i.e. ALL of his hours. If the company bills at $200/hour, the analyst needs to bill out 75% of his hours. Still difficult.

Is the problem a) Consultants should be expected to work more than 40 hours/week. b) Consultants who make $100k should be billing out at more than $200/hour or c) consultants who make $100K are not that profitable. They bring the business in and the less expensive employees are the profitable ones?

I didn’t get much of an answer, but I did get some offers for offline help. After the webinar, I had a queue of phone calls to answer (”Great question, can’t wait to see the answer”) and then a stack of emails to answer with the same comment.

If you want to read the answer that I put together, with Seth Romanow’s help, click through here, to the WAA website. You have to be logged in to the site as a member (but you’re a member, right?) I set up that article so that comments are enabled, and you can comment there. I’m sure there are a lot of other ways to make the numbers work, and hope that a few other people will chime in.

Dear Jared:

Before I say how much I disagree with your recent comments on five second tests and homepages, I should point out that I love your new series of podcasts. Not the stuff you do with Josh or the seven other usability experts who pontificate about what Brown University should do. That other stuff was so boring that I deleted you from my iPod. (Do I win the award for the Queen of Criticism or what?) But as luck would have it, I bought a new iPod and “made” my daughter set it up for me (I am just too busy) and she set it up to get ALL my podcasts. So I was surprised and delighted to hear that you and Christine Perfetti are doing a “usable” series on usability — stuff that I can really use. I listened to home pages and store pages and gallery pages. And then I heard you talk about five second tests.

(Now, for the other thousand people who may eavesdropping: a five second test is when you show a user a page for five seconds, take it away and ask them, “What does this page do?” Reason being: that’s about what users give a page, five seconds, before they decide that they are in the right place or not. It’s fast and cheap and helpful.)

But Jared - I think you are wrong when you say that five second tests aren’t helpful with home pages. I use five second tests with users to test home pages all the time.

In your podcasts, you said something like, “Users already know what the site is about. People don’t need to know what the homepage of UPS is all about, they know what UPS does, they go there to track a package, for example.” Well sure, that’s true if the site is UPS, or JC Penney, or Fidelity Investments, or your favorite site, the Bureau of Labor Statistics. But do you really believe that all municipal sewer engineers are familiar with RedZone Robotics? Do you think that every realtor knows HomeFeedBack? These non-Fortune-500 sites may be found in the organic listing (by their home page) and the user will give the site — what, maybe five seconds? — before deciding to bail. Bloggers and linkers like me almost always link to a homepage, and if someone follows a link, they may end up on a homepage of a site they don’t know, giving it what — five seconds? — before deciding to leave. Or not.

I have yet to start a user test with a five second homepage test and hear the user say, “Oh them. I know them.”

You also pointed out that when someone goes to a site with a specific purpose — like finding instructions on their HP printer — they only care if they can find the link to their printer. They don’t care if the page “looks” like a company that sells printers and computers.

But the example is somewhat lacking. It’s really not any different from the UPS example - the user has a task to complete and knows what site he should complete it on. What about when you are searching, when you are following a link, when you are typing a link in that you heard on the radio?

Recently, we did some user testing on a site that sells do it yourself pieces to construct parts of your home. As part of that test, we did a five second home page test. “I would use this site if I wanted to have the company do the construction for me,” users said, almost uniformly. They were all really surprised that the site was a place to buy the parts so that they could do the construction themselves. In retrospect, it’s clear as the nose on my face, but my nose was way too close to that site. Now, we are going to test the pieces of the home page that gave the users the wrong impression.

All because we did five second home page tests.

Robbin

Don’t you ever wonder about the computations of that little calculator that Google gives you to figure out the length of a multivariate test?

I don’t have any insider knowledge. But I have studied it enough to understand certain issues (and many thanks to Dylan Lewis of the Web Analytics Wiki for confirming my suspicions wrt how it should work.) Specifically, you should need more data to “prove the same thing” if your control has a higher conversion rate, up to a conversion rate of 50%.

So let’s start: why does the GWO calculator ask you to input the conversion rate of your current page? Well, here’s why they care. If you hold everything else the same and tell the calculator that your current conversion rate is 4% instead of 3%, it will want a larger sample (translation: more pageviews, or more time to get those more pageviews) in order to get the statistical significance it needs.

So look at these two examples. All the variables are the same (sort of — I promise I will explain.) However, in the examples below, one conversion rate was 3% and the other is 4%. Notice, also (here is the explanation just promised) that I changed the expected increase in conversion rate. With the 4% test, I have it expected to increase by 25% (so that I will get a one point lift in my conversion — after all, .25*4=1) And with the 3% test, it’s expected to increase by 33.33% (because 3% times .33333 is also a one point lift):

So when the current conversion rate is higher, and you are looking for the same absolute expected improvement, the test takes longer, so that you can get more pageviews - i.e. get a greater sample size.

Why?

Why do we need more data to prove improvement with a highly-converting page than with a poorly converting page? Here, I will use a more extreme example: an absolute increase of one point is pretty low when you are looking at a page that converts at 25%. So we need lots of data to prove that a test will do better than the 25%. But a one point increase — that a whopping increase if your control page converts at 4% right now. So we can prove that our new test is better than our old control with just a little bit of data in that situation.

Here’s the really interesting part: when your control has a conversion rate of 50%, you need the most pageviews, i.e. time to get those pageviews. As you keep going beyond 50%, the time to run your tests starts to decrease. When you get to a conversion rate of 75% for your control, the time it takes for the test should mirror the time it takes at a control conversion rate of 25%. (It’s not perfectly exact for mathematical reasons that are too boring to go into here.) But check it out:

(notice that 25* 10% is a 2.5 point lift, and 75* 3.33333 is a 2.5 lift in conversion rate, also.)

Why?

Why does it all turn around at 50%? And I want to try to explain this without using ps and qs and little hats, since I’m not a statistician. So I won’t use fancy equations. Just simple ones.

All these equations that are behind all these kinds of calculators, they include two events: heads or tails. Conversions or non-conversions. They never say (to the extent that they talk), “Conversion is good.” Only people think that conversion is good and non-conversion is bad. (Those equations also include other stuff, but we don’t have to go there.) In fact, you have to have five conversions and five non-conversions for a combination to show up in the graphical area of the website optimizer (the area where the bars are green and grey and red.)

So when you start playing with conversion times non-conversion, you find out that they multiply out to the largest amount when they are both 50%. Right? .5*.5= .25 but if you now use a little 2% conversion rate instead, you have .02*.98 = .0196. That’s way lower than .25 (and remember — this is not sample size, but is one of the important parts of the sample size equation.)

My fourth grade teacher, Mrs. Petrowski, insisted that I learn all those math laws, and one of them was about “commutativity” — it doesn’t matter what the order is in multiplication, you still get the same answer, she lectured. So we can swap those numbers and say that the conversion rate is 98%, leaving the non-conversion rate to be 2%, and the product is still .0196.

So whether you have a 98% conversion rate or a 2% conversion rate — your sample size is going to be the same. (Remember that there is a lot of other junk that goes into the equations, but this basic principle should hold, even though I don’t have access to the innards of the calculator.) And from all this gobbledygook we learn:

• To prove that a test is 1% better than the control, you need more pageviews if the control has a high conversion rate than you would if the control had a low conversion rate.
• However, once the control has a conversion rate over 50%, you start needing fewer pageviews.
• This is a hard topic. If you didn’t understand, please comment and I will do my best.

Whew. This post took me at least two months to write. Many thanks toDylan, again; to Wendi Malley; to Tom Leung (whom I have driven crazy on this topic); and to EV, the GWO engineer who must be sorry he ever gave me his email address.

Everyone who thinks that change in conversion rate should be viewed as a PERCENT and not as an absolute lift in conversion is welcome to flame in the comments.

Robbin

Every day, I wake up and read the Google Groups for GA. Here are the ten questions I see just about every day, with answers.  (Note: the tiny url for this post is http://tinyurl.com/2m3zrm)

Question #1: Can’t I reprocess the data in some way? Sorry. Google Analytics doesn’t let you go back and reprocess. A very best practice is to think about your needs while you are setting up your analytics, and try to anticipate the future. This includes creation of a profile for every important campaign that you run, whether you define “campaign” as a Google AdWords campaign, a banner ad campaign, or an email campaign.

Question #2: I don’t know how to set up links in my newsletter. Or, I don’t know how to set up links in the paid search for non-Google search engines. Answer: Google Analytics has a nice linkbuilder tool to solve this problem, but I like ROI Revolution’s better. It is more intuitive, and it doesn’t have any typos . You can use the linkbuilder to code the links in any of your marketing campaigns — your email marketing, your Yahoo campaigns, your banner ads. (You don’t have to code Google AdWords, the default in AdWords is auto-tagging — it codes for you.) Once external links are coded, they’ll talk to your GA.

Question #3. Why can’t I get rid of the site overlay? You’ll need to clear your cookies. The Site Overlay activates if the GA tracking code sees the “GASO” cookie when you visit your website. Just use Firefox, choose Tools>Options>Show cookies and type in GASO. Delete that cookie.

Question #4. Why is my site overlay broken? If you make any changes to your Request URI — for example, if you ask to only see page titles and not /mypage.com — the site overlay will not work for the profile where you have the changes made. You should set up a clean profile without those changes. (To be honest, I feel like the site overlay is just about always broken, and I am awed when I see people ask questions about why they can’t get a specific piece of data from it. )

Question #5. My AdWords used to be linked to one GA account. I unlinked them, but now I can’t relink them, for some reason, to the correct account. Why? Answer: Relinking a previously linked AdWords account has to be done by Google. Submit a support ticket and they’ll get right one it. GA knows that you can’t do it without their help, and they are pretty responsive to this problem, I hear.

Question #6. My site has a new name. How should I handle my analytics? Answer: GA doesn’t really care what the name of your site is. So you can take your previous code and put it on your new site. But why would you want to do that? Unless the pages match perfectly (example: you kept the same site but moved from .asp to .php), you will really be comparing apples and oranges. Why don’t you just create a new profile in the same account and put that new code on your new site?

Question #7: Why is my own site my top referring source? Why is it even in my list of referring sources? Answer: Probably because you have more than one domain (for example, you are using a third party shopping cart) and you didn’t take the time to learn how to do the cross-domain tracking. Or because you have an old domain that you are redirecting to your new domain, and you are doing the redirect with javascript intead of with a server side redirect.

Question #8: Can’t I get conversion numbers instead of conversion percentages? Answer: Well, you can get them in the Total Conversions report, but you probably want them in the traffic report. And you can get them by a) pulling out your calculator and multiplying or b) dumping it all into excel and multiplying.

Question #9: Can I mix the secure code and the regular code? Google Analytics has a regular form of code (that includes, but is not limited to, the line http://www.google-analytics.com/urchin.js — and they also have the secure code, which uses the secure version of that, https://ssl.google-analytics.com/urchin.js

There are three ways that you can do this:

Option I) Since the name of your site is probably http://www.mysite.com, you are probably tempted to tell GA that when you sign up. This will give you the non-secure code. You can put it on all of your pages (even the secure pages), but it will generate a security error, possibly scaring off potential customers. So, Option I is a lousy idea.

Option II) You can follow the same instructions as above, but go in to the secure pages and be sure to change http://www.google-analytics.com/urchin.js to be https://ssl.google-analytics.com/urchin.js

Option III) Create a new profile, and pretend that your whole site is secure. To do that, choose the https beginning. GA will then automatically generate secure code for you, which you can put on your whole site. Using the secure site on all your pages - http and https alike — does not matter and will not affect your data. GA will not care that your site is mostly an http site.

Question #10) Why can’t I see the past the question mark in my referrers report? Answer: I really don’t have an answer. I would love if someone else would weigh in on this topic.

Bonus: How do I get the IP addresses of the visitors? Answer: you cannot, personally identifiable information is against GA’s terms of service.

Robbin