Archive for the ‘Surveys’ Category
Posted on May 5, 2010 by Robbin Steif
Earlier this week, I got what I perceived to be the strangest survey from Intuit. The survey was about their online backup service.
They asked me questions like, “Please rank which of these four features factor into your use of our backup service.” And then they named all these strange services that I didn’t even know they had, like disaster recovery, or fixing things up once your employee messes up your accounting data.
Finally, I wrote, “I use your service because my jump drives always broke or got too full.” Simple. Easy.
The problem we (like Intuit) often have is that we have super duper services and special features, and we think that our customers care about them, when in fact, our customers might not even know about them. (After all, we spent a year and a lot of money on that Super Duper feature, and now we want to write about it on our websites, no?)
All of which gets down to why qualitative data — surveys and user testing — are so important as we learn to create better websites.
Robbin
View Comments (1 Response) | Categories: Surveys, Writing for the web
Posted on March 3, 2010 by Robbin Steif
It is amazing what our visitors know and don’t tell us. That’s why we have to keep asking them, and surveying, and looking at our analytics
A couple days ago, I was working with a customer on his website, and saw that as soon as the visitor chose to put something into the shopping cart, she would be confronted with a sign-in requirement. There wasn’t even an option to proceed as a guest.
“Well, that must be a real turn-off,” I said — too quickly, it turns out. Because, the very next thing we did was go look at his Google Analytics. We saw that less than 10% of the visits that got to that page exited.
“Well,” I continued, sure that I knew his customers and visitors better than they knew themselves, “Let’s see what it says for new visitors only. I’ll bet that new visitors, who don’t already have sign-ins, exit in droves.” And you guessed it, we saw that less than 10% of the visits that got to that page exited.
So we get back to the need for analytics, user testing and multivariate testing, because we never know what our visitors really want.
Robbin
View Comments (5 Responses) | Categories: A/B and MVT, Conversion Science, Surveys, Usability, Web Analytics
Posted on March 5, 2008 by Robbin Steif
Thanks to everyone who took our survey. We had 72 responses (and we have about 1500 subscribers, plus tens of thousands of unique visitors who read on a non-subscription basis), so there are definitely issues with statistical significance. I will do a separate post on statistical significance of surveys (i.e. where you only have one numerator and one denominator, as opposed to all those calculators that let you compare tests.) In the meantime, many thanks to Judah Phillips for lending a hand in that department.
We only asked five questions:
1. Please describe your “relationship” with the LunaMetrics blog
2. Why do you read this blog?
3. In the area of Google Analytics, you consider yourself (novice, intermediate, expert)
4. In the area of multivariate and A/B testing, you consider yourself (novice, etc.)
5. Freeform place to tell us what you want
At a simple level, we learned that most responders to the survey are also blog subscribers or read very often (63%), they are mostly interested in GA (46%), although the second area of interest was split between learning about conversion in general and learning about web analytics in general. This one was hard, because I didn’t learn enough to stop beating myself up about doing so much GA stuff and so little conversion stuff. The least important reason people read is to learn about industry trends. (OK, we will remember not to do that. An easy request.)
We learned that among responders, there is a nice bell curve of GA expertise, with the majority of responders (45%) considering themselves at an intermediate level
and experts at 18%, newbies at 32%. On the other hand, the vast majority of responders considered themselves newbies at MVT and A/B. Again, this is helpful information, because it reminds me and other bloggers here at Luna to write at a level that new and intermediate analysts can understand.
I tried to do a lot of neat correlations that fell apart (too little data.) However, I pushed the numbers by hand, and was continually awed at how MVT came in last place to most readers. Of course, there is a strong problem of what causes what; after all, we write about GA so much, that it would be surprising if the audience didn’t self-select and want to hear about it. On the other hand, it always feels to me like MVT goes hand in glove with web analytics….
Although everything was completely anonymous, a number of people left some really great comments, and a few even left their name. And responses were so nice! (Maybe no one knew how incredibly anonymous they were – no IP addresses or anything.) Here are only some of the comments that we got. I copied and pasted (and only added hyperlinks to other places in our blog.)
Freeform answers to, “What is your relationship to this blog?”
I´ve discovered today, reading a spanish analytic blog.
I clicked on the link today while I was visiting your website…just wanted to know more about LunaMetrics…in particular, your CEO, Robin.
I just subscribed after the recommendation of Avinash’s book. I use Google Reader, which i read daily.
accidentally found it yesterday through google and it was the only website to make understand RegEx properly 
Yes, I am embellishing on my answer!! My answer is really A: I subscribe to this blog or read it very often, but I also use some posts as reference guides whenever I need them. Amazingly, I still use Robbin’s Regular Expression guides from 2 years ago, because it’s always good to go back and cover the basics again. [Dear writer -- me too!! There are all these RegExen, like negative lookahead, that I don't use that often, and then when I need them, I go back to the blog and learn about them again - Robbin]
Freeform answers to, “Why do you read this blog?”
I don’t care for the way this question is laid out. I don’t think that I can “rank” the reasons why I read a blog. But I do know that this is one of the blogs that you MUST have bookmarked if you’re in web analytics and consider yourself “staying on top of the industry”. Therefore I have checkmarked that answer, but everything else would come in as 1a, if you will.
I wanted to see how much LunaMetrics knows about paid search.
Interesting insights, How to’s on how to get at the data in GA to answer relevant questions.
because Robbin will ask me if I read it and I must be truthful!!! But your team is so freaking smart I want to read it.
Freeform answers in the place where I said, “Go ahead, tell us everything you wish we had asked, what you wish we would write about, anything at all.”
More than anything, I get to see when the web analytics industry moves to analysis of customer behavior over time (i.e. prior loyal customers have a 20.3% chance of visiting in January 2008, visiting an average of 3.4 times, and has a 32.4% chance of purchasing, spending an average of $125.) When I see this type of information, broken down by Recency/Frequency/Monetary/Channel/Personas, I’ll know that the Web Analytics community has arrived. I believe the folks at Luna Metrics will be among the first to get this.
I like stories about what other people have learned, tried or are thinking about. Since I am a novice, I want to learn as much as I can. I don’t really like the technical posts, but I am not the technical person so I just sort of skim them and move on. I’d also like to learn about how your clients are overcoming organizational obstacles. [Ooh, this is a good one, although Avinash just wrote a nice piece on this.]
I really liked it when you brought up the issues with documentation on the Google Analytics Help Section. I’m all about organization and it would be nice if you pushed that issue some more. Since the issue was brought up, I have seen several changes in the Help Files organization. I also really like it when you conduct experiments (like your Visitor Loyalty experiment from about a month ago). Thank you, Robbin!
You might have asked about your audience. For example, website managers might be interested in a certain level of detail on how to do something, but CEOs might be in more top level stuff and may not have time to slog through the blog.
I *love* that you are putting tools on your blog, like the FireFox extension for copying goals, and the macro for deduping parameters. Thanks!!!
I wish more WA bloggers would write about “Doh!” moments they’ve had – where they’ve spent x amount of time (too much) trying to solve a problem, or configure a profile, or segment a group, or whatever, and after spending all this time, they have an “aha” moment, where they realize the answer. Sharing those moments with other practitioners/blog readers…priceless
View Comments (No Responses) | Categories: Surveys
Posted on January 23, 2008 by Robbin Steif
I was thinking about the LunaMetrics Blog survey. Quite a few people have taken it, but I’d like to get another fifty responses before publishing.
I considered (and discarded) these thoughts:
- You owe it to the blog writers. No, let’s face it: you don’t owe anything. We’ll keep doing great content, and we really don’t know anything about who replies (unless they actually sign their name, which a couple of people did). So we’ll never know if you answered or not.
- You aren’t getting paid for it, so why should you? Good point. So you see why I discarded this one.
- You’ll be able to influence some of the things we write, or at least, how we write them. Well, that’s like saying you will have a chance to win a free iPod. You might win, but your chances are diluted by the number of others who answered. And when was a potentially free iPod the reason that you did something, anyway?
No, it seems to me that there is only one good reason to answer our survey: this is what analysts do. They understand that data is sacred, and that not having enough of it is aggravating. So if creating data is something an analyst can do, we do it.
I hope you’ll understand how important having enough data is, and you’ll take our short, completely anonymous survey.
Take the LunaMetrics Blog survey
Thanks!
Robbin
View Comments (2 Responses) | Categories: Surveys
Posted on January 9, 2008 by Robbin Steif
So yesterday, the new year started in force, and our team sat down to talk about our blog (and what kind of posts we would like to do in the coming months.) “Well, what do our readers care about?” one person asked. “Who are they?” Hmm, I thought, I really have very little idea. So here is our first little survey (and it really is little — just four or five questions, depending on how you count them. None are mandatory, because don’t you just hate when you give freely of your time and they force you to answer things you don’t want to answer?) Please please please take a minute or two to fill this out. We write a lot, but only know that we’re helping when you talk to us.
Take the LunaMetrics Blog survey
Many thanks to Taylor and Traci, John and Jason. I know you guys won’t think that I took any of your advice, but I really did.
- Robbin
View Comments (1 Response) | Categories: Surveys
Posted on August 20, 2007 by Robbin Steif
Did you have questions when you finished Web Analytics: An Hour a Day? I did.
The book was truly amazing. But when I was done, I had written all over it. Sometimes my notes read, “This is awesome, we have to try it.” But sometimes they read, “I don’t understand.” And other times, they read, “I really disagree.”
So I got an interview with the author. It came to nine pages (count ‘em, 123456789), so I am going to reprint it in parts. Avinash, you are truly wonderful for devoting this much time.
So let’s get started. And in my usual “in your face” fashion, I’ll start with a question that most people wouldn’t ask the Guru of WA:
Please allow me a quick interruption. The word Guru is of Hindu (Indian) origin and having grown up in India I have to say that I do not consider myself a Guru. One has to meet an astoundingly high benchmark to get that title and I am very very far away from even the starting point of meeting that benchmark.
For more context on that word here’s the wonderful wikipedia [definition]
In the introduction – why do you write that your book is for everyone? Is it for my mother, who is retired and spends lots of time taking care of my father? Is it for my daughter — the one who can drill down in her Quicken, but refuses to do anything academic? No, of course not. But that’s what customers do. We ask them, “Who is your site for?” and they answer, “Everyone!” So – who is your book for?
You got me.
Perhaps that was overuse of the word everyone.
Here are the specific people / roles that are mentioned in the introduction of the book:
· Mr./Ms. Web Interested
· CEO
· C-level or VP-level or just No-level person
· Marketer
· Sales Person
· Web-Designer
· User Researcher
· Analyst
The introduction describes how the book will be helpful specifically for each role.
As an example, here’s the one for Web Designer: If you are a Web “Designer†then this book will share with you how you don’t have to compromise on the number of ideas you can put on the website to improve the site, that you can have all of your ideas (even the radical ones) go live on the site and measure which one solves the customer (or your company’s) problems most effectively.
Question: I love the idea of surveying, continuously. However, it has not worked out well for me or my customers. We figured out how to delete the pop-up blockers, only to find out that customers hated it. And shouldn’t it be “customers first?” Any advice about the best way to *administer* exit surveys?
In my experience surveys that are shown at the right time with the intent of allowing the customer to express their opinion go ok. Typically we cram so much into a survey that only we care about that a customer looks at it, pukes and exits.
If you ask customers “nicely” they want to tell you about their experience.
My advice:
1) Experiment with different invitation types (pop up, pop under, on exit etc) and see what your customers prefer. And you only have to do this a few days each to get a feel for it.
2) Start with small number of questions (remember the “golden questions” post?) and then expand.
3) Put a really large close button. Make it apparent, clearly, that the survey can be closed. In a very subliminal way it works very effectively and actually gets closed less (if you have done #1 and #2 above first!).
From a mindset perspective you want them to share with you what they think rather than do a quick little interrogation with a battery of questions. Fine balance.
While we are on surveys – how do you feel about surveys that force the visitor to answer certain or all of the questions? It is infuriating to me when I answer a BizRate survey for the chance to get a “free” magazine, and they force me to answer questions (so instead, I just lie.) Thoughts?
I skip it.
I also rarely do any other surveys. I often read them to study them from a knowledge / awareness perspective, but I don’t fill surveys.
Here is the thing. I am not the customer and it is irrelevant what I think about surveys.
The first time we did a survey it was 20 questions and I was positive it would bomb, after all who in God’s name has that much time. Turns out that it had a consistent 18% response rate (compared to an internet standard of 1% response rate for surveys).
My lesson was that I should try not to impose my views and opinions and check them in at the door. Because I am not the customer, no matter how much I think I am. It is a tough pill to swallow because we tend to think we are “experts” because we have so much knowledge and data.
Experiment, it is cheap, see what works and what does not, refine and try again.
Why do you have a Trinity? I really see a duality, clickstream data and qualitative data.
It is qualitative (Experience), clickstream (Behavior) and the third prong is Outcomes.
Some people mix clickstream with outcomes. I choose to break it out for two reasons:
1) I want people to outrageously focus on outcomes. It is easy to be hypnotized by all the clickstream data and reports and forget to set goals or measure in a very hard core way outcomes. Yet the thing that drives action is not all your clickstream analysis, it is the tie to outcomes.
2) I want people to think of outcomes more than conversion. I am not big on obsessing about conversion, which will almost always lead to solving for a minority of your site traffic. Outcomes are improved customer satisfaction numbers, increased task completion rates, increased depth of visit over time, problem resolution rates on support websites, better recency trends on non-ecommerce website.
By putting outcomes as a separate part of the Trinity I am trying to emphasize the importance of understanding outcomes, and different types of outcomes beyond just revenue/conversion.
Do you think that makes sense?
Notice that I didn’t answer his last question (“Does that make sense?”) — I hope some of you will.
Coming next: Part 2, where I ask Avinash about first and third party cookies, where to put the code on the page when you are torn between conflicting needs, and other really “down in the weeds” Q&A.
Part2Â Â Part 3Â Â Part 4
Robbin
View Comments (4 Responses) | Categories: Industry News, Surveys, Web Analytics
Posted on April 24, 2007 by Robbin Steif
I never give any money to the Harvard Business School (HBS). So finally, they sent me a survey, asking why not.
(For the record: I always think that giving to the Greater Pittsburgh Community Food Bank is so much more important than giving to a corporation with a $25 billion endowment.)
Now, I applaud them for actually asking why I never give. But their survey was really lousy. And unlike companies, who may be stuck with a pre-packaged web survey from an outfit like BizRate, this is a school that can afford to buy anything they want — especially when it comes to raising money.
Rarely does someone pay for your time when you fill out a survey. So if I am donating my time, why don’t the people who put together the surveys allow me to skip some of the questions? If I don’t like a question, I’m going to lie, anyway, right? And if you had all the money in the world (like HBS does) you could even create a survey that points out that you missed a question and still lets you skip it (in case you don’t want to answer.)
Well, the HBS survey was one of those incredibly hard to fill out surveys — specific numbers of answers required here and there, not much thought put into usability. So finally, I wrote Bill Sahlman, the senior dean for external relations, since he had sent the original email.
Dear Bill: I tried to take your survey but never succeeded, due to error messages. {Screen shot of error message was included.}
I am a conversion scientist and web analyst. When you require people to answer questions, you lose them. It is even worse when they WANT to answer the question and can’t find the answer.
I wasn’t important enough for Bill to answer personally — despite how important I tried to make myself sound –Â so someone named Courtney Fanning wrote me back.
I am replying on behalf of Bill Sahlman. The error your received is likely due to the limit of options. For some questions, you can only check 2 options. IF you check more than 2, it will not allow you to continue. Thank you for taking the time to complete the survey.
Sorry Courtney. I never took the survey.
End notes: I promise to go back to custom filters for Google Analytics later this week. Many thanks to Stephan Hamel (again) who updated this blog to www.lunametrics.com/blog in three places (so I can thank him at least twice, right?), and to Eric Peterson, who updated us as well.
View Comments (1 Response) | Categories: Surveys, Usability
Posted on August 6, 2006 by Robbin Steif
Eric Mattson sent me this link and is looking for those who have their own blogs to fill out his survey. While it is always nice to help others, this one is particularly interesting because it really forces bloggers to think about blogging issues.
And hey, it’s Sunday. I can get away with a one paragraph post. I think.
Robbin Steif
LunaMetrics
View Comments (No Responses) | Categories: Surveys
Posted on December 13, 2005 by Robbin Steif
I write about exit surveys a lot because surveys are a great way to find out what visitors are really thinking (instead of playing guess-and-check all the time, the way that we web analysts do.) This time, I get to write something nice about a survey. (I know, you think I can only critize…)
My HP printer jammed up, so I finally got on the HP website for guidance. At some point, the site served up this message, “We’d like to ask you about your experience on our website. May we send you email in the next 24 hours?” The message asked for my email address and told me that my address would always be kept private. It was easy to answer “yes” because they didn’t insist that I drop everything to do their survey at that instant, and because they gave me privacy assurances at just the right moment.
When the email came, I had the opportunity to let it sit in my inbox until I was ready. I didn’t particularly love the survey — why can’t companies ask five good questions instead of 25 questions that all seem hard to understand? — but this time, they approached me in the right way.
Robbin Steif, CEO
LunaMetrics
View Comments (No Responses) | Categories: Surveys
Posted on November 15, 2005 by Robbin Steif
I always wonder what the conversion rates of those e-commerce exit surveys are.
As a conversion rate raiser, I take all the surveys that are offered to me. Today, I had the opportunity to do one for PriceGrabber instead of the usual BizRate survey. I was really surprised to see how best conversion practices were ignored. In the survey, they told me to include my order number and admonished, “Invalid order numbers may cause your survey to be rejected.” (Now, who really cares about the survey, them or me?) They asked me for my email address but gave me no link to a privacy policy. And then after I hit “Submit,” they committed the ultimate forms error — they reprimanded me for not filling out a required field, but didn’t indicate which one it was. Since all information from a survey is given freely and willingly, why don’t they take what they are given? When we website owners insist on collecting information that the customer doesn’t need to complete his transaction, we only create an incentive for the customer to lie. And how does that help our data??
Robbin
LunaMetrics
View Comments (1 Response) | Categories: Conversion Science, Surveys
Subscribe to this blog's feed
