Michael asked a great question the other day on the "best survey questions" post and I started to write a reply back in the comments section but it turned out to be too long so here is the answer as a post.
I'm interested in online survey capability. In terms of selecting a vendor for this, what are the important capabilities you should look for?
I assume that you need to be able to specify when and where the survey would be asked, and that this would need to be done as folks were leaving the task you want to measure? Is that right, and is there anything else that you feel is important?
I'll answer this question in two parts. First I'll set some context about surveying and the second part will be my humble attempt at providing tips to Michael about things to consider when buying a survey solution.
Part One: Pre – Pre Survey Solution Considerations
Before you go anywhere I feel like it is my duty to share three important things that you should seriously think about / be aware of….
1) You can do surveys on your own using surveymonkey or zoomerang etc, they are cheap and you can do things yourself. You don't get all the advanced features, controls and the analytical horse power to parse out the numbers. But can't beat the price and you can be up and running for as low as twenty bucks a month.
2) You can engage with a external partner to do this, iPerceptions, ForeSee etc. They come with many advanced features and lots of different ways in which you can target the survey. They do cost more. You have to determine the fit.
Here's a tip, both these companies want to sell to you (why would they not). Call 'em up, request them to present to you the glories of their technology (and ask them what they think about the other). Amazing learnings will follow for you. At they end you may or may not buy but you'll be smarter after that hour investment with each. Watch for specifics about their technology as much as their attitude.
3) There are two types of surveys. Page Level surveys and Site Level surveys. They are both very different and solve for radically different needs, they should not be confused as being the same.
Page level surveys are user initiated ("tell me what you think of this page" "rate this page" etc) and serve the purpose of measuring the experience of the page. You can ask "site" level experience questions but the way they are initiated by users and local level at which initiation happens makes them a sub optimal choice for measuring site experience. They can good for measuring effectiveness of a page.
Site level surveys are usually presented to the users ("please give us a few minutes to answer these questions"), typically on exit, and are a great measure of the site experience ("what got you here today" "what were you able to do" "how much did we suck" "did you find what you were looking for"). They are good at measuring effectiveness of a site.
It is important to be aware of the difference and be smart about the choice you make, I often run into companies that deploy page level surveys to measure site experience and then are surprised that changes have not impact on sales or improvement of satisfaction. Or vice a versa. [If you want to get deeper into this there is a lot more about this in my book.]
Part Two: Pre Survey Solution Considerations
Ok so have done some preliminary soul searching. You have also invited the survey companies to come pitch to you, at the end of which you are not much wiser. :)
From my personal experience here are things that you should think about / consider before you pick the vendor of choice….
#0] Just in case you skipped to this part of the post (!), the first thing to do is read the above and make those decisions!
#1] Mathematical Rigor: No matter what you choose look for a partner that can apply mathematical rigor to your results. I find so many results mis-interpreted because poor math applied in the analysis. Measuring survey results is not simply taking the average of the answers (averages lie!), it is measuring distributions and doing regressions. You don't want to be bothered to apply the statistics and statistical significance, stress test to ensure your vendor does (then you can focus on analysis and not reporting).
#2] On The Fly Segmentation Capabilities: Look for the capabilities that are provided to do on the fly segmentation of your data (aggregates lie!). With some vendors you get no segmentation capabilities, with others you'll get segmentation "on demand" (your ask for it and they'll send it to you later), and with others you get, on day one, access to a online environment where you can slice and dice your data at will.
One way or the other you want to be able to segment the data quickly and efficiently.
There is gold in your ability to pick a particular segment of traffic that absolutely hates you and slice it off and drill down to why they were on your site, what products they own, what did they not find, and… then go fix it fast. Segmentation rocks! You want to be able to do it efficiently, yourself if possible to reduce chances of vendor delays.
#3] Benchmarks & Indexes: Few people in senior management will take action when you tell them "our score on satisfaction (or content or navigation) is 6.0" (or 45, depending on your vendor). But most of them will get off their butt and give your money to take action when you go to them and say "our score on satisfaction is 6.0 and amazon.com is 9.4 out of 10".
Your actual score does not drive action, the difference between that and a industry benchmark drives action (no one wants to look bad by comparison!). And here is another subtle human factor: no one wants your opinion about anything, providing a comparison to a external benchmark depersonalizes the number and it is more likely that it will be "heard".
iPerceptions uses the iPSI (developed by iPerceptions), ForeSee user the ACSI (developed by University of Michigan), both wonderful benchmarks you can compare your scores to and help motivate your management to take action! There are other vendors, look to see if your vendor will provide you with a benchmark or a index to compare.
#4] Open Text "Categorization": No vendor is really awesome at this because it is a really hard problem to solve. Ask your vendor what kind of capabilities will they offer you to parse and categorize the open text responses (where the real gold is in terms of actionable insights).
If they say we can do it all for free and instantly, take that with a grain of salt and ask them to demonstrate it on your website's data.
Remember this is hard to you, and you will want it desperately, and you want to do with the best amongst some less than optimal choices (and not due to the vendor's fault, they are all trying).
#5] Survey Invitation "Type": There are several ways to initiate the survey. When the customer visits, on exit, during session, pop over, pop under etc etc.
In many cases initiating the survey on exit works optimally. A derivation of that method which also works well is that you can invite people when they see the first page on your site and show survey on exit, that works as well.
Stress test your vendor to see what methodology they use and if you can try a couple different ones. Remember what works for your customers might not be the same as my customers. You want to be able experiment.
#6] "Visitor Memory" / Cookie Sophistication: You want to look for companies that have a ability to set cookies so that once a visitor has been served a survey they won't see another one for, say, 90 days, regardless of if they fill it or not. Last thing you want is survey fatigue.
You can also partner with companies that allow you to survey only certain types of customers, say those that have seen x number of pages or those that came from zqi.com or those who…. you get the idea. Ask if your vendor provides you with such sophistication. Some do and others don't, depending on your need this may or may not be important.
#7] Integration With Web Analytics: Does your survey vendor allow you to integrate with your web analytics (clickstream) tool? Ask that question and ask them exactly how they do it (and get one of your smart IT guys to look at it with the FUD detector turned high).
After a while of doing surveys you will want to know: "what did my most satisfied visitors see on the site" "what campaigns / keywords drove the most unhappy traffic" "how do people who see the recommender tool in my clickstream data feel about likelihood to recommend" etc etc.
Integrating your survey tool with your web analytics data is not very hard. You should do it. You just need a friendly vendor and a basket of muffins (to give your IT person).
#8] Pilot Friendly: When you ask the above questions and make a choice it is quite likely that you'll deal with marketing and sales. The best way to learn about reality is to do a pilot. If your vendor is any good (and of course you are of a decent size!) the vendor will be willing to do a pilot with you, say for a month or six weeks.
If they refuse the pilot that will tell you something about them. Even if you have to pay a tiny amount up front to do a pilot it is actually a very good idea to do one. Not only to learn if the tool works for you but to, perhaps more importantly, to see if the people from the vendor are people you can work with.
Remember you rarely buy tools, almost always you are buying a relationship.
That's it, eight very simple things to look for as you choose your surveying vendor.
It is important to point out that the kind of rigor that is being recommended above you typically won't apply before signing up for a surveymonkey account. At $20 per month that is a significantly easier decision to make.
But if you are going to unlock all the power of surveying for learning about your customers and really having a solid "listening post" then you would want to stress test the above eight criteria.
[Disclosure: I am on the advisory council of iPerceptions, one of the companies I have mentioned in this post.]
What do you all think? Please share your tips, tricks, war stories, critique, brickbats via comments.
Couple other related posts you might find interesting: