Web Analytics Tool Selection: 10 Questions to ask Vendors

White GoldJerome N, a reader from the UK, asked me to write a post on the questions that should be presented to a Web Analytics Vendor during tool selection process. My instinctive reply to Jerome was “I think it is more important to know what questions to ask of yourself (your company) before you go into tool selection because the answers to those questions are a lot more critical and will tell you what you are ready for”.

My reaction was that because I feel companies don’t do a good enough job of understanding themselves (in a deeply self critical way) and hence pick a sub optimal solution. But it was also sourced from the fact that in vendor pitches all the answers are “yes we can do it” and it is very hard to discern reality. (If you are a Vendor read this post on tips on doing optimal presentation, if you are a Client read this post on web analytics tool selection Process).

But since Jerome was so kind to ask I wanted to respect his request. It is important to note that most of my suggestions below you would ask the for-fee (paid) Vendors, but you should also evaluate the free vendors in this context. I’ll also fess up that in a vendor pitch or conversation it is hard to get tease out the nuances but hopefully there is enough meat in rest of the post to empower you to tease out some reality.

10 Questions to ask Web Analytics Vendors: Summary:

    1) Main differences with free tools.
    2) Types of versions and flexibility.
    3) Types of data collection options and entrenchment.
    4) TCO (Total Cost of Ownership).
    5) Type of Support (options, pricing, technical or business).
    6) Segmentation awesomeness (post data capture).
    7) Exporting data (options, history, data ownership).
    8) Integration with other sources of data.
    9) What’s up next, the competitive edge.
    10) Types of business lost, why.
    11) Bonus Question.

Jerome here are ten questions you should ask a Web Analytics Vendor before you get too deep into selection……..

1) What is the difference between your tool / solution and Google Analytics? Please share your top five, just five, reports that are different from GA and why?
(In a few months you can replace GA with our friend Ian’s Gatineau analytics package from Microsoft, or if you want log file based solutions then the free ClickTracks Appetizer.)

    Might as well acknowledge the white elephant in the room. If there are good analytics tools out there for free why should you pay for a good analytics tool? Why not focus on following the 10/90 rule for web analytics success?

    The answer you are looking for is not that Google and Microsoft are big and bad and your privacy is under threat. That’s a cop out if there ever was one. You are also not looking for a answer that you won’t get support for a free tool or that they will die and wither away.

    You are looking for specific and tangible examples of reports and metrics that your for-fee vendor will provide that Google (and later Microsoft) don’t provide. Any analytics Vendor worth paying for will have a crisp answer that focuses on differential features, reports, metrics, integration points etc and not scare tactics.

    I want to stress that GA and Gatineau are not right for everyone and don’t have every feature you want. You should carefully consider fit of those tools for your company. But it should be based on facts and not some of the FUD (fear, uncertainty and doubt) that is put forth.

2) Are you 100% ASP or do you offer a software version we can buy and install in-house? Are you planning to do a software version?

    I am on the record saying one of the challenges Vendors will face is that Clients will want to have solutions that are software based and in-house rather than ASP based. Please see Five “Ecosystem” Challenges for Web Analytics Vendors.  Currently most vendors are ASP based with no software based offerings (except for WebTrends and ClickTracks, I believe). That is ok for now, might not be so in a year or two.

    With this question you are trying to probe for how ready the Vendor is for the future in terms of having differentiating offerings for you should you want them (and I think you will, not tomorrow but in the near future). You are also looking for the intangible: how they react to this question, just as much as the content of their answer.

    You can also ask them about first part and third party cookies, which one they use and how much pain, and cost, would it be to use first party. You should almost always use first party cookies, most Vendors enable this. What you are looking for when you ask this question is their reaction. Did they proactively advice you to have first party cookies? Did they insist on it? It shows the mindset of the Vendor.

3) What data capture mechanisms do you use? Javascript only? Javascript and web logs? Web logs only? Others?

    There are many different ways to capture data, javascript tags and web logs and packet sniffers and “sensors” and so on and so forth. Javascript tags and web logs are the most common, each have their merits (if you are stuck between making a choice between them, after having considered all options, here’s a post to help you make a choice).

    In the answer to this question you are looking for a the kind of flexibility that a Vendor has natively in dealing with different data capture formats, you are looking for a Vendor who might evolve beyond javascript tags (or logs or sniffers) as the web evolves and becomes much harder to track (as in flash, flex, RIA’s, RSS, mash-ups and things we don’t even know). No current methodology will survive for a long time, is the Vendor you are considering ready for the near future data capture challenges?

    You are not looking for them to brainwash you that javascript tags are God’s answer to all your prayers (or web logs or packet sniffers or “sensors”). If they try that chalk it up in the sub optimal sales choice column.

4) Please help me calculate the TCO (Total Cost of Ownership) for your tool.

    Most Vendors will tell you that that the cost of their tool will be $30 per month for any site of any size and it is a all you can eat buffet. Ok so I am stretching that a bit, but by not that much. :)

    The Total Cost of Ownership of a web analytics tool can be massively different depending on who you are as a company, what you already have it in place and, mostly, who your Vendor is and what their pricing strategies are. I encourage you to poke and dig for data to get a clear understanding of what the TCO is for each Vendor you are selecting.

    Here are elements or TCO that you would have to consider:

       = Cost Per Page View (since most ASP based vendors charge per page view)

      + Incremental costs beyond initial lump-some (charge if you go over your allocated page views, are there any “advanced” features – say RIA tracking or RSS or whatever in terms of extra modules that would cost more, what else could we buy from the vendor that we might need later – for example Pay Per Click integration with Google / YSM or keyword bidding feature or whatever, better to know now)

      + Annual support costs after year one

      + Cost of professional services (initial install and then post launch trouble shooting or customizations)

      + Any additional hardware you need at your end (pc’s, laptops, web servers, data storage drives etc, could vary by Vendor)

      + Cost of “administration” (manage vendor relationship – could be partial head count, someone to create all the reports and publish them, someone to coordinate between vendor and IT and marketers – all these could be one person, better to know now)

      + Cost of analysts needed to draw insights (you can lump this with the one above if you want but it is important to be aware of the 10/90 rule and realize that you can’t just buy the tool, you also have to hire a relatively intelligent brain to interpret the data – there are vendors who have stated that their tools are so smart that you don’t need Analysts, this would work in your favor but make sure you buy that)

      + Additional head count (partial or full) to maintain the tags, liaise with IT, update pages on the site etc

    Total that up across Vendors and make a informed choice. The only thing above that it not totally in “unique to each Vendor” camp are the Analysts you’ll need, stress test that it is ok to break the 10/90 rule because the tool inherently is so smart.

    Also notice that while some of the above are eliminated with a free tool other things above will still apply even in the case of a free vendor.

5) What kind of Support do you offer? What is included for free and what would cost more? Is free 24×7? What are the limits? How far will you go to help me solve my technical problems and answer “silly” questions from my business users?

    During Vendor pitches you’ll hear that everything is free (and some web analytics Vendors do indeed offer loads and loads of absolutely free support as long as you stay with them). But often there are limits and caveats that are not explicit and you’ll have to dig them out.

    Signing and contract and implementing the solution signifies the start of your problems and not the end of your data problems. It is critical that you understand exactly what you’ll get and exactly how much it will cost to get what you need (say if a Vendor will only provide business hour support during week days then what would 24×7 cost, or if they will only answer questions about the tool and not why the tool is not working with your site then what would that cost – just to stress these are just suggestions to get your juices flowing, you’ll have to make up your own unique questions).

    You will need support and professional services, understand what the Vendor will provide or what the Vendor’s “Authorized Consultants” will provide etc.

6) What features do you have in your tool that allow me to segment the data?

    Ian posted recently asking if we need Power or Simplicity (please cast your vote and help him). I agree with his grouping of sub features in each, except for one. Segmentation is not in Power. Segmentation should be in Simplicity. Segmentation should be everywhere. There should be no web analytics tool that does not allow for massive segmentation, in fact every tool should allow for massive segmentation  in a very simple way!  

    Without segmentation all data is not very insightful, I know that sounds extreme but segmentation is the fastest path to insights. You should deeply stress your Vendor on exactly what options you have to segment data and how easy it is to do it. Try it yourself and see if you can segment data in the tool.

    Also ask for one important point: Do you have to pre-code everything in custom javascript tags on each page on your site to be able to segment the data post capture? Or you can capture data with a standard tag and do segmentation later (even though you did not implement all the segmentation possibilities before launch in custom javascript tags)?

    Most vendors are in the former camp, custom javascript tags on pages to enable any segmentation, and that makes segmentation much harder (how can you think of all the questions you’ll ask of the data up-front before you install the tool?). Again understand where the Vendor is and make a informed choice.

7) What options do I have to export data from your system into our company system? Can I get all the raw data? Can I export processed data? How easy it is for me to export 100,000 rows of Processed (not Raw) data out of your tool into my other company systems? What happens if I terminate my contract with you?

    Ok I admit that’s a lot of questions, but it really is one important question: Who owns the data and if the Vendor stores it and you want to export it do you get the raw logs (huge data files with no intelligence in terms of computed metrics so you have to figure out how to do that) or Processed data (computed data that is much easier to integrate into where ever you are taking the data).

    This question is as much a process of you discovering what you need and realizing that you are not going to get it and then being ok with that (or not).

    Typically most vendors will say you can export everything. As them the specific questions above, understand exactly what you can actually export (remember that you can get a excel dump is not the answer which is why 100,000 rows are mentioned above) and then form a opinion if that is sufficient for your company or not.

    Let me stress that I am not recommending that you insist on getting it all (data) or in a particular way, I am recommending that you ask the hard questions above so you don’t have any disappointment later about what you’ll get should you need it. I have gone into this rather naively and won’t do that again.

8) What kinds of features do you provide for me to integrate data from other sources into your tool?

    There is so much on this blog about the Trinity model and multiple sources of data and how to get a complete picture and all that nice stuff. Bottom-line your clickstream data, no matter what Vendor you use, will feel limiting after a while when you want real insights and you’ll be forced to integrate it with other sources of data. Since the export route (#7 above) is so hard you’ll have to bring that data in. This question goes to the heart of figuring out how easy it is to do that for each Vendor you are considering.

    Examples of data you might want to bring into your tool are: meta-data from other sources in your company, or you CRM data, or data from you ad / search agency, or data from  surveys that has the primary key in there (like cookie values), or a/b / multivariate testing integration so you can measure conversion from the MVT tool but also analyze Site Overlay (Click Density) for each recipe, etc. You have to be able to import data efficiently (without needing humans if possible) and then use it for segmentation or reporting.

    Some Vendors can automatically pick up data from Google AdWords and do direct integration without you having to do anything. Others will require you to do a daily download yourself and upload a text or csv file into your tool. Others still can’t do anything. Figure out what the line in the sand is for the Vendor you are considering for the kinds of data you want to integrate.

9) What are two new features / tools / analytical advancements / acquisitions is your company cooking up that would keep you ahead of your competition for the next three years?

    At the core of this question you are trying to see if your Vendor is good at the today or if they are worried about the tomorrow and have plans to deal with tomorrow.

    It also gives you a sense at how much they know about their own position and that of their competitors (hence we are not asking what are two things you are doing that are good, the framing is in context of competition). As our E-metrics 2006 DC summit experiment proved some Vendors are much much better at having a good reality check about themselves (read: Hello, My Name is Avinash. What is Unique About You?).

    You want to be impressed by atleast one of the two things you hear being a complete surprise to you (which means they are ahead of you, always a good thing), and you want to get the feeling that your Vendor has a good sense for themselves and their competitors.

    If you ask this exact same question across a few Vendors they will talk about each other and the differing perspectives are a source of valuable insights for you.

10) Why did the last two clients you lost cancel their contracts with you? Who are they using now? Would you permit us to call one of them?

    I have to admit this is not one of my questions, a Vendor taught me this question and it is truly fantastic. Patriots AFC '07 Championship LossYou want to be confident that you are making the right choice and there is no better way than to learn why each Vendor recently lost someone’s business.

    Now the reality is that you will probably hear marketing / sales speak and not reality but even hearing the marketing / sales speak can be of value. Of a whole bunch of vendors only two have ever answered this question honestly (both were VP of Sales answering the question) and the interesting thing is that we are doing business with both today, even though in both cases they are not the most awesome Vendor technologically.

    Remember with any Vendor you are actually buying a relationship and not just the tool and on the long run (even near short term) people you can do business with will far outstrip value with than working with the most advanced tool on earth whose people you can’t have a relationships with. 

Bonus question to ask:

11) Do you read Occam’s Razor by Avinash Kaushik and what do you think of his “radical” postings that tend to have a anti-vendor slant?

    This one is only partly fun.  A friend who is a respected industry authority and works at a vendor did say to me that he and the other Vendors got together and the sentiment that was expressed was “what does Avinash have against us”, he actually used even stronger language.

    It was a huge surprise to hear this because this is absolutely not true and some my industry best friends are Vendors. But since you might ask them some of the above questions it is fair for you to ask them question 11 so they can give you their point of view on my stated and unstated biases.

    Just for the record, I don’t have anything against any Vendor (explicit or implicit). We are lucky to have a good crop of tools in our industry, some good, some better, some awesome. I am glad that we can pick and choose and I am glad that my friends at Omniture, ClickTracks, Google Analytics, WebTrends, CoreMetrics, Instadia, indexTools, HBX, Visual Sciences, Unica NetTracker, etc read this blog, subscribe to it and often make public comments and send me private emails. I thank you all and I promise you there is absolutely no anti-vendor slant on any vendor.

What do you all think? Of #11 and all other questions? What am I missing in the list? Do you have your favorite question to ask any Vendor? Please share your feedback via comments.

[Like this post? For more posts like this please click here.]

Comments

  1. 1
    Melinda B says

    Avinash,

    Re: the web analytics vendors (and I know them too) reading your blog–they should be grateful to have a customer/potential customer who cares enough to articulate his concerns so intelligently, clearly and persuasively.

    For too long most vendors have hidden the truth about TCO in these ASP programs and that is coming back to bite them. Also, the page view model does not scale for a corporation financially as it grows (I find myself in the unacceptable position sometimes of not tracking certain pages because it will cost too much–totally counter productive), and as big companies realize this, they are turning away. Some are giving up, some are building their own solutions. As Web 2.0 comes in that will be more of a factor.

    Any company who gets upset about truth from its customers doesn't deserve to be in business.

    Again, keep it up. You represent the views of a lot of us industry practicioners. Thanks for taking the time to do what you do here.

  2. 2

    Hi Avinash,

    Awesome top 10 questions. In regards to #2, may I add another recommended question? Ask your vendor not only whether they offer both ASP and software, but also ask them what happens if your requirements change and you want to switch from one to the other in later years. Can you take your data and metrics with you?

    To be fair, I have to admit that I am a biased vendor myself (Unica) and that I am suggesting this question because our web analytics offerings (NetInsight and NetTracker) are available both as software and ASP.

    Educated customers are better customers, we at Unica agree!
    Akin

  3. 3
    Jonghee Jo says

    What a wonderful post Avinash. I like Question #10 most. Keep up the good work!

  4. 4

    One of your best posts Avinash, direct practical application to our lives (as Users) and if I may say so lots of food for thought for the Vendors (web analytics or otherwise).

    My favorites are #4-TCO, #8-Integration and #10-Judging relationship potential.

    I agree with Melinda, I don't sense any bias. In fact your name came up constantly in our recent vendor discussions and they were quite complementary about your blog.

  5. 5

    All great questions, but don't miss the obvious ones: how long has the vendor been in business, when was the product first released, how many employees, how many active customers, what is their growth rate? Basically you want to be sure the product is reasonably mature and that the company will be in business for a while. If the product is free, figure out where the money is coming from. If you are a company of some size, make sure they have other clients as big as you are. References are so obvious that I assume you've left them off the list for a reason. If you want to get a little more subtle, ask about their problem reporting / bug fix process–partly to see whether they even have real numbers available and partly to get a sense of how responsive they are.

  6. 6

    David: Great additions, thank you. I am not the biggest fan of references. You get the reference names from the Vendor / Agency you are evaluating. They have a vested interest in looking good so they will find references that will make them look good. Now once in a while you talk to a reference and they will give you the real skinny but usually that is not the case. Just IMHO.

    My favorite in your list is the problem reporting one, you are absolutely right about the sense that will provide.

    Thanks again for taking the time to share your comments.

    -Avinash.

  7. 7

    I wondered why my blog traffic had gone up again this week… thanks for another insightful post.

    Weird that the vendors should be moaning about your apparent 'bias' against them. Why would they expect you to swallow their Kool-Aid? Web analytics (or the industry, at least) is full of smoke and mirrors; the questions you pose here are a good tool at blowing some of the smoke away (although #10 does have a certain bowel-loosing quality about it).

    I can say with honesty that when I worked for an enterprise WA vendor, there was always the list of 'ugly little secrets' that we hoped no potential clients would stray onto. For us, it was always performance, with support at #2.

    Ian

  8. 8
    Tim Leighton-Boyce says

    Regarding references. I'd certainly always want to talk to people who have used the system. Even talking to people at the friendly sites nominated by the vendor will often give you an insight into the personality of the implementation team and the support people. You can soon pick up the difference between genuine enthusiasm and just 'normal'.

    As someone who is occasionally used as a reference by one vendor I'm always conscious of a feeling of some sense of responsibility which can make the process a bit restrained. Until we get on to the subject of the area in which they really excelled and then the (favourable) truth bubbles out, I think.

    The other obvious thing to do is to search around and you'll usually unearth users of a system posting about it. In one case (CRM not analytics) I came across some ex-colleagues who were using one of the systems on our short list. I gave them a call and learnt some very frightening details about what keeping it running smoothly involved…

  9. 9

    References are a funny thing. You'd think vendors would hand-pick them and they'd therefore be useless, but that's often not the case. One thing we do is ask for references at sites with needs as similar as possible to our own. If a vendor can't come close, that's a big red flag. Another thing to ask the references which other vendors they considered, and why they picked the one they chose. This can identify some other options, and the reasons can be very enlightening–both about the vendor and about the whether the reference is worth listening to.

  10. 10

    Many many thanks Avinash! This is excellent!

  11. 11

    #6 (segmentation) hits a chord with me which leads me to a revision (or additional) twist to #8 that would be, "what type of segmentation data do you allow me to integrate from your tool into other sources."

    As targeting becomes more ingrained into marketing technologies this type of cross-platform sharing of data clusters will become essential.

  12. 12

    It's funny that you should mention about the "support" issue being the major thing that some San Diego and Utah based companies mention when it comes to differences between their products and Google for example. When clients tell us about discussions that they have with them that is the #1 issue that they bring up as a "difference."

  13. 13

    Hi Avinash,

    Sorry for our late reply (I'm catching up my reading ;-)). Thanks for your post we also relly like your number 10 question. And I think that this question is way better than references ;-)

    we would add a little question regarding their financial situation and warranties regarding the continuity of the product in the case of a merger or acquisition. We are living times of change in the Web analytics industry as the Instadia sale to Omniture has showned us recently.
    Getting back to Instadia we wonder what their clients are actually thinking about their data. The data is currently in Europe and a migration towards Omniture tools would mean a data transfert to the United States. Are financial Scandinavian Institutions confortable about that?

    I take this opportunity to call for support regarding a possible 'privacy' issue and thoughts about a possible possible petition for Omniture to install a data center in Europe who would fall under the European law (and not US). Does any one know womeone at the European Commission that might support such an initiative?

    Avinash, I hope you'll forgive our plug ;-)

    A Rifkin convert couple (The European Dream
    http://www.youtube.com/watch?v=K-rULPjX8VM),

    Aurélie & René
    http://webanalytics.wordpress.com
    http://www.ox2.be

  14. 14

    Do you know a good analytic tool which will allow for real report customization? For example, our customized report is using 2 different Webtrend profile and require calculation for ratio. Webtrend does not provide sufficient flexibility to do this…how about Omniture or others? I just want to build a report which will get the any existing data from the different profiles I have and do the operation I need…I little bit like a spredsheet. Customized reporting seems to be a weakness in this industry?

    Let me know, Thanks :0)

    Thanks,

  15. 15

    Thierry: Take a look at indexTools, it provides probably one of the most flexible interfaces for providing custom reports (any metric by any dimension with any filter – of course with your intelligence applied!). But it does not do the "math" (new metric in column three equals column one minus column two).

    HBX also has a "report builder" where you can do some flexible reporting / extraction, especially for Excel purposes. Unica's NetInsight solutions also provide flexibility in reporting anything by anything.

    Other high end tools allow you more advanced custom reporting. Were you not able to do this with WebTrends Marketing Lab? I think you should totally be able to do it.

    -Avinash.

  16. 16

    Thank you Avinash for your quick answer and great recommendations we will explore.

    According to my information, Webtrends has some limitation. The Marketing warehouse, which I believe is the same as the Marketing Lab you are reffering to, is not the option to meet our needs. The reasons are as follow:

    We have to use the SDC – that is tracking clients by the use of a java script embedded into each page of the site. Many clients turn off js for fear of the scripts including a virus. Those clients with js turned off would not be tracked in MWH.

    Different data sources: To get information from two data sources ie comparing filtered vs non filtered logs, you would have to write a custom SQL script since this is not a setting you can turn on in MWH and hence forth have that data.

    MWH seems to bring more reporting flexibility as a Marketing campaign analytic tool (qualified visitors which respond to campaigns) than as a general Web analytic tools. Only some information qualifies to go INTO the database. Most technical info is filtered out of the database to streamline the system. Which means that without client groups set up, MWH serves no purpose for us.

    Cost & support: MWH is not a WebTrends add-on, but a separate system, it too has its own annual license and support costs (presumably also based on usage of the system).

    Thanks,

    Thierry

  17. 17

    Your posts are always full of energy and information. I can't seem to get enough of your book that I keep visiting your site for more :-)

    Thanks.

Trackbacks

  1. Garçon aka Martin Kopta » Blog Archive » links for 2007-01-29 says:

    […] Web Analytics Tool Selection: 10 Questions to ask Vendors » Occam’s Razor by Avinash Kaushik Differences with… Versions and flexibility. Data collection options. Total Cost of Ownership. Type of Support. Segmentation (post data capture). Exporting data. Integration with data sources. What’s the competitive edge. Types of business lost, why. (tags: analyses) […]

  2. The Analytics Question List – Your Help Wanted…

    With this rush of analytically minded visitors we're seeing today based on the link from Avinash, I thought I'd take the opportunity to solicit some information. So you may have thought you were coming to read a blog, but I'd……

  3. […] Gode tips for alle som er på jakt etter verktøy for webanalyse […]

  4. Web Analytics Books – The Bonanza Begins…

    I just found out that two of the very best web analytics minds have new books coming out shortly: Avinash Kaushik and Jason Burby. This is great news for anyone with an interest in analytics or online marketing. Readers of……

  5. […] Check out Occam’s Razor by Avinash Kaushik. He just published a post comparing web analytics tools and has a great "10 Questions to Ask Vendors" post when evaluating traffic tools. […]

  6. […]

    Web Analytics Tool Selection: 10 Questions to ask Vendors » Occam’s Razor by Avinash Kaushik > Web Analytics Blog
    (tags: analytics webanalytics webbstatistik)

    Web Analytics Tool Selection: Three Questions to ask Yourself » Occam’s Razor by Avinash Kaushik > Web Analytics Blog
    (tags: analytics webanalytics webbstatistik)

    […]

  7. […]
    Viited
    Kümme küsimust tarkvara tootjale
    Kolm küsimust iseendale
    Kuidas valida veebianalüütika tööriista
    Tänud Peebule ja Priidule, kes mind Avinashi saidile juhatasid.
    […]

  8. […] 4/ Web Analytics Tool Selection: 10 Questions to ask Vendors […]

  9. […]
    Qué necesita un web optimizer para poder hacer su trabajo? En primer lugar, necesita contar con herramientas de analítica web y de prueba A-B o multivariante. Además, requiere apoyo y dedicación continua (no en ratos libres que nunca hay) de IT y de diseño ya que hay que preparar los test y las alternativas a probar.
    […]

Add your Perspective

*