As you can imagine choosing a web analytics tool is a critical process because whether you choose right or wrong you will be usually stuck with it for a while. Also because we tend to overstate the importance of historical web data (a post coming on this one of these days) it is quite likely that a quickie divorce will not be in the offing.
So in a world where choosing a web analytics tools seems akin to choosing a wife / husband / significant other, this blog post offers a radically different recommendation on how to choose your mate, sorry web analytics tool.
Summary: The current process of choosing a web analytics tool is fundamentally flawed. Why not change the process to: Implement Free Tool -> Learn -> Upgrade Analytics Skills -> Fix Data Capture -> Evaluate -> Now Choose?
The way tools are chosen at the moment is through a extensive process that usually looks like this:
- Collect all the business requirements (goals, strategy, KPI's, reports, reporting schedule and so on)
- Collect all the technical requirements (site architecture, servers, scripts, pages, IT needs and so on)
- Ensure that anyone who could ever need any kind of access to any kind of web data is contacted (inside and outside the company) and their needs documented
- Put all of above into a RFP (Request for Proposal), and add vendor financial stability, references, etc into the requirements
- Send to vendors, set a very aggressive reply schedule
- Receive RFP’s
- From those weed out the “insignificants”
- Selection of one vendor that meets the requirements by a esteemed committee
- Implement (champagne celebration included)
The search process takes two to four months, implementation one to two, and there is one guaranteed result of this process eight times out of ten: You will pick the most expansive, and usually one of three expensive, vendor.
Roughly the same number of times you would have made a sub-optimal choice and you are looking into three, six, twelve months of stress and having to deal with management questions that sound like this: “how come you are using a quarter million dollars a year web analytics tool and you are not recommending actions”.
The achilles heel of the above process is that the process involved people who asked for the earth and moon in the requirements (most of whom will never even log into the tool) and it was very divorced from the rough and tumble real world of the web, website and web analytics.
The process is too long, too time consuming, the process itself is expensive (just count the people, process, and time commitments from your company) and you’ll always pick the most expensive and expansive tool.
As a antidote to this sub optimal outcome here is a recommendation: 1) Ignore the nine step process above and 2) Don’t do a RFP.
Here is a radical alternative that will set you on a path to finding your right soul-mate (I mean web analytics tool), and do so faster and cheaper:
Step 0: Assign optimal ownership. (Day One)
- Project lead should be the person whose neck will be on the line to deliver web insights (not reports), for this post this person is you!
- Provide him/her with a small group of one to two people who will put the tool to hard-core use.
- Email the entire company (only slight exaggeration) to let them know you are selecting the tool.
Step 1: Implement a web analytics solution. (Day Two)
- If you want to use web logging data go get ClickTracks Appetizer
- If you want page tagging data go get Google Analytics
- If you don’t trust either go get StatCounter (even if it gives you bare bones data that is ok)
It takes one hour to get your hands on any of these tools. Five minutes to implement GA (tag into site footer, hit save, go get a drink and wait for data). Couple hours to implement CT, most of which will be you locating your website log files.
[Going out on a limb the recommendation is to stick to a javascript tag based solution because in the end it is quite likely that that most solutions use it so you might as well take a plunge now. That would be GA or StatCounter above.]
Step 2: Start using simple reports and process of creating a intelligent audience in your company. (Day Three)
- Email your core users (whose job 100% depends on the website, so very few) a report that shows: Traffic (Visitors), Referring URL’s, Search Key Phrases, Top Viewed Pages
- After a week set up a in person meeting to get feedback and questions
- Create second revision of reports
- After a week ask for feedback, in person, go to step 3 (because the feedback will tell you that the reports are not enough, show wrong stuff, they want more! : ))
In three days you went from nothing to actually having real data flow into your company and you are getting smarter by the second.
Step 3: Teach yourself the limitations of web analytics, tagging, numbers not matching, need to go redo your website IA / URL’s / ID’s / Cookies / data providing facilities. (Day 17)
- By now you have found that the reason you can’t answer questions is not the fault of your web analytics tool.
- Make a list of all the problems (you’ll need someone with slight technical skills if you don’t have them), they’ll usually be: URL structures, missing data (need for new url parameters of cookie values), maybe updated javascript tags, etc.
- Prioritize the problems (you plus your core team)
Step 4: IT “rules”! Cross your fingers, dive in. (Day 27)
- Partner with your IT / Website Tech team to roll out the fixes you need to get your website to cough up data
- This is often a painful process, cash in any chips you have (borrow some if you have to!)
- Don’t forget to keep your reporting going and enhance as new data becomes available
- Slowly increase your circle of data users
(Step 4 can take a lot of or little time depending on the size and mindset of your company. I am going to optimistically say this will take a month, YMMV.)
Step 5: Do a honest and deeply critical self review of where you are. (Day 57 or infinity)
After two months is your challenge probably is:
- That you have no idea what to do with the data?
- That you find that you don't know how to use the tool?
- That the tool does not provide you with what you want?
- That web analytics simply sucks?
- That nobody buys anything the tools is telling you?
In five steps you have accomplished the impossible: You actually have a lot of in-depth knowledge of the tool (Google Analytics, ClickTracks or StatCounter), you have data flowing into your company and you are fixing things on your website that would have caused problems for any tool (and you are doing this without a single cent going to your web analytics vendor).
Remember even with time in step 4 you have have moved further faster than in the standard nine step process.
What you do next depends on the output of your Self Review (step 5). My guess on where you’ll be:
- You’ll find out that reporting does not equal analysis and that you need a major upgrade in terms of the web analytics skills in your company [20%]
- You’ll find that data or tool are not your problem, it really is your company culture (both in terms of using data or getting your site tech teams to do things to give you data) [20%]
- You’ll find that Google Analytics or ClickTracks completely fill all your company web analytics needs [20%]
- You’ll find that ClickTracks or Google Analytics are not the right web analytics tools for your company [20%]
- You’ll realize that Web Analytics (clickstream data) is not sufficient to find web insights so you’ll take the money you would spend on web analytics vendors and spend it on Experience / Research analysis (see Trinity) [20%]
If the limitation is truly the tool then now you are ready for intelligent choice for a tough and practical selection process. My recommendations (whenever you are ready to chose a tool):
- Your RFP should now contain the specific and actual problems are are having with the limitations of your current tool (Google Analytics, ClickTracks). And it should only be about the tool. No vendor can come give you a warm hug and solve your problems with your inability to capture data or have people to do analysis. Those are your problems to solve. [UPDATE: For more on how to craft a optimal Request for Proposal (RFP) strategy click here.]
- Select differentiated vendors
- Remember the Big Three are pretty much the same in functionality you need (except in 1% of features that niche businesses will value), so if your final list is the Big Three then you might miss out on a real differentiated choice.
- If you want a real comparison bring in a vendor that is radically different. So consider CoreMetrics, Visual Sciences (not HBX), Instadia, IndexTools, NetInsight, ClickTracks Pro etc, each of whom, IMHO, have something significantly different to put on the table.
- Do a real Proof of Concept (implement the final set of vendors on your live site and compare with the free tool you were using to see if there is real differentiation).
So why should you do what some random blogger is recommending? If you follow the five step alternative process suggested above you’ll benefit from the following outcomes:
- You were not paying through your nose, ears, eyes, etc to first fix the problems you had in your company. (Data capture, basic intelligence up-leveling etc.) If you pick a expensive and expansive vendor you are paying them to simply identify that you have problems, you have the option of doing that for free (and since the process of fixing your problems is months it is a nice chunk of change).
- You have created atleast a core group of people who know what Web Analytics is everything that is frustrating and joyous about it:
- IT: They learn that it is trivial to implement a web analytics tool. Javascript tag, copy, paste, save, done.
- Web Site Developers: They learn all the little things that go into providing data that is business critical for actionability. Parameters to be added to url’s, page names to be updated, duplicated links in pages to be “lid’ed” so you can track them etc etc etc.
- Report Creators: They’ll learn that web analytics is less about report writing and more about analysis and this list might be handy for them to evolve to the next level.
- Web Analysts: They’ll learn that that they can use any tool to find a answer because they are Analysts. And of course that they have ultimate job security and we love them so.
- Marketers: Magic does not exist. Forethought has to be put into doing campaigns and coordination with web site developers and analyst before launch so that things can be tracked after launch. And no the web analytics tool will not make them coffee each day nor give them a massage.
- Business Leaders: Do a true assess of if they have the right skills in their folks, major gaps in their processes and that the true cost of web analytics not tools, it is the people (see the 10/90 rule).
- You will have chosen the best tool for your company, with your eyes open and you would have upgraded your company’s Analytics sophistication in the process.
- You will be promoted to VP of something as a reward and now have new worries to deal with. [There is a very very small chance that you would have been so frustrated that you are ready to quit!!]
Ok now your turn. What do you think? Is this truly a radical alternative? Could you, or I, ever pull this off? Would you change something about this approach? Please share your feedback via comments.
[Like this post? For more posts like this please click here.]
Nice. But I think only "Challenger" brands like yours take this path.
The post reminded me why so many always bought IBM, and why today they're still buying full meal deals from happy-to-sell vendors.
There's another corporate culprit at play. With the hype about perfect data and ROI metrics, many large company managers expect the web analytics people to lead a high profile charge to catch-up, which for many companies = build consensus first + use significant resources. By the time you know anything, the CMO will be gone and the CEO will have another priority.
Put the two together, add a splash of VC bubble bath, and voila, you've got a high growth category!
This isn't so much radical as "sensible and rather essential". Without employing these sorts of steps prior to paying for a WA package, one would never get a baseline measurement of one's company.
I work for a small business and this is therefore much easier to accomplish than, I would suspect, in larger firms.
I would go so far as to say that this is a useful tactic whether or not you already pay for a WA package. In the rush to get data, I would bet many, many companies missed out on why they were doing it and what benefits they expected to receive.
Valuable as ever, Avinash!
It seems to me that if clients followed this method, the enterprise-level web analytics vendors would benefit, too. It is always preferable to work with someone who has seen the value of your offering, and nobody wants a disillusioned client.
Thanks Avinash, I am tagging this article as a go-to reference for people who are wanting to get their feet wet with analytics!
Edward: The not so subtle point of this post was to move this type of a approach beyond the "challenger brands".
To make it more palatable, the whole challenging deal, the stress is on "look we can start for free, fix all the things that need fixing (people, processes, websites), then review, then go buy if it does not work and we'll make a smart decision".
Hopefully that will help "sell" this to a company of any size.
Love the VC Bubble Bath product!! :)
Jen: For Web Analytics vendors I think it is the choice between balancing for the short term or long term.
Some vendors will choose short term revenue to the long term benefit for their customers (give me money know because I have to report it to wall street and if we might not be right for you let's talk about that next year when you have lost your hair and are ready to cancel us).
Some vendors will choose a long term revenue model (go execute "Avinash's Challenger Method" and we'll be waiting for you because we are confident that we are your ultimate right choice and when you buy our product you will do so with open eyes and you'll be with our company for the rest of our existence).
Too pessimistic a evaluation of the current market? Not giving enough credit some place? Do give me feedback please.
You are of course right about the choice a vendor should make, and per your comment some are making.
Thanks to both of you for your kind comments.
I'll second Jill's comments about this being the sensible approach to selecting a tool. It should only take one trip through the selection dog and pony show to realize what a waste of time it is.
Experience also teaches that 95% of analytics tool use is for basic reporting that any tool could do regardless of whether it is free or $250k. This and the commodization of web analytics tools should really help bring people around to the method outlined in your post.
I actually am really happy with both Google Analytics and Net Tracker (running both on two different sites). Cheap, sturdy, and fast. To me the only difference between the two is in collection methods (jscript vs. web logs). Some sites are better suited for tags, while others work better with logs.
Applause!!
Very often the tool is not the problem when it comes to WA! It is culture or something else that causes the issue. So often it pains me to see companies invest in the category killer, spend the 20-100k to get it up and running when i notice that they really aren't using more functions than they could get with a tool that may have even been (dare I say…FREE).
Great post!
Sometimes I read your stuff and just try to understand you as a person. Are you able to go, verbally, where no web analyst has gone before because you are so smart, or because you have the guts to say what everyone else just thinks or because you always turn everything upside down and inside out before you look at it? Maybe it's the last because even your tone and style is so kind yet so irreverant at the same time.
There is such a big lesson here, and I don't just mean with regard to this post but with regard to lots of your posts. So often you are saying, forget what the experts say, forget the common wisdom, throw it out the window and just try to do what you would do if you owned the business and every dollar you spent were your own.
Your Raving Fan –
Robbin
Avinash: On your comment
I definately agree. In fact, that's what we've suggested to some of our clients. Before they can even ask us which Analytics to buy, they need to get their hands dirty in it and find out what's its like before they can even think about the big vendor solutions.
Great Post!
(
Shucks, we've had the same idea for some time. I should of blogged first! :)
)
I guess I just went on record for telling large corporations to buy Visual Sciences. Ok, I did not do all the things that Avinash said we should try first – at least not where I work (because I can't do it there).
If any other type of situation – smaller sites – or at least, companies that don't have so much "change control" Avinash's way might work.
Having my hands "dirty" in data – with headaches and eyeaches and often, not much to show for it – it was not hard to see why Visual Sciences looked so appealing! Read my post http://www.webmetricsguru.com/2006/09/visual_sciences_superior_web_a.html
I'm not even claiming this is workable for any but the most complicated sites – but honestly – do you think you can run sitemeter or statmeter or performancing at most corporations – they'd barf and the issue of letting the data be hosted outside their firewalls. Look, I have been trying to get FeedBurner in and right now we can't – for the reasons I mentioned ….and FeedBurner does not have a licensed version – if they did – we'd buy it in a second – no doubt.
As far as stakeholders and what they ask for …I have some of the most difficult stakeholders you can ever wish on someone – and boy do they work me to death …… and often, lets face it, when you give stakeholders the data – instead of finding out what they wanted to know in the first place – the data just leaves them with more questions than they had before.
Couldn't agree more, Avinash. In my WebAbacus days, the thing we feared the most was an RFP dropping onto the doormat.
This was partly because 9 out of 10 RFPs were the result of the 'typical' 9-step process, with the added spice that another vendor would often have had a hand in writing the RFP, so that we were frequently 'column fodder', as the saying goes.
A related challenge for us in this scenario was that because we took a bit of a different approach to web analytics, we often came out sounding lame or unconvincing in our RFP responses (because our response to so many questiosn that said "Can you do X?" was "yes, with customization"). The Big 3 dominated the format of such RFPs, meaning that different kinds of solutions found it difficult to compete.
Ian
A very insightful post…thank you for covering this taboo topic (free vs. paid analytics). I have heard so many naysayers for GA, but I am hard pressed to find a significant advantage in going paid right out of the gate. Learn for free, then apply what you have learned to your analytics strategy.
Do you have recommendations where one can learn Web Analyst skills?
Joe: Purely due to pressure of work and a lack of time I'll reply to your request by copy – pasting a email. Someone had asked a similar question via email: How to learn web analyst skills? (Or maybe it was where to learn web analyst skills.)
Here was my reply (I hope it is of some help, again apologies for reusing some content):
Avinash, nothing radical there at all. I agree with Ed re IBM. In those days people bought IBM because "you can't get fired for buying IBM" and same went for SAP etc. Thankfully others sought to think differently and today things are different. Here is a radical thought, though: Why not think about the results you want to achieve, at the atomic level, aligned with the business goals and work back from there? Having analytics as a "bag on the side" of an already existing (perhaps flawed) deployment, that only serves to tell you just that, is not a good investment of effort. How the wheel has turned in 25 years indeed !
Hi Avinash,
This post was mystic! Seriously, have you learned to read other people's minds? If so, please let me know where, they did a wonderful job!
Great post, I have already sent the link to my manager. Keep on writing, I am always surprised with your posts' actionability. They just fit the definition of a KPI worth of reporting from Eric Peterson: "All good KPI drive action. This is the polite way of saying, 'any KPI that, when it changes suddenly and unexpectedly does not inspire someone to send an email, pick up the phone or take a quick walk to find help, is not a KPI worth reporting."
Your posts inspire. Thank you.
Daniel Waisberg
As always, a most insightful post. I occasionally advise other companies on web analytics choices–I always tell them to start small and free and only when analytics is part of the culture, to upgrade to a more expensive tool. Your post is one I will refer my clients to.
I also say to my clients that "Implementation is the only thing that matters." I think your readership would benefit greatly if you could address the implementation issue, the importance of data architecture, and why marketers have to care about all this "technical" stuff like .js code and content categories, to get data that matters. :) Too many CMOs, raised in a "brand" environment, don't understand why they need to own the code on a Web Analytics tool, and why it's worth fighting with IT for it, and why every new feature needs a meeting to discuss how it will be tracked in the tool.
In my experience, too many senior managers don't realize that selection of the tool is only the first, and perhaps the least important step in the process.
Melinda
A provocative post, grounded in common sense. Well done.
Before doing anything, I recommend some hard thinking about how (and how much) value is created by the web site and how much more you can get by using analytics. Until you know that, why spend any time or money on analytics? This also helps determine what results are "good". Since most web analyics "failures" are not technical problems but organizational, this goes a long way to set the stage for success.
Jim
Hi Avinash,
"random blogger??" Hardly. Thank you for this great post!
Gradiva Couzin
http://www.yourseoplan.com/
I like your approach and the fresh ideas. But I miss the establishment of the company information need. In my experience this information need described in KPI's is the basis of succesful web analytics
I agree with you that the nine steps by itself will not lead tot a sufficient solution. To have a successful tool the company must know what KPI's will give them actionable insights into there web channel. Only than can they put the information to use and get some return on their web analytics inverstment.
How will your radical approach lead to an overview of this particular information need? Perhaps you could add a step in which the information need is established?
Daniel: In some ways, IMHO, it is a chicken and egg situation (which one came first the chicken or the egg?). I agree with you that it would be great to get good requirements up front. But…..
I have a limited experience in web analytics and in all the companies out there. But I have yet to find a company that can clearly articulate in a vacuum – when they don't have anything (or many many times even when they have something).
So let's say you don't have a analytics tools and I request you to give requirements. I can pretty much guarantee that you will ask for everything because you really don't know what you want. And even if you know what you want (by you I mean your business users) then right after they get the tool they will change their minds (this is ok and normal).
My "radical" suggestion hopes to avoid this problem completely. Don't define what you want (because you will surely not get it right), in 15 mins implement a tool, give it to users, get them to use it for a month, now go ask them what they want and there is a 90% chance that they will give you requirements that outline really what they want.
I hypothesize that it would take you a month to collect requirements, why not during that month give your users Google Analytics for free to use and increase the chance that they will define good requirements.
I stress that this might not apply to everyone and I offer this (and the post) as simply a alternative that folks can consider (something different from the usual).
I hope I have answered your question. Thank you for taking the time to read the post and share your feedback, I appreciate that.
Hi Avinash,
Its really great post for advanced user. I like your idea, “look we can start for free, fix all the things that need fixing (people, processes, websites), then review, then go buy if it does not work and we’ll make a smart decision”.
Thanks for invite me here. I linked your post frm my blog.
Regards
Bala Singam
This is another awesome post by Avinash. Too many corps go with the heavy requirements load, then buy a big, expensive package that takes forever to set up and get working right. Then, when it is not giving them the numbers they really want, they feel tied into it because they just spent a ton of money on it. This alternate approach–free and light then step up as needed is a very good idea. And it will save the customer money in the short term, which most will like. Just have to be careful about the dev time needed to impliment a new solution. Expectations will need to be managed around that.
Avinash:
I recently stumbled across your blog and what a treat it is! I work as a Sr. Web Analyst for a prominent Canadian bank and I found myself nodding at everything that you said. I can't tell you how much we struggled with not 1 but 2 big analytics vendors precisely because of the ask- everyone- and -their -neioghbour's- cat- for -requirements approach that you rail:) against. It was a real sense of deja vu! Thanks for articulating what many people already know but won't talk about.
I've been forwarding your blog left and right to folks I work with and we even plan to use some of your quotes for a presentation we have coming up..with proper citations of course:)
Thanks and keep 'em coming.
And what happens when you want to merge your business data with your user data? why go through all of this – if you have IT resources you might as well tag things on your own, go through your own pain and by the time you need to integrate and merge data to really gain insight to your business you wont have to drop you web analytics vendor and start over.
A lot of sence here – but as a "player", I would say that when you think about differenciated solutions you might look at speed-trap. I have lost count of the number of times that our customers cry "unique", "blinding insight" etc. in response to our solution and we are the ONLY organisations (you can tell from the Patents) which can deliver all the theoretical benefits of a "fully tagged" Web Analytics Solution (and much more besides) WITHOUT tagging your site… and then add real Business Information/Analyical System power like SAS9 from the SAS Institute to integrate your multichannel data and publish the results to your business owners and see the difference!
Give it a go….
Very interesting discussion.
my 2 cents:
* I like the use of free tools idea (now that there some good ones are available). This helps develop some skills and understanding => define what u want => get buy in => come to terms with analytics terms and how they apply to your business (visitor, session/visit, etc). More often than not, each package will leave something to be desired – one will need to make a trade off.
* A word of caution with the 'Free' tools:
1. there is a tendency to get to 'it's free why spend any money'
2. they are free because they help the provider in some way
just some thoughts
Interesting post. The web analytics market space is currently quite a confusing arena, especially for those not from a technical or analytical background.
For many web analytic tools are seen as a panacea to solve business issues by providing 'hard data'. Although this can be the case, it is far more important to have the people in place to make best use of the tools functionality.
We often find that clients already have several tools in place, and that they cheery pick functionality between the various tools. This obviously leads to confusion as each system will likely have its own rules and will therefore produce different results for seemly identical metrics – Conversion Rate is a classic!
The approach mentioned here is a great example of an iterative implementation 'journey'. Web Analytics is at its best when used over short date ranges, decisions are made and actions implemented, and then verified in the following period. Additionally, A/B and multivariate testing allows for a more rapid turn around, and can ultimately lead to self-learning / optimising web sites.
All web analytic tools have their 'killer' features as well as their idiosyncrasies! The key is learning these, and then planning accordingly. In my opinion there are still many circumstances where tool selection is based on who is seen as the largest vendor, rather than the tool providing the best fit for now and the future.
An honest view on what is actually needed, rather than a potential tools feature set is vital. We’ve talked with many clients that only need basic metrics, and have no real interest in further analysis, in which case there are many free or low cost solutions out there. However, web analytics is rapidly becoming an integral part of many business decisions, and as such it is vital that the tools can be tailored to that business’ needs, as well as providing auditable data for comparative purposes.
Overall, with flexibility often comes complexity. As such it is important to think carefully of what is really needed. We have encouraged several potential clients to use free tools initially, allowing them to ‘test the waters’. In many cases, we’ve found that they come back 6 months or a year later, having reached the ceiling or the tools capability. The difference now, if that their company culture has changed from seeing web analytics as an additional MI burden, to an essential data source which must be mined on a continual basis. As a consequence they therefore have a much firmer idea of what they need allowing for a much more efficient evaluation of paid tools.
Finally, there have been numerous posts regarding the hidden costs of ‘free’ tools, as so I won't repeat the arguments here. Ultimately, it comes back to planning and communication, and of course ROI.
Hi all, very interesting discussion.
Generally, the web analytic area is a little confusing for me.
But, your post is helping me very much in my metrics/web analytic journey.
just my 2 cents ;-)
David (from germany)
Yup, I totally agree. The normal process is just too long, and there are too many stake holders in the corporate world for a list of requirements to be achieveable without too much conflict.
In my opinion, most people (I.T. and business inclusive) don't really understand what they want from the analytic tools. Perhaps obligations and competitive pressure from the market is nudging everyone towards it?
I have, in the last month, pulled all the statistics I can together (from our weblogs through our very ancient webtrends v3), signed up for a google analytics account. All this hopefully will give our business decision makers a better idea of what they can expect, and whether this would be sufficient.
The next step would then be to determine what is lacking and then qualifying the possibilities.
This is awesome :) My company is in the process of "shopping for a tool" and some of the candidates are Omniture and Unica. The problem is there are only 2 of us who will probably be using this and I am just a reporting guy who has to do lots of reporting (other than analytics) you know what I mean?
We have GA so I totally agree the first step is to learn the skills and get this FREE thing going. We are using about 20% of its capacity at this point. I am currently working on identifying intelligent audience which is an art in itself (trying to convince my team that we are in the business of web analytics and not web-page production). I am still excited because I am crazy.
Inspired by your book I tried to dive deeper into web log data. Unfortunately ClickTracks Appetizer (also mentioned here) is no longer available for download. Can you recommend a comparable tool which is easy to get, setup and use?
Hello Mr. Kaushik, I really enjoyed your book Web Analytics: An Hour a Day, and now reading your blog, thanks a lot for sharing your knowledge!
Cheers
Arnold