The post on questions to ask WA Vendors started with a observation that it is perhaps more important that you ask yourself a few questions before you get into the Web Analytics tools selection process. These questions would help you understand your company needs in a very unique way which would help you pick the right tool.
The core point of that argument is simple: Most tools selection processes have very little self reflection built in, they might even be cookie cutter processes that might be from some book (or worse from some random blogger!! :)). This is not optimal.
Each company is unique and your approach to picking the right tool should be preceded with a lot of self reflection to ensure you understand who you really are and what you really need. (The “radical” selection process recommended in a earlier post forces this self reflection.)
This post covers three questions that should prompt the kind of critical self reflection that should increase the possibility that you’ll pick the right web analytics tool for yourself.
Summary:
# 1: Do you want reporting or analysis?
# 2: Do you have IT strength or Business strength? Or Both?
# 3: Are you solving for the Trinity or Clickstream reporting / analysis?
Now the questions, context and an attempt at providing some guidance…..
Do you want reporting or analysis?
This is a very difficult question to answer because most organizations have a very hard time being honest about what they need, humans are also poor at being self critical.
The other reason this is hard is that its like asking if you like a $0.99 Hershey chocolate bar or a piece of L’Artisan du Chocolat. It would be a sin not to say L’Artisan du Chocolat.
Everyone wants analysis, yet few organizations (especially ones that are greater than 100 people in total size) actually want analysis. They want reporting.
There are numerous reasons for this, including:
- Decentralized decision making.
- Company cultures (consensus, “cover your back”, layers of management, matrixed, paperwork driven and so on and so forth).
- Availability of tools / features. History.
- Propensity of risk (as in no propensity to take risk and risk taking actually harmful for career).
- Distribution of knowledge in people / teams.
- Availability of raw analytical brain power.
- And more….. (suggest your own in comments below).
The bottom line is that while everyone actually says that they want analysis. Admitting that you simply want reporting would be a sacrilege. Yet most companies simply want reporting. They want a web analytics tool and a web analytics team that simply facilitates reporting that is requested from Marketers, Business Leaders, VP’s, CMO’s and others.
The web analytics team might be told that they should provide analysis (remember given a free choice no one will opt for the Hershey’s bar), but they are not set up to have time or knowledge to provide analysis and if by their free will they do provide analysis then it rarely translates into acceptance and action. Success of the web analytics tool implementation is measured by the number of reports it provides, the number of KPI’s it is pushing out, the number of Marketers who say they are getting reports.
If you are deciding on what web analytics tool to choose you should take a really hard self critical look at your company, its decision making structure, its needs and then be honest and decide if you want reporting or analysis. Then decide on what tools in your selection criteria are good at reporting and what tools are good at analysis. Ignore the category you are not interested in (usually analysis).
This is a really hard choice to make, even harder to justify (because everyone thinks they want analysis). But having the tough conversation will ensure that your company will be happy with the choice that it makes, it will in all probability save money (reporting tools are much cheaper) and it is highly likely that on the long run you will be successful.
You choose the wrong tool (say a true analysis tool), it will only turn 
people off the tool and using data (because it will suck at reporting) and on the long run hurt you a lot more. You choose the right tool for your company (say that is really powerful at reporting) and you will glean a lot more traction since atleast they will look at something and your web analytics people will know what they are getting into. Over time your company, if the culture and org structure and risk taking are all lining up, will get smarter and maybe you’ll be ready to move to a analytical tool.
I am going out on a limb here but currently there are potentially only two true analysis tools: ClickTracks and VisualSciences. They come at massively different price points and have a very differentiated set of features and performance (speed, complexity, depth and breadth, sexiness). They are both right for the right kind of organizations, in case you decide you are it, and both need a deep self reflection on your part before you send a chq.
Other tools are also making great progress getting there, empowering true analysis out of the box to complement their far superior strength in being able to do awesome reporting.
IMPORTANT: Hopefully you’ll see that there is no judgment being made above if reporting is better or analysis is better. Each serves its purpose. The emphasis is on figuring out what you really need and then buying what you really need.
Do you have IT strength or Business strength? Or Both?
Some companies are good at IT (technology), others are good at the Business side of things (marketing, analysis, strategic decisions etc etc). A rare few are good at both, or have environments where the two are the same when it comes to Web Analytics.
Pulling off a successful web analytics implementation is complicated and it is often easy to get it wrong. A professional services company recently shared with me that seven out of ten times they find the tool implementation is wrong (and the client has not known about it for a while, and have been using wrong data).
If you have solid IT (technology) and Business strength in your company then go at it all by yourself and you’ll be fine (here are Web Analytics Technical Implementation Best Practices).
If you don’t have strong IT strength (by that I mean IT who knows and gets Web Analytics and not just standard IT) then go with a partner, say my buddy Justin at EpikOne (Justin let’s talk about my cut of this deal! – all joking aside I have not financial or other ties with EpikOne).
If you don’t have IT strength and you don’t want (or can’t afford a consultant) please consider following this process: How to Choose a Web Analytics Tool: A Radical Alternative.
Here is one last reason to assess IT strength: if you want to do in-house and not asp. If you want to consider hosting the data collection and analysis in-house (say with WebTrends, ClickTracks or Unica NetInsight) then you need some serious IT/Technology strengths (in atleast one person) to pull it all off. Ensure that you are covering this important consideration.
If you don’t have strong Business strength (realize that just by figuring this out honestly and you are already ahead of your peers) consider following this process: Web Analysis: In-house or Out-sourced or Something Else. It provides a great framework for the evolution of a effective and efficient web analytics program.
Are you solving for the Trinity or Clickstream reporting / analysis?
This is a mind-set question. It is a question that tries to judge what you are solving for. It tries to get to help you understand the level at which you are approaching the solution set. It is all about trying to know if you need to buy a tool that will help you “understand clicks” (which is ok) or do a lot more than that. It tries to help you crystallize what your short term goals are and what your long term goals are.
The Trinity mind-set & strategy calls for having a robust Qualitative and Quantitative analysis in your web analytics approach with the goal of: Understanding the customer experience explicitly (via Research), to then influence customer behavior on your site (via Clickstream analysis), leading to win-win Outcomes (via outcomes analysis) for your company.
Your consideration criteria will be vastly different depending on where you are and what your own point of view and approach is in this context. In one case a simple log parser is fine, in another you need a tool that integrates with other data sources and in yet another case you need a tool will pay ball with your data warehouse.
See this post: Web Analytics Tool Selection: 10 Questions to ask Vendors. The questions you’ll ask and what you’ll stress with change with the answer to the question above.
That’s it. Three questions.
A bit of my own self reflection & guidance:
I am on record saying “don’t spend too much time on worrying about web analytics implementation, turn on Google Analytics on your website and follow a simple, but “radical”, process”. Given that I feel a bit embarrassed at publishing two consecutive posts on Web Analytics Tool selection. Both posts combined I suspect highlight how complex the process of selecting a tool can be. To some of you it might even look scary as to how much thought you need to put into selecting the right tool for your company.
To put a nice bow around all this (and to make it easy for all of you), here is how to think about web analytics tool selection & implementation:
It’s that simple.
My apologies if there is any inadvertent confusion. On the bright side regardless of your size and what tools you own now you have a end to end overview of how to go from nothing to something to something awesome. Good Luck!!
What would be your own self-reflection questions? Will the above questions work for you? Am I missing anything obvious? Is this a good use of time? Vendors what else can we ask ourselves before we ask you to bare your soul? Please share your feedback via comments.
[Like this post? For more posts like this please click here.] 
Jerome N, a reader from the UK, asked me to write a post on the questions that should be presented to a Web Analytics Vendor during tool selection process. My instinctive reply to Jerome was “I think it is more important to know what questions to ask of yourself (your company) before you go into tool selection because the answers to those questions are a lot more critical and will tell you what you are ready for”.
My reaction was that because I feel companies don’t do a good enough job of understanding themselves (in a deeply self critical way) and hence pick a sub optimal solution. But it was also sourced from the fact that in vendor pitches all the answers are “yes we can do it” and it is very hard to discern reality. (If you are a Vendor read this post on tips on doing optimal presentation, if you are a Client read this post on web analytics tool selection Process).
But since Jerome was so kind to ask I wanted to respect his request. It is important to note that most of my suggestions below you would ask the for-fee (paid) Vendors, but you should also evaluate the free vendors in this context. I’ll also fess up that in a vendor pitch or conversation it is hard to get tease out the nuances but hopefully there is enough meat in rest of the post to empower you to tease out some reality.
10 Questions to ask Web Analytics Vendors: Summary:
1) Main differences with free tools.
2) Types of versions and flexibility.
3) Types of data collection options and entrenchment.
4) TCO (Total Cost of Ownership).
5) Type of Support (options, pricing, technical or business).
6) Segmentation awesomeness (post data capture).
7) Exporting data (options, history, data ownership).
8) Integration with other sources of data.
9) What’s up next, the competitive edge.
10) Types of business lost, why.
11) Bonus Question.
Jerome here are ten questions you should ask a Web Analytics Vendor before you get too deep into selection……..
1) What is the difference between your tool / solution and Google Analytics? Please share your top five, just five, reports that are different from GA and why?
(In a few months you can replace GA with our friend Ian’s Gatineau analytics package from Microsoft, or if you want log file based solutions then the free ClickTracks Appetizer.)
Might as well acknowledge the white elephant in the room. If there are good analytics tools out there for free why should you pay for a good analytics tool? Why not focus on following the 10/90 rule for web analytics success?
The answer you are looking for is not that Google and Microsoft are big and bad and your privacy is under threat. That’s a cop out if there ever was one. You are also not looking for a answer that you won’t get support for a free tool or that they will die and wither away.
You are looking for specific and tangible examples of reports and metrics that your for-fee vendor will provide that Google (and later Microsoft) don’t provide. Any analytics Vendor worth paying for will have a crisp answer that focuses on differential features, reports, metrics, integration points etc and not scare tactics.
I want to stress that GA and Gatineau are not right for everyone and don’t have every feature you want. You should carefully consider fit of those tools for your company. But it should be based on facts and not some of the FUD (fear, uncertainty and doubt) that is put forth.
2) Are you 100% ASP or do you offer a software version we can buy and install in-house? Are you planning to do a software version?
I am on the record saying one of the challenges Vendors will face is that Clients will want to have solutions that are software based and in-house rather than ASP based. Please see Five “Ecosystem” Challenges for Web Analytics Vendors. Currently most vendors are ASP based with no software based offerings (except for WebTrends and ClickTracks, I believe). That is ok for now, might not be so in a year or two.
With this question you are trying to probe for how ready the Vendor is for the future in terms of having differentiating offerings for you should you want them (and I think you will, not tomorrow but in the near future). You are also looking for the intangible: how they react to this question, just as much as the content of their answer.
You can also ask them about first part and third party cookies, which one they use and how much pain, and cost, would it be to use first party. You should almost always use first party cookies, most Vendors enable this. What you are looking for when you ask this question is their reaction. Did they proactively advice you to have first party cookies? Did they insist on it? It shows the mindset of the Vendor.
3) What data capture mechanisms do you use? Javascript only? Javascript and web logs? Web logs only? Others?
There are many different ways to capture data, javascript tags and web logs and packet sniffers and “sensors” and so on and so forth. Javascript tags and web logs are the most common, each have their merits (if you are stuck between making a choice between them, after having considered all options, here’s a post to help you make a choice).
In the answer to this question you are looking for a the kind of flexibility that a Vendor has natively in dealing with different data capture formats, you are looking for a Vendor who might evolve beyond javascript tags (or logs or sniffers) as the web evolves and becomes much harder to track (as in flash, flex, RIA’s, RSS, mash-ups and things we don’t even know). No current methodology will survive for a long time, is the Vendor you are considering ready for the near future data capture challenges?
You are not looking for them to brainwash you that javascript tags are God’s answer to all your prayers (or web logs or packet sniffers or “sensors”). If they try that chalk it up in the sub optimal sales choice column.
4) Please help me calculate the TCO (Total Cost of Ownership) for your tool.
Most Vendors will tell you that that the cost of their tool will be $30 per month for any site of any size and it is a all you can eat buffet. Ok so I am stretching that a bit, but by not that much. :)
The Total Cost of Ownership of a web analytics tool can be massively different depending on who you are as a company, what you already have it in place and, mostly, who your Vendor is and what their pricing strategies are. I encourage you to poke and dig for data to get a clear understanding of what the TCO is for each Vendor you are selecting.
Here are elements or TCO that you would have to consider:
= Cost Per Page View (since most ASP based vendors charge per page view)
+ Incremental costs beyond initial lump-some (charge if you go over your allocated page views, are there any “advanced” features - say RIA tracking or RSS or whatever in terms of extra modules that would cost more, what else could we buy from the vendor that we might need later – for example Pay Per Click integration with Google / YSM or keyword bidding feature or whatever, better to know now)
+ Annual support costs after year one
+ Cost of professional services (initial install and then post launch trouble shooting or customizations)
+ Any additional hardware you need at your end (pc’s, laptops, web servers, data storage drives etc, could vary by Vendor)
+ Cost of “administration” (manage vendor relationship – could be partial head count, someone to create all the reports and publish them, someone to coordinate between vendor and IT and marketers – all these could be one person, better to know now)
+ Cost of analysts needed to draw insights (you can lump this with the one above if you want but it is important to be aware of the 10/90 rule and realize that you can’t just buy the tool, you also have to hire a relatively intelligent brain to interpret the data – there are vendors who have stated that their tools are so smart that you don’t need Analysts, this would work in your favor but make sure you buy that)
+ Additional head count (partial or full) to maintain the tags, liaise with IT, update pages on the site etc
Total that up across Vendors and make a informed choice. The only thing above that it not totally in “unique to each Vendor” camp are the Analysts you’ll need, stress test that it is ok to break the 10/90 rule because the tool inherently is so smart.
Also notice that while some of the above are eliminated with a free tool other things above will still apply even in the case of a free vendor.
5) What kind of Support do you offer? What is included for free and what would cost more? Is free 24×7? What are the limits? How far will you go to help me solve my technical problems and answer “silly” questions from my business users?
During Vendor pitches you’ll hear that everything is free (and some web analytics Vendors do indeed offer loads and loads of absolutely free support as long as you stay with them). But often there are limits and caveats that are not explicit and you’ll have to dig them out.
Signing and contract and implementing the solution signifies the start of your problems and not the end of your data problems. It is critical that you understand exactly what you’ll get and exactly how much it will cost to get what you need (say if a Vendor will only provide business hour support during week days then what would 24×7 cost, or if they will only answer questions about the tool and not why the tool is not working with your site then what would that cost – just to stress these are just suggestions to get your juices flowing, you’ll have to make up your own unique questions).
You will need support and professional services, understand what the Vendor will provide or what the Vendor’s “Authorized Consultants” will provide etc.
6) What features do you have in your tool that allow me to segment the data?
Ian posted recently asking if we need Power or Simplicity (please cast your vote and help him). I agree with his grouping of sub features in each, except for one. Segmentation is not in Power. Segmentation should be in Simplicity. Segmentation should be everywhere. There should be no web analytics tool that does not allow for massive segmentation, in fact every tool should allow for massive segmentation in a very simple way!
Without segmentation all data is not very insightful, I know that sounds extreme but segmentation is the fastest path to insights. You should deeply stress your Vendor on exactly what options you have to segment data and how easy it is to do it. Try it yourself and see if you can segment data in the tool.
Also ask for one important point: Do you have to pre-code everything in custom javascript tags on each page on your site to be able to segment the data post capture? Or you can capture data with a standard tag and do segmentation later (even though you did not implement all the segmentation possibilities before launch in custom javascript tags)?
Most vendors are in the former camp, custom javascript tags on pages to enable any segmentation, and that makes segmentation much harder (how can you think of all the questions you’ll ask of the data up-front before you install the tool?). Again understand where the Vendor is and make a informed choice.
7) What options do I have to export data from your system into our company system? Can I get all the raw data? Can I export processed data? How easy it is for me to export 100,000 rows of Processed (not Raw) data out of your tool into my other company systems? What happens if I terminate my contract with you?
Ok I admit that’s a lot of questions, but it really is one important question: Who owns the data and if the Vendor stores it and you want to export it do you get the raw logs (huge data files with no intelligence in terms of computed metrics so you have to figure out how to do that) or Processed data (computed data that is much easier to integrate into where ever you are taking the data).
This question is as much a process of you discovering what you need and realizing that you are not going to get it and then being ok with that (or not).
Typically most vendors will say you can export everything. As them the specific questions above, understand exactly what you can actually export (remember that you can get a excel dump is not the answer which is why 100,000 rows are mentioned above) and then form a opinion if that is sufficient for your company or not.
Let me stress that I am not recommending that you insist on getting it all (data) or in a particular way, I am recommending that you ask the hard questions above so you don’t have any disappointment later about what you’ll get should you need it. I have gone into this rather naively and won’t do that again.
8) What kinds of features do you provide for me to integrate data from other sources into your tool?
There is so much on this blog about the Trinity model and multiple sources of data and how to get a complete picture and all that nice stuff. Bottom-line your clickstream data, no matter what Vendor you use, will feel limiting after a while when you want real insights and you’ll be forced to integrate it with other sources of data. Since the export route (#7 above) is so hard you’ll have to bring that data in. This question goes to the heart of figuring out how easy it is to do that for each Vendor you are considering.
Examples of data you might want to bring into your tool are: meta-data from other sources in your company, or you CRM data, or data from you ad / search agency, or data from surveys that has the primary key in there (like cookie values), or a/b / multivariate testing integration so you can measure conversion from the MVT tool but also analyze Site Overlay (Click Density) for each recipe, etc. You have to be able to import data efficiently (without needing humans if possible) and then use it for segmentation or reporting.
Some Vendors can automatically pick up data from Google AdWords and do direct integration without you having to do anything. Others will require you to do a daily download yourself and upload a text or csv file into your tool. Others still can’t do anything. Figure out what the line in the sand is for the Vendor you are considering for the kinds of data you want to integrate.
9) What are two new features / tools / analytical advancements / acquisitions is your company cooking up that would keep you ahead of your competition for the next three years?
At the core of this question you are trying to see if your Vendor is good at the today or if they are worried about the tomorrow and have plans to deal with tomorrow.
It also gives you a sense at how much they know about their own position and that of their competitors (hence we are not asking what are two things you are doing that are good, the framing is in context of competition). As our E-metrics 2006 DC summit experiment proved some Vendors are much much better at having a good reality check about themselves (read: Hello, My Name is Avinash. What is Unique About You?).
You want to be impressed by atleast one of the two things you hear being a complete surprise to you (which means they are ahead of you, always a good thing), and you want to get the feeling that your Vendor has a good sense for themselves and their competitors.
If you ask this exact same question across a few Vendors they will talk about each other and the differing perspectives are a source of valuable insights for you.
10) Why did the last two clients you lost cancel their contracts with you? Who are they using now? Would you permit us to call one of them?
I have to admit this is not one of my questions, a Vendor taught me this question and it is truly fantastic. 
You want to be confident that you are making the right choice and there is no better way than to learn why each Vendor recently lost someone’s business.
Now the reality is that you will probably hear marketing / sales speak and not reality but even hearing the marketing / sales speak can be of value. Of a whole bunch of vendors only two have ever answered this question honestly (both were VP of Sales answering the question) and the interesting thing is that we are doing business with both today, even though in both cases they are not the most awesome Vendor technologically.
Remember with any Vendor you are actually buying a relationship and not just the tool and on the long run (even near short term) people you can do business with will far outstrip value with than working with the most advanced tool on earth whose people you can’t have a relationships with.
Bonus question to ask:
11) Do you read Occam’s Razor by Avinash Kaushik and what do you think of his “radical” postings that tend to have a anti-vendor slant?
This one is only partly fun. A friend who is a respected industry authority and works at a vendor did say to me that he and the other Vendors got together and the sentiment that was expressed was “what does Avinash have against us”, he actually used even stronger language.
It was a huge surprise to hear this because this is absolutely not true and some my industry best friends are Vendors. But since you might ask them some of the above questions it is fair for you to ask them question 11 so they can give you their point of view on my stated and unstated biases.
Just for the record, I don’t have anything against any Vendor (explicit or implicit). We are lucky to have a good crop of tools in our industry, some good, some better, some awesome. I am glad that we can pick and choose and I am glad that my friends at Omniture, ClickTracks, Google Analytics, WebTrends, CoreMetrics, Instadia, indexTools, HBX, Visual Sciences, Unica NetTracker, etc read this blog, subscribe to it and often make public comments and send me private emails. I thank you all and I promise you there is absolutely no anti-vendor slant on any vendor.
What do you all think? Of #11 and all other questions? What am I missing in the list? Do you have your favorite question to ask any Vendor? Please share your feedback via comments.
[Like this post? For more posts like this please click here.] 
Absolute numbers are no very helpful (we had 459,245 unique visitors last month). Trends we have come to realize are better (Dec: 459,249 Nov: 591,067 Oct: 489,419). But there are customer interactions on the websites that results in outcomes for your company that yield trends that are rather difficult to decipher and translate into action.
One factor that is not appreciated enough is that every metric / KPI (Key Performance Indicator) that you report out of your web analytics tool (or indeed from your ERP or CRM or Data Warehouse) tends to have a natural “biorhythm”, i.e. those metrics / KPI will fluctuate up or down and change due to “natural occurrences” that just happen (I can see some of you cringe! :)).
These biorhythms are hard to understand, harder still to predict and since many of us live in the Puzzle world rather than the Mystery world we spin our cycle like crazy to understand the numbers to “explain” them to the management so that they can take some action. Imagine getting a daily / weekly trend and it goes up and down and you have no idea what the heck is causing it, even after you have done your damdest to isolate all the variables.
The result of these natural biorhythms is that it causes Analysts and Marketers to do analysis and deep dive where none is necessary, it causes some of us to look “bad” because we can’t explain the data, and it causes a lack of faith the the ability of data to provides insights.
Here is a great example that illustrates the issues:
It does not really matter what the numbers on this graph are and what the x-axis is. As you look at this at point 7 or 17 or 25 would you know what the trend is telling you and if it is a cause for concern or things are ok and you don’t need to take any action or the high points are causes for celebration?
One wonderful tool / methodology that I have found to be wonderfully helpful in separating signal from noise is from the world six sigma / process excellence and its called Control Limits (or Control Charts). Simply put control charts are really good at applying statistics to assess the nature of variation in any process. Translated into the biorhythm problem in relevant situations control charts can help trigger deep analysis and action.
Control charts were created to improve quality in manufacturing situations (or others like that) but they work wonderfully for us as well.
There are three core components of a control chart. A line in the center that is the Mean of the all the data points, a UCL (Upper Control Limit) and a LCL (Lower Control Limit).
Here is what a trend looks like with control limits overlayed on top:
What are Control Limits really?
Let us understand what you are looking at.
Mean (X): The green line above. A statistically calculated number that defines the average amount of variation in your KPI trend. For example for the above process it is 39.29.
UCL (Upper Control Limit): A statistically calculated number that defines the higher limit of variation in your KPI trend. In the example above it is 45.
LCL (Lower Control Limit): A statistically calculated number that defines the lower limit of variation in your KPI trend. In the example above it is 33.
The control chart above is illustrating a natural biorhythm in the KPI trend that is in between the two control limits, these are points that show natural variation in the metric and tentatively are not causes for doing anything, even though as you can clearly see they vary quite a bit from one data point to the next.
The massively cool thing is that it shows all the points in the trend, think of it as days or weeks or months, when you should have taken action because there was something unusual that occurred. It won’t, sadly, tell you what the heck happened, but it will tell you when you should use your precious time to dig deeper. Isn’t that awesome? Think of all the time you would have wasted solving the Puzzle behind the data points below the Mean, which look like “problems”.
So how do you compute these wonderful Control Limits (UCL & LCL)?
The general rule of thumb for calculating control limits is:
Control limits are calculated 3 standard deviations above or below the mean of your KPI data values. They are not assigned, but rather calculated based on the natural output of your data. Anything within the control limits should be viewed as expected variation (natural biorhythm). Anything outside of control limits warrants investigation. Not only that but if a series of data points fall outside the control limits then it is a bigger red flag in terms of something highly impactful going awry.
In a world where we are tons of metrics, where every dashboard has fifteen graphs on it, control limits are extremely helpful in leveraging the power of statistics to be the first filter of when you should dig deeper or look for a cause. If your metrics and trends have variations from day to day and week to week this is a great way to isolate what is “normal” and what is “abnormal” in the trend.
Control charts also scale very well. It would be easy if for every metric you have there is a clearly established Goal that you are shooting for. That goal can tell you how well, or not, you are performing. That is rarely the case for the massive deluge of metrics you have to deal with. It is scalable for you to apply control limits to all your trends.
Practical considerations in use of control charts (limits):
- Like with all things statistics the more data points you have the better your control limits will be, it would be hard to do a control chart that makes sense with just five data points (you can create it, it just won’t be very meaningful).
- Control limits work best with metrics / KPI’s where it is a bit easy to control for the impacting variables.
For example it would be less insightful to create control limits for your Overall Conversion Rate if you do Direct Marketing, Email Campaigns, Search Engine Marketing (Pay Per Click), Affiliate Marketing and you have loads of people who come directly to your site. There are too many variables that could impact your trend.
But you can easily create control charts for your Email Campaigns and PPC Campaigns or Direct Traffic and it will be very insightful because the variable is just one (or just a couple) and you will find excellent trigger points for performance and in turn analysis and in turn action.
- You do need to be able to understand a little bit of statistics and have some base knowledge around standard deviations etc so that you can leverage this optimally but also explain the power of what you are doing to your Senior Executives.
Practical example of using control limits:
The graph above shows a potential sample conversion rate of a website. Without the Red (UCL) and Blue (LCL) lines it is harder to know each month how the performance of direct marketing campaigns is faring. It is easy to know in Jan 2005 that performance was terrible. It is much harder to know that between March and July statistically there was nothing much to crab about even though the trend goes up and down.
This last point is important, anyone can eye ball and take action on a massive swing. What stymies most Analysts is separating signal from noise for non-massive swings in the data.
Consider using Control Limits on your KPI’s such as cart and checkout abandonment rates, you’ll be pleasantly surprised and happy at what you learn (as will your bosses).
Any decent statistical software will automatically calculate control limits and create these graphs for you. Minitab is the one that is used a lot by folks I know (though it is a tad bit expensive). We have also used our standard business intelligence tools to compute control limits for us (Brio, Business Objects, Cognos, MicroStrategy etc). You can also always simply jury rig excel to compute the limits for you (perhaps a reader of the blog can create a template that I can post here for everyone’s use; Update: Clint Ivy to the rescue! Here’s his blog post and here’s the wonderful spreadsheet he’s created for us. Please download the spreadsheet and plug in your own numbers.).
You can also read a little bit more about Control Charts and try two control chart calculators at SQC Online. In the What section give the control chart calculator for variables a spin.
This is a long and complex post but I hope that I have communicated to you the power of control charts, it is a bit dry and take a small bit of knowledge and patience but it is so powerful in helping your analysis specifically when it comes for separating signal from noise.
Signal -> Insights -> Action -> Happy Customers -> Money, Money, Money! : )
What do you think? Have you used control charts? What metrics do you think they will work best with? Should web analytics vendors include the ability to do control charts as a standard option in their tools? Is none of this making sense?
Please share your feedback and critique via comments.
[Like this post? For more posts like this please click here.] 
