Absolute numbers are no very helpful (we had 459,245 unique visitors last month). Trends we have come to realize are better (Dec: 459,249 Nov: 591,067 Oct: 489,419). But there are customer interactions on the websites that results in outcomes for your company that yield trends that are rather difficult to decipher and translate into action.
One factor that is not appreciated enough is that every metric / KPI (Key Performance Indicator) that you report out of your web analytics tool (or indeed from your ERP or CRM or Data Warehouse) tends to have a natural “biorhythm”, i.e. those metrics / KPI will fluctuate up or down and change due to “natural occurrences” that just happen (I can see some of you cringe! :)).
These biorhythms are hard to understand, harder still to predict and since many of us live in the Puzzle world rather than the Mystery world we spin our cycle like crazy to understand the numbers to “explain” them to the management so that they can take some action. Imagine getting a daily / weekly trend and it goes up and down and you have no idea what the heck is causing it, even after you have done your damdest to isolate all the variables.
The result of these natural biorhythms is that it causes Analysts and Marketers to do analysis and deep dive where none is necessary, it causes some of us to look “bad” because we can’t explain the data, and it causes a lack of faith the the ability of data to provides insights.
Here is a great example that illustrates the issues:
It does not really matter what the numbers on this graph are and what the x-axis is. As you look at this at point 7 or 17 or 25 would you know what the trend is telling you and if it is a cause for concern or things are ok and you don’t need to take any action or the high points are causes for celebration?
One wonderful tool / methodology that I have found to be wonderfully helpful in separating signal from noise is from the world six sigma / process excellence and its called Control Limits (or Control Charts). Simply put control charts are really good at applying statistics to assess the nature of variation in any process. Translated into the biorhythm problem in relevant situations control charts can help trigger deep analysis and action.
Control charts were created to improve quality in manufacturing situations (or others like that) but they work wonderfully for us as well.
There are three core components of a control chart. A line in the center that is the Mean of the all the data points, a UCL (Upper Control Limit) and a LCL (Lower Control Limit).
Here is what a trend looks like with control limits overlayed on top:
What are Control Limits really?
Let us understand what you are looking at.
Mean (X): The green line above. A statistically calculated number that defines the average amount of variation in your KPI trend. For example for the above process it is 39.29.
UCL (Upper Control Limit): A statistically calculated number that defines the higher limit of variation in your KPI trend. In the example above it is 45.
LCL (Lower Control Limit): A statistically calculated number that defines the lower limit of variation in your KPI trend. In the example above it is 33.
The control chart above is illustrating a natural biorhythm in the KPI trend that is in between the two control limits, these are points that show natural variation in the metric and tentatively are not causes for doing anything, even though as you can clearly see they vary quite a bit from one data point to the next.
The massively cool thing is that it shows all the points in the trend, think of it as days or weeks or months, when you should have taken action because there was something unusual that occurred. It won’t, sadly, tell you what the heck happened, but it will tell you when you should use your precious time to dig deeper. Isn’t that awesome? Think of all the time you would have wasted solving the Puzzle behind the data points below the Mean, which look like “problems”.
So how do you compute these wonderful Control Limits (UCL & LCL)?
The general rule of thumb for calculating control limits is:
(Average KPI Value) +/- (3 x (Standard Deviation))
Control limits are calculated 3 standard deviations above or below the mean of your KPI data values. They are not assigned, but rather calculated based on the natural output of your data. Anything within the control limits should be viewed as expected variation (natural biorhythm). Anything outside of control limits warrants investigation. Not only that but if a series of data points fall outside the control limits then it is a bigger red flag in terms of something highly impactful going awry.
In a world where we are tons of metrics, where every dashboard has fifteen graphs on it, control limits are extremely helpful in leveraging the power of statistics to be the first filter of when you should dig deeper or look for a cause. If your metrics and trends have variations from day to day and week to week this is a great way to isolate what is “normal” and what is “abnormal” in the trend.
Control charts also scale very well. It would be easy if for every metric you have there is a clearly established Goal that you are shooting for. That goal can tell you how well, or not, you are performing. That is rarely the case for the massive deluge of metrics you have to deal with. It is scalable for you to apply control limits to all your trends.
Practical considerations in use of control charts (limits):
- Like with all things statistics the more data points you have the better your control limits will be, it would be hard to do a control chart that makes sense with just five data points (you can create it, it just won’t be very meaningful).
- Control limits work best with metrics / KPI’s where it is a bit easy to control for the impacting variables.
For example it would be less insightful to create control limits for your Overall Conversion Rate if you do Direct Marketing, Email Campaigns, Search Engine Marketing (Pay Per Click), Affiliate Marketing and you have loads of people who come directly to your site. There are too many variables that could impact your trend.
But you can easily create control charts for your Email Campaigns and PPC Campaigns or Direct Traffic and it will be very insightful because the variable is just one (or just a couple) and you will find excellent trigger points for performance and in turn analysis and in turn action.
- You do need to be able to understand a little bit of statistics and have some base knowledge around standard deviations etc so that you can leverage this optimally but also explain the power of what you are doing to your Senior Executives.
Practical example of using control limits:
The graph above shows a potential sample conversion rate of a website. Without the Red (UCL) and Blue (LCL) lines it is harder to know each month how the performance of direct marketing campaigns is faring. It is easy to know in Jan 2005 that performance was terrible. It is much harder to know that between March and July statistically there was nothing much to crab about even though the trend goes up and down.
This last point is important, anyone can eye ball and take action on a massive swing. What stymies most Analysts is separating signal from noise for non-massive swings in the data.
Consider using Control Limits on your KPI’s such as cart and checkout abandonment rates, you’ll be pleasantly surprised and happy at what you learn (as will your bosses).
Any decent statistical software will automatically calculate control limits and create these graphs for you. Minitab is the one that is used a lot by folks I know (though it is a tad bit expensive). We have also used our standard business intelligence tools to compute control limits for us (Brio, Business Objects, Cognos, MicroStrategy etc). You can also always simply jury rig excel to compute the limits for you (perhaps a reader of the blog can create a template that I can post here for everyone’s use; Update: Clint Ivy to the rescue! Here's his blog post and here's the wonderful spreadsheet he's created for us. Please download the spreadsheet and plug in your own numbers.).
This is a long and complex post but I hope that I have communicated to you the power of control charts, it is a bit dry and take a small bit of knowledge and patience but it is so powerful in helping your analysis specifically when it comes for separating signal from noise.
Signal -> Insights -> Action -> Happy Customers -> Money, Money, Money! : )
What do you think? Have you used control charts? What metrics do you think they will work best with? Should web analytics vendors include the ability to do control charts as a standard option in their tools? Is none of this making sense?
Please share your feedback and critique via comments.
[Like this post? For more posts like this please click here.]