I read the part about this in your book and this blog post (including all comments) again and basically had/have three more questions:

1) Somebody mentioned using the slope or the most recent periods to filter out the effect of growth. Obviously if you take the data from the last 10 years the average might not be a good idea, but the data from the last year would be more insightful - that is if there’s enough data during that period…thus choosing only the last periods over choosing all the historical data is mostly a balancing act?

2) The use of standard deviations seems to be exactly the same way as when they’re used with other metrics. 3 SD’s = 98% of the data, 2 SD’s 95% of the data…and 1 SD would be roughly 68% of the data? Did I understand this correctly?

3)You mention that having a small sample size can pose a problem (as usual). Would it be right to assume that the control limits would still be “correct” no matter how many data points you have, but if you dont have enough, they’re just not very insightful as the standard deviations would be very big and thus the control limits would be very big, too?

I hope my first 2 questions can be replied to with a simple yes (I hope!), but the last one really got me wondering. If the sample size is too small/too few data points, what exactly would happen that would make it harder to take action?

And is there a way to find out how many data points one would need for these control limits to be helpful? Something like a statistical significance test?!

Or would somethign like a statistical significance for the control limits have to be computed?!

One short coming of fitting a straight line is ths normalizes the data in a very rough manner, a second degree polynomial might work better.

— This is an answer to some questions regarding getting box-plots etc out of the box, try using JMP, this is a statistical tools made by SAS, it is very user friendly with a lot of built in features.

However, most data in the real world don’t adhere to mean-related statistical techniques since they fluctuate a lot for very genuine reasons - which mean cannot capture since its a very basic indicator.

An alternative method would be to fit the data onto a straight line using least square fit and use the slope, rather than the mean, to serve as the basis for control limits. This would take into account the growth/slowdown of products too.

Also, 3-sigma control limits are for eliminating outliers that fall in the outside 2% (meaning, the control limits contain 98% of the data between them). Applying a 2-sigma limit eliminates the outside 5% (meaning, the control limits contain 95% of the data between them).

For those interested or knowledgeable of statistical distributions, the formulae are based on the assumption that the data follows a normal distribution (again, this is not case in most real-world data, though they can be massaged to fit into that assumption).

I have tried this application of upper and lower control limits on the rate of change of email opt in rates per week. I found I needed to eliminate the upper and lower 10% to get a reasonable standard deviation by which to examine a week’s change from the previous week but otherwise it seems to work out pretty neatly.

Does anyone see an issue with using the control limits for this particular KPI? I’d love some feedback before I put this before my boss. Don’t want to get the cross-eyes from him!

Thanks,
Emily

Check it out at http://immeria.net/2007/01/box-plot-and-whisker-plots-in-excel.html

S.Hamel
http://immeria.net

I applaud your effort in bringing some statistical rigor to the analysis of website performance, six sigma is a great tool chest to begin with.
The only aspect that I stress is that these tools provide great insights as long as they are applied correctly.
For instance, I’ve just posted about correlation and how it is normally mis-interpreted. Another great advantage of statistical/stochastic methods is that we no longer run our businesses out of averages but, rather, from expected ranges.
In any case: I am a great fan of your blog and I think this post is a great step towards bringing rigor into performance analysis.

Adelino
adelino.typepad.com

In my opinion, this concept would be perfect for LEADING KPIs instead of LAGGING. Most of the management dashboards I have seen over the past few years, lacked LEADING INDICATORS and in the absence of that; it’s very hard to take proctive decisions by looking at the trend.

Most of the companies I have worked with simply relies on MOVING AVERAGE or to some extent WEIGHTED AVERAGE to compute the TREND but still; the picture is not clear because; they are not aware with the BASELINE figures and until and unless you have MIN and MAX MEAN values; it’s hard to seperate the SIGNAL from the NOISE.

CONTROL CHARTS are worth trying and I would definetely play with it.

My hope is that people will take this recommendation and use it as a mechanism to trigger deeper analysis when they swim in a sea of data that all fluctuates all the time. That’s it, nothing more - nothing less. Though I can totally see how it could be misapplied.

I am a fan of your blog and I appreciate your feedback very much. Thanks.

-Avinash.

The interesting thing that Avinash has done is to take some statistical tools from a very statistically mature framework (six sigma) and apply it to web analytics data.

Because the UCL and LCL are calculated on the fly, with the entire data set, I don’t see how it limits or ignores growth, what it does is pin-point unusual events that need your attention.

Furthermore, if you are concerned about the impact of historical trends on the control limits, then you could limit your calculation of the standard deviation to the last N periods instead of the entire data set.

My 2 cents worth…

-Clint

Adelino de Almeida
adelino.typepad.com

Check out http://en.wikipedia.org/wiki/Box_plot

S.Hamel
http://immeria.net

Therefore, I only used one std dev. so that my LCL was the next minimum above the floor (1).

To your comment, about negative LCLs - you could easily update the formulae to be an IF statement so that zeros are returned if the LCL is negative.

-Clint

While I’m not that familiar with TQM, but I have been thinking about how to apply statistical techniques to web data. I had actually started down the path of creating a standard deviation based graph.

One note, I think the excel spreadsheet that you posted a link contains an error. It has the formula for the UCL and the LCL as only one standard deviation from the average. I believe the correct formula is 3 times the standard deviation. Although I like the idea of showing one and two standard deviations in the graph as well as the UCL and LCL. Another thing the spreadsheet should take into account is that if the LCL is a negative value then it should be set to zero.

Thanks for a great post, Avinash.

I am not sure what top secret NASA data you are pumping into this :) but if you have segmented the data and it still explains “all variability” then in the past I have tried to put in more data points to see if it helps. I am not sure if it will help in your case. Thanks though for trying it, it is great to hear feedback from others.

Anonymous: As sketched in the post it would accommodate for all the data points including the outliers. But when that is observed in a metric / process I have typically done what the wonderful Mr. Morgan suggests in his comment directly above. If relevant in your case please try that (it is not just a matter of ignoring or not the top/bottom percentiles, I would encourage you to think of that decision in the context of the business process / Metric / KPI that you are applying the control limits to, i.e. what are you measuring and what is the business context of the outliers).

Dave: As always thanks for your comment, it is wonderfully helpful and adds to the quality of the conversation. Gracias!

-Avinash.