Excellent Analytics Tip#1: Compute Statistical Significance

We all wish that our key internal partners, business decision makers, would use Web Analytics data a lot more to make effective decisions. How do we make recommendations / decisions with confidence? How can we drive action rather than pushing data? The challenge is how to separate Signal from Noise and make it easy to communicate that distinction.

This is where Excellent Analytics Tip #1, a recurring series, comes in. Leverage the power of Statistics.

Consider this scenario (A):

You do send out two offers to potential customer. Here is how the outcomes look:
• Offer One Responses: 5,300. Order: 46. Hence Conversion Rate: 0.87%
• Offer Two Responses: 5,200. Order: 55. Hence Conversion Rate: 1.06%

Is Offer Two better than Offer One? It does have \”better\” conversion rate, by 0.19%. Can you decide which one of the two is better with just 40 to 50 responses? We got 9 more orders from 100 fewer visitors.

Applying statistics tells us that the results, the two conversion rates, are just 0.995 standard deviations apart and not statistically significant. This would mean that it is quite likely that it is noise causing the difference in conversion rates.

Consider this scenario (B):

You do send out two offers to potential customer. Here is how the outcomes look:
• Offer One Responses: 5,300. Order: 46. Hence Conversion Rate: 0.87%
• Offer Two Responses: 5,200. Order: 63. Hence Conversion Rate: 1.21%

Applying statistics will now tell us that the two numbers are 1.74 standard deviations apart and the results rate 95% statistically significant. 95% significance is a very strong signal. Based on this, and only a sample of 5k and sixty odd responses, we can confidently predict success.

Powerful benefits to presenting Statistical Significance rather than simply Conversion Rate:

1. You are taking yourself out of the equation, it is awesome to say \”according to the God\’s of Statistics here are the results…\”
2. Focusing on quality of Signal means that we appear smarter than people give us Analysts credit for.
3. You take then thinking and questions out of the equation. Either something is Statistically Significant, and we take action, or we say it is not Significant and let\’s try something else. No reporting, just actionable insights.

Is this really hard to do?

No! Simply use the spreadsheet below, which comes to us via the exceedingly kind Rags Srinivasan:

+ Statistical Significance Calculator

[Note: March, 2013: This is the updated version of the calculator. For a bit more context on version 2, please see this comment by Rags.]

In the spreadsheet you get even more bang for your buck. On sheet number one you can apple the 1-tailed or 2-tailed test to your statistical significance calculations. Here are the steps: Choose from the drop down in cell D7. Complete cells B13, C13, B14 and C14 (essentially how many participants or visitors etc were there and how many conversions you got). In cell C18 you\’ll see if the results were statistically significant or not.

In sheet number two, for those of you who are a bit advanced, you can apply the chi-squared test. This test is more optimal for when you see very small conversion rates (not unusual on the web). It is a more skeptical test with a higher threshold for differences. The benefit is that small statistical anomalies don\’t look like real differences.

When in doubt go with sheet number two, the chi-squared test.

Two small tips:

1. This is a best practice but aim for 95% or higher Confidence. That is not always required but it is recommended.
2. \”Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital.\” –Aaron Levenstein

Agree? Disagree? Not really a Excellent Analytics Tip? Please share your feedback via comments.