Thanks a million for your great posts. I just got into web analytics and have found both this blog and your book extremely helpful. You mentioned utilizing the statcalc.xls spreadsheet to measure significance between pageviews. Can you kindly advise on how to do this? Your response is eagerly awwaited. :)

i share similar concerns as jbuser…

for example, 1.74 standard deviations is not 95% significance. 1.96 standard deviations is 95% confidence.

also, for computing standard deviation (s) of 2 or more proportions (in this case, conversion rate), i think it’s a good practice to assume the largest margin of error. in this case, margin of error = 2 * sqrt(0.5^2/n), where n is the size of your smallest sample. this is what pollsters such as zogby do when communicating poll results about hillary/obama, etc.

As a stats guy, I am a little concered with the assumptions behind the model. I downloaded it and the first thing I noticed, was that it makes some pretty large assumptions with confidence levels (anything with z-score (I am assuming) of 1.65 and 2.33 = 95%). I understand that this is probably there to make things “easy” but I think it can be misleading. Also of note, was that IF their is a z-score assumption (which unsure of), there are some other assumptions underneath the covers (which I coudn’t get to), and z-scores are only for known pop means and sta. dev. Do you know what Brian is using? Is it possible to get this information?

Finally, one concern with the “plug and chug” nature of the spreadsheet is what you always must be wary of, and that is making statistical significance a badge of honor. All it tells you is given the values you have, is there a difference between the two. What you must do, more than anything else, is make sure your testing methods are solid BEFORE the test. Otherwise, you are going to be putting in values after values and getting significance or not and making some very important decisions when the whole test could be wrong. Practical vs. Statistical.

Thanks for all your great articles, Avinash!

Note : Thanks very much for the correction Philip! -Avinash.

i’ve also given links for templates in the piece.

pac

I would suggest that in most marketing situations, we are more interested in the former (higher than) than the latter (different from).

In a few cases, we may want to know whether a proposed course of action may harm the response rate, in which case a 2-tail test would be appropriate.

A 5% significance level (95% CL) means there is a 5% probability of being wrong. This is Type I error, i.e., concluding that one RR% is higher than another (statistically significant)when in fact it is not. Acting on this wrong conclusion may result in incurring costs that do not yield revenue or profit to offset the costs.

Type 2 error is when one does not reject the null hypothesis when if fact it is false. In this
case, the cost associated with the decision to not roll out a marketing tactic is the foregone revenue/profit that would otherwise have been generated.

In many situations, the cost of Type II error exceeds the cost of Type I error. Clearly, a trade-off is involved, but a lower CL of 90% of even 80% may not be out of line.

Ultimately, each business needs to decide for itself what an appropriate CL is for purposes of assessing test results.

Would be interested in your thoughts. Hakim

I so much agree on the importance of taking into account statistical significance - an essential part of the “so what” factor!

This is a neat chi square tool to test for statistical significance:

http://www.georgetown.edu/faculty/ballc/webtools/web_chi.html

I do love your blog, bye, Vicky

Wouldn’t it have been more precise to use a two tailed test? If not, why?

You are right, one can get quite sophisticated and get ever better results. The emphasis of the article was how to detect statistical significance in a simple case. I hope to blog more about how we can apply advanced methodologies in testing (to build on my experimentation and testing post).

Thanks for taking the time to post a comment.

Regarding statistical significance, it appears that the reference in your post used a one tailed z test for testing whether there is a significant difference between two sample proportions. Wouldn’t it have been more precise to use a two tailed test? If not, why?

I completely agree with the care around communicating anything that is not of significance, there is always a danger that inspite of your warning the will jump into the lake.

Another pavlovian reaction was to consider that any number of responses under 200 should not be taken into account as it holds high proability of not being representitive.

(For our readers here is something on pavlovian reaction.)

In the world of Multivariate we can detect a strong signal even with small samples. We use somethings like This Page to calculate sample set.

“Is a visitor engaging into A but not engaging into B, converted easier into a lead, than someone engaged into C and B?”

Corelations are important, very, and of course my simply little spreadsheet won’t account for that. Specially for complex web interactions it is important to understand the lower level conversion events might influence higher level (ultimate) goals.

Good to find you blogging, sharing thoughts and experiences. It’s quite some interestign stuff and I hope you enjoy the experience.

I read your different posts on Saturday morning and your thoughts stayed with me for the entire week-end. Thank you.

Yes, statistical significance. I totally join you in the idea and would only add that tests that do not render truely significant results should not be communicated upon. I remember in my first job having warned of the non significance of a test only to find it had heavily influenced a commercial strategy. I vowed never again!

Another pavlovian reaction was to consider that any number of responses under 200 should not be taken into account as it holds high proability of not being representitive. I usually follow this first rule and adapt the variables in order to remain loyal to the statistical representitiveness of a sample. Quite pavlovian, I agree.

And the last thing is that I’ll bare statistical significance in mind but would like to suggest another possible subject: correlation between conversion rates.

Siegert suggested this formulation for a client yesterday:
“Is a visitor engaging into A but not engaging into B, converted easier into a lead, than someone engaged into C and B?”
In other words, you’ve got kind of low level conversion events that influence or not higher goals.
I’m having diffculty formulating this, sorry.
Hope it made sense, keep up the good work, cheers from expensive Brussels ;-)
Aurélie

I too am very happy to see your blog. I found it through Clint Ivy’s blog and I am enjoying reading your posts very much. I find them to be quite informative.

You say above:
“You can easily adapt the spreadsheet, as we have, to compute statistical difference between absolute numbers (say you want to know if the difference Page Views Per Visitor or Average Time on Site between segment One and Two is Significant)”

It’s not obvious to me how to do this. Can you elaborate?

Thanks.

I’m wondering now based on this article, how this model plays in with UX decision when applying A/B tests. We often mind minimal differences and often base our decision the winner between the two.

If we are doing a/b testing (asuming the Success Goal is clearly articulated and measurable and that it is not “impact on brand”) then it would be a sin not to use the spreadsheet in the post above to seperate Signal from Noise. Simply looking at Conversion Rate (or similar metric) difference is very dangerous because of exactly what you say, how much is enough to be confident.

The great news is that most current a/b testing solution (atleast the ones that so “page testing”) already include statistical computations to help us make better decisions.

If you don’t see atleast 90% plus statistica confidence take the results with a grain of salt.

What else do we need to know about the groups involved in the test?

Are they essentially the same group or are they two completely different groups? (I’m assuming you would want to send offers to as many people as possible; hence, they are the same group - less 100 people in the second case.)

In the specific example I used, and the spreadsheet, you control for one thing usually. you can have as many groups as you want. For example you can send one offer to people who live in CA and NY and FL and OR and OH and plug that into the spreadsheet against a control and know which works best.

Alternatively you could try 5, 6, 10 whatever number of different offers to a bunch of folks and see which one converts best.

The problem becomes when you want to test different offers to differnt groups (or many different content in different locations on the same page). Now you are in the world of multivariate and need to apply advanced statistics (think Taguchi).

Doing multivariate is awesomely powerful and yields great results, but beyond my humble spreadsheet.

Do you need a random sample in order to apply principles of standard deviation? 5,000+ is a good size group from which to draw conclusions,

The beauty of using statistics is that the standard deviations required, and amount of Statistical Significance (my suggestion of 95% or higher), will drive how big a sample you need. There is no fixed number (like 5k).

Hope this is the kind of information you were looking for.

It’s excellent that you are giving us real examples of how statistics can be used, and providing tool references. I look forward to additional case studies and discussions.

Will you also be posting about monitoring and managing outside influences? Sometimes the Noise dampens the signal or deflects the signal.

Thanks,
Web: http://www.clickinsight.ca
Blog: clickinsight.blogspot.com