{"id":15,"date":"2006-05-17T00:23:22","date_gmt":"2006-05-17T07:23:22","guid":{"rendered":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/"},"modified":"2026-05-19T15:30:40","modified_gmt":"2026-05-19T22:30:40","slug":"excellent-analytics-tip1-statistical-significance","status":"publish","type":"post","link":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/","title":{"rendered":"Excellent Analytics Tip#1: Compute Statistical Significance"},"content":{"rendered":"<p><img hspace=\"7\" align=\"left\" title=\"Yellow\" alt=\"Yellow\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2006\/05\/yellow.thumbnail.jpg\" \/> We all wish that our key internal partners, business decision makers, would use Web Analytics data a lot more to make effective decisions. How do we make recommendations \/ decisions with confidence? How can we drive action rather than pushing data? The challenge is how to separate <strong>Signal<\/strong> from <strong>Noise<\/strong> and make it easy to communicate that distinction.<\/p>\n<p>This is where Excellent Analytics Tip #1, a recurring series, comes in. Leverage the power of Statistics.<\/p>\n<p>Consider this scenario (A):<\/p>\n<ul>You do send out two offers to potential customer. Here is how the outcomes look:<\/p>\n<li>Offer One Responses:  5,300. Order: 46. Hence Conversion Rate: 0.87%<\/li>\n<li>Offer Two Responses:  5,200. Order: 55. Hence Conversion Rate: 1.06%<\/li>\n<\/ul>\n<p>Is Offer Two better than Offer One? It does have &#8220;better&#8221; conversion rate, by 0.19%.  Can you decide which one of the two is better with <em>just<\/em> 40 to 50 responses? We got 9 more orders from 100 fewer visitors.<\/p>\n<p>Applying statistics tells us that the results, the two conversion rates, are just 0.995 <a href=https:\/\/en.wikipedia.org\/wiki\/Standard_deviation\">standard deviations<\/a> apart and not statistically significant. This would mean that it is quite likely that it is <strong>noise<\/strong> causing the difference in conversion rates.<\/p>\n<p>Consider this scenario (B):<\/p>\n<ul>You do send out two offers to potential customer. Here is how the outcomes look:<\/p>\n<li>Offer One Responses:  5,300. Order: 46. Hence Conversion Rate: 0.87%<\/li>\n<li>Offer Two Responses:  5,200. Order: 63. Hence Conversion Rate: 1.21%<\/li>\n<\/ul>\n<p>Applying statistics will now tell us that the two numbers are 1.74 standard deviations apart and the results rate 95% statistically significant. 95% significance is a very strong <strong>signal<\/strong>. Based on this, and only a sample of 5k and sixty odd responses, we can confidently predict success.<\/p>\n<p>Powerful benefits to presenting Statistical Significance rather than simply Conversion Rate:<\/p>\n<ol>\n<li>You are taking yourself out of the equation, it is awesome to say &#8220;according to the God&#8217;s of Statistics here are the results&#8230;&#8221;<\/li>\n<li>Focusing on quality of Signal means that we appear smarter than people give us Analysts credit for.<\/li>\n<li>You take then <em>thinking<\/em> and <em>questions<\/em> out of the equation. Either something is Statistically Significant, and we take action, or we say it is not Significant and let&#8217;s try something else. No reporting, just actionable insights.<\/li>\n<\/ol>\n<p>\nIs this really hard to do?<\/p>\n<p>\nNo! Simply use the spreadsheet below, which comes to us via the exceedingly kind <a href=https:\/\/iterativepath.wordpress.com\/\">Rags Srinivasan<\/a>:<\/p>\n<ul>\n<strong><font color=blue>+<\/font><\/strong> <a href=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/10\/statistical_significance_calculator_v2.zip\" title=\"Statistical Significance Calculator\">Statistical Significance Calculator<\/a><\/p>\n<p> [Note: March, 2013: This is the updated version of the calculator. For a bit more context on version 2, please see <a href=https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/#comment-546522\">this comment by Rags<\/a>.]\n<\/ul>\n<p>In the spreadsheet you get even more bang for your buck. On sheet number one you can apple the <a href=https:\/\/www.ats.ucla.edu\/stat\/mult_pkg\/faq\/general\/tail_tests.htm\">1-tailed or 2-tailed test<\/a> to your statistical significance calculations. Here are the steps: Choose from the drop down in cell D7. Complete cells B13, C13, B14 and C14 (essentially how many participants or visitors etc were there and how many conversions you got). In cell C18 you&#8217;ll see if the results were statistically significant or not.<\/p>\n<p>In sheet number two, for those of you who are a bit advanced, you can apply the  <a href=https:\/\/en.wikipedia.org\/wiki\/Chi-squared_test\">chi-squared test<\/a>. This test is more optimal for when you see very small conversion rates (not unusual on the web). It is a more skeptical test with a higher threshold for differences. The benefit is that small statistical anomalies don&#8217;t look like real differences. <\/p>\n<p>When in doubt go with sheet number two, the chi-squared test.<\/p>\n<p>Two small tips:<\/p>\n<ol>\n<li>This is a best practice but aim for 95% or higher Confidence. That is not always required but it is recommended.<\/li>\n<li><em>&#8220;Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital.&#8221;<\/em> &#8211;Aaron Levenstein<\/li>\n<\/ol>\n<p>Agree? Disagree? Not really a Excellent Analytics Tip? Please share your feedback via comments.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>standard deviations apart and not statistically significant. This would mean that it is quite likely that it is noise causing [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11,6,9,10],"tags":[],"class_list":["post-15","post","type-post","status-publish","format-standard","hentry","category-advanced-analytics","category-marketing-tips","category-web-analytics","category-web-metrics"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik\" \/>\n<meta property=\"og:description\" content=\"standard deviations apart and not statistically significant. This would mean that it is quite likely that it is noise causing [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/\" \/>\n<meta property=\"og:site_name\" content=\"Occam&#039;s Razor by Avinash Kaushik\" \/>\n<meta property=\"article:published_time\" content=\"2006-05-17T07:23:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-19T22:30:40+00:00\" \/>\n<meta name=\"author\" content=\"Avinash Kaushik\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Avinash Kaushik\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/\"},\"author\":{\"name\":\"Avinash Kaushik\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\"},\"headline\":\"Excellent Analytics Tip#1: Compute Statistical Significance\",\"datePublished\":\"2006-05-17T07:23:22+00:00\",\"dateModified\":\"2026-05-19T22:30:40+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/\"},\"wordCount\":313,\"commentCount\":84,\"articleSection\":[\"Advanced Analytics\",\"Marketing Tips\",\"Web Analytics\",\"Web Metrics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/\",\"url\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/\",\"name\":\"Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#website\"},\"datePublished\":\"2006-05-17T07:23:22+00:00\",\"dateModified\":\"2026-05-19T22:30:40+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\"},\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/excellent-analytics-tip1-statistical-significance\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Excellent Analytics Tip#1: Compute Statistical Significance\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#website\",\"url\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/\",\"name\":\"Occam&#039;s Razor by Avinash Kaushik\",\"description\":\"Marketing | Strategy | Analytics\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\",\"name\":\"Avinash Kaushik\",\"description\":\"Author: Web Analytics 2.0 &amp; Web Analytics: An Hour A Day | Chief Strategy Officer: Human Made Machine\",\"sameAs\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/akaushik\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/avinash\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/","og_locale":"en_US","og_type":"article","og_title":"Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik","og_description":"standard deviations apart and not statistically significant. This would mean that it is quite likely that it is noise causing [&hellip;]","og_url":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/","og_site_name":"Occam&#039;s Razor by Avinash Kaushik","article_published_time":"2006-05-17T07:23:22+00:00","article_modified_time":"2026-05-19T22:30:40+00:00","author":"Avinash Kaushik","twitter_misc":{"Written by":"Avinash Kaushik","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/#article","isPartOf":{"@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/"},"author":{"name":"Avinash Kaushik","@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6"},"headline":"Excellent Analytics Tip#1: Compute Statistical Significance","datePublished":"2006-05-17T07:23:22+00:00","dateModified":"2026-05-19T22:30:40+00:00","mainEntityOfPage":{"@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/"},"wordCount":313,"commentCount":84,"articleSection":["Advanced Analytics","Marketing Tips","Web Analytics","Web Metrics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/","url":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/","name":"Excellent Analytics Tip#1: Compute Statistical Significance - Occam&#039;s Razor by Avinash Kaushik","isPartOf":{"@id":"https:\/\/www.kaushik.net\/avinash\/#website"},"datePublished":"2006-05-17T07:23:22+00:00","dateModified":"2026-05-19T22:30:40+00:00","author":{"@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6"},"breadcrumb":{"@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.kaushik.net\/avinash\/"},{"@type":"ListItem","position":2,"name":"Excellent Analytics Tip#1: Compute Statistical Significance"}]},{"@type":"WebSite","@id":"https:\/\/www.kaushik.net\/avinash\/#website","url":"https:\/\/www.kaushik.net\/avinash\/","name":"Occam&#039;s Razor by Avinash Kaushik","description":"Marketing | Strategy | Analytics","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.kaushik.net\/avinash\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6","name":"Avinash Kaushik","description":"Author: Web Analytics 2.0 &amp; Web Analytics: An Hour A Day | Chief Strategy Officer: Human Made Machine","sameAs":["https:\/\/www.kaushik.net\/avinash","https:\/\/www.linkedin.com\/in\/akaushik\/","https:\/\/x.com\/https:\/\/twitter.com\/avinash"]}]}},"_links":{"self":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/15","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/comments?post=15"}],"version-history":[{"count":1,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/15\/revisions"}],"predecessor-version":[{"id":9973,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/15\/revisions\/9973"}],"wp:attachment":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/media?parent=15"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/categories?post=15"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/tags?post=15"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}