{"id":5726,"date":"2013-04-08T02:17:00","date_gmt":"2013-04-08T09:17:00","guid":{"rendered":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/"},"modified":"2026-05-19T15:32:37","modified_gmt":"2026-05-19T22:32:37","slug":"lean-analytics-cycle-metrics-hypothesis-experiment-act","status":"publish","type":"post","link":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/","title":{"rendered":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act"},"content":{"rendered":"<p><img hspace=\"5\" alt=\"Red Green\" align=\"left\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/red_green.jpg\" width=\"161\" height=\"105\" \/>This should not be news to you. To win in business you need to follow this process: Metrics &gt; Hypothesis &gt; Experiment &gt; Act. Online, offline or nonline.<\/p>\n<p>Yet this structure rarely exists in companies.<\/p>\n<p>We are far too enamored with data collection and reporting the standard metrics we love because others love them because someone else said they were nice so many years ago. Sometimes, we escape the clutches of this sub optimal existence and do pick good metrics or engage in simple A\/B testing.<\/p>\n<p>But it is not routine.<\/p>\n<p>So, how do we fix this problem?<\/p>\n<p>This thought was in my mind as I was reading <a href=https:\/\/www.amazon.com\/Lean-Analytics-Better-Startup-Faster\/dp\/1449335675\/?tag=occsrazbyavik-20\" target=\"_blank\">Lean Analytics<\/a> a new book by my friend <a href=https:\/\/twitter.com\/acroll\" target=\"_blank\">Alistair Croll<\/a> and his collaborator <a href=https:\/\/twitter.com\/byosko\" target=\"_blank\">Benjamin Yoskovitz<\/a>.<\/p>\n<p>The book introduces a wonderful process called the Lean Analytics Cycle, which aims to help you create a sustainable way to pick metrics that matter by tying them to fundamental business problems, creating hypotheses you can test and driving change in the business from the learnings you identify.<\/p>\n<p>I&#8217;ve asked Alistair to share his thoughts on this process with us. In this post, we\u2019ll look at each of the four steps in the Lean Analytics Cycle in more detail. Then, for fun, we\u2019ll look at three real-world case studies where companies put the steps to work so we can see the cycle in action.<\/p>\n<p>Let&#8217;s listen in as Alistair discusses the lean analytics model&#8230;<\/p>\n<p align=\"center\"><font color=\"blue\"><strong>______________________________________________________<\/strong><\/font><\/p>\n<p>\nThe Lean Analytics Cycle is a simple, four-step process that shows you how to improve a part of your business.<\/p>\n<p>First, you figure out what you want to improve; then you create an experiment; then you run the experiment; then you measure the results and decide what to do.<\/p>\n<p>The cycle combines concepts from the world of Lean Startup \u2014 which is all about continuous, iterative improvement \u2014 with analytics fundamentals. It helps you to amplify what\u2019s proven to work, throw away what isn\u2019t, and tweak the goal-posts when data indicates that they may be in the wrong place.<\/p>\n<p>Here&#8217;s a pictorial representation of the complete lean analytics cycle:<\/p>\n<p align=\"center\"><a href=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/the_lean_analytics_cycle_occams_razor_large.png\" target=\"_blank\"><img fetchpriority=\"high\" hspace=\"5\" alt=\"the lean analytics cycle occams razor\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/the_lean_analytics_cycle_occams_razor.png\" width=\"625\" height=\"484\" \/><\/a><\/p>\n<p><strong>[<\/strong>Note: Due to the limited screen real estate, I&#8217;ve redone all the images you see in this post. They preserve almost all original intent, but if you read the book, or see the cycle elsewhere, please don&#8217;t be surprised to see a slightly different version. -Avinash<strong>]<\/strong><\/p>\n<p>While the process above might seem complex, we can simplify it to four key steps that any business of any size can apply to their analytics practice.<\/p>\n<p><strong><font color=\"blue\">Step 1: Figure out what to improve.<\/font><\/strong><\/p>\n<p>One thing the cycle can&#8217;t do is help you understand your own business. That part is your job. You need to know what the most important aspect of your business is, and how you need to change it.<\/p>\n<ul>\n<li>Perhaps it&#8217;s an increase in your conversion rate;<\/li>\n<li>Or a higher number of visitors who sign up;<\/li>\n<li>Or a greater number of people who share content with one another;<\/li>\n<li>Or a lower monthly churn rate for users of your application;<\/li>\n<li>Maybe it&#8217;s even something as simple as getting more people into your restaurant.<\/li>\n<\/ul>\n<p>The point is, it&#8217;s a critical metric for your business. You might need help from the business owner to figure out what the metric is. That&#8217;s a good thing! It means you&#8217;re relevant to the business, and if your trip around the cycle is successful, you&#8217;ll have helped the organization get closer to its goals.<\/p>\n<p>Another way to find the metric you want to change is to look at your business model. If you were running a lemonade stand, your business model would be a spreadsheet that showed the price of lemons and sugar; the number of people who passed by your stand; how many of them stopped to buy a drink; and what you could charge. Right there you have four things that are critical to the business and one of them is ripe for improvement. This is the one metric that matters to your business right now. You&#8217;re choosing only one metric because you want to optimize it.<\/p>\n<p>That metric is tied to a KPI. If it&#8217;s the number of people buying, the metric is conversion rate. If it&#8217;s the number of invites sent, it&#8217;s virality. If it&#8217;s the number of paying users who quit, it&#8217;s churn.<\/p>\n<p>The business model also tells you what the metric should be. You might, for example, need to sell each glass of lemonade for $5 to break even. That&#8217;s your goalpost. It&#8217;s the target for your KPI.<\/p>\n<p>So grab a piece of paper and write down three key business metrics you&#8217;d like to change. For each of them, write down the KPI you&#8217;re measuring, and what that KPI should be for you to consider your efforts a success. Do it now; we&#8217;ll wait.<\/p>\n<p><font color=\"blue\"><strong>Step 2. Form a hypothesis.<\/strong><\/font><\/p>\n<p>This is where you get creative. Experiments come in all shapes and sizes:<\/p>\n<ul>\n<li>A marketing campaign<\/li>\n<li>The redesign of an application<\/li>\n<li>A change in the way you price things<\/li>\n<li>Building shipping costs into a purchase<\/li>\n<li>Changing how you appeal to people<\/li>\n<li>The use of a different platform<\/li>\n<li>Changing the wording on buttons<\/li>\n<li>Testing out a new feature<\/li>\n<\/ul>\n<p>Whatever the case, this is where you need inspiration. You can find that inspiration in one of two ways.<\/p>\n<p><strong>If you have no data<\/strong>, then you can try almost anything.<\/p>\n<ul>\n<li>Try to understand your market. Run a survey, or look at what else they do, or examine customer feedback, or simply pick up the phone.<\/li>\n<li>Steal from your competitors. If someone is doing something well, then imitate them. It&#8217;s the sincerest form of flattery. Don&#8217;t be different for the sake of being different.<\/li>\n<li>Use best practices. Read up on ways that companies are growing their business, from growth hacking to content marketing, and use that as inspiration.<\/li>\n<\/ul>\n<p><strong>If you have data<\/strong>, then figure out what\u2019s different about the people who are doing what you want. Let\u2019s say, for example, that you\u2019re trying to lower the churn rate on an application. Some of your users each month don\u2019t quit. What do they have in common? What makes your most loyal customers different from the rest? Did they all come from the same place? Are they all buying the same things?<\/p>\n<p>Either way, the hypothesis comes from getting inside the head of your audience, asking them questions, or understanding what makes them tick.<\/p>\n<p>The word hypothesis means a lot of different things, but in this context I like this definition from <a href=https:\/\/en.wikipedia.org\/wiki\/Hypothesis\" target=\"_blank\">Wikipedia<\/a> the best: <em>People refer to a trial solution to a problem as a hypothesis, often called an &#8220;educated guess\u201d, because it provides a suggested solution based on the evidence.<\/em><\/p>\n<p>We\u2019re making an educated guess about what could improve the KPI based on what we learned in step 1.<\/p>\n<p><strong><font color=\"blue\">Step 3. Create the experiment.<\/font><\/strong><\/p>\n<p>Once you have a hypothesis, you need to answer three questions to turn it into an experiment.<\/p>\n<div style=\"MARGIN-LEFT: 2em\">\n<p>First: <strong>Who is the target audience?<\/strong> Everything happens because someone does something. So who are you expecting to do a thing? Is this all visitors, or just a subset of them? Are they the right audience? Can you reach them? Until you know whose behavior you\u2019re trying to change, you can\u2019t appeal to them.<\/p>\n<p>Second: <strong>What do you want them to do?<\/strong> Is it clear to them exactly what it is you\u2019re asking them to do? Are they able to do it easily, or is something getting in their way? How many of them are doing it today?<\/p>\n<p>Third: <strong>Why should they do it?<\/strong> They\u2019ll do what you ask if it\u2019s worthwhile to them, and if they trust you. Are you motivating them properly? Which of your current pleas is working best? Why do they do this thing for your competitors?<\/p>\n<\/div>\n<p>On the surface, these three questions\u2014who, what, and why\u2014don\u2019t seem hard to answer. But they are. That\u2019s because they require you to have a deep understanding of your customers. In the Lean Startup world, this is called customer development. The experiment will almost always look like this:<\/p>\n<p>Find out if WHO will do WHAT because WHY enough to improve KPI by the Target you&#8217;ve defined.<\/p>\n<p>This gets to the deliberate nature of the actions we want to take. You start with a great hypothesis, and you\u2019ll get a great experiment. This also keeps you honest, because everyone recognizes the point of the activity beforehand.<\/p>\n<p>In our case studies, below, you&#8217;ll see that the KPIs were things like Property Bookings, Number of Engaged Users, and Daily App Use. If you have access to existing data, take some time to document what the current performance looks like. Of course, it\u2019s possible that you don\u2019t have access to the data (as in the Airbnb case study below.) That\u2019s OK. We have a path for that as well.<\/p>\n<p>Once you have your experiment, set up your analytics to measure the KPI against its current baseline and the goal you\u2019ve set. Then run your experiment.<\/p>\n<p><strong><font color=\"blue\">Step 4. Measure and decide what to do.<\/font><\/strong><\/p>\n<p>At this point, you\u2019ll know whether your experiment was a success. This leaves us with several options:<\/p>\n<div style=\"MARGIN-LEFT: 2em\">\n<p><strong>1.<\/strong> If the experiment was a success, you\u2019re a hero. Celebrate a bit, then find the next metric that matters the most and move on to the next who, what, why exercise.<\/p>\n<p><strong>2.<\/strong> If the experiment failed spectacularly, we need to revisit our hypothesis. It\u2019s time to identify a new who, what, and why, based on what we\u2019ve learned. Remember, as long as you learn from it, failure is never a \u201cwasted\u201d opportunity.<\/p>\n<p><strong>3.<\/strong> If the experiment moved the needle, but not enough to clear the goalposts, we should try another experiment. Our hypothesis might still be valid, and we can try again, adjusting based on what we\u2019ve learned.<\/p>\n<\/div>\n<p>The underlying beauty here is that we\u2019re being smart, fast, and iterative. We\u2019re making a deliberate plan, measuring its results, and circling ever closer to our goal. Identify, hypothesize, test, react. Repeat.<\/p>\n<p>Let\u2019s look at some case studies that will really help to drive the Lean Analytics Cycle home. We don\u2019t know everything about the metrics these companies dealt with \u2014 they\u2019re private companies, after all \u2014 and in some cases we\u2019ve estimated numbers to make the explanations more clear. But even with these changes, the examples will help make all of this a bit more real.<\/p>\n<p><strong><font color=\"blue\">Case Study 1: Airbnb<\/font><\/strong><\/p>\n<p>Airbnb is a hugely popular marketplace for rental-by-owner properties. They\u2019ve found dozens of creative ways to grow, but they\u2019re always judicious and data-driven.<\/p>\n<p><font color=\"green\">Step 1: Figure out what metric to improve<\/font><\/p>\n<p>The metric they wanted to improve was the number of nights that a property was rented. Notice that this is more central to their business than simply measuring revenue: Airbnb does well if its homeowners do well, and for it to succeed, it needs listed properties to be rented often so the homeowners will stick around.<\/p>\n<p>The company knew that to succeed, they needed a significant improvement in rental rates per property.<\/p>\n<div style=\"MARGIN-LEFT: 2em\">\n<ul class=\"noindent\">\n<li>One Metric That Matters: \u201cNumber of nights rented.\u201d<\/li>\n<li>KPI: Property bookings<\/li>\n<li>Target: (not publicly known)<\/li>\n<li>Current level: (not publicly known)<\/li>\n<\/ul>\n<\/div>\n<p><font color=\"green\">Step 2: Form a hypothesis<\/font><\/p>\n<p>We don\u2019t know how Airbnb came up with its hypothesis. But we know it had access to property listings that rented well.<\/p>\n<div style=\"MARGIN-LEFT: 2em\">\n<ul class=\"noindent\">\n<li>Perhaps they had noticed that the pictures of those properties looked, to them, more professional.<\/li>\n<li>\u2022 Maybe they realized that one common complaint from renters was that the property didn\u2019t actually look like the pictures on the site.<\/li>\n<li>Maybe they found that people would most often abandon a listing right after seeing photographs.<\/li>\n<li>Maybe they analyzed the metadata from pictures and found that there was a strong correlation between properties that rented often and expensive camera models.<\/li>\n<\/ul>\n<\/div>\n<p>However they got there, they formed a hypothesis: Properties with better pictures rent more often.<\/p>\n<p><font color=\"green\">Step 3: Create the experiment<\/font><\/p>\n<p>Armed with the hypothesis, it was time to create the experiment. As is often the case, having a clear hypothesis makes devising the experiment fairly easy. Their who, what, and why are as follows:<\/p>\n<div style=\"MARGIN-LEFT: 2em\"><em>Who is the campaign for?<\/em> Travelers looking at listings on Airbnb.<\/p>\n<p><em>What do you want them to do?<\/em> Decide to rent a property more frequently.<\/p>\n<p><em>Why do they do it?<\/em> Because the photographs look professional and make the property look beautiful.<\/p>\n<\/div>\n<p>So for them, the experiment was<\/p>\n<p>Find out if <strong>travelers<\/strong> will <strong>book more properties<\/strong> because of <strong>professionally photographed listings<\/strong> enough to improve the <strong>property bookings by X%<\/strong>.<\/p>\n<p>Notice that in this case, Airbnb didn\u2019t really need any current data. It might just as easily have been a random comment over lunch that led to the hypothesis. But even if the hypothesis isn\u2019t founded in hard data, the experiment design must be.<\/p>\n<p>To run this experiment, Airbnb created what Lean Startup calls a curated minimum viable product. This is like the Wizard of Oz: most of the hard work is done behind the curtains, but the end user thinks they\u2019re seeing a final product.<\/p>\n<p>Airbnb wasn\u2019t sure whether or not the experiment would work, so the team didn\u2019t want to hire a staff of photographers or invest in a new part of the application. But at the same time, they had to have a real test of an actual feature.<\/p>\n<p>Before we go further, there\u2019s an important lesson to take away here. You can do this at Nordstroms, or Expedia, or Unilever. You don\u2019t need to build a magnificent shining castle. You don\u2019t need a beautiful beast to go out and test. You can start small, lean, and mean \u2014 with just the customer-facing pieces you want to test \u2014 and go validate (or disprove!) your hypothesis.<\/p>\n<p>Airbnb\u2019s experiment consisted of something that looked like a real feature, but under the covers was really just humans and contracted photographers. During the experiment they took pictures of properties, and then measured the KPI, comparing properties that had been photographed to those that hadn\u2019t.<\/p>\n<p align=\"center\"><img hspace=\"5\" alt=\"airbnb lean analytics cycle occams razor\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/airbnb_lean_analytics_cycle_occams_razor.png\" width=\"625\" height=\"482\" \/><\/p>\n<p><font color=\"green\">Step 4. Measure performance.<\/font><\/p>\n<p>In this case Airbnb measured the bookings from the few properties that had professional photos and compared the rate of bookings with properties that only had photos taken by property owners. The result? The properties with professional photography <em>had 2-3 times the number of bookings!<\/em><\/p>\n<p><strong>[<\/strong>Remember that the raw number is not the only important part, we would also <a href=https:\/\/www.kaushik.net\/avinash\/excellent-analytics-tip1-statistical-significance\/\" target=\"_blank\">measure statistical significance<\/a>. Airbnb had enough data points to be confident in their results. -Avinash<strong>]<\/strong><\/p>\n<p>By 2011, the company had 20 full-time photographers on staff.<\/p>\n<p align=\"center\"><img loading=\"lazy\" hspace=\"5\" alt=\"airbnb nights booked\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/airbnb_nights_booked-1.png\" width=\"615\" height=\"430\" \/><\/p>\n<p>The graph is impressive, right? There were many other things going right with Airbnb&#8217;s business and business model. But the lean process was a key contributor to improving the bookings rate. Clearly, the experiment was a success. They celebrated a bit, then went on to fix the next biggest problem in the business.<\/p>\n<p><strong><font color=\"blue\">Case Study 2: Circle of Friends<\/font><\/strong><\/p>\n<p>Circle of Friends was a social community built atop Facebook that launched in 2007. It essentially allowed you to create a group of friends who could interact and share content, much as people do today with Google+, before such features were part of Facebook.<\/p>\n<p>As a social network based on user-generated content, Circle of Friends only grew when its users were engaged. The company\u2019s creators wanted users not only to create groups, but also to send messages within those groups, invite others, and interact with Facebook elements such as news feeds.<\/p>\n<p><font color=\"green\">Step 1: Figure out what metric to improve<\/font><\/p>\n<p>The founders had several measures of \u201cengagement\u201d, from whether people attached a picture to a post, to whether they clicked on Facebook notifications, to the length of posts they wrote. Any of these actions \u2014 and several others \u2014 constituted \u201cengagement.\u201d All of this rolled up into a simple KPI: number of engaged users.<\/p>\n<p>While Circle of Friends\u2019 launch was hugely successful in terms of raw attention \u2014 they had 10 million users! \u2014 <em>engagement<\/em> simply wasn\u2019t happening. None of the metrics they used to measure the health of their communities was where they wanted it to be.<\/p>\n<p>The rather nebulous \u201clevel of engagement\u201d was a compound metric, which can be dangerous to rely on. But one of the clear metrics they tracked was \u201cnumber of circles with activity\u201d, meaning that there had been some form of interaction within a group in the past few days.<\/p>\n<ul>\n<li>One Metric That Matters: User engagement<\/li>\n<li>KPI: Number of active users; number of circles with activity<\/li>\n<li>Target: (not publicly known)<\/li>\n<li>Current level: Less than 20% of circles had any activity after creation<\/li>\n<\/ul>\n<p>The challenge, here, was how to kickstart engagement. That\u2019s not an easy thing to do. Fortunately, Circle of Friends had plenty of user data to mine.<\/p>\n<p><font color=\"green\">Step 2: Form a hypothesis<\/font><\/p>\n<p>Circle of Friends had huge volumes of information on its users and how they used the product. They looked at two groups of users: those that were engaged, and those that weren\u2019t. Then they looked at what those users had in common. In other words, they defined what engagement meant (i.e. \u201csomeone who has returned in the last week\u201d) and segmented users into two groups. Then they looked at other things those people had in common.<\/p>\n<p>What they found changed their business. It turned out that the engaged users were much more likely to be mothers. Looking at moms who use the application:<\/p>\n<ul>\n<li>Messages to one another were, on average, 50% longer<\/li>\n<li>115% more likely to attach a picture to a post they wrote<\/li>\n<li>110% more likely to engage in a threaded (i.e. deep) conversation<\/li>\n<li>Those who were friends of the circle owner were 50% more likely to engage<\/li>\n<li>75% more likely to click on Facebook notifications<\/li>\n<li>180% more likely to click on Facebook news feed items<\/li>\n<\/ul>\n<p>Before we look at what they decided to do, stop for a minute and realize that simply having data isn\u2019t that useful. Circle of Friends had all of this information. But asking the right questions of your data is a superpower. In this case, they asked, \u201cwhat do engaged users have in common?\u201d and it absolutely changed the destiny of their company.<\/p>\n<p>From their data, they formed a hypothesis: <em>If we focus only on moms we\u2019ll have the engagement we need<\/em>.<\/p>\n<p><font color=\"green\">Step 3: Create the experiment<\/font><\/p>\n<p>Their who, what, and why were as follows:<\/p>\n<div style=\"MARGIN-LEFT: 2em\"><em>Who is the campaign for?<\/em> Moms.<\/p>\n<p><em>What do you want them to do?<\/em> Engage with other users, invite new users, and create content.<\/p>\n<p><em>Why do they do it?<\/em> Because they find interaction with others rewarding and compelling.<\/p>\n<\/div>\n<p>In the end, the founders made a huge bet \u2014 the kind of bet that only startups with nothing to lose can make. They decided to completely rebrand the company as Circle of Moms, and focus everything on attracting and engaging mothers.<\/p>\n<p>So their hypothesis was:<\/p>\n<p>Find out if <strong>moms<\/strong> will <strong>join, relate and engage in groups<\/strong> because of <strong>a community targeted specifically at them<\/strong> enough to improve the <strong>number of active circles and engaged users<\/strong> by <strong>X%<\/strong>.<\/p>\n<p>They were betting that if they stopped trying to please everyone, and instead focused specifically on mothers, they\u2019d have a higher percentage of engagement, which would in turn lead to the company\u2019s overall success.<\/p>\n<p align=\"center\"><img loading=\"lazy\" hspace=\"5\" alt=\"circle of moms lean analytics cycle occams razor\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/circle_of_moms_lean_analytics_cycle_occams_razor.png\" width=\"625\" height=\"483\" \/><\/p>\n<p><font color=\"green\">Step 4. Measure performance<\/font><\/p>\n<p>It took a few months to carry out the experiment, since it involved re-launching the entire company. The initial result was a huge drop in users, of course, since most of the communities weren&#8217;t made up of mothers.<\/p>\n<p>They lost millions of users.<\/p>\n<p>They feared this would happen, but they knew that if they didn&#8217;t get engagement where it needed to be they might as well close up shop.<\/p>\n<p>If they&#8217;d been focusing on the wrong KPI, such as number of subscribers, they&#8217;d have quickly concluded that the experiment was a disaster. But because they&#8217;d been deliberate about their hypothesis, they knew that the most important metric was engagement. Sure enough, the users that remained were engaged. Overall engagement and active circles climbed significantly, to the point where the business model was healthy.<\/p>\n<p>Of course, having achieved the engagement they sought, the next experiment was about whether they could grow a more narrowly focused community to a decent size. By late 2009, that experiment was a success, too; they&#8217;d climbed back up to 4.5 million users, with strong engagement.<\/p>\n<p><strong><font color=\"blue\">Case Study 3: High Score House<\/font><\/strong><\/p>\n<p>High Score House is a tool that lets parents manage household chores and rewards for their kids. The founders envisioned a world where parents and children sat down each day to identify chores, keep score of what the kids have done, and drool over rewards they might claim.<\/p>\n<p><font color=\"green\">Step 1: Figure out what metric to improve<\/font><\/p>\n<p>In the early stages of launch, the most important metric was the number of families who fired up the app at least once a day. Those who did were considered &#8220;active&#8221;, while those who didn&#8217;t were assumed to have abandoned the application. Unfortunately, while High Score House was getting installations and great press reviews, users weren&#8217;t active.<\/p>\n<p>No active users, no business.<\/p>\n<ul>\n<li>One Metric That Matters: &#8220;Active families&#8221;<\/li>\n<li>KPI: Percent of families who&#8217;ve used the app in the last 24 hours (AKA &#8220;percent daily use&#8221;)<\/li>\n<li>Target: Over 60%<\/li>\n<li>Current level: Below 20%<\/li>\n<\/ul>\n<p><font color=\"green\">Step 2: Form a hypothesis<\/font><\/p>\n<p>The founders went around the Lean Analytics Cycle several times:<\/p>\n<ul>\n<li>They tried changing the look and feel of the application.<\/li>\n<li>They tried sending people messages urging them to come back to the application.<\/li>\n<li>They tried adding new features.<\/li>\n<\/ul>\n<p>None of it worked; fewer than 20 percent of families who were part of the early release used the app each day.<\/p>\n<p>Each time, their hypothesis went something like, <em>if we send a reminder email to parents each day, the percent daily use will exceed 60%.<\/em><\/p>\n<p><font color=\"green\">Step 3: Create the experiment<\/font><\/p>\n<p>The High Score House team went dutifully through the motions of the Lean Analytics Cycle. For the notification hypothesis, the who, what, why looked like this:<\/p>\n<div style=\"MARGIN-LEFT: 2em\"><em>Who is the campaign for?<\/em> Current installed base of users.<\/p>\n<p><em>What do you want them to do?<\/em> Use the app daily.<\/p>\n<p><em>Why do they do it?<\/em> An email will remind them to launch the app and sit with their kids, making it a daily activity that becomes part of their routine.<\/p>\n<\/div>\n<p>In other words, their experiment (for the notification model) was:<\/p>\n<p>Find out if parents will use the app daily because we remind them to do so enough get the percent daily use to 60%.<\/p>\n<p>They ran the experiment.<\/p>\n<p align=\"center\"><img loading=\"lazy\" hspace=\"5\" alt=\"high score house lean analytics cycle occams razor\" src=https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2013\/04\/high_score_house_lean_analytics_cycle_occams_razor.png\" width=\"625\" height=\"466\" \/><\/p>\n<p><font color=\"green\">Step 4. Measure performance<\/font><\/p>\n<p>The experiment disproved the hypothesis. Email reminders didn\u2019t measurably improve percent daily use. Like many of the other changes the team had tried, they couldn\u2019t move the needle.<\/p>\n<p>At this point, Kyle, the founder of the company, decided it was time to ditch the data and go qualitative. He picked up the phone and started talking to users. And this simple, messy, unquantified, anecdotal process yielded an amazing insight:<\/p>\n<p>Many families would set aside a specific day of the week to plan their chores and count their rewards.<\/p>\n<p>Kyle was floored. Families were getting value from the product. They were just using it in a different way from what the designers intended. Kyle now had new information which he could use. But it wasn\u2019t for another experiment. This time, Kyle was going to move the goalposts\/target.<\/p>\n<p>Changing your target should never happen lightly. It\u2019s only OK to change the goal when the customers have given you new, validated learning from which to make the change. In this case, since your goal is tied back to your business model (remember the $5 glass of lemonade?) changing the goal means changing your business model<\/p>\n<p>In this case, HSH was going to have to change the way their application was designed, making it more suitable for week-by-week views and a different usage pattern. But by moving the goalposts, the team had a new KPI: Percent of families who&#8217;ve used the app in the last 7 days (AKA &#8220;percent weekly use&#8221;). And with this new definition, over 80% of the early adopters were using the application regularly.<\/p>\n<p>We\u2019ve seen three case studies. The first one, about Airbnb, shows a straightforward example of a hypothesis and an experiment without data. The second, about Circle of Friends, shows an example of an experiment based on data. And the third one, about High Score House, shows how to adjust the goalposts when they aren\u2019t set properly in the first place. All three are examples of the Lean Analytics Cycle at work.<\/p>\n<p align=\"center\"><font color=\"blue\"><strong>______________________________________________________<\/strong><\/font><\/p>\n<p>\nAmazing, right? Life&#8217;s pretty cool when you have a process!<\/p>\n<p>For me the lean analytics cycle is, at it&#8217;s very core, about driving change quickly, regardless of how much you know about your users or how much data you have. It discourages waiting for perfect data or perfect understanding of every variable or perfect understanding of what the business is trying to solve for (because in all three cases you will never get perfection!).<\/p>\n<p>While it was developed with startups in mind, you can see the attraction of speed, learning and constant improvements for businesses of any size.<\/p>\n<p>It is important to note that the cycle, and the Lean model as a whole, isn\u2019t advocating chaos. They might deal with uncertainty, but they&#8217;re not random.<\/p>\n<p>There is great deliberation in step one to identify the KPI that will be a guiding light for us (embrace &#8220;One Metric That Matters&#8221;). There is a lot of deliberation in step two on ensuring that we have an optimal hypothesis to work from. Then figuring out how we are going to experiment with deep clarity from defining the <em>who<\/em>, <em>what<\/em>, <em>why<\/em>. Finally, understanding whether we actually succeeded by measuring performance. Then, then &#8230; you learn. You internalize. You win.<\/p>\n<p>And how can you be immensely incredible at this? Go get <a href=https:\/\/www.amazon.com\/Lean-Analytics-Better-Startup-Faster\/dp\/1449335675\/?tag=occsrazbyavik-20\" target=\"_blank\">Alistair and Ben&#8217;s book Lean Analytics<\/a> .<\/p>\n<p>As always, it is your turn now.<\/p>\n<p>Do you follow a structured process inside your company to ensure data-driven decisions are being made constantly? If you were to apply the Lean Analytics Cycle, what challenges might cause problems inside your company? As you look at the cycle, is there anything that might be missing, or tweaks you would make to improve it?<\/p>\n<p>Please share your thoughts, critique, ideas, and musings via comments.<\/p>\n<p>Thank you.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Lean Analytics a new book by my friend Benjamin Yoskovitz. The book introduces a wonderful process called the Lean Analytics [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[11,383,384,9,12,10],"tags":[299,430,106,171],"class_list":["post-5726","post","type-post","status-publish","format-standard","hentry","category-advanced-analytics","category-big-data","category-digital-analytics-2","category-web-analytics","category-web-insights","category-web-metrics","tag-actionable-analytics","tag-business-optimization","tag-experimentation-and-testing","tag-key-performance-indicators"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>The Lean Analytics Cycle: Metrics &gt; Hypothesis &gt; Experiment &gt; Act - Occam&#039;s Razor by Avinash Kaushik<\/title>\n<meta name=\"description\" content=\"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The Lean Analytics Cycle: Metrics &gt; Hypothesis &gt; Experiment &gt; Act - Occam&#039;s Razor by Avinash Kaushik\" \/>\n<meta property=\"og:description\" content=\"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/\" \/>\n<meta property=\"og:site_name\" content=\"Occam&#039;s Razor by Avinash Kaushik\" \/>\n<meta property=\"article:published_time\" content=\"2013-04-08T09:17:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-05-19T22:32:37+00:00\" \/>\n<meta name=\"author\" content=\"Avinash Kaushik\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Avinash Kaushik\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"8 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/\"},\"author\":{\"name\":\"Avinash Kaushik\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\"},\"headline\":\"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act\",\"datePublished\":\"2013-04-08T09:17:00+00:00\",\"dateModified\":\"2026-05-19T22:32:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/\"},\"wordCount\":1648,\"commentCount\":77,\"keywords\":[\"actionable analytics\",\"business optimization\",\"experimentation and testing\",\"key performance indicators\"],\"articleSection\":[\"Advanced Analytics\",\"Big Data\",\"Digital Analytics\",\"Web Analytics\",\"Web Insights\",\"Web Metrics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/\",\"url\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/\",\"name\":\"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act - Occam&#039;s Razor by Avinash Kaushik\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#website\"},\"datePublished\":\"2013-04-08T09:17:00+00:00\",\"dateModified\":\"2026-05-19T22:32:37+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\"},\"description\":\"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/lean-analytics-cycle-metrics-hypothesis-experiment-act\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#website\",\"url\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/\",\"name\":\"Occam&#039;s Razor by Avinash Kaushik\",\"description\":\"Marketing | Strategy | Analytics\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.kaushik.net\\\/avinash\\\/#\\\/schema\\\/person\\\/4833e40c5e82d40b6e6d0cc37d026ad6\",\"name\":\"Avinash Kaushik\",\"description\":\"Author: Web Analytics 2.0 &amp; Web Analytics: An Hour A Day | Chief Strategy Officer: Human Made Machine\",\"sameAs\":[\"https:\\\/\\\/www.kaushik.net\\\/avinash\",\"https:\\\/\\\/www.linkedin.com\\\/in\\\/akaushik\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/avinash\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act - Occam&#039;s Razor by Avinash Kaushik","description":"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/","og_locale":"en_US","og_type":"article","og_title":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act - Occam&#039;s Razor by Avinash Kaushik","og_description":"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!","og_url":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/","og_site_name":"Occam&#039;s Razor by Avinash Kaushik","article_published_time":"2013-04-08T09:17:00+00:00","article_modified_time":"2026-05-19T22:32:37+00:00","author":"Avinash Kaushik","twitter_misc":{"Written by":"Avinash Kaushik","Est. reading time":"8 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/#article","isPartOf":{"@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/"},"author":{"name":"Avinash Kaushik","@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6"},"headline":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act","datePublished":"2013-04-08T09:17:00+00:00","dateModified":"2026-05-19T22:32:37+00:00","mainEntityOfPage":{"@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/"},"wordCount":1648,"commentCount":77,"keywords":["actionable analytics","business optimization","experimentation and testing","key performance indicators"],"articleSection":["Advanced Analytics","Big Data","Digital Analytics","Web Analytics","Web Insights","Web Metrics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/","url":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/","name":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act - Occam&#039;s Razor by Avinash Kaushik","isPartOf":{"@id":"https:\/\/www.kaushik.net\/avinash\/#website"},"datePublished":"2013-04-08T09:17:00+00:00","dateModified":"2026-05-19T22:32:37+00:00","author":{"@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6"},"description":"Companies, small or large, fail to iterate fast enough leading to poor business results. Lean Analytics Cycle is a structured process to avoid that fate!","breadcrumb":{"@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.kaushik.net\/avinash\/lean-analytics-cycle-metrics-hypothesis-experiment-act\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.kaushik.net\/avinash\/"},{"@type":"ListItem","position":2,"name":"The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act"}]},{"@type":"WebSite","@id":"https:\/\/www.kaushik.net\/avinash\/#website","url":"https:\/\/www.kaushik.net\/avinash\/","name":"Occam&#039;s Razor by Avinash Kaushik","description":"Marketing | Strategy | Analytics","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.kaushik.net\/avinash\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.kaushik.net\/avinash\/#\/schema\/person\/4833e40c5e82d40b6e6d0cc37d026ad6","name":"Avinash Kaushik","description":"Author: Web Analytics 2.0 &amp; Web Analytics: An Hour A Day | Chief Strategy Officer: Human Made Machine","sameAs":["https:\/\/www.kaushik.net\/avinash","https:\/\/www.linkedin.com\/in\/akaushik\/","https:\/\/x.com\/https:\/\/twitter.com\/avinash"]}]}},"_links":{"self":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/5726","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/comments?post=5726"}],"version-history":[{"count":1,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/5726\/revisions"}],"predecessor-version":[{"id":10149,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/posts\/5726\/revisions\/10149"}],"wp:attachment":[{"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/media?parent=5726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/categories?post=5726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.kaushik.net\/avinash\/wp-json\/wp\/v2\/tags?post=5726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}