![\"Shine\"](https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/09\/shine.jpg\")
With a plethora of digital media channels at our disposal and new ones on the way every day(!), how do you prioritize your efforts?<\/p>\nWith a plethora of digital media channels at our disposal and new ones on the way every day(!), how do you prioritize your efforts?<\/p>\n
How do you figure out which channels to invest in more and which to kill?<\/p>\n
How do you figure out if you are spending more money reaching the exact same current or prospective customers multiple times?<\/p>\n
How do you get over the frustration of having done attribution modeling<\/em> and realizing that it is not even remotely the solution to your challenge of using multiple media channels?<\/p>\n Oh, and the killer question… if you invest in multiple channels, how much incrementality does each channel bring to your bottom-line?<\/p>\n Smart Marketers ask themselves these questions very frequently. Primarily because we don’t live in a let’s buy prime time television ads on all three channels and reach 98% of the audience<\/em> world any more.<\/p>\n We have to do Search Engine Optimization<\/a>. We have to do Email Marketing<\/a>. We have to do Paid Search<\/a>. We have to have a robust Affiliate network. We have to do Social Media<\/a>. We have to do location-based advertising to squarefour people. We can’t forget Mobile advertising<\/a>. We have to… the list is almost endless. Oh, and in case you had not noticed… the real world is still there. TV and radio and print and… Oh my!<\/p>\n The reality is that we can’t do all of those things.<\/p>\n Smart Marketers work hard to ensure that their digital marketing and advertising efforts are focused on the most impactful portfolio of channels. Maybe it is Search, Email and Facebook. Maybe it is Affiliate and Paid Search. Maybe TV and Twitter and Newspapers. Maybe it is five other things.<\/p>\n How does one figure it out?<\/p>\n Controlled experiments!<\/strong><\/p>\n What’s that? This: You understand all the environmental variables currently in play, you carefully choose more than one group of “like type” subjects, you expose them to a different mix of media, measure differences in outcomes, prove \/ disprove your hypothesis (DO FACEBOOK NOW!!!), ask for a raise.<\/p>\n It is that simple.<\/p>\n Okay, it is not simple.<\/p>\n You need people with deep skills in Scientific Method<\/a>, Design of Experiments<\/a>, and Statistical Analysis<\/a>. You need the support of the top and bottom and middle of your organization (and your agency!). You need to understand all the environmental variables in play (a hard thing under any scenario) not just in context of your company but also as they relate to your competitors and ecosystem.<\/p>\n But if you have access to some or all of that (or can hire good external consultants), then your rewards will be very close to entering heaven. Marketing heaven that is.<\/p>\n To make the case for controlled experiments I want to share with you one simple, real world example I was involved with.<\/p>\n My explicit agenda is to spark an understanding of the value of even simple controlled experiments (that might need only some of the horsepower mentioned above).<\/p>\n My secret agenda is to illuminate the power of this delightful methodology via a simple example, and get you to invest in what’s needed to move from good to magnificent.<\/p>\n Ready?<\/p>\n The Setup.<\/font><\/strong><\/p>\n This is a multi-channel example. The company truly has a portfolio strategy when it comes to marketing. We are going to simplify the example to make it seem like they only do two things. They mail catalogs and they send emails. The purpose of both is also simple: Get people to convert online (website) or offline (call center).<\/p>\n The question was, should they do both? Should they love one more than the other? Is this digital thing here to stay or is the thing that has worked so well for 150 years – catalogs – the thing that still works (“the internet is a fad!”)? What is the incremental value of doing email?<\/p>\n To answer this question the company took their customer lists (catalog and email) and identified like-type customers. Like-type as in customers that share certain common attributes. For your business, that could be people who have been customers for 5 years (or 5 months) or those that order only women’s underwear or those that live in states that start with W or those that order more than 10 times a year or only men or people who were born on Jupiter or… (this is where design of experiments comes in handy :).<\/p>\n Then they isolated regions of the country (by city, zip, state, dma pick your fave) into test and control regions.<\/p>\n People in the test regions will participate in our hypothesis testing. For people in the control region, nothing changes.<\/p>\n It is also important to point out that I am keeping the data simple purely to keep communication of the story straightforward. We’ll measure Revenue, Profit (the money we make less cost of goods sold), Expense (cost of campaign), Net (bottom-line impact).<\/p>\n What is missing in these numbers is the cost of…. well you. The people. A little army in your company runs the TV campaigns. A larger army is the catalog sending machine. A lone intern is your email campaign people cost. A team of five are your paid search samurais. When you do this, if you can, include that expense as well.<\/p>\n Enough talk, let’s play ball!<\/p>\n The Experiment and Results.<\/font><\/strong><\/p>\n Here’s the outcomes data for the control version of the experiment. This group of customers was sent both the catalog and the email. Nothing was changed for them – this group was treated exactly as they were in the past.<\/p>\n If the company did both things, revenue was $12.<\/p>\n Because revenue is very often a misleading way to understand impact on the company’s bottom-line, most smart people prefer to go for net impact (the result of taking out cost of goods, campaign expenses etc.).<\/p>\n In this case, that amounted to a bottom-line impact of $2.59.<\/p>\n [If you want to learn how a focus on the bottom-line, especially net profit can change your life, and I mean that literally, please see this video: Agile, Outcomes Driven, Digital Advertising<\/a>. Parts two and three, Rockin’ Teen and Adult (Ninja!).]<\/p>\n Here’s the data for variation #1 of the experiment… this group of like-type of customers were only sent the catalog – no email. The marketing messaging and timing and all other signals for relevancy and offers used for this group was exactly the same as the control group.<\/p>\n Compared to the control group, revenue went from $12 to $10. Company expense went down a little bit (email campaigns after all are not free).<\/p>\n The net impact went down from $2.59 to $2.00.<\/p>\n 17% reduction in revenue, 23% negative net impact to the bottom-line.<\/p>\n Does that help you understand the incrementality delivered by the campaign that is missing in this variation of the experiment (email in this case)? You betcha!<\/p>\n No politics. No VP of Email vs VP of Catalog egos and opinions involved. No you are trying to mess with my budget<\/em> spit on your face. No but that is not what Guru x at a conference said<\/em> or but that is not what people on Twitter think<\/em>. None of that. Just data.<\/p>\n How sweet is that?<\/p>\n Here are the results of variation #2… this group just got the email. The killing of trees, filling of recycle bins, and breaking the backs of postal carriers was paused. :)<\/p>\n Again, and I can’t stress this enough, all else was equal.<\/p>\n Compared to our control group there was a whopping 29% reduction in revenue. OMG!<\/p>\n But, a bigger OMG is coming: the net impact on the bottom-line of the company was a measly 2%! OMG!!<\/p>\n So the incremental value delivered by combining a catalog campaign with an email campaign is an increase of 2% on the bottom-line of this company.<\/p>\n Not for every company on the planet. Not even for all campaigns you do. But for this campaign and these types of customers you can confidently say: “Yes there was a drop in revenue and if you care about that, oh beloved HiPPO, then let’s send more catalogs. But at least now you know the net incrementalism delivered to our bottom-line from doing that.”<\/em><\/p>\n If your HiPPO is smart, and in my experience many HiPPOs are smart and well-meaning, shewill ask you this: “Is that 2% ($0.05) sufficient to cover the salaries, pensions, health benefits of everyone we employ to do catalog marketing?<\/em>“<\/p>\n Controlled experiments also allow us (Analysis Ninjas) to do some subversive work. A question that came to my mind was: What is the incrementality of doing any marketing at all? What would happen if we do nothing, and we retire all our marketing people? Would the company go under?<\/em><\/p>\n Now it is rare that questions like those get asked. But it is too tempting not to use this methodology to get a sense for what the answers might be.<\/p>\n So for variation #3, no catalogs or email were sent to the customers in the test group. Here are the results…<\/p>\n It turns out if you completely stop marketing, and you are an established company, the impact is not that your revenue goes to zero! :)<\/p>\n Revenue in this variation went down 58% (pretty big). The impact on net to the bottom-line was a reduction of 42%. Both not great, but not zero.<\/p>\n So now you have some sense of what is the incrementality of all the people in marketing (their salaries, pensions, expenses etc.), and what you have to compute is if it is less than or greater than $1.09 (the loss in net impact).<\/p>\n Talking just a smidgen more seriously, eliminating catalogs and emails (and all marketing) might not make the company bankrupt immediately. But that is simply an outcome within the confines of this experiment. And it is easy to imagine how the impact might just get worse over time. The nice thing is that you can also test that!<\/p>\n Good lord I love this stuff!<\/p>\n The Lessons from this Controlled Experiment.<\/font><\/strong><\/p>\n It is possible to compute incrementality of adding or removing marketing strategies.<\/p>\n It is possible to go back and use this incrementality to make solid, long-term new decisions for the business (and not to keep doing what you have forever until your business goes bankrupt).<\/p>\n It is possible to take politics, bickering, back stabbing and all that ridiculous stuff out of the picture. Okay maybe not all of it, but a lot of it.<\/p>\n It is possible to determine the value of doing Paid Search campaigns for brand terms where you already rank #1 via SEO. It is possible to understand if you should invest in Facebook at all. It is possible to understand how much to support your TV campaigns via Yahoo! display campaigns. It is possible to specifically nail down every incremental dollar added to the bottom-line of adding YouTube to your Search campaigns and then adding radio campaigns and then adding magazine ads and then adding Twitter. And along that chain it is possible to understand exactly when you’ve reached diminishing margins of return!<\/p>\n Important: The lesson you should not take from this is that catalogs don’t work. They may work for you, they may not. All you should take away are the possibilities outlined above.<\/p>\n Closing Thoughts.<\/font><\/strong><\/p>\n Here are some important bits of context, and a few more lessons I’ve learned from having done this a bunch of times…<\/p>\n * The results you see above are raw end results. The team did the normal modeling to ensure that the results were statistically significant<\/a> (large enough sample set, sufficient number of conversions in each variation).<\/p>\n * It is not always easy to get exact replica (like type) customer sets. There are always things that are a little bit beyond your control. Do the best you can.<\/p>\n * Work as hard as you can, and then some, to ensure that there are as few “disturbances” in your test and control group. In the middle of the experiment don’t start a new paid search campaign or tweeting like a crazy duck to the same set of customers. Shout loudly until the entire company knows what you are up to (and beg for their co-operation).<\/p>\n * No answer is ever definitive, so act on the results immediately.<\/p>\n * In the same spirit the best companies in the world know that you are in a constant testing mode. There are so many factors that can affect your results. Seasonality, shifting consumer behavior, competitive landscape changes, disruptive product introductions, new technologies, legalization of illegal things, so many more things.<\/p>\n So you test, learn, rinse, repeat, become awesomer.<\/p>\n<\/div>\n If you want to learn more about controlled experiments, and see more examples and a case study, please jump to Chapter 7 and page number 205 in your copy of Web Analytics 2.0<\/a>.<\/p>\n Bonus:<\/font> Here’s one of my favorite articles… all the way from 2007 but chock full of pithy valuable lessons for all of us regardless of our field: 41 Timeless Ways to Screw Up Direct Marketing<\/a><\/p>\n![\"marketing](https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/09\/marketing_profitability_analysis.png\")
<\/p>\n![\"marketing](https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/09\/marketing_profitability_analysis_catalog_only.png\")
<\/p>\n![\"marketing](https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/09\/marketing_profitability_analysis_email_only.png\")
<\/p>\n![\"marketing](https:\/\/www.kaushik.net\/avinash\/wp-content\/uploads\/2011\/09\/marketing_profitability_analysis_no_email_no_catalog.png\")
<\/p>\n\n