Analytics On The Bleeding Edge: Transforming Data's Influence

Analytics teams are named for the silos and limitations within which they trap themselves.

Paid Media. Owned Media. SEO. BI. Customer Service. Data Warehousing. Email. And, a thousand other silos (depending on your company size).

One outcome of this reality is that while every team works hard to do their very best work, it is rare that they earn strategic influence from their work. That’s not really surprising, if your view of your scope is narrow… Your impact will be narrow as well.

The other dimension to consider is most Analtyics teams kick into gear after the campaign is concluded, after the customer interaction has taken place in the call center, and after the funds budgeted have already been spent. When you only look backwards, it limits your ability to have an impact.

Finally, few analytics teams obsess about predictive analytics in a way that allows them to dictate future action. This is a huge miss… Left to their own accord, how many companies will make the same decisions data would recommend? Astonishingly few.

Transforming Data’s Strategic Influence.

The above-observed realities were on my mind as I took on a new role to lead Global Strategic Analytics. This time around, my goal was for the analytics team to chart a very different path… To solve for expansive influence, before, during, after, money is spent by the organization.

A key part of how this manifested in our work was doing truly super-advanced machine-learning powered analysis to answer hard questions that few can successfully. This is of course exciting and very cool.

But the difference in the team’s impact comes from the combination of an audacious vision and putting together the people-process-structure that powers our desire for data to have an expansive influence across the company’s decision-making needs.

I lovingly call our strategy analytics on the bleeding edge. It is powered by the union of:

1. intelligent analytics initiatives
2. application in data to new and unexplored areas
3. upgrades to processes to create deeper integration with Finance & Strategy teams
4. power real-time actions and future decision using automation and algorithms

While our team’s journey had many, many, miles to go, I want to share the broad strokes of our vision and execution in the hope that it’ll help spark your imagination – perhaps you can use our core philosophy to reflect on your organization’s current status and future path.

I’ve boiled our approach into four smart clusters. It is most directly applicable to larger companies, but some components do apply to companies of all sizes.

Let’s look at them one at a time… Ready?

Smart Lessons | Analytics Cluster.

Like every good analytics team, we started doing work that you’ll recognize.

Executive scorecards, post-campaign analysis, some limited data puking (only when we absolutely can’t get away with it because someone who influences our existence is asking!), using Google Analytics in a smart way, setting good behavior standards like always having Targets (pre-set, always pre-set) and using methodologies that don’t suck and 90% or higher significance, etc., etc., etc.

Smart Lessons | Analytics Components

What may or may not be as common, but is an integral part of our analytics strategy is the extensive use of controlled experiments to answer life’s hardest questions.

Is campaign strategy x better than campaign strategy y? Because they are both different, both have different executive sponsors and it is insanely hard to know based on data we have which one is better.

Does advertising really have a long-term business impact? Surely you’ve been asked that one before, and there is a short-term answer but the long-term one needs a sophisticated controlled experiment.

What does the diminishing returns curve look like for TV GRPs for our company? More shouting is not really better – and it is expensive!

This is very hard to do, we now have a proven seven-step experimentation process, with one of the coolest algorithms to pick matched-markets (normally the kiss of death of any large-scale geo experiment).

Underpinning our Smart Lessons work is the very basic – incredibly complex – art of picking the right Key Performance Indicator. It underpins every dimension of success.

Hence, if your assessment is that you are messing up Smart Lessons, it is because of this simple reason: Wrong/bad KPIs.

The choices our team helps make are powered by another awesome innovation: The Impact Matrix.

Smart Start | Analytics Cluster.

It has always gnawed at my soul that most companies only turn to data after all the money has already been spent. After the campaign is done. Long after your CRM-powered emails have been sent. Long, long,  after your TV ads have stopped running.

On those non-normal occasions when the campaign did not quite work, it pained me that we were learning after the money had already gone poof!

Hence, in this role, in particular, I've been deeply obsessed with using data before any money was spent. For data to provide a degree (high!) of confidence that, if we spend the money, it would most likely deliver a positive impact on the business.

This cluster, a part of our awesome win before you spend strategy, is called Smart Start.Smart Start| Analytics ComponentsThe first component is a gloriously scaled global creative pre-testing program.

Creative is the thing you see in the ad. The text. The goats. The slow music. The repeated mention of the product (hopefully). The use of a celebrity (or not!). Yada, yada, yada.  It turns out, hold on to your seats, the creative has approximately 60% influence on the ultimate success of your campaign!

Not the audience, not your ad stack, not your targeting strategies, not your obsession with other little bits that are currently sucking up 98% of your attention.

The creative is what matters, and, unfortunately, few people who do modern analytics focus on the creative.

We pre-test pretty much everything in an online labish environment, and predict whether a piece of a TV or Billboard or Radio or YouTube or Facebook creative will be successful. With the support of our progressive CMO, we spend money on creative that passes pre-test.

In a unique feature of our analytics strategy, we practice a trust, but verify approach to lab testing. We routinely put failed in-lab creative in the market and use both passed in-lab and failed in-lab to see how they do in the real world. This helps us understand the quality of signal we get in-lab (it is around 67% for Yes and 89% for No).

Now, our Marketing teams know before they spend money if the campaign’s creative will deliver success.

The other gloriously scaled global practice is our pre-flight check, which we call the Minerva Check – named for the Roman goddess of wisdom. :)

The Minerva Check is a collection of media plan minimums required for success. Reach. Frequency. Duration. GRPs. Passed creative rates. Tactical strategies. Ad unit types. Etc.

We identified these minimums from the massive amount of data we have for our past campaigns. Meta-analysis. Matched market tests. Product/regions/channels. Throw everything in there, and out comes a list of things that every media plan has to meet to be greenlit.

Now, our Marketing teams know before they spend money if the campaign’s creative will deliver success.

It is not perfect (I like perfect). It is not enough (I want more). But, with incredible certainty, we can now say, before any money is spent, that the chance that Media Plan X will deliver success is 3% and the chance that Media Plan X-modified (with Minerva incorporated) will deliver success is 97%.

Intelligently applied data proving its value when it really, really matters – before the company budget is spent.

I’ll give you one guess as to how much our VP of Finance loves this capability. :)

Smart Execution | Analytics Cluster.

For the last year, the thing that I have been obsessed with, along with our small team, is the next cluster: Smart Execution.

Real-time data in the hands of humans is a colossal waste of technical, financial and human resources. It takes too long for humans to process the data through themselves, their team, the bureaucracy, the agencies, and the technology stack to convert real-time data into action.

(From 2006: Is Real-Time Analytics Really Relevant?)

Yet, there are, often literally, tons of signals coming off your ad and analytics stack that you can use to identify if things are going right or wrong and take quicker action – by eliminating humans from the process!

My deep love for this cluster comes from the competitive advantage you can build for your company…

Smart Execution | Analytics ComponentsIn our in-flight optimization journey thus far, we have worked to identify signals that are believable, and identifying at which point they become believable (ex: statistically significant).

The next step was to create a collection of decision trees. It sounds complex, it is not.
Here’s an example. The benchmark for the beautiful metric AVOC is 15.3%. The decision tree is: If it is about 20% for our campaigns, then sing happy birthday.  If it is between 10 – 20%, then raise a flag. If it is below 10%, then stop.

Then, automate the execution of this decision.

Now repeat this across many, many metrics, for many dimensions, in the three clusters you see above.


You have the start of a fabulous in-flight optimization engine.

We’ve now made data influential and useful while we are spending money!

You can imagine that excellence in the orange (Smart Start) and blue (Smart Execution) clusters means there is more green coming out of our green cluster (Smart Lessons) than red (bad news).

Another secret agenda: By being excellent at Smart Start (win before you spend) and Smart Execution (win while you spend), you end up making Smart Lessons (the thing that occupies so much of your present life) utterly boring!  By the time you see the results, you pretty much already know what they are going to say. :)

Smart Future | Analytics Cluster.

I hope you got the sense that I am very, very passionate about each cluster above and that they are my favorite children.

That would be a misleading conclusion.

Yes, it is all fantastic work that is obsessive about making data more useful in more novel ways for our company than is common in other companies.

But my favorite child is cluster four: Smart Future.

It answers the hardest questions a CMO asks:

What is the true incrementality of all my marketing spend?

What is the bottom line (ex., sales) impact of my brand marketing?

How does the portfolio of all our activities – owned, earned, paid media, promotions etc. – work together, and how do the channels complement each other?

How effective are our efforts in the context of all the actions our competitors are taking to impact our company?

I see you nodding your head. You are being asked these questions, and you know the depth of analytical difficulty.

Our innovative approach…

Smart Future | Analytics ComponentsDue to non-public IP, it is a cluster I can’t tell you much about.

Except to say that most companies, when they attempt to answer the aforementioned questions, take the approach of using existing statistical approaches that require explicit programming. We’ve chosen to use machine learning algorithms that learn from the underlying structures inside massive amounts of our datasets without explicit programming. That’s the magic.

As the second box indicates we not only use this approach to look backwards. Rather, delightfully, we also look into the future. We have the ability to model scenarios, budgets, channel allocations to maximize effectiveness and efficiency for our future campaigns.

It is very unique and difficult work.

Analytics on the Edge.

So what does a strongly proactive and truly influential data strategy at the bleeding edge look like?

Machine learning algorithms help to create the optimal budget and channel allocations that flow into Smart Start programs, ensuring core elements of marketing are pre-wired to deliver success. The data then flows through automated decision trees, making decisions in close to real-time maximizing success in the real world. The end result is data stories and scorecards that help our leaders get a unique view into marketing’s impact.

Data not as a side-show. Data as an influential core.

Analytics on the Edge | Smart Clusters

Cool, eh?

We are not perfect, we are not complete, we have miles to go, but I am so proud of the work the team has done and the business impact delivered.

There is so much packed in each box, I could write a 98-page book for each. :)

A reflective assignment for you.

How expansive is your analytics strategy? How much influence does it have on the marketing (or product or customer service or HR or whatever else) that your company undertakes?

Print out the image below.

Sit down with a pen for when you have a calm 20 mins to think.

Write down across each cluster how and where your analytics efforts are being applied today.

Smart Clusters Worksheet

If you see that you have something filled in each of the Smart Start, Smart Execution, Smart Lessons, and Smart Future cluster, raise your hand up because I’m high-fiving you at that very moment!

Well done.

If you found that not all the boxes are filled, no biggie. Start to sketch proposals to take to your leadership.

If you see that your efforts are mostly centered in the green Smart Lessons cluster, be happy. That is where most companies are. But, also be hungry because you want more.

I hope that in this note you’ve found enough very specific starting points, informing you about what you need to do in order to get to the bleeding edge or of analytics and, ultimately, implement a strongly proactive and truly influential data strategy.

If you want my recommendation for the best next cluster to focus on, it is Smart Start. Influence the company with data before any money is spent, and untold love will shower on you (along with promotions and extra headcount for your team!).

Bottom line.

It is very hard in a modern corporation, which, still, primarily runs on opinions, to make data influential. (Not the be-all and end-all, influential.)

Yet, we have so much data, we have such vast opportunities as Analysts to have a material impact on the company’s profitability and even the direction of the business. We just need to unlock our imagination.

It is not easy. But, nothing worth having is.

Carpe diem.

As always, it is your turn now.

Please share your feedback on my ideas and lessons from your anatlyics journey, via comments below. Thank you.


  1. 1
    Dan Jackson says

    This post is why I'm glad you are back to regular writing on Occam's Razor.

    In some ways this is so far and beyond what our analytics team is doing at our large company. Initiatives like creative testing or the minerva checks are not something we've considered, and yet as you describe so crucial to ensuring Marketing's success.

    In the future, I hope you'll share more about Smart Future as you've certainly piqued my curiosity!

    Thanks Avinash.

  2. 2

    This is just another LTBP (Love-this-blog-post) article.

    What bothers me is the Minerva block (how it's built, what data was used in the decision trees for every step :), the min. amount of data needed/used and the look-back window of data used in the start, especially for the Smart start media plan.

    Can you share a little bit more information about it?

    • 3

      Miroslav: Always lovely to hear from you.

      We take into account the last few years' worth of a channel's performance to build the Minerva Check for the minimums a media plan has to hit to have a chance of success. We take into account observed data (say surveys or clickstream), experiments we've run to test various hypotheses, and models we've built (advanced statistics or machine learning) using data we have or data sometimes we buy for panels that are out there (to include longitudinal as well as competitive data).

      In the end for each channel, we'll have the criteria to pass the Minerva Check. For example, here's part of the Minerva Check for a Television media plan in the US:

            >= 4 weeks @ xxx TRPs
           ya%/yb% of campaign reached 6+ 
           At least zz% of TRPs on broadcast/sports
           Minimum mmm TRPs per creative concept
           >=75% of budget behind passing creative

      All those numbers in red are real numbers – sadly I can't share them with you.

      Our Minerva Check covers all major channels on the planet (from YouTube to Billboards to Magazines to Facebook and more).

      I hope this is of value.


  3. 4

    Hi Avinash,

    Just a few points. You're right in saying that analysts can build their credibility by being all-in early in the process.

    Having media plans built around predictive analytics is an excellent approach to reducing risk with certain approaches.

    Also, understanding diminishing returns from increased audience % reached is something that is definitely worth exploring early in the planning phase. FB Ads has a Campaign planner while Google has Reach Planner. In FB ads Campaign planner, it shows curves where the growth in audience % reached begins to taper off even as more money is pumped into reaching the audience. We've even had cases where we've suggested saving marketing $ as it didn't seem worth the extra push.

    • 5

      Adil: Great points, thank you for sharing them.

      One of the things we've found transformative is to tie reach-based planning tools to outcomes we can predict.

      This way, are not just saying hey you can reach x type of people in y numbers but also add and we believe this campaign will deliver a +5 lift in Consideration at a Cost Per Individual Lifted of $6. That helps spark new, interesting conversations about what's worth it and what's not.

      Fun times. :)


  4. 7
    Michelle Lawson says

    Avinash: I'm not sure if I should be super-excited or really depressed. :)

    On one hand, I feel like I've chosen the right career because this post really unlocks just how wide and deep the impact of an Analytics team can be.

    On the other hand, I feel so far away from ideal in my current role, and perhaps knowledge.

    Thank you for consistently pushing me, our industry, forward with your vision. I'm off to think about win before you spend work at my job!

  5. 8
    Tricia Mumby says

    Thank you Thank you for sharing your mind with us!

    Could you describe the "labish" testing?

    • 9

      Tricia: We have several strategies we use to test our creative (or other assets) to allow our target audiences to share their reflections.

      A primary strategy is to create immersive mirror environments of digital platforms to allow in a test/control audience groups to experience our ads and share their reflections.

      Another strategy is to be where people are (say, malls) and in a physical space have people share their reflections.

      More like these. Not quite a lab with one way mirrors environment, but as good as we can make it with the tech at our disposal.

      Hope this helps a bit.


  6. 10

    Thanks for sharing Avinash! Very helpful!!

    I request if you can share a little more details on the match markets test? Is it something more on lines with the brand and conversion lift studies? Any articles/reads you can recommend would be helpful.

    Thank you!

    • 11

      Pranav: Conversion lift studies are certainly a variation of an experiment. Usually they are less market based and more audience based, still super cool and super useful.

      Matched Market Tests are simply a sexier way, :), of saying Geo Experiments. For questions related to, say, incrementality, diminishing returns, other hard questions, we can fragment the DMAs (US) into matched markets (based on anywhere from 6 to 18 criteria, leveraging some synthetic algorithms to do it optimally). This allows us to run large-scale hard-question experiments.


  7. 13

    Magnificent, much more than a man burdened with a multitude of tasks will be able to do, but that's what he wants to do immediately after reading!

    One wisecrack remark: Minerva is _Latin, Athene would be the Greek version (but don't ask me about Rama vs. Krishna, or Ramanujan vs. Kaushik, btw)

  8. 14
    Anubhab says

    Thanks, Avinash for sharing this analysis.

    Thanks a ton.

  9. 15

    I would love to see the analytics on what happens when people buy Twitter followers, and how it affects their overall engagement rate.

    Obviously, at first, the engagement rate would drop as the fake accounts don't engage with you.

    However, if you end up benefiting from Social proof of having so many followers, how many more people that DO follow you would end up engaging with you more, hoping to siphon off your followers – if nothing else.

Add your Perspective