Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics

Off-CenterEveryone likes chasing big shiny objects all the time. What's not to like. They are big. They are shiny. :)

But a lot of progress in life comes from doing the everyday small things better. A small improvement, every single day, to move the ball a little more forward.

A best practice I've developed is to take a step back and reflect on if I have a good balance between chasing shiny objects and making incremental progress on small every day things.

Today's post is from the latter category. It is a story about dealing with a complicated everyday challenge in an incrementally better way.

It is not quite about web analytics, not quite about data you can just get from WebTrends or Google Analytics, and not quite about just making yet another pretty chart. Rather it is about trying to step back and to think differently about a particular problem and, this is so scary, fighting to not simplify things to the point of uselessness.

I deeply wish that the problem we confront in this post did not exist at all. I wish we did not have this desire to pulpify a bunch of metrics to produce something recognizable. But it does exist, and so let us figure out how to incrementally improve the insights we can deliver.

Here's a summary of our journey in this delightful post…

Ready? Let's go…

What are Compound Metrics?

Compound metrics are everywhere. An example from the world of sports is the NFL Passer Rating (don't worry, no one really understands what it is). In a government context, Consumer Confidence Index and Consumer Price Index are good examples. One that you likely care about a lot is your FICO credit score. Perhaps the most famous digital compound metric is PageRank .

In a digital analytics context a compound metric might be…

Visit Quality = [(% of downloads + % of visits with more than 68 pageviews – % of visits with less 2 pages + % visits with Facebook shares + % of visits with store searches)/5)*1476]

The goal would be to communicate, perhaps, the quality of the site or the quality of each visit or quality of the outcomes, to the Dear Leader in your company using an "one simple easy metric."

I understand why people want to create compound metrics.

We have too much data on the web. We can measure way too much nuance, and sub-nuance and sub-sub-nuance. Because there is so much, and so much of our leadership is full of simple minds (not a ding, we should not expect them to be Analysts), the instinct is to "make it all simple, make it into one simple number" in the hope that they'll understand what's going on.

The intent behind creating compound metrics is good, sadly the outcome never is.

The Challenge with Compound Metrics: Social Media Edition

Go back and look at the Visit Quality compound metric someone's defined above. Do you see the problem? Do you see why it would be nearly impossible for you to look at the number that get's spit out every day/week/month and learn almost nothing from it – even if the number moves every time ?

For a bit more on the other reasons please see: Actively avoid insights: Use Compound Metrics

Here's a summary: When you cross-breed a bunch of metrics to produce a, I'm being charitable here, hybrid "simple number," the process, by design, hides insights, hinders the ability to understand performance and almost never allows the management team to identify root-causes.

Let's look at a practical example that demonstrates why I'm allergic to compound metrics.

Tool X below computes a social influence score for me. I'm a 83.

(Even if you can recognize this tool can I please request that you not name it in comments? They are hardly unique in creating compound metrics in the social analytics space. Thank you.)

social score avinash

So what does that 83 mean?

Nothing.

I just know that I'm 83 and a peer of mine is 87 and another one is 76. So what? More importantly, what do I do now?

There is a trend in the image. I can see it goes up and down.

Why does it do that? Not clear at all.

What contributes to 83 that causes the trend to go up and down, or for that matter what causes the 87?

No clarity.

So what do we do with the 83? Well… It depends on what you are measuring. Oh, it's not clear what it is measuring. Hmm… so it is useless?

They do share how much of this influence is from my presence on Twitter, Google+, Facebook and Wikipedia. Oh… Maybe that helps?

social influence breakdown

No. Not really.

So what's the 83 and what can you do with it?

Nothing.

This is the problem with compound metrics. You don't know what causes the final number to happen. You don't know the focusing factors, hence you are blind to why things go up/down and what actions you can take to improve.

In this case this Tool X is deliberately hiding all the focusing factors because they don't want you to game the system. I respect that. But for me as the recipient of this "boiled down from all the complexity simple metric that anyone can understand" it is a little less than useless.

Compound metrics. #arrrrhhhh

Let's look at another example and bring the problem closer to a company level challenge.

(Again, I just pulled a random example. If you know the tool or the provider, please do not mention it in comments. They are not unique in being guilty. Thank you.)

Tool Y shares this comparison of two companies.

15

What can you learn from it? In their social presence each company wants more of the Engagement Rate, what insights can you derive about each company from Engagement Rate performance above?

Yes. One seems high and other other seems low, in extremely tiny percentage terms.

But can you learn anything about what Mercedes-Benz is doing right or wrong? How about Audi? What might they be doing right or wrong?

Nothing. Zip. Nada.

You don't even know if 0.670% is awesome or completely sucky. You might if you knew that the upper limit or benchmark for Engagement Rate is 100%. That would give you incredible context. But you don't get that above. That in turn implies you don't even know if Mercedes-Benz is just a little less than awesome or really totally completely sucky!

What if you were kindly told by Tool Y that this is the formula used to compute Engagement Rate…

facebook engagement score attempt

(Remember, don't mention the tool or the company in comments. EVERYONE does this, let's not single them out.)

Does it help?

Now can you tell me what 0.290% and 0.670% mean?

And, always more importantly, why 0.670% became 0.670%? What caused Audi's number to be higher than MB?

Let's say this was not Audi and MB.

In April your Website Engagement Rate (WER) was 29 and in May your WER was 67. Would the score of that compound metric be able to tell you anything about what you did better or worse? Would your management team know what action to take when you plonk this compound metric in a dashboard?

Extremely highly improbable. Hence my allergy to compound metrics.

[PS: The above formula is not even the most sub-optimal one. There are social/search/website engagement/quality/experience compound metrics that are much, much worse. Those exquisite beauties contain weights, multipliers and other sundry "values" attached to each element in the formula. "Values" that are sourced from nothing more than our biases and blind spots. The end result is unique type of awful that you have to experience to believe.]

Compound Metrics "Decrapification" Strategy: Social: Standard Approach

If you can avoid compound metrics, that is the optimal. Rather can creating unrecognizable pulp, use the Digital Marketing and Measurement Model process to identify the best direct key performance metric. Use them to deliver insights that directly drive ultimate business profitability.

In our social scenario above I'll take inspiration from the Digital Marketing and Measurement Model process to create an alternative simple approach to using compound metrics.

Rather than "simplify" things and put five metrics into a blender and puke out an "easy to understand" number, my strategy would be to expose the focusing factors in order to encourage our leadership to look a little deeper to understand performance.

Instead of calculating the "Average Social Engagement Rate," I would much rather (as recommended in the Best Social Media Metrics post) show the management team this…

best social media metrics

Four metrics, not "one simple easy metric."

The first three, Conversation Rate, Amplification Rate, Applause Rate, show, wait for it…. engagement (!) on the social channels. As you look across you can see how well or badly you are doing on individual channels and the last row beautifully shows how much money we make off each social visit to your digital existence.

Sweet, right?

[If you would like to get the above view for your social media efforts please checkout True Social Metrics.]

What about Audi and Mercedes-Benz?

Instead of the cute infographic compound metric view with two other value deficient metrics, number of fans and total "interactions," would it not be magnificently been better to show this…

alternative to social engagement rate compound metric

Yes, it is neither cute looking nor "one simple easy metric." It is also a little harder on the eyes. The recipient will have to think a little harder.

But would you agree that the above view makes it really, really, easy to understand social engagement AND social's impact on the business of each company?

So, if you are willing to trade a small amount of pain (the discussion you have to have with your Dear Leader that a "one simple easy metric" might actually not be so simple or easy or valuable, and that you'll show they a bit more data), then it is entirely possible to completely avoid having to use a compound metric. Imagine the waves of pleasure that will wash over your body because you no longer have to create a crazy formula based on little more than biases and blind spots (exemplified by the Engagement Rate formula above).

Oh, and this works beautifully even if you look at just one company and their performance over a period of time.

Let's say we work for Audi (hurray!) and want to show social performance. Rather than saying the Average Engagement Rate was 0.67%, 0.68%, 0.57%, we can just show this table….

alternative to social engagement rate compound metric 2

Yes, more data. But so much easier for the Dear Leader to understand performance, brainstorm on root-causes and discuss fixes with the leadership circle.

Progress. Prosperity.

Compound Metrics "Decrapification" Strategy: Social: Awesome Approach

What if you are in a situation where your good intentions are insufficient?

What if you are a little minion, but a poor humble soul, who no one cares about and who's being whipped into creating compound metrics?

It does happen. Sometimes you simply can't avoid it. There is too much pressure to "simplify" things.

What do you do? How to ensure that you still deliver something of value?

Here's my suggestion: Give them want they want, but package with it what they need. Oh and while they are at it, include the one disruptive thing that will give them critical performance context!

Show them this….

audi social engagement score

We have the "one simple easy metric" up top. (Happy?) Then we have the focusing factors (contributing elements) clearly outlined. Further more we show them performance of each element.

But we don't stop there! We also include something that is incredibly valuable for Dear Leaders around the world in trying to understand performance: We include critical context to illustrate if the performance seen is good or bad.

For the last part (the big gray box) I've used 1. an available benchmark for the metric or 2. an upper limit I'm aware of or have computed from existing data or 3. a target we as a company are shooting for.

Now the Dear Leader has what they wanted, but we've also provided what they needed to go from data to action.

It should be quite obvious why your company will make smarter decisions now.

Oh, and you don't have to use my best social media metrics . You can use anything you want for your formula that computes your compound metric.

For example you can do this….

audi social engagement score 2

I don't think it's better. But what do you care. : )

The point is, you should embrace the framework. You can use any reasonably relevant collection of metrics, for the compound metric you are trying to create. Just make sure your focusing factors (metrics you use) are as complete as possible, and ideally independent from each other.

Switching back to my original four focusing factors… you can now see exactly how this would look like if we wanted to compare the Social Engagement Scores of Audi and Mercedes-Benz…

mercedes audi social media savvy comparison

So, so, so much better right?

Even though the compound metric is there on top, you'll deliver a deeper understanding of why Mercedes is at 0.290% and Audi is at 0.670%. Both companies can learn from the other and figure out how to fix their social strategy to rock more.

And remember the above visualization could be for the month of May and June for one company. Still works beautifully in explaining the performance of the "one simple easy metric."

Compound Metrics "Decrapification" Strategy: Search: Awesome Approach

As I'd mentioned earlier, this approach could work in other contexts as well – any time you end up with a compound metric.

You follow the same three step process for decrapification outlined above:

1. Show the compound metric.
2. Show the contributing metric's performance.
3. Show that against a benchmark or upper limit or target.

Step three above is something I've started to do a lot recently, my tiny incremental innovation if you will.

Let's look at one more example to really solidify this three step process for you. Since we've beaten Social to death already, let's look at another exiting area: Search.

This example is one where I was unable to run away from a compound metric. I had to create one. Boo!

The mandate was to try and figure out how to show any company their "overall Search performance" using a "one simple easy metric."

How is that even possible given the insane complexity that is the digital Search business? But mandates demand that we deliver, so off we go.

Here's my humble attempt at a Search Performance Index (SPI)…

zqi search performance index

It boils the entire complicated paid search business into four key levers (four focusing factors, four key metrics). Keyword Relevancy, Bid/Budget Savvy, Ad Creativity (copy, sitelinks, offers), and Geo/Targeting Smarts. It uses those levers to create an index (score on top).

I'll let your imagination run wild as to how the score itself is computed. (If there is interest, I'll share it in a future blog post.)

The cool part is not that there is an overall score that the Dear Leader can look at and be thrilled with. The cool part is that she/he will be forced to glance, even if for a moment, at the focusing factors that explain why the Search Performance Index is at 643. The coolest part is that they'll know which area needs more attention because of the distance between the current performance and what's possible (the benchmark/upper limit/target).

Now it is easy for them to prioritize where we should devote resources to improve the SPI. Action!

That is, dear blog readers, to borrow a phrase, priceless.

I'm sure you noticed that while there is a lot of data behind the simple visualization above, there is almost no data overtly present. Just a bunch of bars.

That is by design. Sharing a more nuanced view of reality does not have to be complicated. Take everything away, until you just have the things you absolutely need left.

Once you have that nailed the focusing factors and the compound metric formula, it is easy to do cool stuff with it (if you have access to the data).

You can look at the compound metric across competitors and smile (or cry)….

search performance index competitive indexing

I'm beating this to death, but how much more fun is it that you know why the SPI for each company is where it is? Rather than just know the number all by itself?

And even if you are ZQ Inc (643) you can see that you are at least good at one thing (ad creativity) and you can figure out how to use that strength – even as you fix the other focusing factors .

You can/should do this to understand your own performance over time. And you don't have to use my choices above. I want to stress the process and not prescribe the metrics you should use.

Internalize the process. Spend some time understanding your own unique business needs, get advice from experts out there and create an extra-special magical formula that you believe will deliver glorious insights. Remove the metric you don't need above, plonk in your new BFF.

The next time your Awesome Search Performance Agency is doing their quarterly review, ask them to open their presentation with this view of the SPI….

search performance index quarterly

Dear Leader is happy, she/he has the "one simple easy metric."

You are happy because you snuck in just enough information to make the index easier to understand.

Your Agency is delighted because you are asking for a much more sophisticated understanding of reality to show the value of your huge investment in the Agency and Search. Agencies want their feet held to the fire, they love accountability.

Win – win – win.

Summary

So often we report (puke) data, we diligently produce (puke) dashboards, we lovingly create (puke) slides. But it is rare that our efforts drive action. It's not because our puking was not great, it is simply because we are unable to succinctly bring forth the underlying focusing factors and present them in a way that is accretive to quick internalization of which of the focusing factors need killing and which need feeding.

Solve for that.

And now you know how.

Ok, it is your turn.

Do you agree that compound metrics are sub-optimal? Are there compound metrics your company uses that you've simply failed to kill? Or other compound metrics that are your favorite? How do you deal with the insights that delivery actionability challenge that mashing five metrics into one to produce a paste presents? Do you think showing the focusing factors beneath the "one simple easy metric" will work inside your company? How can I make the above approach even better?

Please share your ideas, critique, praise, improvements, examples in comments below.

Thank you.

Comments

  1. 1

    Hi Avinash,

    A nice post. I think we need to dig deeper into the user of the Compound Metrics and how much priority they are giving to this part of the business.

    I would like to take an analogy of the Capital Markets. Compound Metric in this case would be an Index (Just like Dow Jones/ Nasdaq/ etc.). Now whether I should just look at the final index value or I should look at the underlying stocks would depend on what my role is.

    – If I am an analyst with the ECM (Equity Capital Markets) then definitely I should be digging deeper. In fact I would have to not only dig into each stock, I might be required to dig deeper into the financial statements of each of the companies in that stock as well.
    – If I am the head of ECM, I would like to have atleast a birds eye view of all the companies in the the index
    – If I am the head of the Investment Bank, though I may want to, but the time will not allow me to dig deeper! This is just a part of my business and I have to play Golf as well! ;-) Now whatever may be the presentation of the components of the index, I might not have the time to look into the details.

    I would agree with your point that Indices in the Intern domain are not too useful right now. There are a few reasons – It is early times and not enough research has gone into designing the compound metrics and the weights given to each of the numbers. So you have your own index that you like, I have my own and my grandmom might have her own. There is no standardization and no science behind designing. When we look at the historical trend or compare with peers, it might not mean much as these might not remain stable. The social media domain is just shaping and world is in too much flux. But if the field becomes stable (big and important as well), then there would be people who would design the indices in such a manner that they do tell us a good story!

    In the meanwhile, I would focus on the end result (which is again a kind of index)! If your end goal is sales, focus on revenue generated. Obviously to get revenues, you would have to focus on 10 different parameters and metrics. But as an executive, I would focus on the end result and let the team run after the 10 parameters!

    • 2

      Paramdeep: Please see my reply to Anthony, he had something in a similar vein.

      And I completly agree with James' recommendation on the greatest compound metric of all time!

      :)

      Thanks for sharing your perspective, I do appreciate that so very much.

      Avinash.

      • 3

        @Avinash: Thanks for the comments.
        I think if a lot of such Multi – billion $ CEOs, start focusing (Given my assumption that this field becomes huge as traditional marketing channels) on Digital Media (specially Social Media), I would say that these compound Metrics (Indices) would be a lot more thought through and researched (As is the case with most financial services indices).

        As Indicated by James, Revenue/ Profit can be the simplest ones. But they are again compound metrics and depend on a lot of factors apart from the ones that are under your control.

        PS: I think my comment was stuck in the moderation review queue for quite some time and hence you might seen it later in your inbox! :-) But that is not an issue at all. Given the thoughtful comments, I anyways go through most!

  2. 4
    kidakaka says

    This is so true, I hate it even more when that metric is not transparent (like the Adwords Quality Score, or Google's PageRank). No one knows what is the exact formula, people talk about multiple things, and giving general 'buy low, sell high' strategies which make me want to pull my hair out (or even better their hair out!!).

  3. 7
    Anthony says

    Compound metrics are dangerous, and they do mask insights, and I agree with the suggestion to present them with complimentary information.

    However, I think the issue is a bit more complicated, and I think it really depends on if one is managing vs. advising. Often times the very reason such a metric exists is for advisers to obtain a pulse of the business (e.g., board members). These stakeholders aren't interested in every detail of the business, they just want to know high level performance of key areas (up / down / flat).

    Sure the compound metric is masking the details for a given area, and it would be impossible to improve the business solely looking at the compound metric, but that is not the advisers job. Their job is to gauge high level progress, and dig lower where / when needed, and perhaps when they start to dig lower, that is when one should pull out the details of the compound metric.

    • 8

      Anthony: I do agree with you that sometimes you just want to inform, you don't want to deliver any deeper understanding of the business and you don't want the other person to find any insight or identify any action.

      If that is the case, perhaps a compound metric is fine. I would argue that in that case even a email saying "things are fine in business area x" might be just fine.

      But the moment you want to move beyond inform, even to the CEO of say a $5 billion annual revenue company, the moment you want the CEO to understand anything, get the slightest clue about what's working well, the moment you want to get even a tiny amount of their attention/help at solving a problem…. compound metrics will kill you.

      Please do note that I'm not arguing with you that you have to summarize. This past friday I spent time with a CMO whose Marketing budget for the current year is $1.6 billion dollars! If ever there was someone who does not want detail, it is him. Yet, I would (and did!) give him the SPI. I did not just want to inform, I wanted understanding that lead to budgets being moved.

      I showed him just three summarized "stories" (Search Performance Index was one), each with focusing factors. In the future I would keep it to just three, but I might switch which story depending on where my expert analysis suggested I wanted attention/help/action.

      -Avinash.

  4. 9

    I like the standard approach. Not sure of the awesome approach for both social and search. When you present the compound metric at top and the 'focusing/contributing factors' just below, the recipient(s) of your report will try to correlate the focusing factors with compound metric. But since there is little to no correlation between focusing factors and the one number that you get from the computation of the compound metric, it is going to be very hard to prove that the 'focusing factors' are in fact the factors worth focusing on.

    For example it is possible and quite common that the conversation rate increases over time but the social engagement score still goes down or the conversation rate decreases over time but the social engagement score still goes up because of the innate issues with ratio metrics. For example: 5/1000 is 0.5%, 50/10000 is 0.5%, 210/4196 is 0.5%, 345/68848 is 0.5% , 392/78000 is also 0.5%. If we consider numerator as conversation rate then the conversation rate has increased by 7700% but since the focus is on the compound metric, there is virtually no improvement.

    As long as the compound metric is not a ratio metric, presenting 'focusing factors' just below the compound metric make sense otherwise it creates confusion and can create shadow on whole of the analysis. Does it make sense?

    • 10

      Himanshu: I'm afraid I don't know if I quite understand your comment, please accept my apologies.

      But in general… as the post says…. the focusing factors/contributing metrics, under the "awesome approach" for a compound metric should be complete and independent. I.E. They should explain key elements that go into a computing the metric and at the same time not have overlaps/related areas.

      I hope this helps reduce any confusion.

      Avinash.

  5. 11

    Such a wonderful, sound method. I often find higher-ups want simple figures to summarise performance and I love the approach of framing such numbers with the focus factors.

    Also, I'd be keen to learn about your search formula; paid search would benefit from a coherent scoring system to help identify strong advertisers in a given market.

    • 12

      Damion: Thanks for the feedback Damion!

      I promise to share the SPI formulas once I get a little more testing/validation under the belt, at the moment it is very much a work in progress even if I think I've got the core of it set up properly.

      Avinash.

  6. 14

    I've never known what to think about these composite metrics, Avinash, but I have steered clear of them because I've never identified one as a KPI as telling me everything I want to know.

    Your "decrapification" example should make it very clear for all that you sacrifice insights and perspective when you skip thinking critically about what you want to learn from your marketing efforts.

    Great post!

  7. 15

    Hi! Great post!

    Really keen to learn the search formula. Pls share!

    Thanks and keep up the good work :)

  8. 16
    John Y. Nielsen says

    You can categorize metrics into simple or compound types.

    By "simple," I mean singular and direct. These metrics stand alone; they are not combined with other metrics and are for measuring individual attributes of the entity of interest. An example would be the time required for a particular loan processor to process a bank loan. The entity (processing a bank loan) is measured individually (without regard to other loan processors) for the attribute "time." When averaged over time, you could see the performance standard for that loan processor, and his or her variance from that standard. '

    The metric is direct: that is, not derived from other metrics nor evaluated relative to the behavior of other loan processors.

  9. 17
    tjamesjones says

    Best compound metric is profit!

    • 18

      James: Ha, ha!

      Well said, and I hope that every single person that is trying to "summarize the business in one simple metric," first answers the question why they can't simply use profit as the metric. Social Profit. Search Profit. TV Profit. Just. Show. Profit!

      Wait. That is actually hard. Ok, forget it. I'm off to create a compound metric. :)

      Avinash.

  10. 19

    Hi Avinash,

    I discovered your blog a few weeks ago. I usually never write or comment any article because 98% of what you stumble upon the Web is just a content filler.

    But your no-nonsense tone and the amazingly valuable insights and clear analysis you just give at every blogpost just impresses me. It is just so good I even comment to compliment you. I even have the feeling I was doing everything wrong before.

    Thank you for inspiring me at each post.

    Victor

  11. 20

    You won't believe it Avinash, I'm working on something with exactly the same situation right now any my challenge is to 'simplify'. Your post comes at a great time, did you target it?

    Whatever, I loved the post and indeed now it's my turn. Thank you!

  12. 21

    Hi Avinash, another great post. Sad thing is, all of the decision makers I know go for the pretty dashboards. How do we change this? Julio

    • 22

      Julio: I agree with you, it is sad. But the good news is that we can give them something they want, pretty and one one page, along with what they need, smart insights.

      One of my strategies I've already explained in this post – sweat your understanding of the focusing factors (the ones that are most material) and segment the data to ensure they deliver a deeper understanding (pretty and in one page).

      The other one is to take 30% of the page to write data identified challenges, opportunities in English on that pretty one page. By that I don't mean say: "In the first graph you can see the blue bar is 18% YOY compared to the red bar for metric/segment x." The executives are not idiots, they can see that. We have to figure out why the blue bar is 18%, what are the underlying factors, then write in English what the challenge/opportunity is. Ask for action.

      Most of the time the Analyst does not get to the "pretty one page" first. It is sent out by the machine or a data puker, sorry I meant report writer. Then we should expect zero understanding or action – because of the missing 30%. The part in English where some adds the analysis of the data processed via their smart brain.

      -Avinash.
      PS: Bit more here: Five Rules for High Impact Web Analytics Dashboards

  13. 23

    Nice post, Avinash.

    Similar discussion to what many of us, you included, were talking about in 2007 regarding the Engagement metric and how we needed to deconstruct metrics rather than make them more complicated.

    Worth bringing up again as more people are working in social media.

  14. 25

    Great post Avinash, I like the simplicity of the Search Performance Index, viewed in conjunction with key business drivers, Satisfaction, Leads & Revenue it would certainly assist me to drive buy-in and improve efficiency with our agency search team.

    Like others, I would also be keen to understand your SPI formula, please don’t make us wait toooo long

  15. 26

    You are one of a handful of people who freely offer actual help … you literally show folks how to do something. This is rare.

    Much praise!

  16. 27

    I'm always happy when you take apart some common practices in reporting. SEO tools suffer from this as much as Social. Seems like every tool has it's own user, page, keyword or domain scoring & you can rarely deconstruct them.

    Thanks

  17. 28

    I disagree – I think compound metrics are very useful. I've working on inflation and pricing indexing for the UK government and the compounds are necessary and highly accurate.

    The fault is with the analyst not adding comparison and the 'why/what next' explanation section.

    The point of a compound metric is to take many, many metrics and give an overview. But if the analyst is using them incorrectly – like mixing unrelated metrics so it means nothing – or not adding the why behind the data then the fault lies on the analyst.

    For example, I like your table showing each of the months. Add a 'what do we do now' comment box and that's great data for the business leaders.

  18. 29

    Avinash,

    Great post. Really useful. Thanks for sharing.

    Interested in knowing how the search score is computed…you asked for more work :-)

  19. 30

    Great post.

    When working with multiple (possibly conflicting) metrics, how would you simplify to allow A/B testing, which require working with a well-defined, single KPI?

    • 31

      Alex: By an amazing coincidence I've just replied to a comment by Brendan on this precise topic. Please check it out: http://goo.gl/EpZXx

      In a nutshell, focus on the metric that helps you measure the job that the page is trying to do. The most important job.

      Avinash.

  20. 32
    Brent Sitterly says

    Interesting conversation.

    On the one hand I absolutely agree that compound metrics can be misleading. On the other hand stand alone metrics can be just as misleading. Thematically I agree that the important thing to do is to identify the "most" important factors and present this information in a constructive way. The one thing I think compound metrics are better suited for than stand alone metrics(based on some psychological biases, see some of Daniel Kahneman's work) is uncovering the relationship between metrics, an oft talked about but rarely practiced analysis task: understanding the dynamics of the system you're analyzing.

    For instance, Conversion Rate, its a compound metric by any measure. However it is considered, generally, more "valuable" than either visits or conversions. This undoubtedly is due to the assumed importance of the relationship between these other two metrics.

    What is important is that its easy to abuse compound metrics and easy to misunderstand compound metrics. So when they are used there should be as much transparency as possible.

    One thing that does bother me about using stand alone metrics is the loss of complexity, Having too much complexity is bad, but having not enough is bad too. Time on site is a good example of that. Just measure something like time on site is problematic b/c low time on sight is beneficial(you get the user what they want efficiently) and high time on sight is good(more exposure to your brand).

    What did old William of Occam say, "Plurality must never be posited without necessity", that without necessity piece is what I believe differentiates complexity is bad from too much complexity is bad.

    Cheers

    • 33

      Brent: I don't consider conversion rate to be a compound metric. The definition is: Outcomes/Visits. Or orders/visits. I personally prefer outcomes/visitors. As you can see, two clear variables, on compounding with other metrics.

      You are right about holding one single metric to be the holy god. That is sub optimal.

      In Web Analytics: An Hour a Day, my first book, I'd strongly encouraged folks not to present any metric without a BFF. For example never present just Average Order Value, also show Revenue because it is entirely possible to have AOV keep going up while overall revenue keeps going down!

      I can find a BFF for any metric, and we should always use that concept.

      Thank you for reminding us of this important nuance.

      Avinash.

  21. 34

    I read this a few days ago (enjoyed) and just opened up a slide deck from an agency touting a "simplified" compound metric! LOLZ, indeed.

    It's one I haven't come across, but it's basically compound of a bunch of intent-to-purchase micro-conversions. It's called something like "road to purchase," meaning that the KPI shows what % of traffic micro-converts and is on the road to purchase. It doesn't seem outrageous, but my problem is that I'm tasked with running site experiments, and those compound metrics are too squishy to test against.

    Welcome your thoughts, now or in future, on compound metrics + testing :-)

    • 35

      Brendan: I'm not sure I completely understand what they are doing. But I they have a way to infer who is closer to a purchase decision, that is perhaps worth considering.

      Intent is always hard to pinpoint.

      For A/B tests… the page or the offer or piece of creative etc has a singular job to do. Try to pinpoint that. What is the job that you want the thing to be tested to do? Then use the metric that helps you understand which version of the page/offer/creative does that job better.

      It is always possible that the page/offer/creative does other jobs too. That is great. But when it comes to testing, hold the variations you are testing to do that one singular job.

      Avinash.

  22. 36
    Yehuda Raizner says

    Hello Avinah,

    As usual a great post.

    If I define this my way, compound metrics are just going the opposite way of measuring your behavior properly.
    Instead of going down from the top (segment), compound metrics moving up from bottom (aggregate).

    Is it the right definition?

  23. 37

    Again, fantastic post!

    Compound Metrics are similar to many statistics, being that they're often 'bent' so to speak.

    I thoroughly enjoyed reading this, and many others, because of your ability to identify real life problems, offer creative solutions (which are at the same time realistic), and make us smile through your humor.

    Thanks again Avinash!

  24. 38

    It is very difficult to get the right mix of a compounded metrics and we all know a lot of "scores" that are flying around claiming to be the best indicator. Worst part is that these metrics actually lead to miscalculation as an agency need to fit in the numbers rather than projecting the right numbers for the betterment.

    But in the end people love to see a complex number which they don't understand because it gives them a feel of happening which in tern provides the oxygen to all these scores.

  25. 39

    Another great post! Avniash, I love your posts. You provide an honest and refreshing view. Thank you!

  26. 40
    Khushboo Garg says

    Hi Avinash

    I am a new follower of your blog and I absolutely loved this post.

    You have illustrated a great intuitive concept that I believe can applied to any sort of Performance metrics in a business environment.

    I do have a question for you – Should the "contributing factors" be unique to what the organization cares about or should we look at what contributing factors have been defined as the standard by the industry? I ask this in order to make sure I am making an apples to apples comparison between myself and a competitor for example.

    Thanks for a simple and amazing post!!

    Khushboo

    • 41

      Khushboo: Let's unpack a couple different things.

      The compound metric should be unique to what the organization, or their management, really cares about. The contributing/focusing factors should be unique to the compound metric. The contributing/focusing factors should describe the compound metric as completely as possible (so you account for most of the important variables in play) and, this is very important, they should be independent of each other.

      In some cases you'll create a compound metric as a Industry Analyst or a Tool Provider or a Government Agency etc. In all of those scenarios the above guidance applies, and to the extent possible we should make sure the contributing/focusing factors are 1. measurable across the measured 2. in a standardized manner.

      -Avinash.

  27. 42

    Thank you Avinash,

    I think that compounded metrics are only good if they are validated. They have no meaning unless you prove that they are predicting other results.

  28. 43

    I'm divided!

    You tell the truth about the danger and poorness of reducing complexity to simple things: Reading wikipedia entry about Othello may help understand the big picture but you'll miss the essence a mix of deepness, feelings and pleasure. It's the kind of the same for complicated metrics.

    But sadly I am also slightly disappointed because when mentioning Search, I was expected at least something associated to SEO and no, you just talked about Adwords ;-(

    Like Star Trek, please, don't let us into darkness!!! Do you think Search future is into PPC?

    • 44

      Antoinet: I agree with you, the suggested compound metric is only applicable to Paid Search. But I do hope that the approach outlined can be quite easily used to define one for Organic Search, or even for all of search.

      And to answer your question…. the future of Search is Organic + Paid, anyone who thinks otherwise is making a career limiting choice.

      Avinash.

  29. 45

    This post is the reason I read every single word you publish. Great job.

  30. 46
    Luke Middendorf says

    Great information, thank you for sharing.

  31. 47

    Hi Avinash,

    This is my first visit to your blog. It's very informative and nicely presented.

    However, I couldn't find index of your blog posts. Please direct me to it. Else creating an index of blog posts grouped by topic would be helpful for readers.

    Thanks,
    Sudeep M Jamkar

    • 48

      Sudeep: Hi. Please see the knowledge tab in the top nav. It contains a categorized list of all my posts.

      It is fairly updated, though some of the latest posts might be missing. You'll find them in the Latest Posts in the bottom nav.

      Avinash.

  32. 49

    Hi Avinash,

    Thank you for the nice post.

    I think that the compound metrics has its purpose. According to the math, compound metrics is same as dimensionality reduction. We reduce the dimensions of the dependent features to either improve performance or improve on visualization of the output metrics.

    Such reduction works best when the correlation between the input and the output is retained. That is "the variance is retained 95%".

    Best regards,

    Alex

    • 50

      Alex: On paper you are right, it really is as simply as that.

      The challenge in real life is that compound metrics are rarely constructed with sufficient amount of knowledge/intelligence, even when they are they magnificently hide the truth (first part of my post) and hinder the ability to communicate insights.

      That is simply the reality. Hence my macro recommendation that they should be avoided at all costs. Just show the key metrics.

      But if we have to, we should follow your advice and ensure we identify key dimensions. My humble addition would be to ensure completeness and independence of said dimensions.

      Thanks!

      Avinash.

      • 51

        Hi Avinash,

        Thank you for responding to my comment. I think you hit the exact point.

        I agree, the problem is not that the method of rolling up multiple metrics into a single compound metric is bad, the problem is that often more analysis/knowledge should be put upfront. There should be a purpose for doing that.

        One useful goal I can imagine may be comparison, ranking. We need this single simplification for purpose of data visualization. For instance Google PageRank is a pretty useful metrics to sort the pages based on a compound value of rank which in turn is calculated based on multiple dependent metrics.

        My humble note would be that every instrument has a purpose. At times we need a simple metric to present certain data, at other times we would like to analyze the data in more detail and, perhaps precision.

        best regards,

        Alex

  33. 52

    To quote Damion " I often find higher-ups want simple figures to summarise performance and I love the approach of framing such numbers with the focus factors. "

    We run into this all the time. Our clients want hard, clear numbers. We end up spending a lot of time explaining the meaning behind said numbers – especially when they don't match a client's expectations.

    However by moving beyond the data and explaining other factors, we are able to paint a better picture of what the information means.

  34. 53
    niroshan says

    Hi Avinash,

    I agree with the social media metrics with their actions. But in most of the cases visitors don't engage with the activity and they buy it offline(higher percentage in my country). End of the day social media effort increase sales by increasing branding. Do we have a method to measure brand awareness through social media?

    For an example I do a local ice cream business and promote through the social media. But visitors don't engage directly but they are going to the shop and purchase ice cream. Finally my effort directly links with the brand awareness.

    Regards,
    Niroshan

    • 54

      Niroshansam: Two quick thoughts… it is possible often to move from hypothesis to using data to validate opinions.

      Depending on how you want to define "brand awareness," it is easy to a little hard to measure that. There are a ton of tools that provide a ton of social measurement today. So data is there, your challenge is to define what you are trying to measure.

      Measuring offline impact of online is a little hard, but totally possible. Here's a list of my recommendations: http://goo.gl/QwOhW

      Speaking of this… I've just written a mini-blog post on "Social Surprises" and the effectiveness of social as an acquisition channel. Check it out here: ! http://goo.gl/jtCZN The two analysis done there are most excellent.

      -Avinash.

  35. 55

    I disagree on one point: compound metrics are useful insofar as spectators can apply them to decision making.

    Credit score: useful to lenders, not useful/actionable to the consumer getting scored.

    Social influence compound metrics: useful to rank influencers, not useful/actionable to the person being ranked.

    PageRank: useful to Google, not useful to the websites being ranked

    (Football) QB Rating: useful to teams, not useful to the QBs

    The trouble is where designers of metrics confuse these two use cases and design compound metrics for those who can take action on the metrics, not just because of the metrics.

    • 56

      Steven: Thank you. I agree with the separation of use cases. And in some of them perhaps it is a quick yea/nay for the "spectators."

      But I dare say that except for the most quickie of cases, these metrics are still not useful.

      Take Social Influence as an example. You are a 87 and I'm a 80. A spectator know knows you are more "influential" But if the spectator does not understand how the score came to be, they might still not know what to do.

      We can see how it applies to the other cases you mention as well.

      I want to reiterate, that I do agree with you on the two use cases and there are cases where the spectator might not care about how the score was arrived at as they decide action. A perfect example of this is your mention of Credit Score. The car dealer, bank, might not care, they just use. :)

      Avinash.

  36. 57

    Avinash: another brilliant article – thank you.

    "Always question authority." — Even if it means questioning the "authority" of the ones who are dispensing the "authority" (and influence factors)?

  37. 58
    Aaron Reddall says

    Hi Avinash,

    High appreciations for the information and the efforts you put in to get the best insights into analytics.

    Your articles are always a welcome!

Trackbacks

  1. […] Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics (Occam's Razor by Avinash Kaushik) […]

  2. […]
    Avinash’s Occam’s Razor blog posts are always educational and insightful. But for many of us tiny businesses, the discussion gets into complexities that we rarely think about. When it comes to compound metrics for social media, most of us are not yet looking at any social media metrics. In fact, when I look at our customers, the majority don’t work social media at all. But, even within the post Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics, valuable lessons are reinforced.
    […]

  3. […] Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics, http://www.kaushik.net […]

  4. […]
    Avinash Kaushik posts “Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics” at Occam’s Razor.
    […]

  5. […]
    Avinash Kaushik https://www.kaushik.net/avinash/analytics-tips-improve-search-social-compound-metrics/
    Main Takeaway: The areas that need more attention in a social strategy (or any strategy) can’t be gleaned from one compound performance metric. Show the company leaders a breakdown of the factors that contribute to the compound metric – force them to see the areas that need work by introducing a benchmark or target.
    […]

  6. […]
    Most analysts are SO SMART and have amazing ideas, but they can’t convey their genius ideas to others. This is the biggest problem in analytics today. One of my favorite bloggers who can articulate his ideas clearly is Avinash Kaushik. The only problem? His ideas are so awesome his posts are a mile long, but I promise they are worth the time.
    […]

  7. […] Decrapify search, social compound metrics […]

  8. […]
    I’ve just read an enlightening (and well-written) blog about compound metrics. You know, those “simple” products of data amalgamation that demonstrate… nothing. Metrics like “Visit Quality” and “Engagement Rate”. What are they? They are attempts to make sense of our complex digital world for our bosses, the decision makers. The problem is, these metrics are so over-simplified they provide no information on which to base any decision. I’ve borrowed the above image from Avinash Kaushik. I hope you’ll visit his blog, which addresses this issue in detail, and more importantly, provides a win-win-win solution.
    […]

  9. […]
    There is a good reasons why statisticians hate “compound metrics” or worse, “grades”. These metrics, based on multiple variables, hide all the details and make any interpretation hazardous, if not impossible.
    […]

  10. […]
    The whole problem with “drill down” is that BY DEFINITION you have to be looking at compound metrics in order to drill down. I am in full agreement with Avinash Kaushik’s idea that compound metrics need decrapification as he discusses in Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics. Yes, it sounds like seeing something amiss and being able to drill down to more detail is a great idea. In practice, you just typically start out with crappy compound metrics that don’t make you want to drill to anywhere. Avinash says it well in his conclusion
    […]

  11. […]
    Excellent Analytics Tip #25: Decrapify Search, Social Compound Metrics by Avinash Kaushik This may be old news to some of you, but compound metrics suck. And yet, people still use them without thinking about what they mean. (A compound metric, in case you were wondering, is a measurement that combines multiple criteria to yield a single number, like PageRank.)
    […]

Add your Perspective

*