Occam’s Razor by Avinash Kaushik

There are lots of different tools, many of which measure the same thing, numbers don’t tie, there is confusion and a lack of trust and…. well I am getting ahead of the myself.

The story starts with a email asking for thoughts on a issue a lot of us face on a daily basis.

Why don’t I let the original author tell the story……..

Can I ask you a question? I’m fighting mightily to make HBX able to measure marketing channels—pay per click, organic, email, banner ads, and onsite merchandising. But individual channel managers simply do not trust the data.

I think that data overall is damn accurate—total conversions by product is within 10% of in house measurement. I have a sneaking suspicion that web analytics tools are NOT the best measurers of true conversion by channel.

The data never matches what they get from their pay per click tool, their email tool etc. They blame the web analytics tool, when I think it’s just that they are expecting something that can’t be done.

Have you had any experience with this phenomenon and do you have any advice on how to dive in and tackle to diagnose?

This is a tough problem. Very real, very tough.

[Before we get too deep I wanted to highlight that this is as much a data problem as it is a organization problem.

For now Web team is usually in a silo, then Acquisition tends to be in a different silo and further more it is not uncommon for different "Channels" to sit in their own homes that may or may not work / collaborate with each other. The corrosiveness that organization structure can cause is often underestimated.

Data gets blamed and worse any possible positive outcomes don't, well, come out. I wanted stress this important element.]

So what to do?

Perhaps the most important thing to realize in this case is that your web analytics tool is often not measuring the same thing as your offsite tools are (for example: ppc, banners, affiliates etc). Even success (conversion) is not measured the same way.

Examples of issues causing differences:

# 1 : Often many “off site” tools measure impressions (completely missing from web analytics tools) and clicks. For the latter if you code correctly then clicks come through to your web analytics tool but remember that there you are usually measuring “visits” and “visitors” and not clicks.

# 2 : A good example is conversion. A ppc / banner / affiliate cookie is usually persistent and will “claim” sales even after ten visits of the customer, including for scenarios where the customer might have came back repeatedly through different channels (first paid then organic then banner than paid again then affiliate then….. you get the idea).

Most web analytics tools will credit the first channel or the last one creating one more bone of contention with “off site” tools.

[As a bonus in this scenario, #2, your paid search vendor and your banner vendor and your affiliate vendor all "claim" credit for conversion even though it was just one customer that converted!!]

My approach at solving is really quite simple. Five easy to execute steps:

Step One: Understand what each tool is actually measuring and how it is doing it. This means really getting down and dirty with the data and the vendors and pushing them hard to explain to you exactly what they do. If you are not a smidgen tech savvy then take your friendly IT Guy with you.

Step Two: Document. Create a slide / email outlining all the reasons why the numbers don’t tie. Be intelligent, be creative. Here are two slides I had created from a long long time ago, a real blast from the past….

The slide above provided a summary of why the two data sources were providing numbers that were 30% off. As you can imagine took a lot of work.

[Click on image for higher resolution version!]

This slide explained one of of bullet items (1.II.b), how each tool dealt with the sessionization when it came to search engines (and with increasing dominance of search engines this turns out was a big problem)

Notice the “pretty picture”, it shows a very complex process with great clarity.

Step Three: Educate your users (in this case the data skeptics). Do dog and pony shows. Present to the senior management teams, or anyone who will touch the data. Leave them easy to refer to presents such as the slides above. Make sure that your audience is now smarter than you are because that will build trust (both in the data sources and, more importantly, in you).

Step Four: Start to report some high level trends between the tools, while doing your best to hide/remove the absolutes from any report / dashboard. Especially in comparisons one month won’t be good enough and it might even be distracting.

Often it is not uncommon for me to index the numbers them in some way and compare trends, rather than show the raw numbers . So xxx campaign / program went up in comparison to others by xx points (or xx percent) etc. To take the focus away from numbers.

It will take a while for them to get comfortable with the numbers, depending on the organization complexity, politics and how good a job you have done in Step Three.

Step Five: Pick your “poison”.

This is a key step. You need to wean your organization from relying on multiple tools for the same data. The goal is not to kill different tools (unless they are utterly redundant) but rather not to have the same Metric from different tools.

You need to pick the best tool for the best metric.

It is ok to let your Acquisition team use DoubleClick / Atlas / Whatever to report end to end on your banner / display / digital media campaigns (make sure they are following customers through conversion).

Ditto for your Pay Per Click (PPC) / Search Engine Marketing (SEM) team. If you are big then your agency probably has lots more data in its own possession than you could have, and they probably want to move faster than you could. Don’t compete with them.

Some web analytics tools do promise that you can bring in any data and merge it with your web analytics tool and it will work wonderfully. Sometimes it does, often it does not.

Just pick your poison, best cleanest possible source of data for each job and use it.

If you are the Web Analytics person should you just quit and go home? No.

Focus on data that you have from your web analytics tool that they (other tools / teams) don’t have.

For example: What is happening on the site after these skeptics deliver Visitors to your site? Use site overlay, bounce rate, content value, testing, all things they can’t do, don’t have access to but in the end help them understand what is going on on the site and how they, skeptics, can make more money from your data.

Most siloed tools (PPC, Banners, Affiliates… etc) don’t can’t actually do deep site analytics, so forget the tactical, they will do that better, focus on the strategic. Rather than become combative be “collaborative”.

You do that and you have your hook! No one will argue with you about differences in data.

In summary: While this is a tough problem that we all face, it is possible to bring about some sanity to the existence of multiple data sources. It is not possible for now to just have one tool for all your needs, but if you follow the above five step process then you’ll be able to bring out the best in each and benefit from it.

Ok now its your turn!

Please share your perspectives, critique, additions, subtractions, bouquets and brickbats via comments. Thank you.

[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]

This post has been in my head for atleast 18 months. Nothing like a flight from San Francisco to Washington DC to get things out of one’s head.

Measuring “engagement” seems to be this long standing quest on the web. There was so much we could measure and so little. As Marketers we have been frustrated with the near constant 2% conversion rates for our websites. We would like to have another metric that justifies our existence, and of course that of our website.

And that’s just when it comes to e-commerce websites.

The fervor for measuring engagement is even higher for non-ecommerce websites because there is little in terms of Outcomes to measure there.

So there has been a lot of proverbial ink used up in defining “engagement”. Pundits have pontificated. Bloggers have blogged. Guru’s have spoken from their perches. Industry Analysts have given their brains to the cause. Vendors have…. well tried. Hard.

Yet not much to show for all this collective effort.

Engagement is not a metric that anyone understands and even when used it rarely drives the action / improvement on the website.

Why?

Because it is not really a metric, it is an excuse.

An excuse for an unwillingness to sit down and identify why a site exists. An excuse for a unwillingness to identify real metrics that measure if your web presence is productive. An excuse for taking a short cut with clickstream data rather than apply a true Web Analytics 2.0 approach to measure success.

Does that sound a tad bit tough-lovish?

The desire to measure “engagement” with customers is a good one. But let’s try to understand why in the context of web analytics so many efforts at measuring “engagement” have yielded almost no results:

1. Each business is unique and each website is trying to accomplish something unique. Think of all the reasons a website exists, now imagine what engagement could be for each.
   

   
   Result: It is really hard to generalize, and often turns out to be a comparison of apples to monkeys to whales. That translates into a poor understand of what is being measured.

2. It is nearly impossible to define engagement in a standard way that can be applied across the board. Definitions that exist are either too broad (to cover every nuance) or too narrow (hence very unique).
   

   
   Result: Few people understand what you mean when you say “engagement”, and even fewer can then translate it to apply to their sites. Unlike clicks, visits, conversions, recency, ip addresses etc when you tell your management “engagement” it is hard to know what it is/means.

3. At the heart of it engagement tries to measure something deeply qualitative.
   Yet most efforts to measure it in our world tend to be hard core quantitative (translate that as: “we have clickstream, let’s take our interpretations of what could possibly be happening, now find clicks that can carry the burden of our personal impositions, voila! here’s engagement“).
   

   
   Result: That mismatch is ok for a couple months, but as you measure it over time you’ll discover that it does not indicate true customer intent and hence is doomed to have sub optimal impact.

4. One of my personal golden rules is that a metric should be instantly useful. This one is not. Say you measure engagement. It could be a % or a absolute number or a ratio or whatever (in fact it can be any or all of those at the same time). You fire off a graph or a excel spreadsheet with trends. You repeatedly get asked: What are we measuring?
   

   
   Result: Little action. It is not most important but we should always try to have metrics that are instantly useful, you look at ‘em and you know what it is and if going up is good or bad. It is rare to find a measure of true customer engagement for a website that does not required a partial PhD to understand what is being measured.

5. Most of all engagement is a proxy for measuring an outcome from a website. Conversion is not enough, as mentioned above, so we try something else. The problem that we’ll define engagement as a measure of some kind of outcome but we won’t give it the sexy name of engagement.
   

   
   Result: Confusion and delay (tip of the hat to Thomas The Tank Engine). If we are measuring page views divided by unique visitors as a proxy of engagement (more pages per visitor means more “engagement”) they why not call that metric page view per visitor? Atleast that will make it clear what you are measuring and then some smart person will question that it is not a very good definition!

In Summary: The reason engagement has not caught on like wild fire (except in white papers and analyst reports and pundit posts) is that it is a “heart” metric we are trying to measure with “head” data, and engagement is such a utterly unique feeling for each website that it will almost always have a unique definition for each and every website.

“So what you are saying is that we should not measure engagement.”

I am saying you should very very carefully consider the above points, then not take a short cut (or as the American’s say, a cop out) and actually define the metric as a Outcome metric (see element three of the trinity ).

Here is a process you can follow:

Simple easy to follow process that should help identify the critical metrics for your business and force your business leaders / stakeholders to help identify the real success metric that otherwise might have been hidden behind “engagement”. And now it will be actionable across your organization becuase people will understand exactly what it is.

To stimulate your thought process here are some metrics you can use to measure “customer engagement” (that visitors are engaging with your website):

• “Are you engaged with us?”
  (exact phrasing of a site level survey question - let your customers interpret it as they will, after all why is your interpretation better then theirs)
  

• Likelihood to recommend website
  (another site level survey question - would you recommend our website to your friends / family members / lovers :))
  

• Use primary market research
  (similar to the first one, but in this case use good old market research to get a feel for how engaging your website is - and measure it every three months to compute the trend)
  

• Customer retention over time
  (on a ecommerce or non-ecommerce website, do people come back and how often - here’s a helpful post on how to measure it)
  

• # of Visits per Unique Visits, Recency of Unique Visitors
  (recommended as a last resort - I am really not in favor of using quantitative metrics to measure qualitative outcomes - but you can use these to see if your website is “engaging” enough to pull people back and more frequently)

I am sure you’ll have other metrics that you can think of in the spirit of the ones above. The above list is to share with you how I think about it.

We all want to engage with our customers. But as analytics practitioner our goal is to use the right metric by working hard to get to the root cause (rather than making a excuse) and sharing that with clarity with our decision makers. Then and only then will it be actionable.

In Summary :

• When most people measure “engagement” they have not done due diligence to identify what success means for their online presence. In absence of that hard work they fall into measuring engagement, and then measure something that is hard to action or something that will rarely improve the bottomline. Avoid this at all costs.
  

• Think very carefully about what you are measuring if you do measure engagement. If engagement to you is repeat visitors by visitors then call it Visit Frequency, don’t call it engagement. Don’t sexify, simplify! :)

• If you want to measure “engagement” then think of new and more interesting ways to measure that (see list above). Engagement at its core a qualitative feeling. It really hard to measure via pure clickstream (web analytics data). Think different.

Ok now its your turn. Please share your perspectives, critique, additions and subtractions via comments. Thank you in advance.

[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]