Occam’s Razor by Avinash Kaushik

Often we present data without thinking about it too much.

We might actively think about the metrics we are computing (and avoid rookie analysis mistakes).

But it is rare that we, “Web Analysts”, actually think, I mean think, about the story we are telling.

I think that’s because that is not our job (I mean that in all seriousness).

Our job is to report data. On good days it is to understand and segment and morph and present analysis.

But we don’t think about the implications of the data in a grander context and we don’t think about the role we can play in connecting with the Business, the Marketers and be so bold as to try and change behavior of decision makers. Change company cultures.

This blog post is a short story about my small attempt at changing the culture and setting a higher bar for everyone. Using data.

The Use Case:

The data in question was survey data. This one was specifically about a day long conference / training / marketing event for current and prospective customers.

On a five point scale for each Presenter the Attendees were asked to rate “how satisfied were you with the presentation and content“.

Quite straightforward.

Here are the results:

But you can also imagine getting this kind of data from your free website survey, like 4Q from iPerceptions ["Based on today's visit, how would you rate your site experience overall?"].

Or if you use free page level surveys from Get Satisfaction or Kampyle ["Please share your ratings for this page."]

In those cases you would analyze performance of content or the website.

The Data Analysis:

On surface this is not that difficult a problem to analyze.

Here is a common path I have seen people take in reporting this data, JAI! Just Average It! :)

The actual formula is to take the average of the last three columns (satisfied through extremely satisfied).

This is ok I suppose.

I find people have a hard time with smaller numbers and then you throw in the decimals and you might as well call it quits.

Your boss, Bruce, eyeballs this and says: “Looks like everyone performed well today, let’s uncork the champagne.”

Meh!

Those a bit more experienced amongst you know this and what we might see from you is not averaging but rather a more traditional Satisfaction computation.

The formula is to add the three ratings (satisfied through extremely satisfied) and divide that by the total number of responses. For Jonny: (6+12+0)/18

A bit better from a communication stand point.

6.0 is a number hanging in the air naked, without context, and hence is hard to truly “get”.

100% on the other hand has some context (100 is max!) and so a simple minded highly paid executive can “get” it. Jonny, Chris and Apple did spectacularly well. Will and Brian get a hug, Guy was great (come on, 89% is not bad!!).

The Problem.

Well two really. One minor and one major.

The minor problem is (as you saw in Guy’s case immediately above) percentages have a certain nasty habit of making some things look better than they are. From a perception perspective.

My hypothesis is that in general human beings think anything over 75% is great.

So maybe we should not use percentages.

My major problem is that this kind of analysis:

1. rewards meeting expectations
2. does not penalize mediocrity

Both are a disservice in terms of trying to make the business great. I have to admit they are signs of business as usual, let’s get our paycheck attitude.

Think of mediocrity. Why in the name of all that is holy and pure should we let anyone off the hook for earning a dissatisfied rating? So sub optimal!!

Consider “meeting expectations”. I was upset that our company was not shooting higher. Accepting a rating of Satisfied essentially translates to: “as long as we don’t suck, let’s accept that as success”.

What a low bar.

I believe that every business should try to be great. Every interaction should aim to create delight. It won’t always be the case, but its what we should shoot for.

And its what we should measure and reward.

Why?

Because our way of life should be to create “brand evangelists” through customer interactions that create delight.

You like us so much, because we worked so hard, because we set ourselves such a high bar, that you will go out and tell others. Be our Brand Evangelist.

Why?

So we don’t have to do that.

The Solution.

Now it is very important to point out that worrying about all of the above was not in my job description. As the Manager of a small team of Analysts (or as an Analyst) I am supposed to supply what’s asked for (sure with some analysis).

But I made two major changes to the calculation, and one minor.

1. Partly inspired by the Net Promoter concept I decided to discard the Satisfied rating.

   
   When we spend money Marketing (/Sales / Teaching / Advocating) I am aiming for delight.

2. Decided to penalize us for any negative ratings (even if slightly negative).
3. Index the results for optimal communication impact.

I call the new metric: Brand Evangelists Index. (Ok so its a bit wordy.) BEI.

The actual formula applied was:

{ [ (Very Sat + Ext Sat) - (Not Sat + Not At All Sat) ] / # Responses } *100

The Results.

Here’s what the success measurement looked like:

The result was a radically different understanding of quality and impact of each Presenter.

Not obvious?

Check out all three measures next to each other:

Superyummylicious!

You can see how the Brand Evangelists Index separates the wheat from the chaff so well.

Compare Jonny’s scores for example. Pretty solid before, now a bit less stellar.

In fact Will who initially scored worse then Jonny is now 11 points (!!) higher than Jonny.

That’s because the BEI rewards Will’s ability to give a “delight” experience to a lot more people (as should be the case).

Compare the unique case of Guy Berryman.

In other computations Guy was dead last but there was not much difference between him and say Will and Brian. Just a few points.

But the Brand Evangelists Index shows that Guy was not just a little bad, he was badly bad.

Sure he got a couple bad ratings but Guy failed miserably at creating delight.

He failed at creating Brand Evangelists.

And if we invest money, in these times or in good times, we demand more. Guy can’t do, or has to do a lot better.

Note that Apple could also use some mentoring and evolution.

The Outcome.

Initial a few people said what the freak! Some stones were thrown.

But I took the concept of the Brand Evangelists Index two levels higher and presented it to the VP and the CMO.

They adored it.

The reasons were that the Brand Evangelists Index

1. demanded higher return on investment
2. it set a higher bar for performance and
3. it was truly customer centric

The BEI became the standard way of scoring performance in the company.

[In case it inspires you: That year I received the annual Marketer of the Year award (for the above work and other things like that). Imagine that. An Analyst getting the highest Marketer award!]

The Punch Line.

When you present data think of not just the data you are presenting but what are you measuring really and how you can lift up your company.

You have the data. You have immense power.

Now your turn.

What do you think of the Brand Evangelists Index? How would you have done it better? Got your own heroic stories to share? I would love to know how you used data to alter a company’s culture.

Thanks.

PS:
Couple other related posts you might find interesting:

• Make Your Analysis/Reports “Connectable”
• Seven Steps to Creating a Data Driven Decision Making Culture
• Lack Management Support or Buy-in? Embarrass Them

An Analysis Ninja, let’s call him Philip Walford, asked a delightful question. Philip wanted to know if the impact of a faith based initiative in his company, product demo videos, could actually be measured using data.

Hurray!

Faith is good. Data is better. : )

[And before you flame me: know that I love my religion more than you love yours. Wait. That did not come out right. Let me rephrase that.]

In this thanksgiving week 2008 post I’ll share Philip’s question about how to identify value of video product demos on an ecommerce site, and my answer about involving customers.

Here’s Philip. . . .

We are a large retailer with a lot of product on our site. In the past we have invested lots of dollars and time producing things like demo videos for our products, or adding other features and tools to our website to provide more information about a product. Our goal is to inspire customer confidence in their purchase (by giving them as much information is possible).

The question is, what are the KPIs of things like a demo video.

My recommendation was to measure conversion rate for the segment that views the video. If conversion is higher then the videos are bringing value. Others in my company have presented the hypothesis only customers that are a lot more invested in buying the product are likely to click on the video link and hence “pre qualified”, hence that segment would have had a higher conversion rate regardless.

I understand their perspective but I feel they are reading too much into the situation but I don’t know how to argue this point. There are several directions we could go with this but I wanted to see if you could share some guidance on this issue.

My answer to Philip. . . .

This is a complex problem, more than might be apparent on the surface.

It is also an example where it can be easy to jump into bed with your web analytics tool to get satisfaction but you wake up in the morning feeling. . . . well. . . . less than satisfied.

But before we go there I have to give a ton of credit to Philip and his crew for being skeptical of reading too much into their own opinions or biases.

I firmly believe that people who work for a company rarely (never!) represent customers. They are too close to the company and too different.

Just because I work for Microsoft and use a Zune (yes I do!) does not mean I can be a effective customer representative of Microsoft Zune customers. Company employee opinions rarely reflect those of customers. Do please be aware of that.

So when looking to make decisions, look for data (quant or qual).

I’ll present Philip with three solutions / options as he battles the challenge of figuring out if the investment of muchos dineros in creating product videos is worth it (besides the fact that these videos ooze sexiness!).

1) Use ClickTracks (Compute Contextual Influence)

There are two challenges with using clickstream data and the “typical” measure of conversion rate to determine success.

A] You might be looking at a “biased” segment (as challengers to Philip’s recommendation mentioned). I.E. Only the highly motivated people.

B] By comparing all people who converted and viewed the video with those that converted and did not see the video you are not comparing fair segments. You are also lumping all other “convince our visitors to buy” tools into one large bucket. Tools like Comparison Charts and Product Screenshots and Product information and Customer Reviews and more.

It is quite possible that those other tools might be getting people to convert at a much higher rate and by dumping them all together you are not being fair.

And of course you’ll get a wrong read on conversion impact of the videos.

So even if you use your web analytics tools (your Google Analytics or Omniture or WebTrends or CoreMetrics or whatever) try to compute “contextual influence” (value of each feature in context of the others).

It is actually very hard (damn near impossible) to do this in all those tools (even for the Paid solutions, even after you plunk down half a million dollars for the mandatory Data Warehouse “add on”).

ClickTracks is the only tool I know of that can do this out of the box, using its terribly named “funnel report”. No data warehouse. No extra tags or variables or sprops or wt_&*#$. In fact not even much IT, I just need admin access to my tool (not site, web analytics tool).

Its easy to use. Create a hierarchy of your website. Add individual or groups of pages into each stage (notice I did not say step because you can jump steps here). Add an outcome (in my case say “Thanks for placing your order” page). Click Calculate.

Boom!

[You are not supposed to be able to read the analysis, sorry, privacy dictates that.]

What I want you to note is two things.

This is a site where each stage means a view of the site (and like a traditional funnel how many people get in, get out, move on etc).

Secondly note that each box (which represents a page/’s or a tool – videos, comparisons, reviews etc) has a different stage of blue.

What this lovely report does for you is compute “the influence”of each of those pages/tools in driving the ultimate outcome – purchase here. The darker the blue the more “influential” that piece of content. [Influence is defined by the existence of that piece of content in the visitor session, regardless of what path the visitor took, regardless of when the content was seen.]

Ain’t that super sweet?

The analysis you see above is for a real ecommerce website. What it proved to us, delightfully, was that the product videos, we had created at a cost of over one hundred thousand dollars, yellow star above, was the least influential tool we had on our site.

The most influential, sexy pink star above, was a tool that had cost us $8 to produce – it was a page that compared different versions of the product (information that was handily available in the company).

We used actual customer behavior. We analyzed contextual segments. Ultimately it allowed us to put our precious few resources in the right area.

Of course it is quite likely that everyone who came to the site and did not buy (convert) might have loved the videos and rushed to stores to buy our products (one HiPPO actually said that!). There is no way to prove that using just the web analytics data.

What we did is proved impact on online buyers.

As to the HiPPO. . . . read on. . . .

2) Use Surveys (Actively Collect VOC)

When in doubt (or confronted by a HiPPO, remember don’t run) what better way to go then gather some Voice of Customer. Dare I say the voice of god? :”)

Two things I have tried (of many!) that work a lot of the times. Each covers one unique bucket of visitors to your website.

A] Consider sending a simple post purchase email survey to customers who have purchased on your site and ask them for the key influencers of their purchase.

You could share with them the various tools you have on your site (product information, comparison tools, images, videos, customer reviews etc etc) and simply ask them to rank order them in order of importance.

Don’t ask them to tell you how much they like them, or choose ones they like, they tend to pick all. :) Just ask them to rank order. Or use a tactic similar to that.

This tells you want works for those who buy.

For the 98% that will never convert on your website. . . .

B] Consider a onsite survey like 4Q (though 4Q can only be customized so much so perhaps you want to use either your own or one of the big daddy paid survey tools).

This will go to a small random sample of people who are on your site (who may or may not buy). You’ll ask them three or four questions about why they were there (primary purpose) and then what tools/features of your website they liked (rank ordered if at all your survey company can do that).

That will give you what you want.

Since this can also be thought of as a page level problem, you can also use something passive, a page level survey / poll, like Kampyle on your product pages and ask people to quickly rate the various features. There is a Site Content feedback topic in Kampyle which you can customize.

Now you have the most important piece of data you need, your customer’s. Few website owners / marketers / hippo’s can argue with this. Leverage this advantage.

Finally one last option for you. . . . hopefully one you’ll use before you write a chq for a hundred grand to create your videos. . . .

3) Use… wait for it….. Testing! (Measure Actual Customer Behavior)

I am sure this does not surprise you. Run a A/B or Multivariate Test and let your customers help inform you of the value of these features.

For 30% or 40% or whatever %, don’t show the product demo videos and for the rest show the product demo videos and see the impact on the data. Boom (!) you have your answer, without any biased opinions.

It is certainly going to take you a small amount of effort, get the Website Optimizer, talk to your IT folks, create version of the page with no product tour link etc.

But you are making a very expensive decision for your company are you not?

And here is the additional benefit of testing. You are free to use any kind of “conversion”.

You can measure success as conversions (submit order).

You can measure success (of the test) as number of people abandoning from the product page.

You can measure success as the time people spend on the product page. [There is a very cool javascript code that does this with the Google Website Optimizer, it is especially helpful for rich media / flash sites. Without a doubt other vendors can do this as well, just ask.]

You can measure success through your survey tool if it is integrated (this is some extra work sadly, but for big bets I recommend it).

You can integrate your analytics tool with your testing tool (say Google Analytics with Website Optimizer) and use other metrics to measure success such as bounce rate or electric shocks etc :).

[For GA and GWO ROI has integration instructions .]

The bottomline is that you can define success and then let the customers tell you.

That’s my answer to Philip.

Sounds exciting?

Am I the only one who thinks when you do this kind of analysis you are in a nearly orgasmic state?

Yes these methods are some small amount of work. But nothing in life worth having is easy. The tools might be free, but that does not eliminate your need to investing your time and effort! :)

And on the positive side with a recession looming people who involve customers in making decisions, rather than their opinions, will win big. The “guessers” will not win big. They might even win small. Or fail.

Plus if you do this you’ll be a Analysis Ninja, not a Reporting Squirrel.

Ok now your turn.

Have you tried to analyze the features like Video Demo’s on your website? Or perhaps other complex features you have launched? What works for you? What totally failed? In my recommendation to Philip, what did I overlook?

Please share your feedback, critique and hurray’s.