Yes, data is important. Helps make marketing better. Makes for smart organizations. Blah, blah, blah.
You know the drill: Measure. Find insights. Take action. (Or die trying.) Ascend to corporate heaven.
While there is a great deal of appreciation for the power of metrics/data, I've come to realize that Sr. Leaders don't quite appreciate the deep, and often corrosive, consequences of choosing metric x over metric y as a key performance indicator (KPI).
[Sidebar] A key performance indicator is a metric that helps you understand actual performance against preset business objectives. [/Sidebar]
The metric you choose communicates to your organization what's important to you (the POWERFUL person). It communicates to them how their personal success will be measured. That translates directly into what they prioritize when it comes to your digital initiatives.
Choose the right metric and they'll create the most glorious digital experience in the universe, the perfect acquisition campaign, the most amazing customer service channel. And they will shock you with the profits they deliver.
Choose the wrong one and they'll create self-serving, sub optimal, non-competitive, tear-inducing outcomes that will, slowly over time, bleed the business to death.
It really is that stark. Simply because it all comes down to the incentives you create.
Don't believe me?
Let's look at six corrosive metrics and their angelic twins, which illustrate this challenge – and magnificent opportunity – quite vividly.
1. Page Views vs. Visitor Loyalty
Is there anything easier than measuring Page Views? This metric has been in every tool since we started torturing web server logs to measure hits (!).
What does Page Views measure? It kinda sorta measures consumption. It is hard to know if a lot of Page Views per visit is a good thing ("The visitor loved our site so much that they read 23 pages of content!") or a bad thing ("Our site is so horrible that it took 23 pages for the visitor to find what they were looking for") or a horrible thing ("After 23 page hunt the visitor gave up, cursed us, abandoned the site, and went on to tweet to 23,000 followers that we stink").
When you look at 23 Page Views, how do you know which of the above three was the outcome?
But it gets worse.
Most content sites are currently monetized using display advertising, most commonly on a Cost Per Thousand Impressions (CPM) basis. When you are paid on a CPM basis the incentive is to figure out how to show the most possible ads on every page ("mo ads mo impressions!") and…. ensure the visitor sees the most possible pages on the site ("mo ads mo impressions mo page views mo money!").
That incentive removes a focus from the important entity, your customer, and places it on the secondary entity, your advertiser.
It does not take a degree in rocket science to see what happens next. The web is littered with examples of this awfulness.
Here's one simple example.
Photo slideshows are a great way to engage and delight customers. Yahoo! News has them. Except that they neither engage nor delight. Monetization on content websites, including likely Yahoo!, usually is on a Page View-driven CPM-incentivized mechanism. The way this model manifests itself is that every time you click on the Next Photo button (arrow thingy) they load a new page. The new page has the next photo and lots of new ad impressions. Even on a pretty fast connection that means waiting, often for seconds. Every photo should deliver delight. Instead, every time you click on the Next Photo button, all you remember is the pain of waiting. [I'm ignoring the fact that in this day and age the photos themselves are tiny.]
Would it cause you to think positively of Yahoo! News? Or Business Insider? Or Forbes? Or all these other sites that impose a Page View-driven CPM-incentivized experience on you? More importantly: Would such a poor experience cause you to go back to these sites?
In that single session Yahoo! News made some of its Page Views quota and some of its CPM earnings. But it failed from a macro perspective. Short term gain; long term loss.
Now consider photo slideshows on (my beloved) news site, the BBC.
The BBC photo slideshows don't deliver small doses of pain every time you click the next button. Instead, they deliver small moments of joy.
In that single session the BBC created fewer Page Views for itself, smaller CPM earnings. But it created joy and delight from a wonderful user experience. That directly translates to me using the words "my beloved" every single time I talk about the BBC website, visiting the site a lot more often (5x a day at least), consuming a lot more content, and in the long run actually seeing (and clicking on) a lot more ads. Short term loss; long term gain.
The metric the BBC is focused on is not Page Views, it is Visitor Loyalty.
Visitor Loyalty is not in every single report in your Digital Analytics tool. But it is there. It is a standard metric. And it measures not what happens inside a session (short-term incentive), but rather behavior across sessions (long-term incentive). It forces the designers, editors, merchandisers, IT team, and everyone in between to trade tawdry sensational stories delivered via slow-loading, pain-inducing pages, for a focus on customer (not advertiser) delight.
Ironically, that actually means more ad impressions in the long run. It means becoming big.
Take a look around you. Most content sites, be they thesun.co.uk, xinhuanet.cn or nydailynews.com, have home pages that are (and I'm being kind here) link pukes. On average these sites have 500 links on their home page. Why?
If the web analytics dashboard prominently measured Visitor Loyalty, would they still create link pukes?
Would they not think: "Even my mom hates our site, how can I earn her love, the thing that has eluded me all my life?" Would they then not focus on relevance and not generic link puking? Would they not buy simple behavior targeting solutions to use past behavior to customize some of the experience to deliver delight?
Would they not buy a solution like JumpTime to, in real time (!), look at the FloPower of every link and economic value it is delivering (still in real time!) to go from 500 to just 200 links? Would they not obsess about speed because both mom and dad despise waiting?
I believe the answer to every single one of those questions is yes. Yes, they would.
All from anointing the right metric, Visitor Loyalty, as your KPI. It forces a focus on the long term and on the right entity (the customer and not the advertiser).
Friends don't let friends measure Page Views. Ever.
2. Revenue vs. Economic Value
Ecommerce/lead gen type websites are typically incessantly focused on one-night stands. "Hello, so nice to see you, now take off your clothes and jump into bed with me!"
Of course they don't say that exactly. But the "buy now, buy now, buy now, buy now" design and merchandising on their websites makes that amply clear. Just try visiting orbitz.com or macys.com or petsmart.com. Sometimes this one-night stand obsession is subtle, sometimes it is obvious in what is presented to you when you land, but it always becomes more transparent as you go deeper into the site.
That is a reflection of a deep obsession on Revenue. It is reflected in the obsession with Conversion Rate. Every web analytics tool in the market measures single-session conversion rate, so if visitor, your potential customer, does not convert in that single session (i.e, refuses the one-night stand), the visit is marked as a failure!
Guess what that encourages? An ever-harder obsession about getting better at scoring more one-night stands.
Most people don't want one-night stands. I know, I know, you are super cute and awesome. Still.
Most people want to visit your site, do some research, go away, visit other sites, come back to yours, get more confidence about your brand, go back to Google and compare reviews/prices, come back to your site and add the product/service to the cart, go and ask their spouse/boss for permission, come back and buy from you (or the other site).
That was 7 dates.
When your KPI is revenue, you are focused on trying to get as many single-session conversions as possible. You make bigger Buy Now buttons. You pimp product specs (ugh!). You do sub optimal things. You ignore delivering what's expected on the first six dates.
Sure, some people will have a one-night stand with you. But most won't. Then how you do grow your business? How do you move beyond the standard conversion rate of less than 2%?
Shift to caring about Economic Value.
So when someone visits your site and signs up to receive email, and does not buy anything, that is not a failure. That is a micro-conversion because that first date will lead to a second, a third and a seventh (if you play your cards right!).
When someone comes to your site and watches a video, that is a micro-conversion.
When someone clicks on the product reviews tab, that is a micro-conversion.
When someone clicks on the "Send Page View Email" link (to get permission from wife/spouse), that is a micro-conversion.
Etc., etc., etc.
Every micro-conversion creates economic value for your business. It engages in the awareness, consideration, comparison, purchase slow dance. It delivers higher macro-conversions (revenue!) over multiple visits by the same person by incentivizing you to behave optimally, in sync with your customers and at their speed. It gently encourages everyone in your company to obsess about the micro-conversions by saying they are of business value, to create better designs, more prominent placement of content/images/stuff customers want.
Over the long term it shifts your company from the corrosive single-session, conversion obsession (for that is what Google Analytics, SiteCatalyst, WebTrends measure) to a pan-session, way-beyond-a-one-night-stand experience that delivers higher Economic Value.
Rather than just focusing on 2% success, and 98% failure, you are now focused on 100% success!
Do please note that I'm not saying don't worry about Revenue. As you saw above, the definition of Economic Value includes Revenue. I just want you to obsess about macro plus micro as THE way of being massively profitable. And as in the first case above, by delivering delight.
Pick Economic Value, your parents will be proud of you.
3. Time on Site vs. Task Completion Rate
Over time (ironic, right?) I've developed distaste for the time on site metric.
1. Can't measure time spent on the site if you only see one page, and
2. Can't measure the time spent on the last page of the visit
These sad realities make that metric even more suspect. Maybe suspect is too strong a word. The above two make it very difficult to infer exactly what the performance is reflecting.
Is 7 mins time on site awesome? And should we assume that the visitor spent zero seconds on the last page, or 28 minutes? What is the implication?
[Bonus] How are Time on Page and Time on Site calculated? [/Bonus]
It is not completely valueless. But it is not worthy of being crowned a KPI.
So, what are we actually trying to measure when we use Time on Site?
We are trying to infer whether the visitor had a great experience ("Wow, they spent 92 mins on the site! Man we rock!"). We are trying to infer if they consumed enough of our content (to make them happy and make us money). We are trying to figure out where they had problems ("What? The avg time on site is only 2 mins? Golly we suck!"). We are trying to figure out if our latest redesign was a success ("Look, time on site moved from 3 mins to 900, awesome!"). We are trying to…
This is the operative word: Trying.
The reality is that there is a vacuum there. We are not (yet) sitting inside the brain of the visitor. So we take our biases (also called experience :)), our opinions, our psychological issues, and all that and try to fill that vacuum.
We have no idea who Kim Watkins is and what her 6.3802146 time on site means. So we say: "Look, the average is 2 and Kim spent 6.3802146 mins so that was an 'engaged' visit." Hurray.
Why infer? Why be so arrogant as to believe that our biases, sorry experience, will interpret Kim's visit accurately?
Why not just ask Kim?
Two simple questions. The first gives primary purpose. The second is a yes or a no.
Kim will let us know she was there to buy a pair of Manolo Blahnik pumps. And no, she was not able to complete her task after 6.3802146 frustrating minutes because neither your navigation nor your internal site search engine got her to the right page.
And no, it was not a very "engaging" experience.
When you choose time on site as your KPI you are encouraging your organization to apply inference, and make changes that are, at best, wild guesses with a 1/100,000 chance of fixing the core problem.
When you choose task completion rate as your KPI you are encouraging your organization to put their ear directly next to the horse’s mouth, listen, feel the breath, then go fix the problems the horse has identified.
You'll agree that only one of these methods improves business profitability, results in customer-centric experiences and reduced losses from failed expeditions to chase mirages identified as issues.
And no, Ms. Watkins is not a horse. She is fine young woman. :)
Don't infer. Ask.
4. % of Search Traffic vs. Share of Global Search Volume
This one is more subtle. It is a matter of which lens you want to look at your performance.
% of Search Traffic: This measures the percentage of traffic you receive from search engines, in context of all other traffic sources.
How do you get it? You log into Baidu Tongji (or Yahoo! Analytics) and create a little pie of your Search, Campaign, Direct, Referral and Other traffic sources. That shows you that 45% of your traffic is from Search. [Given how people use the web to seek information, at least for now, around 50% seem to be about the optimal number.]
You feel proud because you started with just 5% of the traffic from search engines. You've worked on a robust search engine optimization and pay per click programs to steadily grow your search traffic. Bonuses have been distributed.
This is a cause worth celebrating and, unlike other metrics in this blog post, given the deep importance of search this metric can be promoted to a KPI. It will incentivize the right behavior. Working ever harder on understanding your content, CMS and business strategy to do ever better SEO and PPC. It will drive the % of Search Traffic graph to go up and to the right (bigger piece of the pie). That 45% is now 500,000 visits a month from search! It is pretty good.
The problem is that we can often get stuck just looking at our own data, and in doing so we miss a chance to understand the real opportunity. We might completely miss the boat even as we celebrate what looks like huge success (moving from 5% to 45%).
You received 500,000 visits from Google.com. There were 209 million searches in your category (say pets) on Google.com originating from the US.
So Share of US Search Volume = 500,000/209,000,000
Gives you a different perspective right?
Some questions are simple. "OMG we have such a tiny share of the visits, what do we need to grab an ever bigger share?" Sure, not all 209 million will end up on your site, but you define the pets category! You have to get more than that tiny number of referrals. This will have huge implications on your paid search strategy, your valuation of clicks you get from Google and Bing. You might have to go out and hire new people, get a new agency, experiment with the long tail, buy some behavior targeting solutions, so much more. Sure we went from 5,000 to 500,000, but that will simply not do. The opportunity is too large and too relevant to ignore.
Other questions will be much harder. "OMG we spend mmm millions on TV, Radio and Magazines trying to create demand by interrupting people. For the most part we don't even know if they care about us, our products or our ecosystem. And here are millions of people behind 209 million queries a month who are raising their hand to say they want our products and services, they are interested in our ecosystem! We are spending ttt thousands on search. Should we rethink the balance between 'interrupting to possibly create demand' and 'welcoming with open arms people who want to hear from us'?"
This is a very, very hard discussion to have. Egos, politics, years of doing the same things, opinions, and genuinely believing that the current path is the best one … all come into play.
But if you want to be an agile, nimble competitor, it is a discussion you have to have. Even if in the end the TV budget stays 5,261% higher than digital. The debate is important. Making deliberate choices is important (even if you make the wrong choice). Because deliberate choices can be revisited. Data can be analyzed. Course changes can be plotted.
If you never deliberate, you slowly silently reach the point of no return and file bankruptcy protection.
Perhaps you'll get lucky and that won't happen to your company.
But changing the lens through which you view success can ensure that you watch the right thing, you debate and deliberate, you choose to slowly experiment, you shift budget. Step one? You use a metric like Share of Global Search Volume to incentivize the people in your company to look at the right thing and then power the right discussion.
Like everyone else, I love TV. I'm not advocating that the TV budget above should be 0%. But it is profoundly sub optimal to have this mismatch: Let's spend all our money on a channel where we, at best, kinda sorta feel users with the right intent are and let's ignore the one where 100% of the users with the right intent exist (and are looking for us!). That is a unsustainable life threatening strategy for everyone. Unsurprisingly it results in a weakening of your brand and profits. Yes, even for you.
Go change your lens.
5. # of Followers (or Fans or +1s) vs. Conversation Rate
One of my most retweeted quotes about social media goes like this: "Social media is like teen sex. Everyone wants to do it. No one actually knows how. When finally done, there is surprise it’s not better."
That probably says it all.
And how do we compound the problem? As major brands we proceed to measure one of the most useless measures of success: The number of Likes we get on Facebook.
Or the number Fans or Followers or +1s on Twitter, Google+, RenRen, Vkontakte and other lovely social channels.
When your digital dashboard measures Likes/Followers/+1s, what are you incentivizing your Agencies to do?
Use every legitimate and illegitimate technique out there to beg/cajole/lead/mislead people into pressing that button. Very little thought given to what happens after the button press (no incentive!).
What is the medium or long term strategy to engage with the audience? Where is the plan to ensure your social contributions score higher on Facebook's EdgeRank algorithm? Where is the structure that will ensure you build out a real credible asset for your company?
You have a lot of Likes, but you never get to creating a robust Earned media channel for your company. [An optimal inbound marketing portfolio will have balanced Owned, Paid and Earned channels.]
To seekers of Likes and Followers, social media "strategy"ends up being something lame, like sweepstakes, polls and pimping your latest press release. That barely works in the real world. Why would it work in an ADD environment like social media?
So how do we incentivize the right behavior? Look beyond the +1s, Followers and Likes, and leverage social channels to build out a community of like-type and like-sized :) people around you, a community that converses, shares, amplifies and, over the long term delivers economic value to the company. Leverage what the channel is really, really good at, close one too many connections based on conversations and value.
I've defined four metrics (Best Social Media Metrics) that incentivize the right behavior.
I've defined Conversation Rate as: # of Audience Comments Per Social Contribution
You can compute it for every social channel on the planet.
With TV you don't know who your audience is or if they are interested in you or what they care about., In social channels, you know all of those things. You can use that knowledge to participate in and initiate conversations. You can build a better connection (social equity? :)) and you can deliver value (by sharing valuable tips, answering questions, linking to good deeds by your competitors, creating special unique content, etc., etc.).
Conversation Rate incentivizes you, or your proxies (agencies), to really understand what social contribution is causing your audience to add their voice, to have a conversation with you. That will help you optimize your contributions, force you to understand your audience, and deliver value to your audience and your company.
Get zero replies per post/tweet/status update?
Your million Likers/Followers are telling you something. Stop. Reboot.
As your agency/company moves away from a Likes quest, you'll be astonished at the incentive Conversation Rate provides your employees. That in turn, slowly but surely over time, create a credible Earned media channel for your company.
So do the right thing. Converse. Don't shout. Don't pimp. Don't sweepstake.
I was advising a stealth mobile application company (hello future one billion Facebook dollars!) and this example comes from that experience.
If you've ever created a mobile app you know that from version 0.1 all the oxygen in the room is taken up in trying to figure out how to get your first 100,000 installs, how to score the Editor's Pick etc.
That is understandable. There are fifty million apps in iTunes and Play.
So naturally, # of Installs becomes the KPI that goes on top of the dashboard.
The problem with # of Installs is that it does not provide deeper insights about the value of the app to the users. It does not say anything about what the engineers got right or wrong. There is nothing in # of Installs that drives an obsessive understanding of the customer, the app experience/value, product development and all those other more valuable strategic parameters.
My advice to the team was: "Let's keep # of Installs as a metric we track, but let's make 30 Day Actives as our key performance indicator – the thing we really, really focus on."
There are so many amazing incentives from a focus on 30 Day Actives.
First, the company deemphasizes short term win — installs — and emphasizes the long term win — retention.
Second, employees care a little less about hundreds of new installs and start to care about 50% of people who uninstall the app in the first 24 hours.
Third, the company comes together to focus on the customer in every facet of their execution.
What promises are our sales/marketing programs making? What does the post-install process look like?" "Is the app instrumented to collect the right usage data? What is the optimal number of ads in the app that causes fewer 30 Day Actives? When people cancel, what does that experience look like? How do we go about releasing updates to ensure higher retention? Do we need a loyalty program? What can do to empower our customers to spread their stories about us?
And so much more.
When the focus is on the # of installs it is not hard to imagine that there is no overt incentive to consider the above questions, or to assign a high priority to getting those answers.
Use 30 Day Actives as your KPI. Build a stronger profitable business.
It is important to point out that I'm not advocating that you stop measuring page views, revenue, time on site, % of search traffic, # of Likes, or # of installs. They are all fine metrics. You'll most likely use them as diagnostic measures when you analyze the metrics I do recommend you shift to.
I'm advocating that you not make them KPIs, don't crown them God, don't allow your employees to solve just for the primitive six. Because none of these six metrics incentivize optimal behavior or business outcomes.
You become what you measure, so why not solve for what actually matters?
Let me close with a quote on incentives, from the inimitable Steve Jobs…
"Incentive structures work. So you have to be very careful of what you incent people to do, because various incentive structures create all sorts of consequences that you can't anticipate. Everybody at Pixar is incented to build the company: whether they're working on the film; whether they're working on a potential direct-to-video product; whether they're working on a CD-ROM. Whatever their combination of creative and technical talent may be, we want them incented to make the whole company successful."
No one could have framed it better than Steve.
Incentive structures are not a web analytics problem. They are an organization design problem. But in choosing the optimal metrics to crown as heroes we can use data to incentivize the right behavior, value creation for a company, and deliver happiness to customers.
As always it's your turn now.
Do you use the primitive six as KPIs in your company? Have they incentivized you, your peers, to solve for optimal business and customer outcomes? Do you have other suggestions for primitive metrics? How about suggestions for metrics that incentivize optimal focus? Got a favorite "OMG I'll die if we can just measure metric x"?
Please share your feedback, suggestions, critique, huzzahs via comments below.