Creating relevant engaging digital experiences is the quest for so many of us. It is a huge part of my job. I love creating experiences that delivers delight and happiness.
Measuring "engagement" seems to be an even longer quest for Marketers and Analysts. There was so much we could measure and so little. As Marketers we have been frustrated with the near constant 2% conversion rates for our websites. We would like to have another metric that justifies our existence, and of course that of our website.
And that's just when it comes to e-commerce websites.
The fervor for measuring engagement is even higher for non-ecommerce websites because there is little in terms of Outcomes to measure there.
So there has been a lot of proverbial ink used up in defining "engagement". Pundits have pontificated. Bloggers have blogged. Guru's have spoken from their perches. Industry Analysts have given their brains to the cause. Vendors have…. well tried. Hard.
Yet not much to show for all this collective effort.
Engagement, that phrase / name, is not a metric that anyone understands and even when used it rarely drives the action / improvement on the website.
Why?
Because it is not really a metric, it is an excuse.
Even as creating engaging experiences on the web is mandatory, the metric called Engagement is simply an excuse for an unwillingness to sit down and identify why a site exists. An excuse for a unwillingness to identify real metrics that measure if your web presence is productive. An excuse for taking a short cut with clickstream data rather than apply a true Web Analytics 2.0 approach to measure success.
Does that sound a tad bit tough-lovish?
The desire to measure "engagement" with customers is a good one. But let's try to understand why in the context of web analytics so many efforts at measuring "engagement" have yielded almost no results:
-
Each business is unique and each website is trying to accomplish something unique. Think of all the reasons a website exists, now imagine what engagement could be for each.
Result: It is really hard to generalize, and often turns out to be a comparison of apples to monkeys to whales. That translates into a poor understand of what is being measured. -
It is nearly impossible to define engagement in a standard way that can be applied across the board. Definitions that exist are either too broad (to cover every nuance) or too narrow (hence very unique).
Result: Few people understand what you mean when you say "engagement", and even fewer can then translate it to apply to their sites. Unlike clicks, visits, conversions, recency, ip addresses etc when you tell your management "engagement" it is hard to know what it is/means. -
At the heart of it engagement tries to measure something deeply qualitative.
Yet most efforts to measure it in our world tend to be hard core quantitative (translate that as: "we have clickstream, let's take our interpretations of what could possibly be happening, now find clicks that can carry the burden of our personal impositions, voila! here's engagement").
Result: That mismatch is ok for a couple months, but as you measure it over time you'll discover that it does not indicate true customer intent and hence is doomed to have sub optimal impact. -
One of my personal golden rules is that a metric should be instantly useful. This one is not. Say you measure engagement. It could be a % or a absolute number or a ratio or whatever (in fact it can be any or all of those at the same time). You fire off a graph or a excel spreadsheet with trends. You repeatedly get asked: What are we measuring?
Result: Little action. It is not most important but we should always try to have metrics that are instantly useful, you look at 'em and you know what it is and if going up is good or bad. It is rare to find a measure of true customer engagement for a website that does not required a partial PhD to understand what is being measured. -
Most of all engagement is a proxy for measuring an outcome from a website. Conversion is not enough, as mentioned above, so we try something else. The problem that we'll define engagement as a measure of some kind of outcome but we won't give it the sexy name of engagement.
Result: Confusion and delay (tip of the hat to Thomas The Tank Engine). If we are measuring page views divided by unique visitors as a proxy of engagement (more pages per visitor means more "engagement") they why not call that metric page view per visitor? Atleast that will make it clear what you are measuring and then some smart person will question that it is not a very good definition!
In Summary: The reason engagement has not caught on like wild fire (except in white papers and analyst reports and pundit posts) is that it is a "heart" metric we are trying to measure with "head" data, and engagement is such a utterly unique feeling for each website that it will almost always have a unique definition for each and every website.
"So what you are saying is that we should not measure engagement."
I am saying you should very very carefully consider the above points, then not take a short cut (or as the American's say, a cop out) and actually define the metric as a Outcome metric (see element three of the trinity ).
Here is a process you can follow:
Step Two: If you did a great job with it then the above statement contains the critical few metrics (three or less) that will identify exactly how you can measure if your website is successful at delivering against its purpose.
Step Three: If you have a ecommerce website then revenue or conversion is probably one of your critical few. But one of the critical few is what your senior management might call engagement. Work hard to define exactly what that metric is (see below for ideas).
Step Four: Don't call that metric engagement. Call it by its real name. Don't hide behind a pretty moniker.
Simple easy to follow process that should help identify the critical metrics for your business and force your business leaders / stakeholders to help identify the real success metric that otherwise might have been hidden behind "engagement". And now it will be actionable across your organization becuase people will understand exactly what it is.
To stimulate your thought process here are some metrics you can use to measure "customer engagement" (that visitors are engaging with your website):
- "Are you engaged with us?"
(exact phrasing of a site level survey question – let your customers interpret it as they will, after all why is your interpretation better then theirs)
- Likelihood to recommend website
(another site level survey question – would you recommend our website to your friends / family members / lovers :))
- Use primary market research
(similar to the first one, but in this case use good old market research to get a feel for how engaging your website is – and measure it every three months to compute the trend)
- Customer retention over time
(on a ecommerce or non-ecommerce website, do people come back and how often – here's a helpful post on how to measure it)
- # of Visits per Unique Visits, Recency of Unique Visitors
(recommended as a last resort – I am really not in favor of using quantitative metrics to measure qualitative outcomes – but you can use these to see if your website is "engaging" enough to pull people back and more frequently)
I am sure you'll have other metrics that you can think of in the spirit of the ones above. The above list is to share with you how I think about it.
We all want to engage with our customers. But as analytics practitioner our goal is to use the right metric by working hard to get to the root cause (rather than making a excuse) and sharing that with clarity with our decision makers. Then and only then will it be actionable.
In Summary :
- When most people measure "engagement" they have not done due diligence to identify what success means for their online presence. In absence of that hard work they fall into measuring engagement, and then measure something that is hard to action or something that will rarely improve the bottomline. Avoid this at all costs.
- Think very carefully about what you are measuring if you do measure engagement. If engagement to you is repeat visitors by visitors then call it Visit Frequency, don't call it engagement. Don't sexify, simplify! :)
- If you want to measure "engagement" then think of new and more interesting ways to measure that (see list above). Engagement at its core a qualitative feeling. It really hard to measure via pure clickstream (web analytics data). Think different.
Ok now its your turn. Please share your perspectives, critique, additions and subtractions via comments. Thank you in advance.
[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]
Via 
great article, thanks! i think it summarises perfectly well your methodology and approach to analytics.
as for me, engagement is not only difficult to measure, but difficult to just MENTION, because no such word as "egagement" exists in Spanish… life is miserable :)
Avinash,
Thanks for opening the can of worms :–)
I agree with some of your comments that engagement index does not replace your core KPI's. I think engagement can be an additional metrics to understand what is happening in the ecosystem. Engagement can be an aggregate metrics with weights assigned to certain activities. Again this will be unique for each organization.
Example, if you have an ecommerce site and have ratings and reviews, blog, a link to recommend the site to another user, a login into certain properties….etc. Engagement index can be measured by measuring each individual activities and calculating the sum of values. Engagement index will be based on weighted averages (you can assign different weights to each activity). This would be specific to certain initiatives and organizations. Trending is key for engagement index and in my mind, it should be measured on a weekly basis or monthly basis.
You might find certain co-relation with brand value and upward trending engagement index.
-Rahul
Avinash, I have had your Web Analytics 2.0 concept swimming around in my head for over a week now. With this post it dawned on me that you are applying engineering principles to WA, and that's a wonderful and powerful thing to do.
When I was in business school, there was a tendency for people to look for a magic formula for everything: you plug in a couple of numbers, perform some simple arithmetic and presto, out comes "the answer" without regard to whether the answer reflects reality or not. I see the same thing with the "engagement" discussion.
In contrast, the engineer gathers requirements (why do we even have a website? what are my visitors trying to accomplish?). Then they apply some math — maybe even *lots* of math — and the way it's applied is very particular to the situation. The loads, boundary conditions, materials, environment, etc. all matter in engineering in the same way that site type, design elements, marketing efforts, and target market make a difference in how you analyze a website.
And the biggest engineering principle I see you advocate is testing: engineers perform qualification tests in order to see if their mathematical models truly represent what happens in real life. You are suggesting that we actually ask our visitors if they are engaged… we can then use that answer to calibrate our metrics, to see if they reflect what's happening in real life.
We all have noticed how a poorly performing campaign suddenly become "brand awareness". Could it be the same with "engagement"? When everything else fails, let's measure engagement! :)
Rahul : Fair enough.
My recommendation is to apply the WWJD test (also known in my book as the "three layers of the So What" test).
At the end of the index creation process ask:
1) Is this easy to understand and is it "instantly useful"? It does not have to be but it sure helps.
2) If the trend moves up or down is it easy to know what happened, or even where to start looking for the problem?
3) If it is a engagement index then how much customer voice is in the index and how much of it is your own (tortured) interpretation of the customer voice? Does not have to be all VOC but some, or a bunch, would sure help.
If it passes the above three questions with a B+ (not even a A+ :)) then you have yourself a actionable index. If not then it is just you and I playing with data.
Thanks for the nice "brain poke", I like thinking. :)
-Avinash.
Avinash,
Thanks for a good read — again:-).
I have always treated 'Engagement' not as a KPI, but as a 'SI' (Service Indicator). The analogy I can think of is your car. You car has a few SIs (some of the newer ones has one too many), for example 'Electrical' or 'Maintenance'. So when the light comes on, you know something needs fixing but you don't what it is until you take it to the mechanic.
Similarly, 'Engagement' to me is a composite score made up of quite a few underlying metrics/scores both quantitative and from qualititative inputs including conversions and/or views/visit, and/or likely to recommend etc. Based on your business and what your site is trying to do, one could come up with what the underlying components should be and what the cut-off engagement should be before the 'light comes on' (like you said one cannot generalize the definition for Engagement across Industries and focus areas).
Once the 'light comes on', you have to look at the underlying components to figure out what is cause for that dip.
I also agree with you that don't just re-label a metric and call it engagement for buzzword sake. "Confusion and Delay" is what you will get as you said -being a Thomas watcher -courtesy my 4yr old :-) -, I had to mention it. However, not sure I would call engagement an excuse :-)
-Ned
File – print as pdf – email boss@acompany .com – message: “read this to realize what is wrong with us” – sit back and enjoy some peace!
I have been reading your blog for a long time and I can always count on it to provide real world actionable information. You say do this or don’t do that, and you also provide specifics and real alternatives. It was great to read why engagement is a excuse but it was even more helpful get your tips on the process to follow and what metrics to consider. I think that’s what makes your blog unique.
Thanks Avinash!
well, that'll put the cat among the pigeons! :)
what I really like about your posts, Avinash, is even when you really dislike something (and I've read/heard you take issue with "engagement" on many occasions over the last year) you still provide a positive or at least alternative view/solution that people can use/think about, rather than simply have a rant, which unfortunately is the case on many blogs.
I work in the public sector and have had several conversations with colleagues and management about how to measure engagement, and believe me it is an extremely nebulous concept when dealing with predominantly content sites.
Your point that engagement is more of a qualitative concept than quantitative is great, I've had to torture data in mindbending ways in an attempt to show some measure of engagement, but never with really satisfactory results. Thanks for providing a different perspective (again)!
Jon
Avinash & Rahul –
We've been working with a similar "Site Performance Index" number that comes from weighting behaviors we've designated as positive. (To avoid counting lift in a negative behavior as "a good thing.")
We probably pass your test with a B to B-, Avinash, as we have yet to incorporate our survey data. Where we have had some success is using this one number – with a scale visual – to communicate with a broader group and our leadership.
It's my job to know the details – find out why we're moving up/down, but my leadership team can use that one summary number in the corner of the scorecard for a quick "gut check". The rest of the details in the scorecard give them more.
That said, I still find it incredibly frustrating when I ask for objectives and hear, "We just really want this experience to engage the visitor." As opposed to those other experiences that we want to drive people away from the site?
Great post!
In my experience most companies find it very hard to start acting on web analytics data. As a result a lot of data is just used for a "gut check", as Kirsten put it.
The problem with gut checks is that they hardly ever lead to action. Nothing is ever at stake: if the gut check turns out good, there is no need to act. If the gut check turns out bad, it is just a gut check.
It has been a long time marketing tradition to use measurement for excuses. Terms like brand awareness, brand value or brand engagement are much older than the web analytics industry. These 'KPI's' are still being used to prove effectiveness, when there is no better way to prove it.
To me, your post, Avinash, makes this point clear again. Web Analytics practitioners have to break the marketing traditions and make managers aware that they need to act on data. A good start would be to accompany every gut check with a plan of approach on how to improve results.
I disagree with many of your points and think this post is misleading. Yes engagement is not easy to define, but used correctly I have saved my clients millions in the past using engagement metrics. I have answered some of your points here.
http://blackbeak.conversionchronicles.com/2007/10/02/finally-i-disagree-with-avinash/
Outstanding! Thanks again for a brilliant post.
Avinash, my friend and fellow guru panel member at the upcoming eMetrics Summit, I believe you and I substantially agree on this issue. What you refer to as “head and heart” I have referred to as “physical and emotional”:
http://blog.jimnovo.com/2007/08/02/webtrends-score/
Much of the engagement focus so far as been on the activity and the metrics talked about simply more complex versions of the activity metrics we have, e.g. Views per visit. It all seems like an awful lot of effort to generate what might be interesting but not highly actionable data. The advice you give to the people focusing on these more complex activity-based ideas is right on – forget calling them “engagement”, call them what they are, especially since they will tend to be very site-specific in nature and application.
Where I disagree, or perhaps I misunderstand or need to clarify, is on the idea that you can’t behaviorally measure the “heart” or “emotion” vector. In offline database marketing, the heart / emotional idea of “engagement” has always been called “likelihood” – it’s a predictive model. I’m not sure why online folks have to rename everything, but it sure does create a lot of thrashing around, doesn’t it?
In other words, measuring the likelihood of an action indicates emotional attachment to the action; clearly, if someone is unlikely to repeat an action, they can’t really be engaged, can they? Does a person who last posted on a blog 6 months ago have the same level of engagement as a person who posted yesterday? It’s the exact same activity, what differs is the person’s likelihood to repeat the posting activity. Or their engagement with the activity, if that’s what you want to call it.
I would argue that a visitor can create a ton of complex activity on a web site, be a “node”, have lots of friends etc. but if the visitor has not been to that web site for 6 months, they simply are not “engaged” by any stretch of the imagination.
You can derive these likelihoods simply, reliably and universally for any action you are talking about using time as the base: time since event, time between events, or duration for some specialized applications.
I, for one, would be very happy to stop talking about "engagement" and go back to "likelihood" but I think this discussion opens up a window that is extremely important to the WA community – reporting on the future as opposed to always reporting on the past.
Those interested in going more deeply into this topic should read my Measuring Engagement series here:
http://blog.jimnovo.com/2007/04/25/measuring-engagement/
P.S. I suspect the huge base of installed web analytics apps that are not capable of measuring Recency and Latency is partially responsible for the tendency for people to focus on the activity side of this issue. After all, it’s a lot easier to measure “stuff that happens” than “stuff that does not happen” and people have to work with the tools they have. That does not mean it’s the correct approach, it just is what it is; many are still waiting for the right tools.
I've definitely seen "excuse" engagement metrics, but I think there is a place for using "squishy" metrics as proxies for other measurements that are more difficult/costly to make. The caveat is that you need to validate your squishy metrics to make sure they mean something useful.
For example, I have one client that does exit surveys on their support site, which gives a lot of hard and soft data about visit "success" and "satisfaction" (two nice squishy measures like engagement). We have had found some very useful insights working backward from survey responses to web analytics data (what happened during "unsuccessful" or "frustrating" visits). We can define metrics based on those visits that can be applied to the entire visitor population, not just the small fraction that responded to the survey (with full awareness of sample bias issues). And then segment the heck out of those folks to figure out who they are :).
There is no single metric can define engagement, but this is vital for all the sites, especially the non-commerce ones.
Here are some tips to value your engagement:
If your ranking on unique visitor is 100 in ComScore, and if other metrics like visitor/visits ratio, visitor/page view ratio and time spent rank lower than your unique visitor ranking, then you can conclude that you are not competitive in engaging your users. Which means users are likely visit other sites than visiting yours.
Another metric you should look at is your search with zero results from your analytics tool. This tells you that you do not have content user look for. If this number is higher than 1%, then you have problem engaging your user.
Today's users have higher expectations on a destination site. If your site search engine fail to bring the right results,then you are not engaging your users.
Engaging user the first step of providing good user experience. For a content site, content is still the king, but user experience is the king maker.
I hope this shed some lights as why I think the assertion of Engagement is an excuse can dangerously misled your readers, especially the ones are new to the field.
Jim : It is always a honor to have your perspective (though I wish it were more than once in six months!! :)). You have more experience, in multiple channels, then the rest of us combined and I truly appreciate the wisdom.
I accept and agree that onsite behavior based measurement falls in the "likelihood" based family of future looking metrics. Especially as we use various surveying instruments.
For reasons similar to ones you outline in your comment I am also all for actually asking people if your website experience is "engaging" to them. By asking I mean using primary market research. Why add, subtract, multiply, and divide if you don't have to! :)
I really appreciated the feedback Jim, thank you.
Peter He : I am afraid I am going to have to politely disagree with you.
ComScore is a extremely poor indicator of "engagement", using any of the metrics you suggest. There could be dramatic differences in site purpose and experience that render the four metrics you mention useless at guessing if the experience is optimal.
Here is a not smart example: The best designed site will have the least time on site, is that bad "engagement"? I am sure there are other better examples.
% of zero search results I put under: "cost of doing business metric". Having results come up on internal search is the price of doing business, having results show up does not in any way guarantee that it will be / was / is an engaging experience.
I agree with your thought that engaging (translate as not bouncing) the visitor is step one and that user experience is king. But the suggested metrics do a sub optimal job of reflecting if the user experience was positive or negative.
I am sorry for disagreeing.
Everyone : If you have not had a chance I highly recommend reading Ron Shevlin's blog post in reply to this one. You'll find it at:
Ron disagrees with me a little bit, but does so in the best way possible: by providing specific examples and extending the conversation with a cogent set of thoughts (and a wonderful 2 x 2 matrix to boot!).
I'll go out on a limb (and Ron can bat me down) and say that in the end we are more close in our perspectives than apart. I appreciated Ron's six example dimensions and taking the conversation to online and offline implications.
Please read his excellent post, and I thank Ron tremendously for adding his valuable perspective to the conversation.
(I feel privileged to learn from 16 absolutely wonderful perspectives, thank you all!)
-Avinash.
Avinash,
Again, a very well written post. However, I disagree with you on this one. My comments are posted on my blog http://webanalysis.blogspot.com/2007/10/engagement-is-it-metric-or-excuse.html.
– Anil
Bravo Avinash great post,
This has been on my mind as well. Though I think a company can define metrics such as Satisfaction, Engagement, Interaction etc it is very site/company specific, though. I'm definitely in Jim's camp on this
I have had customers talk about measuring engagement and I was even blown away when an ESPN analyst mentioned it. So I would say there is something called Engagement but it is a very complex artifact that has many drivers as Jim is pointing out.
I honestly think if you want to start looking at engagement you need to start doing Structural Equation Modeling because this type of analysis starts looking at customer Customer Perception vs Actuals = Reality, but to be honest if someone isn't doing Multivariant analysis they aren't ready for SEM.
See you in a few weeks.
all the best
-raf
President Web Analytics Association, SAS Institute
I like Ron Shevlin's post on engagement, as it does cover the extent to which this metric is important well outside the traditional realms of web analytics.
I think the conversation around Engagement has become convoluted for the very reasons you site Avinash, and the responses to this post confirms that! Lots of disagreement and different opinion.
My opinion (just to add to the confusion) is that there are two distinct types of engagement – behavioral and emotional. And you have to be clear what one you are talking about.
Behavioral engagement is your immersion in a task environment – the website/product/service allows me to achieve a goal. It allows me to achieve this fast, easily and efficiently. It's a means to an end – my goal. "Engagement", as we traditionally think about it, is perhaps a bad word for this. As there is no real emotional connection (maybe a residual one, but not strong).
Emotional engagement is generating a 'connection' with users/visitors based more on content and vibe than tasks. This falls in the realm of branding. It's a tough one to define, and a metric that is only really accessible via surveying. In a nutshell, it's the feeling you have when your experience meets or exceeds your emotional expectations. These are different from your goal or task oriented expectations. Sometimes they aren't even that important, sometimes they are. As you point out Avinash, it all depends on what your goal is!
The only problem I have with Ron's post is that, apart from referrals, all the other metrics assume some type of emotional engagement from task behavior. Which is not really correct. I was a frequent user of a range of services from my last bank, and by any task metric probably looked emotionally engaged. I left them because I just never felt like their worldview and my worldview were the same. I believe Banks are an example where emotional engagement IS important.
But I don't agree that you can ask consumers 'are you engaged'? That's just not going to result in useful answers. Leaving the definition of a term to a consumer doesn't work. They are no better at it, and sometimes much worse.
At the end of the day, I think your general theme is the correct one though, and I agree with you – you need to define your purpose and go from there.
Measuring engagement is like asking how long is a piece of string – you only really know once you grab a ball of twine and cut a piece off.
Avinash, you have a wonderful experience in the web analytics industry and your practical experience and theoritical experiments are really exciting.
One thing he might be lacking here is ignoring a fact that any value add is value add. There are times when a metric is used to get some value it brings, if used properly, then searching for someother. Based on your suggestions, I would love to research a better metric to work like engagement but I dont agree that engagement is a excuse.
I had similar discussion with Adelino when he wrote against Life Time Value.
What I say with both of you is a sense of perfectionism. It is good but for whom and when. If a metric is giving me $30 dollor value add while the ideal metric would have given me 100, then I would go forward and use the metric and get that 30. May be you chose to chase the remaining 70.
Just a point to note here. Most of the time the best knowledge does not yield benefit. If we go chasing the loses, We may end up being Karl Marx. He wrote the largest book on capital and never earned his living fine.
Therefore, I would call people not to agree with you here on engagement and continue make money. The better would be to form a new engagement strategy based on your recommendation and have the new metric named, Avinash Engagement Metric (AEM).
One more thing to be noted is you are making Business Manager's life difficult by giving him many numbers in place of a Engagement Score. He may not like the idea and your idea may not get implemented at all.
Bhupendra
Could you define whether you're talking about customer engagement or visitor engagement? if you are talking about visitor engagement (people whom are not yet customers but have visited your site) that is where I disagree. See my previous comments.
However it seems to me that Jim Novo and Ron Shevlin are talking about customer engagement. In my view this is a separate argument. You're then talking about nurturing the customer not persuading the visitor.
As I understood your post however you're defining "customer" as "visitors that engage with your website". Are you talking about retention of customers (ie someone who has bought something) or engaging with visitors who may have arrived for the first time?
Respect to you for the amount of thinking you have had me doing the last couple of days either way.
Steve : The nuance parsing is great. It indicates the conversation is moving forward, and in the right direction. Thank you.
I have to admit that I was not thinking of Customer Engagement, as you define it below. Mostly because most websites don't have forced login or pan session unique individual (not cookie) tracking which means that that "Customers" are hard if not nearly impossible to track which makes the customer engagement conversations moot on pure web data. On some sites (amazon comes to mind) it might be possibly possible, not for most/rest of us. When I think of customer I think of customer (a unique person who has a relationship with you already).
Now let's flip to Visitor Engagement.
Most Consultants and Agencies that are looking for "any other metric to justify the website's existence 1) beyond conversion because conversion is so low or 2) because the site does not "sell" anything" try to sell "engagement" to their clients.
In this case measurement thus far has focussed on taking web analytics data from a website to compute Visitor Engagement. Those attempts go something like this:
* page views / unique visitor (what!!)
* website that send me traffic that reads my pages (yes! I love these sites)
* websites that send me traffic that buy my book (oh you are so engaged!)
* percent of site traffic that comes back often (hence engaging)
* percent of people who see more than a page (hence engage)
You get the idea. Yes they are all potentially indicative of "engagement of some sort". But why do we need to sexify them? The metrics above are Depth of Visit, High Quality Referrers, High Quality Conversion Sources, Recency, Non Bounce Traffic.
We could hire a Consultant who will spend three months and create a complex "Engagement Index" for you that has weighted average/whatever and put all the above metrics in that Index.
They walk out of the door with US $200,000 and you are left holding a "Engagement Index" that no one understands broadly and when the Index goes up or down no one has any idea why and what needs to be done.
The Consultant is happy, you are left holding the bag with a hole (where the $200 grand was).
Hence my recommendation: Figure out your site purpose, figure our what it means to talk to your visitors and have a conversation with them and get them to visit again and again, identify the right metric for you, call a spade a spade, let your entire organization understand what you measure and let everyone big or small drive actions from insights that only a simple metric can give.
Don't Sexify, Simplify. And conquer!
Now here is a final piece of irony: I am a consultant.
(And yes I'll be happy to help you create a "Engagement Index" for your website for a lot less than two hundred grand, email me.)
I thank you Steve for the willingness to engage in a non-entrenched open minded conversation, it is rare. I humbly reciprocate your kind respect for having stretched my own thinking (along with, as you mention, Jim and Ron).
-Avinash.
PS: You have mentioned that your company, Satama, has been extremely successful in measuring Engagement for your clients and provide lots of value from it. You can chalk my perspective down to the fact that your methodology is a gap in my knowledge resulting in me simply not "getting it", as the Americans say.
Hi Avinash, No problem on engaging in a conversation, this is what I get paid for, to think and you have certainly made me think, so again thanks for that.
I still disagree with you and so will give you an example to illustrate.
We defined a simple engagement criteria of a 3 minute session. So we segmented all the visitors that had been on the site for 3 minutes (enough time to make a purchase). By segmenting I mean totally separating the behavior of all the visitors to an index of visitors who had become engaged as per our definition (Engagement Index).
SATAMA deal with a number of very big clients that spend a lot on marketing every quarter and we saved this particular one €2million on advertising spend that would have been entriely wasted.
A direct action came from the analysis and instead of spending €2.5 million, they spent €500,000. They were extremely impressed with what they learned. If you ask them whether measuring engagement was important they would give a resounding yes.
You have global visitors (hundreds of thousands of them a month) coming to a particular product page which could be bought online via a shopping cart. The end goal of these campaigns were sales.
You had maybe 20% of those visitors which were part of the engagement index. As it stands right now, I agree, not very useful.
However because we put the visitors into a segment (remember we were using a tool that could seperate global visitors from our engagement index) we could see how effective the campaigns were and work out not only a cost per conversion, but also the cost per engaged visit.
Now this was extremely useful. All visitors who bought a product were by definition engaged. It took about 3 minutes to purchase anyway, so it stands to reason that you need to have visitors that became engaged at some point with the campaign site if you want them to buy.
Now I could say whether Google was driving more engaged visitors than banner ad placements or rich media ads and which one cost us the most. Search engine marketings cost per engaged visit was much much lower than the banners in this particular case.
Based on previous campaigns, the shopping cart abandonment for this product and engagement numbers I had in front of me I could predict that running banners in the *same way* as previously was going to be a disaster from a sales perspective. (Einstien once said, "The definition of insanity is doing the same thing over and over and expecting different results")
The abandonment was incredibly high (another engagement metric – though we call that shopping cart abandonment like everyone else) and the engagement from the campaigns incredibly low in comparison to other campaigns. In the size of the company I'm discussing €2.5 Million is chicken feed so when i raised my hand and told them not to do the initial plan (banner placements) they tried it anyway. My prediction on cost per engaged visit and cost per acquisition was out by about 10% but regardless the numbers on the first €100K spent was a disaster. (SEE INSANITY above ;o) This was when they started paying attention to what I'd been trying to tell them.
So we tried to show them how to improve.
Google engagement costs were much lower for the same amount of traffic coming from banners but they were still quite high (we thought) based on experience.
We looked into trying to improve on this and the banners (took action). The keywords used by visitors to find the product could also be separated based on the engagement index and when we had split which ones were engaged and which ones weren't we could optimize search campaigns based on those keywords.
Sales and engagement rose.
We also looked at the banners down to the ones which engaged and the ones which didn't and directed media spend to the ones which did, again.
Sales and engagement rose.
Overall instead of burning €2.5M on banner ads (which had been done in the past with really dubious success metrics loosely based on generating awareness) they optimized the engagement from all the campaigns and learned a a lot from the journey.
Had I not had the engagement index, I would have not known which keywords those guys used in comparison to global traffic. Nor would I have known which banners worked better than others.
All I would have known was traffic numbers from each source and sales data but my optimization routes would have been more limited to expensive qualitative studies. The data I am talking about all came from a web analytics tool.
We hit the same targets set for a much lower cost than €2.5 million. It cost us in total €500K. So while I didn't charge anywhere near $200K to set-up and run with the engagement index, had I done, it would have still been a huge cost saving for them. (In retrospect maybe I am too cheap! ;o)
Is this a proxy for sales? Could I have segmented based on visitors who became customers? In this case no, the data we got was too small – worthless – we tried.
Now what Jim Novo and I believe a number of others reading your original post may have thought is that you were talking about customer engagement, in terms of a person you have recorded buying something. I don't disagree with anything Jim said because I believe he is discussing the nurturing phase, more focused around CRM and customer analytics. In the case I've described we could also have done this, however the time/resources weren't there.
I on the other hand am talking about visitors who have yet to buy and can't be segmented out of the noise as a customer. Basically people that may or may not be interested in your products and services and filtering out what might be useful.
I hope this helps clarify my position.
Wow. I've been following this discussion with the greatest interest as someone who mostly works on e-commerce sites but is just now getting to grips with an 'information only' site.
I've been sitting quietly learning valuable lessons (thank you Jim in particular) up to now. But the last few exchanges between Steve and Avinash prompt me to speak up and simply comment on how stimulating I find this. Thank you to all involved.
This is an interesting article indeed. I’m not sure I see why engagement has to be just one metric. I think, as you suggest Avinash, it is in fact more nebulous than that. I think context is important in considering engagement. I also think overall engagement (however it is measured) varies wildly depending on the source of visitors and respective weights of referring source.
IF web analytics data was the ONLY thing I had to go on I’d look at average time spent and average pages viewed per visit and trend both over a specified time period. I’d then take these trended averages and look at them in the context of a third metric – in the case of an e-commerce site it would (probably) be site conversion. Ideally I’d also like to segment each of these three metrics by referring source. A visitor from a banner ad will quite probably have a lower level of interest than a visitor from a Google PPC ad.
By trending all three and seeing what changes with average time and pages viewed per visit when conversion goes up or down I’d hope to get a better idea of what I would expect engaged behaviour to look like when things are going well. By extension it would also be worth looking at average time and pages viewed per visit against a segment of all visits that completed a desired task.
I would also be thinking about what the site does / sells; is it likely to be an involved experience or a no-brainer? i.e. How would I expect a visitor to behave and what level of involvement would I expect.
I know my method is plonky and I can see it has holes in it when you come to some “web 2.0” sites and non e-commerce sites, but on the grounds that engagement is probably as much down to the individual as the web site it seems to me to be a reasonable starting point.
Steve : Two words for you: Segmentation Rocks!! :)
Here a example from a post on May 2006 that used time to segment and highlight valuable sources and measure trends:
Satama is using 3 mins or more and in this example it used five seconds or more. If I translate it into consultant speak, since I am one, it was to capture "instant engagement", though that word was not used (and now you know why! :)).
Let me once again express my deep appreciation to you for engaging in a open minded discussion sans hidden agendas. I am very grateful for it.
-Avinash.
I can't let this lie yet ;o). When I used engagement index for the visitors who were segmented into a 3 minute index it was to simplify my conversations and explanations with the client. Not to sexify it.
I had to use some term to clarify which segment I was talking about as we were using a bunch of different segments. I used something the client could easily understand and say rather than the precise web analytics term "the segmented visitors who've been on the site for more than 3 minutes" in every conversation.
I would argue that in your example instant engagement was a good name to differentiate your segment from others. So what terminology did you use instead of Instant engagement?
Steve : Something raw and unimaginative: "non bounced traffic".
It is not sexy but everyone understood what it meant immediately, making it easy for even for someone who knew nothing about web / web analytics in the company to find insights in the trends.
Thanks so much for the comment,
Avinash.
Avinash,
I still disagree with you but in my view now (As Jim Novo indicated to me a few days ago) it's purely a semantics thing. I can agree to disagree now that I know we are just discussing terminology.
You're doing the same thing as we are and don't disagree on the principles, (indeed in your example it rocked!) except you choose not to group the metrics under any name that had anything to do with engagement.
The way I now understand it, you're measuring engagement in it's various forms anyway, you just don't call it "engagement".
My point (and hence various responses) is that in your original post you didn't say that which is why I reacted in the way I did and called the post misleading.
Instead you suggested we use qualitative metrics, which were all good advice if you could afford to do it to measure what I'd call nurturing. Basically using qualitative data (such as surveys) to find out why people don't engage. I don't have a problem with that and would also encourage it if the resources are available.
I did get your point about using good engagement as an excuse for poor sales figures. That's just bad analytics strategy in my opinion not the problem of the metrics name.
All that said, this was a great open debate! ;o)
Cheers
Steve.
Since you mentioned visitor recency in this post…One thing to be aware of when dealing with this report.
In GA, Visitors that came back to your in less then 24 hours from their last visit will be recorded as : 0 days ago!!!
Do you understand what that means? when the 0 Days ago bar gets greater, what does that tell us ? we can not know if it is because visitors come more often to our site or is it because they stoped coming and as a result the % of new visitors got bigger.
This report should be handled with care… or modified.
Amazing post Avinash, you have done a great service to your readers by instilling skepticism about measurement of Engagement on the Web.
Just finished reading old posts on Engagement Index on other sites. I could not believe how complex the proposed methodology was, and after applying your filters here how useless it was.
-James.
PS: Good discussion with Steve, my take away was that in the end he made your point about why not to use engagement rather than the other way around. :-)
"Few people understand what you mean when you say “engagement”, and even fewer can then translate it to apply to their sites."
Agreed.
So ah…
What is engagement?
Why isn't that wrote anywhere in this article?
Honestly I stopped reading after I read what I quoted above and still didn't have a definition.
About time to discuss it, Avinash.
In all this debate, I can't find a good definition of what is "Engagement". We use "a set of criteria" to qualify our visitors (sales folks have been doing qualification forever). If conversion from the "qualified set" is consistently higher than all visitors, then our set of criteria is acceptable. Further, we look at ROI of our marketing spend from both Visitor & Qualified Visitor perspective.
So, the set of criteria to define "Qualified Visitor" may be different for each company, but the metric still applies.
A very old blogpost, which I have only just read, but with the benefit of reading all the replies and other posts, I tend to agree with Ron Shelvin's view of engagement.
I have seen the benefit of measuring engagement across multiple channels, not just the website, and using it as one of the decision tools for customer segmentation.
16 months on from the initial post, and does the debate still remain?
Thank you for the insightful article. I see a lot of great points and tips for the use in any maketing analytics situation.
Nice point.. 'Engagement' is a flexible term and tends to combine a number of traditional online measures and helps simplify the evaluation of user experience for marketers.
What happens when your website is not your primary tool you use to 'engage' and sell to your audience? Also, let's face it, you can drive a bus through many linear measurement techniques. It has backed businesses into a corner and the number who still insist that Search is their primary driver sales is staggering, especially when the majority consist of Brand related searches.
I know you wrote this post almost four years ago, and then it made sense. In my metrics table I have a coloumn for engagement, these are interactions with content on typical social media sites. I would count likes, comments, retweets and also clicks (the later does have to be a link but not sales link, could be anything).
Why am I looking at engagement metrics? They influence my total impressions and hence the likelyhood of getting leads.
Just my two cents… Thanks for writing a great blog Avinash
/Joakim Nilsson
Joakim: Let met try to clarify this, as perhaps it is not coming across clearly in the post.
Attempting to create an engaging experience is great. Measuring if you are engaging the traffic that comes to your site is great.
It is not great to take a collection of metrics unique to you and call it engagement. That word, in context of measurement, means nothing and everything. When you measure be clear and specific about the behavior you are measuring.
So if you are measuring tweets and likes and all that call it: Social Shares.
If you are measuring…. call it what it is. Don't hide behind the metric "engagement".
Thank you for sharing your comment!
-Avinash.
Yes and no :) I totally see your point and the issue with using somewhat loaded terms such as "Engagement", let's not talk about "influence" then :)
In the purpose of this particular measurement, the metric groups could just have been called A, B and C. Contender include "User interactions with social content cross all channels, metrics are different per channel". But "Engagement" was easier.
But I agree with you and take the feedback as being careful what to call the metrics not to confuse the reader of the report.
"Influence" is an other highly debated term since Klout got bigger. I think the Klout metrics are valuable when looking at them on a trend reporting together with other metrics, such as sales for example. Are they are measurement of "influence", of course not who can measure real influence? But what else should we call it?
Unfortunately I don´t agree with this post.
We should notice that in many cases the users are not buying the first time they visit the website. So "engagement" metrics help to find out if users are coming back, and by hence, there is better probability that they finally purchase.
Even more important is to track users buying more than one time, that´s also engagement, and its qualitative metric but also quantitative. You may have many people buying, but if they only buy once,… you have a bounce rate of buyers,… and in the long run that´s not good, right?.
In some websites with no online shop, they are looking to increase user entertainment, and one way to do it is by measuring how long they stay visiting the site, or how many page views we have per user. That´s another "engagement" metric related to how engaged the people are to the site.
So it depends very much on what type of business you have with your website but from my experience "engagement" metrics are important in many cases.
And… yes,… we may have a word for "engagement" in Spanish, and this probably "fidelidad"… :)
Antonio: The post overwhelmingly suggests that creating engaging experiences on the web is mandatory. There is absolutely no disagreement there.
The post attempts to reject the use of the word Engagement for a metric's name.
What is engagement on your site? On Twitter? In your mobile application? On a content site vs ecommerce site vs government site? Many different things.
So let's measure that engagement is occurring. But of the sake of clarity and driving action based on data, let's call the metric what it is in each case. So if you measure Page Views Per Visit, call it that. If you measure # of Retweets Per 1000 Followers, call it that. If you measure # of +1's clicked, then call it that. Just don't call them engagement because no one will know what you are measuring.
Good luck!
Avinash.
The Wikipedia article on "propinquity" gave me an idea or two about engagement metrics to consider.