What would make you cry of happiness in a Web Analytics report?
What would make you cry of happiness in any report / presentation that you got from a analytics practitioner or consultant or your mom?
This post attempts to sort through the good, the bad and the ugly and answer that question (except that Mom bit, that will require therapy!).
It will also help you win contracts, prizes, company bonuses, and generally give you Superwoman (/man) like powers to impress people with your awesomeness in presenting complex insights that simply drive actionability.
Some context first.
I had the honor of helping judge the winners for the WAA Championship (and the SEM Scholarship Contest ). That made me think a lot about what makes great analysis.
When looking at so many wonderful entries, how does one decide the winner? Are there specific traits? How do you know who deserves to sit at the kids dinner table and not the adult one?
Thus this post was born. It was my attempt, before I judged the contest, to create a framework that would help me identify real analysis and separate the Squirrels from the Ninjas.
Daniel said it would make a great post, and so this one's for him.
[UPDATE: Thanks to permission from the WAA I was able to add the top four winners to this post. Please see links at the bottom of the post, the contain great learnings.]
My hope is that it will help you identify what makes for magnificent analysis and in your day to day job (as Marketers, Usability Professionals, Consultants, Analysis Ninjas, Reporting Squirrels, …) present your thoughts on a set of data and have the maximum impact in terms of insights and action.
Seven Filters That Help Identify Great Analysis:
After you are done with any analysis, and before you present it to your client / peers, apply these 7 filters to ensure that what you are sending out is real gold. . . . .
1) No data pukes.
A summary of the data from the tool is not enough (no matter how pretty). Period.
Often "analysis" that is submitted is essentially a small table of data, which is essentially a "mini me" of the large table from which it came. This is not analysis, it is just a smaller report.
Here is another thing that people consider analysis: x,xxx visits to the championship page almost a xx% of the x,xxx visits for the period of 06/01 to 06/14.
That's the "table" in English. It has the additional disadvantage of forcing me to do math two or three times and try to even graph it in my head. Too much work for anyone to do from a "analysis".
This might be a bit harsh but as I read any "analysis" here is what's going on in my mind:
A] "What's your point?" Give me value, not data.
B] Based on your point, "what do you want me to do?"
C] If relevant, and usually only if asked, give me the data (and please please please don't make me think or have to compute 19% of 8,296 Visits!).
Remember its the first one that I want the most. A drives action.
Bonus points: If you did a good job with the graph, you should not have to repeat in English underneath the graph what it is showing.
If there are any data rivers in your data, please consider redoing your analysis.
2) Hard tie to business outcomes. Always.
If you have learned anything on this blog then it is probably my insane obsession with Outcomes (see Trinity, Web Analytics 2.0).
At the end of the day every analysis needs to solve for the business outcomes. So you have to have some understanding of the goals going in (this is much harder than you imagine).
Many people just jump into the data, find interesting trends and patterns, convert those into "insights" and off it goes. The problem? You are sending things out you think matter, rather than what the business actually cares about.
Revenue, leads, increased customer satisfaction, brand value, friend invites, loyalty, bounces, website engagement :^), job applications, ads clicked, task completion rate and …. and … and .. You get the idea.
Awesome analyses always have hard tie to outcomes.
It was interesting that for the WAA championship there were no real business outcomes provided to the Analysts. Just loose guidance: give us stuff. :)
Yet that did not stop the superstars of web analytics. They simply assumed what the site's desired business outcomes were.
Many of them opened with their interpretation of the three goals of the website (or the bold ones even said something like: "you should actually be driving xxx outcomes but you the WAA are focused on silly things" – now that is chutzpah I can admire!).
Is your analysis focused on clearly established business outcomes? If not by your boss / client, then by you?
3) Usage of other tools (True Analytics 2.0).
Another little thing I obsess about, trying to always advocate the use of more than one source of data to ensure people understand more than just the "what". . . .
I think it was the fifth analysis that I saw that used something other that he web analytics tool the Analysts were provided (Google Analytics).
Such a shame.
Google Analytics (or for that matter Omniture or WebTrends etc) are great tools. You need the What. But it is so limiting. You need the Why and the What Else and more. True Web Analytics 2.0 to get robust answers that have deeper customer and competitive insights.
Sure the WAA does not do surveys to understand how to serve their customers better, and does not have "direct" competitors and WA 2.0 is hard work.
But the enterprising Analysts (and the winning team) went out of the box. They used the AdWords Keyword Tool, they did searches on other search engines, investigated Web Analyst's search behavior on Google Trends, checked how many corporate WAA members members link to the WAA site (a measly one!), one of them did their own survey directly to members (!!), checked the DMOZ, compared the site to IAB etc.
Now that is almost orgasmic. They did not take the lame excuse that the client did not give them data sources. They used all the tools at their disposal and executed a 2.0 analysis.
Do you?
Bonus: Remember you don't have despair about what the client as. Use free services like Compete and mine Google for press releases by various organizations (like shop.org) that contain relevant info and more.
4) Not boring. Please.
Ok this blog is a exception (!), but let's admit it: Web analytics is boring.
Analytics of all sorts is boring. To lots of people (not you and I of course!).
Most web analyst report outs, glossing consultant analysis, make you want dramatically hurt yourself, rather than read them. They are all the same, data pukes, pretty graphs that tell you nothing, no tie to outcomes and descriptions and summaries that would make the IRS proud.
Let that not be you.
Look that these nice folks, they read all the analysis (check out the thick stack!), and just look at how excited they are!
I have to read lots of reports and summaries and briefs. I am always looking for people who made it interesting to read their submissions. Do they have a interesting way of framing the analysis?
In the real world this quality stands over all other "consultant" / "analyst" reports.
The analysis of the winner of the WAA Championship was essentially a series of email exchanges between people, with each email they revealed their ideas, insights and methodologies. No graphs. No tables. Just a quirky sense of humor (and deeply delightful analysis).
They stood out from the polished nicely templated graphics rich submissions of everyone else.
They made web analytics unboring. They made it fun.
Sexy wins. :)
5) Connect insights with actual data.
This might sound absolutely surprising but in many of the analysis it is really hard to see what the connection is between data and the insights derived from that data.
It seems along the way we have all developed "best practices" and preferences and "what works" and what does not and so on and so forth. Hence as we look at websites and data we sometimes simply jump to making recommendations based on what we know and think and feel rather than staying grounded in data.
Often I read something like: "Redesign the navigation", and my first thought is why? based on what?. Or "Internal search should be every where" – why? surely a best practice, but why for this site?
Lots and lots of people did this in the WAA Championship, especially those that were from decent sized agencies or consulting companies. They have the curse of knowing lots.
Me? I always put that secondary. My recommendation: Tie your recommendations to the data on hand. Include your feelings in a appendix, but in the main body, tie to data.
6) Meet the "expectations of scale".
This is perhaps a personal bias (especially in competitions). I am not going to, sorry, have the same set of expectations from Michelle Chin as I do from Jaume Clotet as I do from Zaaz.
Each of those comes with massively different set of experiences and resources. The bigger you are the more I expect (and please remember not more data pukes, more analysis!).
More in terms of insights, more in terms of rigor, more in terms of everything.
If you are "big" or you have written a book (!!) then you are playing the game at a different level when it comes to expectations. Michelle has to be just so good to beat the bejesus out of you (and I know Michelle, she can!).
Look at your size. Do your analysis reflect the depth that your size should? It better.
7) Have something unique. Enough said.
Remember that if you are going for a RFP or a contract that 99% of what you will have access will be the same, 70% of the analysis that you will end up doing will be the same as your competitors, you might have read the same books and attended the same conferences.
Do something that makes you stand out.
And I'll let you into a secret, it is not the formatting of the text you deliver or 3d charts. That has been done to death.
And its not that hard.
Here's a example, everyone will report that a metric (say conversion) was 53% for keyword z and it was 56% for keyword q. Why don't you compute statistical significance between the two? Rather than reporting those two numbers out you can show how much confidence there can be in those numbers.
See how easy it was to stand out?
Or here's another one. One of the Analyst started by stating that they were leaving out a time period that could distort the data. Interesting that they thought of that.
You could likewise eliminate from your analysis sources that reflect "one time only not repeatable events". Why bother?
Or try this, measure offline impact of the online activity! It is hard to do and you'll stand out!!
Business life can be a contact sport (competitors certainly are) and if you want to win then you have to have a UVP – a unique value proposition.
Never let a analysis leave your computer without making sure that there is something unique in it that will stand out.
See that was not hard?
Here is a summary of the "Avinash Filters for Awesome Analysis Presentations":
1) No data pukes.
2) Hard tie to business outcomes. Always.
3) Usage of other tools (True Analytics 2.0).
4) Not boring. Please.
5) Connect insights with actual data.
6) Meet the "expectations of scale".
7) Have something unique. Enough said.
Do you agree with the list? Have something to add? Would you like to "puke" :) on something, or simply disagree? Please share.
From your experience are there techniques in presenting analysis (or conducting them) that have worked particularly well for you? It would be awesome to have your insights and lessons. Thank you.
PS:
Happy Birthday to Nelson Mandela! An icon, an amazing human being, a true leader. Here is a great four minute audio and photo tribute, please check it out: Nelson Mandela at 90.
PPS (Bonus!):
Due to popular demand, and thanks to the WAA's permission, here are the top four winning entries from the WAA championships. This is a great way for you to learn more about how to present great analysis (and they each took a different tact).
-
#1. ClickThinking
#2. Predicta
#3. Jaume Clotet
#4. User Intelligence
[It is quite gratifying to me at some level that three of the top four are international entries. Validates for me the superior analytical sophistication that is outside the US.]
Via 
Hi Avinash
Will the winners be making their entries available for viewing?
Good points in here, I know I have been guilty of data pukes. I'm mending my ways and I work closely with some great content writers who challenge me to make more of a story out of all the data, so the person who reads the analysis can use it straight away without having to come back and get an explanation of the content.
cheers
Jon
Avinash,
This particular post, above all others you have written, ties it all together from a analytics consumer point of view the best. Certainly you've talked about these things before, but this is a wonderful, quick reference for how to wow.
The most surprising part was "No graphs. No tables." for the winner. That goes to show that analytics might deal with numbers, but the true deliverable is ideas.
Making it simpler an interesting!!
I am not sure if i can add pics and funny comments to my reports, i have smug bosses here.. you want me to risk my job!
Avinash,
Great post. I can't say that I currently practice all that you preach/teach in this posting, but it's certainly given me something to think about.
The only other thing that I can think of is to cater your presentation of your analysis to your audience. Some people just want bullet points. Others want all of the supporting data so that they can validate the analysis you have presented themselves. They might want to use some of your data in another presentation that they create monthly. Some people like power point slides…others word documents. Obviously, you aren't going to know what your audience likes when you first start working with them. If you have a long term relationship, they will give you feedback on what they like and don't like. This might not have been helpful in the WAA Championship contest.
Quick question. Do you have any recommendations on readings for computing statistical significance and making those connections…like about keyword conversions and such?
Thanks again!
Hi Avanish:
I think this one might touch all the bases – but is is our kind of web analytics.
http://tinyurl.com/58xm8d
Combines deep analytics with execution, measurement and ties out to sales.
TO'B
Hi Avinash,
The WAA Championship was a great learning experience, and your post suits as an informative follow-up of analytics best practices. I don't get to work with direct eCommerce that often; so viewing monetized KPI's in Google Analytics was a plus =) I also think it would be helpful to view the winning submissions if available. Thanks,
~helen
Nice post Avinash.
I think you are the only one who can get away with saying using analytic tools are an orgasmic experience.
Tying to businesses outcomes is probably the most important of them all. Id say this also includes tying insights to explicit actions to take, doing some quick impact the action could have, calculating opportunity loss and even prioritizing which to do first (ie which has the greatest lost if not acted upon). I'd also say this works backwards that if you can't take action on the insight from the analysis you're doing, then your time might be best served elsewhere.
Very honored, thanks for the post ;-)
Hi Avinash! Nice post! ;)
Avinash,
This post is really up there in terms being on my all-time favorites — of course, me being a confessed chronic analyst :-). I think folks can raise the quality of their analytics just by following even one of your filters.
The one footnote I would like to add on #2 (Hard tie to Business Outcomes) is that not only should you always have a clear idea of how the analysis can/should be used and the impact it is going to generate, but also a certain knowledge of who in the company would most benefit by getting the results from the analysis. Often I have seen good analysis going to the wrong audience which pretty much kills any impact it might have had. At the end of the day, we should not be [as Naisbitt said] "drowning in information but starved for knowledge"; on the contrary, we should have it processed and organized and available to the right people in a format fit for decision making.
Great post!!
Avinash,
Thanks, as always, for shearing your brilliant thoughts and ideas, my favourite:
"4) Not boring. Please."
It was very fun for me to participate in the WAA Championship.
Web Analytics, I love this game! ;-)
Jaume
Avinash, month-long listener, first time poster. Excellent write-up.
My question is this: I'm wholly on-board with your emphasis on outcomes- these reports are useful only insofar as they allow us to change our behavior for the better. We're a 6-person startup so we don't collect pretty graphs and present them to anyone – we're the end audience. We use analytics to test our guesses about how people are using our site. Then we make changes and test again. So far so good…
Where I get hung up though is in deconstructing the events that lead up to a desired action (ie. a sale) into a clear sequence of sub-events that I can study. For instance I know that we're doing something right when the orders are flooding in, when time spent per visitor goes up, or when the avg number of pages per visitor goes up but that's the extent of our use of analytics- small tweaks and observation if it helped. We don't have an "exploratory" approach though, we have a "test this hypothesis" approach. I'm aware there are "click paths" and "funnels" and other things I don't yet understand and perhaps the answer lies in one of those tools, but is there anything from your experience with Analytics that tells us "the best thing you could do to improve conversions right now is xxx?" Maybe that's too pie-in-the-sky or even the role of hiring a 3rd party analyst to interpret your data, but I feel like Analytics is this powerful tool and we're using it more like a magic 8-ball to answer yes/no on hypotheses we come up with instead of listening to it and having it tell us what our site needs. Does that make sense?
sean
Thank you for sharing the winning analysis documents! Great opportunity to learn from the pros!
Dear Avinash:
I love this post!
It's so much fun doing Analytics the way you always teach us, but I totally agree with Alice Cooper's Stalker.
People sometimes want bullet points. Most of the times they want nice graphics, slides… even excel tables.
And I had a colleague who told me about how he almost loses her job because one of the hippos (sorry, bosses) thought she was doing his job, pointing out what his department was doing wrong and what could do to fix it.
Some people just want the web analyst to be an analyst and provide data. Point.
In my job I find people who love evolution graphics, other like that "next steps to became better" report, other need data to illustrate power point slides… I don't get bored and I try to give them whatever makes them happy. Even I improve sometimes…
But I think each hippo is a big big world :)
Have fun in the summer! Thanks again.
Alice CS: I like the idea Andrew expressed in a comment above, "the true deliverable is not the analytics (/data), it is the ideas". I love that sentiment. I'll keep that in mind as I compile my analysis.
You are of course right about the presentation style, being aware of what the client might find more effective is optimal. But to me the winning entry was anything but traditional, but that is what made it compelling and worth reading. But I do believe that if they present to C level staff they might have to move to a ppt version. :)
Ned: I really like the idea of finding the right audience for the analysis.
But increasingly I am also pushing this vision of tools morphing to "empower a data democracy" – that anyone in the company who needs data has access to it AND the data/insights are organized in such a way (with application of intelligent algorithms, filters, segments) that they can understand what to do (say 70% of the time).
I think that needs to happen to eliminate this problem of data mismatch that kills companies today (this mismatch is the reason most of online marketing is a faith based initiative!). Its a vision, hopefully it will come true a little bit at some point.
Sean: What you are saying makes sense, please don't worry about that. If I summarize: you have access to all this data and you don't know how to find anything worth actioning.
That requires the art of analysis. It is a art, a acquired skill, and a little bit of science.
But that's what is missing.
It means being decent at segmentation (Here is a simple one: Do new visitors to the site behave differently from returning visitors. Or is there a difference between Days to Action for visitors that come under "brand" or "category" terms. And so on and so forth.)
It is realizing that path analysis is a terrible use of time (except for 100% structured experiences).
It is doing multidimensional analysis using something like the Google Analytics Keyword Position Report to optimize bids on campaigns to get the right average position for your ad.
It is knowing your business well enough that you can ignore data that does not matter and focus on data that does.
I could go on. But I think you catch my drift.
In the near term if you don't have the analytical experience the simplest way to fix that is to get consulting horsepower, tell the consultant what your business goals and let them help you find the insights (while teaching you "how to fish"). Over time you'll get to know what buttons to press, but better still you'll get comfortable doing analysis and that will get you out of the bind you find yourself.
I want to stress that you should not feel bad, every jobs I started I felt exactly as you do now. It takes time, it takes practice, it takes lots of failing. But it is a incredibly rewarding. I can tell you that from experience.
Gemma: Dealing with reality is hard, having spent 11 years of my 12 year career working as a practitioner I can completely relate to your friend whose boss wanted him to be a "reporting squirrel" and not a "analysis ninja".
Been there, done that.
I first want to point out that the core thing a Analyst has to deliver is ideas that need to be actioned, that can have a impact. Not data. (Referring to Andrew's comment above).
To your friend I would say, there are 19 times more jobs available in Web Analytics than there are people to fill them. If he wants to be a Analysis Ninja, he can easily find another job!
In Spain perhaps it is not so easy as I say. But you get my point about what a Analyst should do (while, as you rightly say, trying to balance for the Hippo).
-Avinash.
Hi Avinash,
This time i will not say that it is a great post as usual, indeed it is the best post among the all other posts. Because of those four PDFS about how to present great analysis (and they each took a different tact).
Actually this is a best idea to represent it to blog readers and web analyst practitioners.
Thanks for publishing those four winners in the above post.
Thanks & Best Regards,
Bhagawat.
Avinash,
If weren't for you Analytics would be a single threaded craft owned by a "select group of people". You have broken that barrier and made it truly mainstream. In the process you've built a sacrosanct place for all things pertaining to analytics (not just web related).
Best wishes
And as always – Mr Kaushik presents another post that guarantees i will get no work done today as i pore over every detail of yeat another fascinating post.
I just printed off the winning entries and will read them, but must say – i love the back and forth email format – maybe because it reminds me of "E" by Matt Beaumont But also – i just really like the conversational aspect of it, the "good point,so what you mean is…" aspect works for my way of thinking )
I started helping out someone sort out his analytics and of course – the first thing i told him? "visit Occam's Razor – the guy makes sense!"
Time for some coffee and light reading :) great work Avanish – and hearty congrats to everyone that took part in the WAA!
Very insightful post! … and especially helpful for all those who are trying to provide truly effective, meaningful analyses day-to-day :)
Hi Avinash,
Thank you for the opportunity and for judging the competition. It's the first time that our digital research teams work has been labelled as "sexy" but we will claim it all the same ;).
We thoroughly enjoyed the competition, we stuck to our conventional methodologies and embraced a different approach in terms of presenting our findings with a bit of humour. Proud as punch that we topped the list at the end of the day and look forward to the next one.
We will continue in our quest to make web analytics and conversion strategies exciting for our clients. we thoroughly enjoy reading your material. Keep up the great work.