I get a lot of emails with questions, atleast 10 to 15 each day. Some are easy, others hard, and some mind boggling (due to their length, complexity or audacity!).
"Dear Avinash" is an occasional series where I share some of my answers that might benefit the greater ecosystem. I'll only share the questions that might be universal, and ones where the source would be impossible to identify (to preserve confidentiality).
The first one covered career advice for Stressed "Agency Analysts" and Search Robots messing up your life.
In this post we'll cover a topic near and dear to all our hearts, comparing trends of our Key Performance Indicators and two specific strategies you can use to drive action. One that focuses on presenting data, the second an approaching to analyzing trends.
Let's go. . . the question first. . . . .
We have an ongoing debate and I was wondering if you can shed some light on the issue. We make some reports that track month over month change and others with year over year change.
One argument is that if there is a holiday in a given month or new products come out in that month then month over month will give a distorted picture. (For many companies the same product types usually come out in the same time period each year.) So year over year is what we should look at
On the other hand there should (and usually is) always an increase in year over year. And year over year is less actionable. If I want to know what product/page types need more attention, last year’s types are long gone.
I have tried to find some metric that mixes Y/Y and M/M, but to no avail.
Do you have any thoughts on this matter?
I have often advised that the cheapest way to give context to your performance is to use comparisons to other time periods.
Here's me comparing performance of this blog over two years for the Visitors metric:
Overall happiness reigns, I think. [The blue line is '07 and red '08.]
And that in some sense is the catch.
The best that a comparison of historical trends can do is give you some initial context (yea or nay) and the next best thing is that it is a good way to raise initial set of questions. "Hmmm what happened over there?" "Why don't the peaks line up?" [Digg effect]
But, as is pointed out in the email question, it is the context around changes that makes things more valuable. The graph itself won't answer the questions "why is that up or down" or, perhaps more importantly, "is our current performance better than last year's".
Product sales for you might peak each Thanksgiving (in the US). But if Thanksgiving this year was $15 mil revenue and last year was $10 mil then is that good? More importantly, is that good enough?
This is where tribal knowledge comes into play. What is different about this year and last (or this month or last)? Have you doubled the team? You have free shipping this year? This year you spend a lot on AdWords? Or you just hired me to do consulting for you this year at $10 per hour? Etc etc.
Because of that it would be nearly impossible to come up with a perfect historical trend that will be "clean". So the first thing is to realize that and then not expect too much :), except that comparing trends is a good thing and that its purpose is to just raise questions.
Next I have two suggestions:
# 1: Collect the tribal knowledge and annotate the graph.
Rather than the graph above, I'll present this one (in this case to myself! :) . . . .
When I send out the above graph or present it in the meeting everyone will say "ah ha, we can discount that peak, oh we did not do as well at that point, and we need to do more of this thing over here". I.E. important actionable conversation.
So talk to your Marketers, Boss, Cleaning Lady, the dudette you replaced at half her salary! Get the tribal knowledge, paste it in.
One of these days my hope is that Web Analytics vendors will A] Make it easier for us to add the annotations and/or B] Mine other sources and automatically add context / tribal knowledge as Google Trends does today.
[And one of these days the term "avinash kaushik" will have enough search volume to show up on the top graph! Miraculously it does show up in the 2nd graph above - though it is quite likely that my friends at google are just drawing a "pity line" for me! :)]
But if you are presenting in excel (or powerpoint) then consider annotating your data as you go along. That will be fantastic at providing some immediate answers.
# 2: Segmentation to the rescue!
In aggregate trends can hide insights and hence "dirty" the data. If you want to compare "clean" trends then your best option is to compare different segments within your data. [In all scenarios segmentation rules!]
For example you could just look at Organic traffic trends. Or performance of email campaigns. Or everyone who comes to your site from Florida. Or number of people who see more than five pages. Or % of Direct (free!) traffic. Or…. You catch my drift.
The benefit of comparing segmented trends is that you are able to go from trying to figure out which of the 1,800 variables is causing a impact to having to investigate just a couple of variables. This means you'll understand cause and effect (what you did and what was the outcome) much faster.
Here is an example. This graph shows, for the same time period as above, the Visitors from Organic Search for the keyword "avinash kaushik". . . .
Now the cause and effect can be understood much faster and actionable insights delivered intelligently.
Let's say I did lots of things to drive SEO from Nov 2007 to May 2008. Well clearly it worked, I wrote a lot less content yet my traffic increased very nicely (even better YOY).
So I could summarize that paying Matt Cutts $1.6 million to help me with search engine optimization of my blog's URL stems was a genius investment! [QQ for Matt: Why is my Page Rank still 4? Tears. :]
Do the same for your business. No no no, not hire Matt, segment your trends when you compare them.
One of my other favorites was segmenting out Direct Traffic. That is so very cool because Direct traffic (non campaign, non search, etc) was usually less influenced by other things (acquisition related thing you do – places you spend money) and hence served as a great barometer for over health of the site.
Are you getting better at getting free traffic? Do people remember your site and just show up? Do you have enough engaging content (in case of pure content non-ecommerce sites) that people return again and again each month? Etc etc.
To me that is a "clean" segment / trend to look at. For your websites there will be others. Dig and poke.
Ok its your turn now.
What are your tips when it comes to comparing trends? What things fail spectacularly? What works really well? Do you dread these things or love 'em? Care to share your war stories? What is right about the approach above? What's wrong? We would love to have your thoughts.
Couple other related posts you might find interesting: