Make Web Analytics Actionable: Focus On "What's Changed"

New Here is a dirty little secret:

Unless you are a exceptionally content heavy website And your content changes every day, it is quite unlikely that your Top 25 of anything changes all that much.

So why do we insist on stuffing our dashboards with top x or this and top y of that and top z of whatever? Let me explain with an example.

Here are the keywords driving traffic for one month:

top search keywords-month one

And here are the keywords driving traffic to the website in the second month:

top search keywords-month two

Notice something amazing? While traffic is very different in those two months the actual keywords that sent traffic from search engines barely changed. In fact even the order in which the keywords sent traffic is astonishingly similar!

I bet most of you (ok atleast you Analysts) are not actually surprised at this because you get dashboards, reports, data pukes and it all seems to be the same all of the time.

The reason for this is also not very surprising. Most websites are in stable businesses, they tend to do the same things when it comes to their acquisition strategy and there is a heavy concentration of the few things visitors want from them.

Yes there are some things happening "under the water" but they can be hard to find (take time and effort!) and our web analytics tools don't make it easy for us to identify those elements.

This leads to a few very sad outcomes:

1) We, Analysts, dutifully keep shoveling data over the fence (and that in our hearts of heart does not make us happy).

2) The Marketers and Decision Makers are frustrated because they think web analytics means stuffing their Inboxes with things they don't want to read.

3) Takes a really long time to take any action because the core things that matter are not apparent. Unhappy bosses.

So what's the solution?

whats changed report

Focusing your analytical and reporting efforts on "What's Changed".

ClickTracks, mentioned often here and in my book :), is a great web analytics tool and one of my all time favorite reports is the What's Changed Report.

It is quite brilliant in its execution.

You can choose any two time periods and ClickTracks will churn through all your website data, apply intelligent mathematical algorithms (statistical significance etc) and present a report to you that shows the most significant changes along all your data dimensions. It's Dr. Stephen Turner and his team at their best (IMHO :).

Awesome, because rather than looking at the top x of whatever you look at every day now you can look at just the data that matters.

Search key words worth paying attention to. . . .

Here is how the What's Changed report looks if I compare two time periods:

https://www.kaushik.net/avinash/wp-content/uploads/2008/04/whats changed report-search keywords

[Click on the image above for a higher resolution picture.]

Isn't that fantastic!

You can separate the wheat from the chaff by pressing just one button and now you know, instantly, which key words / key phrases need your attention because they have started sending a lot more traffic in the current time period. Maybe these need to go directly into your AdWords or AdCenter campaigns.

You also know which keyword are "falling", becoming less important in your data (perhaps you should drop 'em, perhaps you spent a lot of money to no effect, perhaps customers are moving on). You are SEO'ing your poor heart out, are your efforts working?

Both sides of the report very actionable because they uncover things "below the surface", things that web analytics tools are typically bad at highlighting.

Once you pick your time period you can get this view for all your data.

Referrers worth paying attention to. . . .

clicktracks whats changed report referrers sm

[Click on the image above for a higher resolution picture.]

Which campaigns are working, which are not. What websites are new surprises, on the left, and which ones are on their way to becoming disappointments, most likely on the right. Who should I start relationships with (say new affiliates) and which ones I need just let go. You get the idea.

See what I mean when I say it is absolutely actionable?

[Special Note: Please do not type in all the url's above and check 'em out. Some of them are from websites of an adult nature and might not be office worthy. The data is real for my blog, and no you can't ask me why they link to a web analytics blog. Your guesses are welcome though. :)]

Content consumption, pages, worth paying attention to. . . .

clicktracks whats changed report visited pages-sm

[Click on the image above for a higher resolution picture.]

Again all of the above questions and techniques apply in making this valuable.

To me there were two surprises in the above picture.

On the "Rising" side one post, on Personas and Non-Line marketing, became so much more popular. It was written months and months ago. Suddenly it popped. I immediately rushed to check out Entrance Keywords and Entrance Sources and found the reason and I am taking action so that I can get other posts get that same bang!

On the falling side the surprise was that I did not realize in the last month three of those posts (besides the first) were that popular, all were written months and months ago. So in this case I am realizing what I should have known a month ago! If I were doing any sales or marketing or had key topical (important now) calls to action I would go add to these post.

Other reports / data. . . .

You can get this report along many other dimensions telling you what campaigns are working better or worse, what products are selling more or less (statistically) and other such great trends.

Very useful, very actionable, access to trends and insights that are normally hidden. Access to this kind of report helps you do the 90/10 that Tiger Woods is advocating. . . .

tiger woods focus on what to do next-accenture

Today sadly with our Web Analytics tools we spend 90% of the time reporting on what you/company did and send masses of reports out, and 10% on what we / you / Marketers should do next. [My thanks to the Accenture corporation for creating such a great ad.]

What's Changed changes that.

Recently our good friends from Juice Analytics have taken a step to help the Users of Google Analytics create their own What's Changed report for Referrers. You'll find all the details at: http://is.gd/5Kw

You go to the site , spend nine seconds install GreaseMonkey in your firefox browser and then spend another nine seconds installing the plugin, two seconds restarting your browser and boom! You get this. . . .

google analytics juice analytics what's changed plugin

A new button in your Top Referrers report.

Now go ahead and press that button, it will change to a button called Loading…. and after a few seconds (please be patient) you get two additional reports on that page.

#1: Sites with more than 50% growth in traffic in last 3 days:

google analytics what's changed report-referrers

Sweet! Now compare that to my Top Referrers report. . . .

google analytics top referrers report

You can easily see how the first report is more valuable than the default report, for one thing top referrers stays the same most of the time.

I can look at the first report and go identify who my real friends are :), why they are sending me traffic, what can I do to get them to my top referrers etc.

#2: Sites with more than 50% growth in traffic in last 3 days:

google analytics what's changed report-new referrers

I won't repeat the message again, you can see how immediately valuable this type of information it (not the least to realize that my friends at Zaaz might not be telling the complete truth when they tell me they read my blog every day! Chris, Jason, Shane I don't believe you! :)).

Go run and install the plugin from Juice Analytics right now and see how quickly you can spot new things in your reports.

The "what's changed" functionality in Google Analytics is not quite as robust or pervasive or flexible as it is in ClickTracks. Hopefully someone from Juice Analytics or the Google Analytics team is reading this post :) and perhaps there will be more in the future. But for now I am thrilled that users of Google Analytics can atleast take a small step forward in understanding their data. For that I am grateful to the folks at Juice.

[Sal: I would love to have this "button" on the Search Keywords reports! Please.]

Update: 04/23: Sal kindly obliged and we have a very useful button now in the search keywords report as well to look at "What's Changed". Please see: Keyword Trends in Google Analytics. Here is now it looks (KW that sent 20% more traffic). . . .

google-analytics-whats-changed-report-organic search keywords

Web Analytics is actionable, I hope that all the Vendors will take a pause from simply helping us Puke data out and consider implementing mathematical and statistical techniques that help us all become the Tiger Woods!

So here's my question to you: When you do reporting of your web analytics data are you on the What's Changed train?

Ok now your turn. Would you care to share some of your insights? Big or small? What has worked for you? Are there other techniques that you have tried that work for you? Any templates or pictures you could like to share with us? Please share your own strategies and success stories. Thank you.

PS:
Couple other related posts you might find interesting:

Comments

  1. 1

    I only have time for a brief comment…because I need to install Greasemonkey and try out the Juice Analytics script! Awesome post, Avinash!

    Your comment about how ClickTracks "applies intelligent mathematical algorithms (statistical significance etc)" in crunching the What's Changed report really jumped out at me. I've never used ClickTracks, but this seems like a feature other tools should add to their product roadmaps.

    Many WA tools have he ability to show a side-by-side "% Change" factor when comparing one time period to another. And, they allow sorting by that % change. The problem is that the highest raw delta %s come from the long tail — a page that went from 2 visits to 4 visits has a 100% increase…largely due to noise. So, that requires some export-and-then-manipulate-in-Excel to try to get to a remotely meaningful list of changes. And that was still using some awfully crude logic.

    Thanks for the great observation and pointers to some useful toys (ahem…I mean…tools…)

  2. 2

    Nice post, Avinash, and thanks for the link to the greasemonkey 'what changed' script.

    Off-topic question: what do you use to generate the "ripped edge" effect on your images?

    Cheers

    Alan

    http://www.rkgblog.com

  3. 3

    you are definitely going to be in juice analytics' "what's changed" report! Great plugin, thanks for sharing.

  4. 4

    Alan Rimm-Kaufman: amazing, I was just asking myself the same question !

    Avinash:
    1) Your 90-10 description (of too much time spent on reporting, and too little on finding action-items) strikes to the point.
    2) G-R-E-A-T link for FF greasMonkey script. I also hope GA will enhance the "compare dates" with some "what's changed" features.

    Thanks for the good post and writing !

    p.s: I wonder if someone else will also note the special case demonstrated of "occam's razor" on your third screen shot at – :Search key words worth paying attention to. . . ."

    Tal.

  5. 5

    First: to Alan and Tal…get Snag-It. You'll enjoy it. Robbin Steif pointed me to it. Its pretty neat.

    Second: Avinash, you've actually hired a team of analysts to be creating this stuff right? I mean, no human has the time necessary to create the work and play with the toys you seem to find in every corner. Great work. I love the concept. Stagnating is a terrible thing. Looking at similar numbers every week its a natural progression to settle in and chuck out the same statements. I'm thrilled to see that these reports are starting to be composed in tools. Drawing data and creating them in excel or access is exciting at first, but then updating linked tables and reference cells gets to be a dedicated resource task of its own.

    On another side note, I'm disappointed we won't be co-presenting in San Fran at eMetrics, however, I look forward having a discussion or two if possible. I'm excited, on another note, that I won't be competing for a time slot with you either.

    Best Wishes…

    Daniel Shields

  6. 6

    Hi Avinash,

    I need to process this "what's changed" notion a little more before I make any comments on it, but one thing I wanted to add in here is that it can be a GOOD thing to see that your numbers are consistent on a weekly basis or whatever the timeframe measured (of course, only if the consistency shows that what you are doing is good). It's when they are not consistent and/or show that you need to improve something that you have to really start being an analyst and figure out "what to do next".

  7. 7

    Everyone : Hello from my beautiful homeland: India. Agra is hot! Literally.

    Tim : Amen! Are any vendors listening? :)

    You are very right about not simply looking at the largest % difference, they may or may not be statistically significant. That's what's so awesome about what CT does.

    Alan : I use BlogDesk to write my posts, it allows you to apply a few picture effects, all very lovely. I love BlogDesk for more things beyond just the picture stuff, I highly recommend it as a writer.

    YE : ^)

    Tal : Don't over analyze my personal reports! :)

    Daniel : One of the greatest benefits of the blog – its ability to teach and also encourage me to learn. I don't think I would be as modestly smart as I am about RSS and SEM and Social Media Metrics and so much more without this blog. Also life analyzing just page views is boring. :)

    Thanks for the comments everyone, I am thrilled that you all continue to find my stories here interesting.

    -Avinash.

  8. 8
    Alice Cooper's Stalker says

    Avinash,

    Pretty much every one of my comments to you starts out with 'Great Post!' and this is no exception.

    I think that the heart of your post is Trend Analysis and having the tools to quickly filter away the data that is not indictive of trends…either upswinging trends or downturning trends.

    As you are so fond of pointing out in your posts on Web Analytics 2.0, it's all about gaining actionable insights. Being able to highlight and bring huge and measurable swings into the open provides us with actionable insights that can be acted on.

    If there is a sudden surge in interest in a specific content topic, you have the power to provide additional postings on that topic and respond to visitor interests. Downturns in interest can be equally as insightful and impact site or company offerings.

    I agree with Tim when he mentions that product managers of analytics tools should wake up and incorporate tools that can help highlight trends more easily.

    The problem is…I don't want to wait for the next release of my tool. I want those tools, NOW!!!

    PS: Yeah, I like the Tiger Wood creative in this case too.

    Alice Cooper's Stalker

  9. 9

    Hey Avinash

    Welcome to India!!

    Please pay a visit to Delhi too, and hold a seminar/workshop on WA for us. I'm sure we all are dying to hear you here.

    Cheers!

  10. 10

    Hi Avinash. Neat stuff!

    I am responsible for one of those "content-heavy" websites, and even though I get plenty of change in my "top X" reports, they don't lead to many insights.

    The capability I absolutely can't live without on the "What's Changed Train" is groupings, both from a content perspective and from a referrer perspective. A change in exposure of a particular page of a particular book doesn't mean anything, but relatively more views to book content, as opposed to other types of content on my site, could signal something significant. For referrers, I use groupings of known blog URLs, web-based email referrers, affiliate sites, customers, etc. Individually, a mention on a blog would be buried too far down in a Top X report to ever see the light of day, but when like referrers are grouped together it's obvious when something's changed.

    angie

  11. 11

    Avinash I will help you in your efforts about humility and not tell you what a great post is this :P

    One little thing: I think in your second #2: Sites with more than 50% growth in traffic in last 3 days: You maybe wanted to say: "new referring sites appearing in last 3 days."

  12. 12

    Enjoy your 'hot' vacation Avinash :-).

    Great tips for any analyst, especially the 90/10 rule.Too often we work overtime to analyze what we did wrong (in the past) instead of using that time to think about how we approach the future. Also, in addtion to being on the 'What has changed' train I think one should also have the right resources lined up for alternative transportation in case he 'missed the boat' on any critical things :-).

    And lastly, about certain referrers directing traffic to your blog — I would think that was an easy guess with you mentioning the likes of Angelina Jolie in your blogs :-)

  13. 13

    Hey Avinash,

    I agree this is one of the most valuable reports as it cuts through the clutter of 100s of data points into something you can learn from. Somehow I'm tied to GA and usually takes 30mins to pull this off in excel..so got to get on the juice :)

    Though once you find out what's changed, the next question to answer is why did it change? Since nothing happens in a vacuum, a change on pageviews came from a change in site visits, and many times both direct and search (which can help to answer the why) It would be nice if these change reports were able to do the linkage between metrics to get closer on what was the trigger that made this all happen…

    Maybe thats why they call it web analytics 2.0…because it's all the features you want and no tool to provide it ;)

    anyways nice find!

  14. 14

    Interesting Post Avinash! (again…. ;-) )

    One wonders about the format display of the what's changed in your examples. It's still… hard to see what matters and note the significant changes.

    For myself when doing these what's changed style, I find a stacked percentage graph to work very nicely. X Axis == Time. Y Axis == 0-100%.

    If memory serves one of the ultra few things I did like about webtrends many years ago, was that it did give you such a graph on keywords. Thus I could see keyword percentages change over the week.

    Sun/Mon would be things like "hernia" or "groin strain". Thurs/Fri would be "stress" or "mental health".

    Gee I wonder why….. ;-)

    Cheers!
    – Steve

  15. 15

    Juice Analytics amazes again! This is very very cool stuff. And a great lesson on where to focus. Personally I still go with the wacky Excel route for the reasons Tim Wilson mentions way above. I can kindof build in my own significance factors, since my tools don't do it for me.

    BTW, you probably know this, but your weird referrals are referral spam. One of the plagues of blogs (along with trackback and comment spam). The cool thing about referral spam (imho) is that it goes to show that people are looking at some web analytics data: referrals! Bring on the onsite search spam! It wouldn't be effective spamming practice if no one was looking ;)

  16. 16

    Brilliant article Avinash, it's definitely convinced me to buy your book because if your free advice is this good, the book must be even better!

  17. 17

    Keyword Trends just as requested.

    -Sal

  18. 18

    This is an absolutely amazing post about the real things to watch in your web analytics! Too many online marketers miss this.

    Great post! Sharing with my audience immediately.

    Maria Reyes-Mcdavis

  19. 19

    You make the point in the first sentence of your blog that if your site is not changing then obviously the top 25 things in your site wont change as well.

  20. 20

    Somebody ( in comment #2 ) asked how you create the torn edge effect on some of your images; I didn't see an answer. But it's a very attractive effect … if you feel inclined to share, I'm also curious. Sorry for the tangent.

    You asked for guesses as to why porn sites are linking to you … I don't think they are. I see a lot of the same thing in my own referral reports: sites I could be fired for viewing at work sending traffic to a landscape photography blog. It makes no sense.

    Except that a lot of web servers are configured to publish reports ( to the general public ) by default. Some people run software to create "referral spam" or to visit your site and spoof the referral header. The goals are to get you to wonder who this site is, sending you traffic, and glean a click, but also to get a link from your stats. At least that's what I've deduced from my own experience…

  21. 21

    Hi Forrest, Avinash uses (blogdesk.org) in order to create the torn edge effect on images;

    I gave that question – and He answered by Email, And I guess he would love for everyone else here to also know – so now we all know :)

    Tal.

  22. 22

    Angie : If memory serves me right then with ClickTracks you can create content groupings (using Advanced Segmentation) and then it would compute for you the "What's Changed" metric.

    Nick : No matter how smart the tools get you will still need humans (mercifully) to figure out the Why. The What's Changed report makes the Analyst's life easier, raises the right questions. Then the real work starts.

    Even with smart tools the 10/90 rule still applies! :)

    Jen : You are a exception, not the rule!

    On the other point, one of the referrers is a spam site, but it was the soft core website I did not want people typing. And that is a legitimate link, turns out they have a great deal of interest in Google and linked to my Working For Google post. Who can blame them. :)

    Forrest : In comment #7 (mine) above I do mention that the "torn effect" is from BlogDesk, my blog writing software.

    See immediately above for the explanation of the soft core link, it was legit. I researched. :)

    Thanks everyone!

    -Avinash.

  23. 23

    Avinash, Another home run hit out of the ballpark. I love the idea of looking at "what changed" and connected to this post from the very beginning. Before the acquisition, the top two search terms that brought people to our site – month in and month out, was "maxamine" spelled correctly, followed by "maximine" misspelled. Everything else paled by comparison. Our name and spokesperson are a bit more memorable these days :-)

    Debbie Pascoe

  24. 24

    Outstanding post! The ideas and insights are very worth reading. You really gave me valuable information. Thanks for sharing it!

  25. 25

    Great post…I ran into your post while I was looking for materials to extract more value out of the Coremetrics reports. CoreMetrics has the so-called "What's Changed" feature as well.

  26. 26

    […]
    Make Web Analytics Actionable: Focus On “What’s Changed” – “Must Read!”
    […]

  27. 27

    […] Instead, you should hunt for the insights and be “Web Analytics Ninja” as Avinash Kaushik stated in his presentation “Hunting Actionable Insights“: […]

  28. 28

    Thanks so much! I've been looking for something like this…, wondering why it wasnt a standard part of Analytics. I just Twitted your post :)

  29. 29

    Seems ClickTracks is now part of Lyris and only available as 'integrated' solution with web hosting, cms etc. Very unfortunate. Are there any alternatives out there?
    Thanks!

  30. 30

    Funny how such a valuable report continues to be overlooked by most web analytics vendors. We'd hate to see it disappear, so we built one for Google Analytics users. It runs inside Excel, and we've tried to make it as flexible as possible so people can tailor it to their unique website objectives.

    http://www.nextanalytics.com/whats-changed-report

  31. 31

    Hi Avinash,

    I know I'm a little late to the Clicktracks party but it now seems to be taken over by a company called Lyris. Does anyone know a similar solution which doesn't fall under the Lyris umbrella?

    Thanks,
    Michael

  32. 32

    The Firefox plugin mentioned on this webpage and in your book is no longer available. Is there "what's changed" functionality in Google Analytics now, perhaps as a custom report or dashboard…

Trackbacks

  1. […] Make Web Analytics Actionable: Focus On "What's Changed" | Occam's Razor by … Focusing your analytical and reporting efforts on “What’s Changed”. (tags: seo tools analysis internet marketing) […]

  2. […] sending you new traffic and which new keywords are being used to find your site. Check out how to create a "What's Changed?" report here (additional resource here). It helps you find out which data is actionable (i.e. which sites to […]

  3. […] And finally, Avinash Kaushik looks at the other end of the equation, asking how you find what your customers care about on your site. Neat. […]

  4. […] Make Web Analytics Actionable: Focus On "What's Changed" – You can always count on Avinash to provide a great read. This post is about noticing recent changes in your analytics and how to act on that data. […]

  5. […] Avinash has a great post up on Occam’s Razor about making web analytics actionable by focusing on what has changed. He points out that most sites, even blogs that are posted to regularly, change very slowly over time. Despite traffic numbers, you will find that your keywords generally stay the same from one month to the next. By tracking what does change, however, you can learn a lot about visitors and your site. […]

  6. […] Make Web Analytics Actionable: Focus On “What’s Changed” (Avinash Kaushik) […]

  7. […] I post di riferimento, per leggersi l’intera documentazione, sono questo e questo. Scopo: andare ad aggiungere un utile pulsantino nelle schermate di Google Analytics relativamente ai siti di provenienza ed alle parole chiave che hanno fatto registrare accessi dai motori di ricerca. […]

  8. […] Make Web Analytics Actionable: Focus On “What’s Changed” – jak opisuje post ważną częścią efektywnej analizy używalności witryny internetowej jest obserwowanie zmian w poszczególnych wskaźnikach tej strony. W artykule opis tych wskaźników, w których zmiany powinny być dla nas szczególnie ważne. […]

  9. […]
    Instead, you should hunt for the insights and be “Web Analytics Ninja” as Avinash Kaushik stated in his presentation “Hunting Actionable Insights“:

    * Analyze “What has changed”
    * Try to share the “Context” of these changes next to the numbers
    * Give insights, advice, recommendations or steps to take

    This will help you in taking action based on the actual behavior of site visitors and as such support the decision making processes.
    […]

  10. […] At the suggestion of Avinash Kaushik, the new script works for keyword data, helping you see how organic search traffic is changing. An increase in a keyword may indicate a general change in user interests and/or improved performance on search results. […]

  11. […] This, my friends, is a real blog post about the real uses of web analytics for planning your next move in the online marketing game. Detailed and thorough, a great read for newbies and pros alike. Make Web Analytics Actionable: Focus on What Changes! […]

  12. […]
    “It’s fine to see the top 50 keywords by clicks, or the top 10 ad groups by revenue” he said, but what is much more powerful is to see what’s changed – the keywords making money today that were not making money yesterday. Or the campaigns that performed well last month, and are not performing well today.”

    There is a great post on the subject of this approach to reporting on his well respected blog, written about the time this conversation took place.
    […]

  13. […]
    Avinash Kaushik of Occam’s Razor fame suggests, in not quite so few words, a mix of descriptive and prescriptive web analytics reports.

    Descriptive reports should cut to the chase and tell us what’s changed. Such reports can be auto-generated from most tools.

    Prescriptive reports require human intervention, and tell us what should be changed to meet mutually agreed goals.
    […]

  14. […]
    These reports showcase a core feature of ClickEquations, the ability to compare performance between any two periods and to very easily see the difference between performance in those two periods. It’s a feature that was actually inspired by an earlier post on Occam’s Razor, and was the direct result of a conversation we had with Avinash early last summer.
    […]

  15. […]
    Make Web Analytics Actionable: Focus On "What's Changed" | Occam's Razor by Avinash Kaushik
    Unless you are a exceptionally content heavy website And your content changes every day, it is quite unlikely that your Top 25 of anything changes all that much.
    […]

  16. […] by Glenn Friesen – read the full story where this content was originally published, at Make Web Analytics Actionable: Focus On "What's Changed" | Occam's Razor by Av… Share and […]

  17. […]
    Que ce soit pour votre propre site, ou encore mieux pour une campagne de relations publiques en ligne, cet outil permettra de vous ajuster constamment et affiner vos stratégies de communication.

    Plus d’informations sur le script dans le post original de Juice Analytics. Si avec ça, vous n’êtes pas encore soulagé, je ne peux que vous suggérer cet excellent article de Avinash Kaushik, évangéliste Google en web analytique.
    […]

  18. […]
    For a deeper look into understanding what has changed, please take a look at a post in the Occam’s Razor Blog. In this post, the author does a great job in reminding the analyst that looking at recurring reports may not have much value at all. As a matter of fact, you may find that typical views of incoming key words, referring links, and content consumption pages, may all look very similar from a week to week or month to month basis.
    […]

  19. […]
    Practice predictive analytics, look for patterns in your data. If you notice a trend, you will be prepared for changes and have your resources ready when you need them. For example, if your social efforts show steady increase in conversions through paid advertising, you'd better figure out how to adjust your PPC budget. One way to identify a trend is to track changes in data patterns.
    […]

  20. […]
    Practice predictive analytics, look for patterns in your data. If you notice a trend, you will be prepared for changes and have your resources ready when you need them. For example, if your social efforts show steady increase in conversions through paid advertising, you’d better figure out how to adjust your PPC budget. One way to identify a trend is to track changes in data patterns.
    […]

Add your Perspective

*