My instinct was to use the wry observation uttered by US Supreme Court Justice Potter Stewart in trying to define
po rn: "I know it when I see it."

That applies to what is analysis. I know it when I see it. : )

That, of course, would have been an unhelpful answer.

So here I what I actually said:

If you see a data puke then you know you are looking at the result of web reporting, even if it is called a dashboard.

If you see words in English outlining actions that need to be taken, and below the fold you see relevant supporting data, then you are looking at the result of web data analysis.

Would you agree? Got an alternative, please submit via comments.

I always find pictures help me learn, so here are some helpful pictures for you. . .

This is web reporting:

And so is this, even if it looks cuter:

And while you might be tempted to believe that this is not web reporting, with all the data and the colors and even some segments, it is web reporting:

See the common themes in all the examples above?

The thankless job of web reporting, illustrated vividly above, is to punt the part of interpreting the data, understanding the context and identifying actions to the recipient of the data puke.

If that is your role, then the best you can do is make sure you have take the right screenshots out of Site Catalyst or Google Analytics, or charge an extra $15 an hour and dump the data into Excel and add a color to the table header.

So what about web analysis?

The job of web analysis mandates a good understanding of the business priorities, creation of the right custom reports, application of hyper-relevant advanced segments to that data and, finally and most importantly, presentation of your insights and recommended action using the locally spoken language.

See the difference? It's a different job, requires different work, and of course radically different skills.

Examples of web analysis? I thought you would never ask. . .

This is a good example of web analysis:

[And not only because it is my work! Learn more about it here: Action Dashboard.]

Notice the overwhelming existence of words. That's not always sufficient, but I humbly believe always necessary.

When you look to check if you are looking at analysis or reporting look for Insights, Actions, Impact on Company. All good signs of analysis.

Here's another example of really good web analysis:

[Click on the image above for a higher resolution version.]

Ignore how well or badly the business is doing. Focus on approach taken.

Here are some things that should jump out. . . . A deliberate focus on only the "movers and shakers" (not just the top ten!).  Short table: just the key data. Most of the page is taken up with words that give insights and specific actions to take.

Another example that I particularly like, both for the style of presentation and how rare it is in our world of web analytics. . .

[Click on the image above for a higher resolution version.]

No table, no rows, no pies. And yet data holds center stage with clearly highlighted actions.

Normally, we all do the column on the left (it might look different, but we have it). Unfortunately we don't appreciate is the power of the middle column ("segmentation reveled"). That is super important because it gives the recipients exposure to the hard work that you have done and in a very quiet ways increases their confidence in your work. Guess the outcome of that? They take the actions you are recommending!!

Analysts constantly complain that no one follows any of their data-based recommendations.  How do you expose your hard work? In a garish Las Vegas show girl fashion where all the "data plumes" are, unsexily in this case, hanging off the body? Or, in quite concise ways? Only one of those two work.

One more? Okay here you go. . .

[Click on the image above for a higher resolution version.]

Diana has loads of observations, supported by visuals (sometimes it really helps to show the search results or the emails or the Facebook ad) with highlights (actually lowlights) in red, and finally recommendations.

And note the tie to outcomes (another common theme in all examples above). In this case, the search improvements are tied to the increase in donations I can make because of sales of my book. 1.5 extra smiles per month! (All my proceeds from both my books go to charity.) A good way to get attention from the "executive" and get him or her to take action.

Do that. A lot. Be creative. Yes it is hard work. But then again glory is not cheap, is it?

Exceptions to the rule.

Not every output you get from your Analyst, or "Analyst" :), with loads of words on it, instead of numbers, will be analysis. Hence my assertion that "I know it when I see it." Words instead of data pukes is just a clue, read the words to discern if it actually is analysis or a repeation of what the table or graph already says!

In the same vein not every output that is chock full of numbers in five size font, with pies and tables stuffed in for good measure, is a representation of web reporting. It is hard to find the exceptions to this rule, but I have seen at least two in nine years.

Top 10 signs that you are looking at / doing web analysis.

Let's make sure this horse is really and truly dead by summarizing the lessons above and using a set of signs that might indicate that you are looking at web analysis. . .

#1. The thing that you see instantly is not data, but rather actions for the business to take.

#2. When I see Economic Value I feel a bit more confident that I am looking at the result of analysis. Primarily because it is so darn hard to do. You have to understand business goals / outcomes (so harrrrrd!) and then work with Finance to identify economic value, and then you have to configure it in the tool and then apply advanced segments, and then figure out how things are doing. That is love. I mean that is analysis! Or at least all the work that goes into being able to do effective analysis.

#3. In the same vein, if you see references to the Web Analytics Measurement Model (or better still, see it in its entirety on one slide up front), then you know that the Ninja did some analysis.

#4. Any application of algorithmic intelligence, weighted sort, expected range for metric values (control limits), or anything that even remotely smells of ever so slightly advanced statistics is a good sign. Unknown unknowns are what it's all about!

Also mere existence of statistics is not sufficient. All other rules above and below still apply. :)

#5. If you see a Target mentioned in the report / presentation, then the Analyst did some business analysis at least. See the top right of the picture immediately above.

#6. Loads and loads and loads of context! Context is queen! Enough said.

#7. I have never seen web analysis without effective data/user segmentation. I think this statement is in both my books. . .  "All data in aggregate is crap." Sorry.

#8. If there is even a hint of the impact of actions being recommended then I know that is analysis. It is hard to say: I am recommending that we shift this cluster of brand keywords to broad match. It is harder to say: I am recommending. . . and that should increase revenue by $180,000 and profit by $47,000. Look for that.

#9. If you see more than three metrics in a table you are presented with then you might not be looking at analysis.

#10. Multiplicity! If you see fabulous metrics like Share of Search (competitive intelligence) or Task Completion Rate (qualitative analysis) or Message Amplification (social media) then they are good signs that the Analyst is stepping outside Omniture / WebTrends. I would still recommend looking below the surface to ensure that they are not just data pukes, but the good thing is these are smarter metrics.

User Contributions:

#11. From Carson Smith: If someone looks at your analysis / report / presentation / dashboard and has to ask "and… as a result?", then it might be reporting. What happened should be obvious.

[I love applying the "Three Layers of the So What" test to any analysis I present or see. I ask "so what" three times. If at the end of it there is no clear action to be taken then I know it is just web reporting, not matter how great it looks or how much work went into it. Ask "as a result?" or "so what?" to your work!]

#12. From Chuck U: 1) If it can be automated, it's probably not analysis 2) If your data warehouse team says they can automate it for you, then it's definitely not analysis. [#awesome! -Avinash]

Can you think of other signs? Please share your suggestions via comments. I'll add the best ones to this list.

In the list above, and in the examples in this post, you see my clear, and perhaps egregious bias for business analysis and business outcomes and business actions and working with many parts of the business and business context. But I've always believed that if you and I can't have an impact then why are we doing what we do?

I hope you've had some fun learning how to distinguish between web reporting and web analysis. It is a fact of life that we need both. The bigger the company, the more they want data pukes, sorry, reporting.

But if you have "Analyst" in your job title then you perhaps now have a stronger idea of what is expected of you to earn that title. If you have hired a "web analysis consultant" and are paying them big Rupees then you know what to expect from them. Don't settle for data pukes, push them harder. Apply the rules above. Send their "analysis" back. Ask for more. Raise your expectations!!

I hope now "you'll know it when you see it," and have more datagasms!

Okay, it's your turn now.

How would you answer the question about the difference between web reporting and web analysis? What signs do you look for when evaluating the work of your Analyst or Consultants?

Please share your thoughts via comments below.

Thanks.

PS: In case you are curious here's the current official definition of po rn, as outlined in Miller v. California:

(a) whether the 'average person, applying contemporary community standards' would find that the work, taken as a whole, appeals to the prurient interest,

(b) whether the work depicts or describes, in a patently offensive way, sexual conduct specifically defined by the applicable state law, and

(c) whether the work, taken as a whole, lacks serious literary, artistic, political, or scientific value.

The Difference Between Web Reporting And Web Analysis is a post from: Occam's Razor by Avinash Kaushik

It is possible to create truly data driven organizations , but for my "Guru" talk at eMetrics summit in Washington DC tried to tackle a much more solvable problem: Creating a data driven boss (or the boss's boss or the boss's boss's boss's boss or….).

My hypothesis is that not only is this a solvable problem, but that it is also a way for you to be in an environment where you can be challenged while adding value to your big / small organization.

Before we dive in I must say that it is my assumption that you actually want to do this and that you have the passion to fight the good fight. Glory and higher salary will be the obvious end rewards, but by themselves they are not motivation enough.

You have to have the passion to want to roll the big boulder uphill, here are your weapons………

Six Steps to Creating A Data Driven Boss:

# 1: Get Over Yourself

# 2: Embrace Incompleteness

Many of us come to the world of Web Analysis with experiences in traditional analytics where things can be counted to the last drop. On top of that we are classically trained to not to take risk and to only make decisions based on data we can swear on to be "accurate".

The problem is that we live in the most perfect imperfect medium in the world: the Web.

For now it is impossible to collect data perfectly. It is ugly, it is dirty, it is incomplete and no matter how hard you try it is not going to get perfect.

Yet we can't resist.

We obsess about silly things like cookie deletion rates. We go down rat holes of chasing down the last 5% of the deltas.

Not only is that effort not worth it, it is futile. Simple reason: You can't ever know what the total number of Page Views were on your site, much less every thing else.

And that's ok.

Now here is the silver lining in this dark sky. We live in the most data rich environments were we can find a ton of actionable insights.

Think about it. Taking our ads in Fortune Magazine is a completely faith based initiative. Based on the number of subscribers the magazine has you assume there will be an outcome. Or maybe we do some primary market research.

You can do much better than that on the web.

You can see exactly how many people got the "magazine", you can see how many of them read each "story", you can see the ads they were "exposed to", you can….. and that's just the basics. You can see were each person came from, what drove them to have a interaction with you, what the outcome was, did it result in a positive brand impression and so on and so forth.

Yet we obsess about the last 2% perfection, and we waste the opportunity to use 80% great stuff we have.

So make sure you are on first party cookies, tag as many campaigns as you can in a clean way, collect data and make decisions. Resist the temptation to be perfect, it is the enemy of good enough.

Embrace incompleteness and it will set you free. Both you and your boss.

# 3: Give 10% Extra

Organizations run on reports and so does your boss. They ask questions and you give spreadsheets. Then they ask for more and you automate the production of spreadsheet.

The other day someone was recommending that the only tool you should use for web analytics is excel! Then they proceeded to share best practices on how to write complex macros and reference cells and reduce the pain of creating spreadsheets.

Here is the problem with that: You were initially providing data, and now you are providing data without even having to look at it!

How is anyone going to find actionable insights?

Your boss is fifteen steps removed from data, you are closest to it. Yet now you have become a reporting squirrel (I was going to say monkey but that sounds rude).

As the person closest to the data is it not your job to Look at it? Look and understand and make recommend?

Make a conscious choice what job you want to be in: Reporting Squirrel or Analysis Ninja ?

[Bosses: When you pay people / consultants to do "data work" also make this conscious choice - are you paying the consultant to be a RS or a AN? Don't hire a Squirrel expect them to do a Ninja's job.]

Next time Your boss asks for a report, give them 10% extra.

You do that by actually looking at the data. Stare at the table. Go visit the website, click around, experience it, then go back to the data, connect the dots that only you can because you are the smartest person in the room.

Now give your boss 10% extra: Your insights that your boss did not ask for.

Make a recommendation. Tell them what is working. Tell them what is broken. Tell them that xxx or yyy is a better metric to answer the question.

You create a data driven boss by giving them something they can drive from data. Not by giving them spreadsheets or reports or only what they want.

At the end of each week ask yourself, did you give your required 10%? 10% extra is all it takes.

Got Ninja?

# 4: Become A Marketer

I find that great Analysts are not simply "data people". They are "customer people".

Yes they have all the qualities that we have talked about before (critical thinkers , curious, common sense, etc). But a delightful quality I have found is that are Marketers.

The reason is that different parts of the organizations care about different things but Marketing cares about the business with a very unique perspective.

If you want to change your boss and your company then you'll have to become a Marketer, someone with an understanding marketing principles, someone who can be a customer advocate / champion, someone who can evangelize the purpose of data in creating customer centric decisions.

Sales cares for selling, IT cares about keeping servers and sites up, Engineering cares about building things that can hopefully be monetized, Marketing cares about customers and, most of the time, they care about a longer term success and not simply meeting this month's quota.

No matter what organization you are a part of, you have to become a Marketer. Think like a Marketer and execute with that mindset.

Your job is to "market" your data in unique and innovative ways that solve for the customer. Get an understanding of marketing.

Take a course in marketing at the local university, read up on it (I subscribe to Seth's RSS feed), partner with the Marketers in your company and absorb.

Your boss will love you. Your career will soar.

# 5: Business In The Service Of Data. Not.

Lots of companies are data rich and tools "richer". In fact in many of them extensive data efforts to mine the logs and extract and xml and data warehouse it and mix it and merge it and clean it and build for scale and BI it and……..

Nothing.

No insights come out.

In the obsession about capturing, processing, storing, moving, shaking, baking data the core reason for doing all that is forgotten.

When the question is asked, rarely : What has all this complexity delivered for the company? The answer usually is: We have lots of reports and know up to the moment exactly what is going on via our blackberrys.

Classic sign of a ecosystem were the business exists to produce data to employ people to do all of the above.

The business does not exist to produce data. Doh!

Data should exist to serve the needs to the business and provide insights that can be actioned. Get that mindset if you want to change your management's mindset about how decisions should be made.

Do an inventory, ask around, how many decisions have been made based on data that can be traced directly to have added value to the bottom line revenue numbers? [Bosses: Great filter to apply for Consultants you hire as well, ask them that last question.]

It is important, nay critical, to constantly check yourself and ask if the business is really serving the needs of data or vice a versa.

So what does this mean?

When you undertake data projects apply this advice: Do small, deliver in a month, measure if it had an impact on the bottomline (even if small). If yes continue to invest more. If not dump it, time to do something new.

Traditional IT projects tend to be long multi year undertakings that used to deliver in the traditional worlds. That does not work on the web.

On the web things happen too fast, they get complex too fast, and every data project you undertake starts to decay almost immediately. Embrace speed and flexibility and 80% good enough. Implement, measure value, if yes move forward, if not kill.



We don't kill enough, we plan too far, we "implement" for too long, we don't think smart and move fast . Change.

Data driven decisions are not made when you spend 95% of your time in "getting" data rather than analyzing what little (or a lot) you have. You want a data driven boss? Spend 80% of your time analyzing data and producing insights.

# 6: Adopt A New Mindset, Expand Horizons: Web Analytics 2.0.

One final bonus tip. Expand the data you use to make decisions, move away from clickstream clickstream clickstream all the time.

ClickStream data is good at the What. We have tortured it to find insights. We have done the best we can with just knowing the clicks. It has worked ok, but it has not done spectacularly. Hence we have tried to take it to the next level by adding a bunch more clicks together and making it more complex. That is not stuck at all.

The other problem with a clickstream only strategy is that your bosses don't get it. There is lots of confusion, still, about Visits and Visitors and Unique Visitors and Sessions and…. So when you and I add / divide / subtract / multiply our clicks and page views and present analysis it does not have quite the impact we want, because at some level our bosses still don't get it.

That's quite ok.

Web Analytics 2.0 is your friend.

There is one important reason for that: with Web Analytics 2.0 you are talking your boss's language, that of Customer Voices and Competitors and, get it (!), Money!

Most web analysts focus on analysis with Omniture and WebTrends and Visual Sciences and HBX and Google Analytics and Coremetrics etc.

If you are one of those consider expanding your skills and experience to understanding and executing Surveys and Remote Usability Testing and A/B – Multivariate and Testing and competitive intelligence and so on.

Doing that will mean that you can represent the customer voice back to your bosses with qualitative data. It will mean that you can fight the HiPPO driven opinions with data beyond clicks. It will mean that you can kindle a small fighting fire in your boss by showing how your competitors are doing.

You boss might not care about clicks, but you bet your bottom she/he cares about customers and competitors.

As you create your own execution strategies do a Web Analytics 2.0 checklist. How many cylinders (above) are you firing on?

Can you believe that I did the above presentation in 12 minutes with eight slides? :) [If you were there share your feedback! Looooong twelve mins?]

Creating a data driven boss is not difficult. All it takes is some or most of the above six things. I wish you all the very best, may your days be brighter and your bosses more data driven!

Ok now it's your turn.

Please share your perspectives, critique, additions, subtractions, bouquets and brickbats via comments. Thank you.

[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]

Six Rules For Creating A Data Driven Boss! is a post from: Occam's Razor by Avinash Kaushik

Ask a famous blogger, a published author, a random twitterer or your mom how to succeed in web analytics, or how not to be a Reporting Squirrel. The answer will invariably be:

Before you provide the data, ask the requestor what is the business question they are trying to answer. Then fulfill that need.

It is a good answer.

Most of the time they, Marketers /bosses /HiPPO's, ask and we puke data out.

The result is also almost always the same.

After a while of doing this you, poor Squirrel, want to jump off a building. As if that was not painful enough, during the course of your employment the company made no actual decisions based on web analytics data. Ouch.

I call it a lose-lose.

If the outcome is so obvious. . .  why do we still have this problem? Why is it that we don't ask for the magical thing you were told to ask since the day you were born into this world? Business questions.

Part of it is that we might not be in the position to ask for that question (I don't buy this 80% of the time, sorry).

Part of it is the case that we don't often understand the difference between a business question and a report request. Even for experienced Analysts / Consultants.

Let' us solve this problem today.

I really want you to become an amazing Analysis Ninja and help ensure that this issue never comes back to bite you in the butt.

Come with me.

Why Ask For Business Questions?

One word: Context.

I am sure this will not surprise you, but it turns out you are a very unique person. You are distinct from all your other siblings, even your twin.

And it turns out every business is unique, every website is unique. There is one and only one of its kind in the world.

Even if you completely copy and paste someone else's website and start executing, your business is probably unique in terms of the individuals you have collected around you and how they work together.

Or perhaps while you sell via the retail channel like everyone else, your strategy is different in its focus on driving purchases through the web, or maybe you are obsessed only about offline store sales. Or perhaps while you have 100% copy pasted GroupOn's business and are executing it in Russia. It is 100% likely that you are solving the problem completely differently than the US GroupOn site you are copying.

Or maybe you are HP and are obsessed and 100% focused on solving for Customer Satisfaction on your eCommerce site and your closest competitor Dell is obsessed 100% on getting more and more conversions.

This is a long preamble to tell you that the strategy we normally execute in measurement is wrong. We read books / blogs on Metrics & KPIs and we think we know what to do for an ecommerce site or a blog or support site etc and we start reporting what the blog / book recommended.

Then we are shocked no one cares one bit about the data.

If you are unique, why should you crack open a standard analytics tool with its standard reports and metrics and get going?

Or why simply respond to a "report request" and start data puking? The person at the other end is probably uninformed about Analytics and Segmentation and what is possible (even as they are supremely qualified to do their job in Marketing / Sales / HR).

You need business questions because

1. Rather than being told what metrics or dimensions to deliver you want business context: What's driving the request for that data? What is the answer the requestor looking for? Then you apply smarts because you have context.

2. Best practices are highly overrated. If this is your first day on the job, sure go ahead and puke out what "industry experts" recommend. But know that it won't impress anyone because you don't actually know what the business is doing / cares about / is prioritizing.

Convinced you need to only accept business questions?

I am glad.

Three Attributes of Business Questions.

We have done a pinky swear that you are going to start your daily web analytics journey by asking the business what questions they want answered.

I have to warn my budding Analysis Ninja that what you will typically hear is:

+ I want to know how much traffic is coming to our website

+ I want a conversion rate

+ I want a path analysis for our visitors (oy vey!)

+ I want to the list of top exit pages on our website

+ I want to know how many leads we got on website this month

+ Or give me a report that shows click-through rates of our home page promotions

All of these are requests I am imploring you to rebel against. They are not business questions.

Take a deep breath. Simple. With your eyes, not your mouth, say: "I really want to help you."

At this point what we really want to do is refocus the discussion and increase the likelihood that you can be something more than a reporting squirrel. With a twinkle in your eyes politely say:

We are executing a true Web Analytics 2.0 and Multiplicity in our company. In as much as we have access to many sources of data that can help answer What, Why and What Else questions quickly and efficiently.

Based on my expertise I can help you pick the right tool and metric if you could share the question you are trying to answer. What problem are you are trying to solve?

That will probably take all of 20 seconds to say (don't forget to twinkle your eyes).

They'll be struck by your sincerity, and shocked that you want to help this much.

The trick now is to make sure that you are able to recognize if what comes out of their mouth (or in words over email) is actually a business question and not a rephrased report request.

Business questions have these three simple characteristics:

#1. They are usually open-ended and at a much higher level, leaving you room to think and add value.

#2. They likely require you to go outside your current systems and sources to look for data and guidance in order to measure success.

#3. They rarely include columns and rows into which you can plunk data you already have.

If what you hear fails any of the above tests then you have to go back and try again.

Try not to be impatient or show off how smart you are or pick a fight. Keep the twinkle in your eyes, highlight what was different about what they said compared to the first time around, and then gently ask them a specific follow up question.

If you read the three characteristics carefully you'll notice that they are encouraging the best from your requestor (context, business priorities, problem framing), while at the same time encouraging the best from you (your knowledge of data, systems, analytical strategies).

That is the basis for the magic that converts you and me from data puking Reporting Squirrels to Analysis Ninjas who leverage custom reporting, advanced segmentation, statistics, surveys, competitive intelligence tools and to deliver reports with specific insights.

Guess what's the result?

People who just wanted data are now running around taking action based on your insights. Why? Because you did not provide data. You answered questions that were important to the person, tied to business priorities.

In rare cases your requestors might not even know how to reply when you ask them for business questions. Let me send you off with a little gift for the times when that will happen. . . .

Sample Business Questions for Ninjas & HiPPO's.

Look the person in the eye, still with the twinkle, don't forget the twinkle, and say:

My dear friend Avinash Kaushik asked me to share these sample questions with you. He said it would help us identify what's most important for our business.

If you hear the questions below you've hit the jackpot, because these are questions, have the three characteristics outlined above. . . .

• How can I improve revenue by 15 percent in the next three months from our website?

• What are the most productive inbound traffic streams and which sources are we missing?

• Have we become better at allowing our customers to solve their problems via self- help on the website rather than our customers feeling like they have to call us?

• What is the impact of our website on our phone channel?

• How can I increase the number of customer evangelists by leveraging our website?

• What are the most influential buckets of content on our website?

• If we could only do one thing to increase revenue on our website what would it be?

• What is the incremental impact of our display ad campaigns?

• Are we building brand value via activity on our website?

• Do fully featured trials or Flash demos work better on the website?

• What are the top five problems our customers face on our website?

• What is the cost for us to earn $1.00 on our website?

• What is the effect of our website on our offline sales?

Open ended. Force you to go to new sources. Don't contain columns and rows.

That's when you know you are on the right track.

It is quite likely that you will run into other business questions that might be more pertinent in your case.

But the theme that you are looking for is tough, highest-level business problems that you can help solve by analyzing the data you have (or data you don't have but will figure out how to get).

These are the questions that give you goose bumps. These are questions that give you a drugless high. These are questions that validate your decision to come into this field. These are questions that get you out of bed in the morning and feel excited to be alive. (Ok so maybe that is just me. But you'll see what I mean!)

Questions that rock your world, and coincidentally make for a truly data-driven org.

Parting Words of Wisdom from a Practitioner.

For the longest time in the web analytics world we have been content to do one of two things:

1. Provide the data we have in our applications in the hope that in the deluge of visitors, page views, referring URLs, time on site, and exit pages, there is something that marketers and business stakeholders will find of interest and take action on.

2. (Eager to please.) Take requests for reports, create them, and figure out how to email them or publish them on the intranet.

Reality is rather messier as a result of this.

Business leaders feels frustrated that they are not getting insights that they can act on. On the other hand this can't be fun for you. It can't be easy for you to hold the title of Senior Web Analyst and you are reduced to running reports.

Hence the most important foundational element of any effective web analytics program is to ask real business questions, understand those business questions, and have the freedom to do what it takes to find answers to those questions by using Web Analytics 2.0 strategies.

So. . . .

If you are the Business Honcho, bare your soul and share the questions (samples above) that keep you up at night or the priorities that you think are required to go out and win against your competitors (again these are not reports you want).

If you are the underling, seek to get a peek into the said soul and understand the strategic questions that the business wants answered. When you learn what the questions are, go get answers, one at a time.  You will now be on your way to truly adding value to your company.

If you are a powerless underling, provide the reports, puke the data being asked, play the useless non value added game that you are being asked to play. But all the while know that real glory lies some place else. Be on a question quest at every given opportunity, with a twinkle in your eye.

Identifying business questions is a journey.

As you solve one set, the next will come up. Or you may be in the middle of solving one set, and suddenly that set will become irrelevant and there will be a new set.

This evolution and change is a sign that you are actually answering business questions and not just doing reporting, because business is always evolving and changing and you have to simply learn to change with it.

From the bottom of my heart I wish you all the very best.

Carpe diem!

Ok now its your turn.

How do you spend most of your time at your job or with your client? Providing data rich columns and rows and reports with pretty font, or answering business questions? If you have tried this strategy already what are the biggest barriers to you being asked questions rather than data? If a strategy has worked very well for you in this context what is it? Twinkle in your eyes? [I knew it! :)]

Please share your experience / feedback / pain / joy.

Thanks.

Rebel! Refuse Report Requests. Only Answer Business Questions, FTW. is a post from: Occam's Razor by Avinash Kaushik

Why just measure conversions as one purchase or conversions just as a submission of a lead or opening of an account on facebook / twitter / what ever?

Why not measure Visitor behavior after that first purchase / lead / membership sign up (or the first super poke)?

Why obsess about the "quickie" that is opening an account? Why not the Visitor behavior in the 30 days after sign up?

For your Search campaigns.

Or your Display ads. Or Affiliate programs?

If you do that, you are not just measuring the "one night stand quickie" what you are measuring is something more of value: Was there a second date? Perhaps a proposal in five months? Maybe a marriage in nine months (because that's when the baby showed up). : )

We do this so rarely.

Success for campaigns (Search, Social Media, Display, TV and what not) is always measured based on that one visit or what happens when the campaign runs.

Yet it is likely that you are measuring incomplete success.

Let's do an example.

Facebook, in its quest for world domination, runs a world wide campaign (of whatever sort, really, make one up).

Success is to increase the membership on Facebook and thus making it a more "valuable" property (so that even in a weaker economy they can raise another billion dollars of cash).

So they get another 500k memberships.

Success?

How about this….

Rather than measuring membership sign ups why not wait 30 days and look at the Recency report for Visitors that came to the site as a result of that campaign?

Did that just blow your mind? : )

No?

Ok, what you are measuring in the case of Recency is not simply people who signed up BUT rather the behavior of the visitors who signed up.

You are measuring if people you acquired from this campaign are visiting the site frequently lot in this 30 day time period after the campaign launched.

My hypothesis being that simply signing up is not enough. That's not enough success (not in this economy baby!). Real monetizable success is:

1) Do those people visit facebook again? And this is key…
2) They visit every day, or every other day, or every few days.

The more frequently people visit facebook.com, the more valuable they are to facebook. They see more ads, they super poke more, they friend more, they add more apps, and so on and so forth.

All activities (sexy translation: "visitor behavior") that is multiple multiple times more valuable then simply signing up.

When you look at campaigns not just from a "quickie" view (did they sign up?) you are not sure if you really added value to your business.

This is the kind of analysis you'll do. . . .

And here's a specific example of what you are looking for:

Campaign 1: 100k sign ups. "Top Bracket" Recency: 10% (i.e A month after the campaign 10k of these Visitors visited in the last 23 hrs on the 31st day.)

Campaign 2: 50k sign ups. "Top Bracket" Recency: 50% (i.e A month after the campaign 25k of these Visitors visited in the last 23 hrs on the 31st day!.)

Which one is more valuable?

With the "quickie" method it's campaign #1.

With the "post quickie" method (visitor behavior) it's campaign #2.

Super Sweetness: Combine looking at the Recency report with the Loyalty (Frequency) report (below) and you have a gold mine of insights. You'll now how exactly how many times Campaign 1 and Campaign 2 Visitors visited. That's helps your truly find your BFF campaigns.

Now the reality is most of us will follow the traditional "lets make sure we measure conversions" path and rarely do this.

Usually because we don't know that free or paid tools now allow you to do this reporting standard off the bad. AND that they are so easy to use that even my 4 year old son can do these segmented views (while, and I am proud of Chirag for this, remembering to filter "New Visits" out!).

This is one other hidden reason: Marketers are impatient. They want to know now what happened so they can collect their bonus (not really their fault, its how the compensation works).

But if you as HIPPO's and Leaders put the right structures in place to incentivise the right behavior and then encourage the right measurement than Queen Success will be yours!

If you buy into measuring Visitor Behavior then another fantastic metric to measure your campaign "conversions" is Visitor Loyalty.

Your display campaigns are purely for a "branding" impact and the measure of success is nothing.

Ok just kidding (am I? :)).

Your display campaigns are geared towards driving people to your brand and not bounce. Great. Measure that.

But also measure this, do these campaigns results in a increased likelihood that people will come visit your site again? And which display campaigns (on Yahoo or New York Times) result in Visitors that after a 90 day period come again and again and again?

Campaign 1:

Or "glorious" Campaign # 2:

Clearly New York Times display campaigns (#2 above, as an example :) are terrible and should be abandoned because they are not resulting in a longer term relationship with the acquired Visitors.

This is what I mean by measuring "latent conversions" (by that I mean using visitor behavior as a "conversion").

More marching orders:

• If you are a Social Media website (with no discernable outcomes) don't use "membership acquisition" as a measure of success.

  
  Show your investors and potential advertisers that your property is driving the kind of "sticky" / "engaging" / "viral" / "insert buzzword here" actual visitor behavior. They will love you more.
  

• If you are a non-profit like NTEN (the "craigslist of non-profits") then use Visitor Loyalty metrics to measure success of your AdWords campaigns.
  

• Even if you are a ecommerce website take the data from your web analytics tool (not easy with Google Analtyics at the moment, maybe with the API, but you have Omniture so you should be doing this already) and measure not just the first purchase based on campaign traffic.

  
  Look back 90 days and see which campaigns drove more repeat purchases. Value those campaigns higher, even if initial conversion was lower.

And more stuff like this (add your examples in comments).

Any Reporting Squirrel can report instant conversions. Only a true Analysis Ninja will do this kind of "post quickie" analysis and help the business make thoughtful and cost saving intelligent decisions.

Good things come to those who are patient, and those who focus on Visitor Behavior.

Ok your turn now.

Do you measure "latent conversions"? How about Visitor Behavior as a success? Especially pan session visitor behavior? Have I missed any other examples of analyses that can be done (it is late in the night)?

Please share your thoughts and feedback via comments.

[NOTE: I am using a new new wordpress plugin that will allow you to edit your comment for a couple hours after it has been posted. In case you make a mistake.]

PS:
Couple other related posts you might find interesting:

• A Primer On Web Analytics Visitor Tracking Cookies
• Standard Metrics Revisited: #5 : Conversion / ROI Attribution
• Eight Rules for Choosing Web Analytics Key Performance Indicators
• Excellent Analytics Tip#6: Measure Days & Visits to Purchase

Excellent Analytics Tip #15: Measure Latent Conversions & Visitor Behavior is a post from: Occam's Razor by Avinash Kaushik

"I have done web analytics for five years, I have mastered Omniture, WebTrends and Google Analytics, I provide analysis and not just reporting. I feel like am an Analytics God.

What would be your advice for me in terms of next steps for my career? My goal is to climb the ranks and increase my salary."

Let me hasten to add two things.

Michael is not his real name.

Modesty aside, :), Michael is good at what he does.

I get many emails in the spirit of this one and thought it was about time I wrote a proper post about it.

Another reason for writing the post now is that it is always a good time to think about your career path, but never more so than the current economic circumstances. Some of you face tough times, some might get laid off [see end of this post], some might make opportunistic leaps. Either way good time to ponder, do some self reflection and make a conscious choice.

Before we get going some assumptions I am making:

Every Web Analyst (or really Business Analyst) of any sort finds themselves at that critical point. Have been doing analysis for a while, now where does my career lead me?

The first and perhaps most important thing to realize that you have to make two very important very critical very life impacting choices:

Choice 1: Business or Technical.

Choice 2: Individual Contributor or Team Leader.

Each choice will help propel your career in a different direction (slope and length). Typically we don't think that you have those choices (we all want to jump to Director / VP and get the chq, not so fast buster!).

Do some introspection.

Here's the Avinash Kaushik Web Analytics Career Introspection Guide! Answer these questions:

#3. Are your true "Analyst" skills your massive mastery of how to solve every technical problem with every tool and how to implement anything and you could decode and reconstruct the debugging tool WASP in two days? You can hack the Google Analytics tag to capture people's underwear size and color.

#4. Are your true "Analyst" skills your understand of your company's business strategy, your mastery at translating "measure something" from a VP to three Critical Few metrics that bedazzle her, your ability to understand the long tail and get a ah ha moment that revolutionizes how you understand and measure of your search campaigns?

I am getting to how you can increase your salary part. Please stick with me.

Answering the above four questions honestly and critically is much harder than you think. Trust me on that.

At Intuit I truly learned the value of self awareness. Steve Bennett prioritized Management development (I'll be eternally grateful to him for that) and my friend Scott Wilder really got me going on this path by doing my Enneagram assessment (MBTI is too shallow).

To me self awareness is the process of figuring out what you are truly good at, and really truly knowing (and accepting) what you are not good at. It takes time to get going, and is more of a lifetime journey and less a destination. It comes with great benefits. For example, I have learnt to maximize my being in roles / situations where my strengths will boost success.

Your answers to the four questions will ensure that you don't end up in a job that 1) you'll hate or 2) where very quickly you'll rise to your level of incompetence.

There are jobs, based on your choices above, either in companies or, many people forget this, at web analytics vendors.

If you want to have a move your career forward in web analytics (from a Metrics Analyst) here are the four options for you (and yes they all will help you make more money, some more than others):

|1| Technical Individual Contributor.

A lot of people wrongly believe that to make lots of money you have to get into Management (technical or business). This is totally wrong. For the longest time, for example, I was well compensated for being a Senior Individual Contributor.

Roles in this category would include: Sr. Project Manager. Sr. Architect. Implementation God. Sr. Tech Lead. Tech Demo God (usually at a Vendor). And more.

For this role at a client (company) world the common theme in this role is that you report up to a Director or a VP and you get to set policy, rules and regulation, only have, if at all, the barest of dotted line responsibility for project implementations, you might be the master liaison with the business team and your vendor to make sure technically all that is supposed to be happening with the technical tag hacking and tool hacking is happening.

This role in the vendor world you get to go to various clients and show off cool detailed stuff that your VP of Marketing consistently screwed up so far and answer technical questions from wise guys. You might also be the one man army tapped to do rapid prototyping to prove you are better than Google Analytics (!), or it is likely that you are the point of contact for the first sixty days for a new client when your company is trying to impress the client by providing fast help (as the payment chq has not yet cleared). Make no mistake this can be fun, you get to travel, meet new companies and people.

It is quite likely that you'll sit in the IT (CTO / CIO) function.

Career Prospects:

$$$ Prospects:

Long Term Job Title Growth:

|2| Business Individual Contributor.

If you are a Analyst today you are in a individual contributor role on the business side (if you are a Web Analyst in IT the best career move you could make for yourself is to get moved to Marketing – or a business function, it is really hard to have a strong Analyst – not reporting squirrel – role in IT).

Roles for you on the business side in this category would include: Sr. Analyst. Internal Evangelist. SPOC for CMO / CEO dashboards (supreme analysis). Central Business Liaison (for a large business, focus on getting people to implement web analytics and get going). Strategic Solutions Consultant (clearly with hype like that this a role at a vendor!). Product Genius (at a vendor, perhaps Apple :).

For this role at a company (client) the role can report to anyone from a Director to the CMO. Your job is heavily business focused – understand various businesses and their strategy and provide über analysis (pan business function) or create dashboards or be in charge of rolling Omniture across 90 business sites (beat them up until accomplished).

There are a rare few roles where you can become the internal Analytics Champion, I did this for a while. You are good at your analytics "game" but you are also a strong business person (I hate to say this but MBA / "strategic" type). You get to go around and work with VP's, CMO's and Sr. Leaders and identify measurements strategies for their impossible to answer questions (often they don't know how ease these are so you totally look like a hero). As an Evangelist you pull your organization up by the bootstraps (quite gratifying).

This role in a vendor world can mean you are a product manager of the analytics product, you are a project manager for certain features, you are a professional services rep (sorry, "Strategic Solutions Consultant") and roles like that. No one does Marketing (with a pinch of hype :) like Omniture, this excellent page on their site will help you understand what a business individual contributor role might look like at any vendor: Omniture Consulting.

For many of you there is also a option of a role I play now, an Evangelist. A business individual contributor role with significant influence over the vision and the product, as well as an opportunity to impact the external Analytics ecosystem.

Career Prospects:

$$$ Prospects:

Long Term Job Title Growth:

|3| Technical Team Leader.

Roles in this category would include: Manager, Analytics Implementation. Sr. Manager, Website Analytics. Group Manager, Web Operations Reporting. And still rare but sometimes: Manager, Web Analytics Data Warehouse (Steven I did not forget you!).

In the early genesis of web analytics, in the good old days, it was owned by IT this role was a lot more prominent in companies. Having WebTrends almost mandated that. The shift to ASP (javascript based) solutions (crediting HBX here) caused a shift of web analytics to the business side (an excellent outcome). It also eliminated the need to have a large IT staff to support web analytics.

An example is six years ago when I took over web analytics there was a four person team (one leader, three direct reports) in IT supporting just running WebTrends internally and churning 200 reports out. The shift to a asp based solution meant only one job remained and it became that of a Sr. Technical Individual Contributor (and I was lucky to have a very good one!). The other jobs were evolved or replaced with people who did analysis not reporting (an efficient use if there ever was one).

Roles in a company setting would be reporting up to Sr. Manager or Director (or rarely VP/CIO levels). Often you'll find yourself in the Business Analytics team in the CIO / CTO function, you take care of that "web data". :) In some companies there is also sometimes a role in the Web Analytics team that is in Marketing (/business) where you can carve out a nice technical team lead career (reporting to the Director of Web Research & Analytics :).

Roles at a vendor probably have a lot more technical team lead opportunities. Managing technical aspect of the analytics product or managing the technical army of consultants or things like that.

It goes without saying that this role requires something really really hard: Your ability to leave your leave your lone ranger mentality and the deep rooted habit of just doing all the technical stuff yourself (yes you are sooo good at this stuff and you don't trust other people's code).

It is harder for technically oriented people to blossom into people managers, but really that's what you are signing up to do. You are going to have to be comfortable with some of your awesome hacker skills getting rusty as your leadership skills (and delegation!) mature.

Career Prospects:

$$$ Prospects:

Long Term Job Title Growth:

|4| Business Team Leader.

When people think of making more money in web analytics jobs, 99% of the time this is the role they are thinking of. [Might I just quickly again encourage you to use the Avinash Kaushik Web Analytics Career Introspection Guide first.]

Roles in this category include: Sr. Manager, Web Analytics. Director, Web Research & Analytics. Manager, Web Metrics. Team Lead, CoreMetrics Reporting. Group Manager, Analytics & Optimization. Etc.

This role in a company setting is increasingly reporting to Sr. Directors, VP's or, in companies that get it, the CMO. Ideal candidates were Analysis Ninja's of supreme kind and have shown streaks of good people leadership skills. They are motivators, can inspire confidence, are inherently unselfish (key for leading people) and have the ability to charm the pants of Sr. Management (though come to think of it that might be a HR violation!).

I cannot overstate this enough: Ideally you have grown from a Reporting Squirrel to an Analysis Ninja, but your hard core technical skills are vastly overrated in this role. As is your ability to be, say, a Excel Master Blackbelt. Remember, you inspire and you lead.

This role, in a pure web analytics leader fashion, is a lot less needed or visible or available in a Vendor setting (unlike the other three above). Vendors need to mostly sell. Perhaps to analyze nedstat.com or unica.com for internal use. Or perhaps as a business lead for the $300 per hour consulting arm.

Career Prospects:

$$$ Prospects:

Long Term Job Title Growth:

That's it. Four different job families. Each unique in its job, salary and future prospects.

My fondest hope is that as you evaluate your career that you'll now be empowered beyond the normal job stuff that you often read. What you have above is the output of my humble experience in multiple multiple roles working as a Practitioner, Author, Evangelist, People Leader.

There is one job / role / career choice I have not covered here. Becoming a Consultant. Perhaps another day. For now let me share just this advice, the most common reason for Web Analytics Consultants failing (or not succeeding as much as they should) is that they believe technical skills are enough or just being good at business (analysis / understanding / savvy) is enough. It is not. It is hard to find both is one place as well. If you want to do that find a partner. You be the strongest Analyst on earth and let her be the Technical Goddess. Now you are set for greatness.

Before I end, I had promised something about being laid off in a down turn.

My Story.

I have had three professional jobs in the US. I have resigned from only one, the last one. The first two jobs (4.5 yrs and 2 yrs respectively) I was laid off.

The first in worst of personal financial times (market going down, first baby on the way etc). It was deeply stressful.

In hindsight though each layoff was the best thing that could have happened. Allowed me to start fresh. Each bumped my career trajectory in ways that would never have happened if I had stayed at the job.

I hope your job is secure. But if it is not I hope you take some inspiration in my humble experience of being laid off twice from good jobs (that I was good at) and the longer term results.

Ok your turn now.

What has worked in managing your own web analytics career? Anything above in my Web Analytics Career Introspection Guide resonates with you? What about the four job families? What did I not consider or get wrong?

Please share your stories, feedback and encouragement.

I appreciate your attention, thank you.

PS:
Couple other related posts you might find interesting:

• Nine Rules To Work / Live By
• Three “spire’s” of Great Leadership
• Multiplicity: Succeed Awesomely At Web Analytics 2.0!
• How To Excite People About Web Analytics: Five Tips.
• Web Analysis: In-house or Out-sourced or Something Else?
• Make a Great Vendor / Agency / Consulting Pitch – Win Big Contracts

Analytics Career Advice: Job Titles, Salaries, Technical & Business Roles is a post from: Occam's Razor by Avinash Kaushik

One is in the business of providing data.

One is in the business of providing, to use a old fashioned word, information.

This one of the core reasons why most dashboards are "crappy", i.e. they are data pukes that provide little in terms of context and even less in terms of actionable value.

Here are some examples of sub optimal dashboards, sub optimal in my mind from a actionable perspective. . . .

Perhaps the most common type is above. Lots of data, even drill downs included, but you can't look at it and go: "Wow we need to do . . . ". No sirrie bob you can't.

I wanted to point the above out purely because of a common feature of 80% of Web Analytics Dashboards, in excel with a billion tabs to look through. This is not a dashboard, it is the result of a massive sum of money paid to a Consultant who is trying to impress you with his / her excel skills – without actually telling you anything.

You are walking down the street. You look at someone from behind and you think "hmmm she's / he's pretty". So you speed up and overtake them and in the process you sneak a glance at them (yes you are married but looking is still ok :), and you are hugely disappointed. Not pretty. That's the dashboard above. Very sexy and Web 2.0'fied and a ton of data there, but a lot less actionable than you might have hoped.

Why is this so? All the above efforts are well intentioned, took lots of honest work and probably took months to put together. So why?

Here are some hidden (corrosive) reasons why most dashboards tend to stink when it comes to helping the Executive make any decisions:

1. They leave the interpretation to the Executive (/ customer / requestor / other Squirrels). This is a fatal flaw because most dashboards are highly aggregated views of any KPI and are missing all the nuance and analysis (that only you as Ms. Ninja have, and you don't go with dashboard).
2. Most Executives actually want insights / action recommendations but they don't trust the Squirrels / Ninjas / VP's / Data Providers. So they ask for numbers. We dutifully cram as many of them on to a A4 size paper in 3 size font and send it along with a magnifying glass.
3. Most Squirrels / Ninjas live in a silo. Going out to collect enough tribal knowledge to actually know what is going on to then make recommendations from the data is not something that we do, nor are we encouraged by our Executives or our organization structures. This incentivizes data pukeing.
4. Often dashboard creators tend to be "outsiders" (Consultants, Experts etc) and they often don't have deep practitioner experience that would allow them to understand the human / "below the surface" issues like the above three. That leads non-Practitioners to make the common mistakes like creating the above three dashboards.

If you want your Executives / Customers to take action, you have to give them information and not data. It takes effort to get there, it will take all your charms (though no violation of any HR intimacy policies), and it will take some time.

Step one as always is to become aware of the above three problems.

Step two is to get a possible solution from the Occam's Razor blog. :)

My attempt at solving this problem was to try and attack it from a human psychology perspective: How can I create a "dashboard" that will incent the right behavior from the Squirrels / Ninjas while giving Executives the information they need to make decisions (rather than engaging in a bitchfest which is the typical outcome).

Recommendation #1 was to move to a Critical Few philosophy for Executive reporting: Only report the three or five (at most!) metrics that define success for the whole business. Kill all the ancillary metrics that were nice to know (and my kill I mean let lower levels worry about it).

Recommendation #2 was my humble, admittedly ugly, attempt at a "Action Dashboard":

4Q. (Sorry Jonathan! :)

Each quadrant representing a solution to a human problem that lead to crappy dashboards.
(Apologies for having to redact some of the data above, to protect the innocent.)

Let me walk you through it.

First very up top a clear identification of what the Critical Few metric was, who was responsible for that metric from a business perspective (translate into "head on the line") and who was responsible for the analysis.

Also note the little red dot. That here indicated trouble. It can have two other colors, yellow for don't fire anyone yet but get ready and green for send someone a big hug and a box of chocolates. Next. . . .

The first quadrant (the graphic) shows the trend for the metric. Ideally segmented (as is the case here, cart abandonment is illustrated for four key customer segments).

This quadrant is to satiate Executive curiosity that you know what you are doing, it will be glossed over (and that's good!).

The second quadrant (Key Trends & Insights) is to add value by interpreting the trends and adding context. It says there that some things are up or down (in english :), and it also warns which data might be bad etc. You are starting to do your job here.

This quadrant is the one that Executives will read a lot initially, over time they will gain confidence in you, they will love that you share context (hello Ninja!), over time they will gloss over it (a good thing).

The third quadrant, clockwise, (Actions / Steps To Take) is force the shy Web Analyst to get out and talk to Marketers, Website Owners, VP's, Whomever it takes to get all the tribal knowledge, identify root cause for the trends in the metric and recommend solid action to take. The Analyst will rarely be able to do this by themselves. It will require human contact with others, it will require conversations, it will mean identifying solutions collaboratively. It is a fantastic opportunity to become smart about the business.

This quadrant is key to driving action. No longer do you leave things to interpretation or let's blame people etc. You are recommending what actually needs to get done. Your Executives will kiss you and over time this is the only quadrant they'll read. It will also mean that monthly meetings will move from bitch fests to deciding who does what. Amen!

The fourth quadrant, (Impact on Company/Customer) exists in case it is not clear to the Executives why they need to take action (listen to poor old you the lowly Analyst). I feel it is the key thing missing from any dashboard, they are normally missing the kick in the rear end and this quadrant delivers it. It is the answer to this question: "As a result of this trend (up or down) what was the impact on the company and its customers". It also forces you, Marketer / Analyst, to do hard work to estimate the impact and put it on paper.

This is the killer quadrant, if nothing else drives action this will, knowing exactly how much money was lost, how many customers were pissed, how much opportunity was wasted. Now when they ignore you they do that at their own peril and with their butt on the line. Trust me action you recommend will be taken.

See how simple it is?

You fix the human problems, you address the flaws in the system today and you actually become much smarter about the whole business (thanks to q3 and q4).

Win – win – win.

Over time you'll gain a lot more trust from your Executives and all the crappy dashboards can die and be replaced with one that looks like this one. . . .

Now you are asking your Executives to simply layer their own judgment on the recommendations and help the company take action. Who needs to see the numbers? They pay you and I to deliver actionable insights.

I stress that it won't happen overnight, but shoot for that nirvana state.

May the force be with you.

Ok now your turn. Care to share your own learnings and battle scars? Your success stories? Perhaps critique my "Action Dashboard" (sorry could not think of a better name, do you have suggestions?). Your perspectives are most welcome and would be greatly appreciated. Thank you.

PS:
Couple other related posts you might find interesting:

• Five Rules for High Impact Web Analytics Dashboards
• Web Analysis: In-house or Out-sourced or Something Else?
• How Should Web Analysts Spend Their Day?
• Seven Steps to Creating a Data Driven Decision Making Culture.
• Six Data Visualizations That Rock!

The "Action Dashboard" (An Alternative To Crappy Dashboards) is a post from: Occam's Razor by Avinash Kaushik

Measuring top line success of ecommerce websites is not very complicated, all the sweet revenue based outcomes are there (at the least).

Measuring non-profit websites is a bit complicated, but not really all that hard because we can, with a small amount of love, figure out outcomes to focus on (donations, # of sign-ups for the protest in DC, # of petitions signed, volunteer applications, etc).

Measuring government websites is a bit more complicated, if for no other reason than that it takes a pinch of effort with a dash of imagination to figure out what one is solving for. What are the desirable outcomes one can focus on to measure success?

The above question came to mind from a kind note I got from Ines Jans who is a part of the team that is responsible for www.belgium.be

Ines and team were just starting to think about analytics (because the love their customers!) and asked for some thoughts.

My first question to Ines was, (surprise!):

Q: Tell me a bit more about what your site does, like what are the real goals (or give me some ideas about it) and what challenges you face, what do you expect people to get out of it?

[Best Practice: Always, always, always start any measurement conversation with the above inquiry. The answer will be key to insights, and without it you'll simply be a glorified Reporting Squirrel.]

The answer, which might fit most government websites was:

A: The goal of our site is to be a portal to all the official information there is about Belgium and make information easy to find. Visitors should be able to figure out which Ministry is responsible for what tasks.

[Best Practice: Don't be surprised in your Analysis Ninja quest if you get answers that just start the conversation, rather then give you a prescription for what you need. Squirrels will despair here, but Ninjas will take clues from what they hear and visit the site and come up with a set of important measurable outcomes.]

Based on the answer above and some time spent on the English language site as well as those in other languages (Google Translate!), I came up with the following five questions I could ask data to measure success.

Let's take each of these questions one at a time and figure out the best way to answer each using a true Web Analytics 2.0 strategy.

Q1. Are Visitors able to find the information they are looking for?

Given the singular purpose in life of this government website is to be the one stop shop for all the information one could possibly need, it should be pretty obvious that the very first, and magical, thing we would measure is if Visitors to the site are able to find what they might be looking for.

So would you use a web analytics tool?

Here's the first surprise: No!

There are certainly tertiary ways in which you can answer this question using Omniture's Site Catalyst or Google Analytics or other wonderful web analytics tools.

But the best way to answer this question?

Ask the Visitors!

Using a simple survey that pop-up on-exit (when Visitors leave the website) you can ask your customers to tell you if they were able to complete their task. No interpretation required.

4Q from iPerceptions, available in 18 languages, is a free on-exit survey you can use. If you don't want to use a external survey build your own, ask four questions, analyze the data for:

"Were you able to complete the purpose of your visit today?"

The answer to this question becomes the #1 Key Performance Indicator (KPI). You are going to watch it like a hawk, you'll post it on all your bulletin boards, you'll set up custom alerts to ensure that your team gets a small electric shock every time this number drops below 65%!

The overall number is good enough, but the data that will be awesomely actionable will be, if you use 4Q: Primary Purpose by Task Completion Rate. . .

You see the second question in 4Q is "Which of the following best describes the primary purpose of your visit?" and a standard report in 4Q will paint the above picture.

Now you not only know if people find what they are looking for, but you also know which tasks are hard to complain.

You need to fix "Complain about the French" : ) because the Visitors are already upset and only 5% are able to complete their task, resulting in them becoming even more mad!

Remember: You don't need to show the survey to everyone who comes to your site. You can sample just a small percent of your Visitors. You only need 300 responses in a month to get a statistically significant sample of data, and 1,200 if you want to do segmented analysis.

Q2. Are the Visitors satisfied with their experience?

HiPPO's (the "highest paid person's opinion") in the organization, even in the government will love to have a more direct (than task completion rate) answer to the question: Are our Visitors happy with our website?

That's were it is prudent to measure Customer Satisfaction.

4Q and other surveys of course measure that quite easily: Based on today's visit, how would you rate your site experience overall?

Measure it. Trend it. Report it. Correlate the trend over time with changes you have made to the site and identify insights (any causal connection between site improvements / campaigns and customer satisfaction?).

An alternative, or additional, way to measure satisfaction is to count and analyze the Contact Us submissions. . . .

Start with the number of submissions. Trend over time.

Drill down into the type of complaints and do atleast rudimentary sentiment analysis (i.e. read & categorize) of actual messages to gauge customer satisfaction.

Remember: When you do surveys you don't have to torture your Visitors with billions of questions! In researching this post I went to US government sites and I got a ugly 34 question on one single page looooong survey. 34 questions! Most were irrelevant. I would have answered a few, but this showed a fundamental disrespect for your customers. In the end your Visitors are upset and you suffer from a lack of data.

Only ask what you can action.

Q3. What is the most popular content on the site? What area can we prioritize higher than it currently is?

It is not unusual for content sites to produce content. It is even less unusual for them to produce content that they think potential visitors to the site might want.

What is rare is the analysis of what visitors to the site are actually consuming on the site.

Here's a simple analysis I had learned from Tim Hart (who was with the J. Paul Getty Trust): Measure the distribution of content in each section of your website and the percentage of Visits to each section.

On the y-axis is each of the sections on the belgium.be website. In blue is the amount of content in each section. In red are the percentage of visits where that content was consumed.

Is it not awesome! Insights galore!!

If this were their data, and it is not, it would be pretty obvious that there is huge interest in content about Housing and Economy tiny fraction of the site's content is about Housing and Economy.

The balance for Family is a lot less lop sided.

While the government might love Justice, Mobility and Health (and boy do they love Environment!), Visitors to the site are a lot less interested in those pieces of content.

Action? You know what people want, how about giving them more of that content?

When it comes time to prioritize the next set of web pages or videos or podcasts, how about giving higher priority to those big red lines?

Sweet right?

You can also do segmented version of this analysis, see what Visitors to English, Dutch, French and German sites prefer. Or within Family what group of content do people like. Etc etc.

Two more ideas to get into your Visitor's head. . . .

Measure Downloads:

There are a ton of downloads (pdf's mostly) on the belgium.be website. Forms, applications, useful guides (like how to marry a belgian or how to prepare for your first job) etc.

It is a trivial cost, analytically, to track these downloads using your web analytics tool. Do it. Measure what your Visitors are most interested in.

Yes, yes, yes I see my technical squirrel friends raising their hands and saying you can only track that someone clicked on the download link and not that the download was successful. I know.

For our analysis here just intent is fine.

In fact unless a vast majority of your Visitors are connecting using dial up it is safe to assume the download of small files went through. I know that does not make the squirrels happy. I am sorry. You keep squirreling while we make decisions about how to improve the site.

Outbound Link Tracking:

Another thing you'll notice about the website (see why it helps to surf a site you are supposed to analyze?) is that there are a ton of links on the site that point to other government websites.

Track 'em!

Of course the above is not their data :), it's just an illustration of how absolutely easy it is to track this data.

From the report it is very easy to then figure out what links your Visitors click, which is a great, positive, indicator of the fact that they found what they wanted and also what they were interested in.

Remember: It is not very hard to do any of the above three types of analysis. All you need to get into your customer's head is move away from "Top Pages Viewed" and "Page Views Per Visitors" and think a bit more creatively.

Q4. How long does it take for someone to find key information they want?

There are some pieces of content that are so darn important that they are heavily linked (say latest news in case of belgium.be) right from the home page, or that you really do want people to find them asap (in the Health section for example the pdf about how to deal with H1N1 virus in belgium).

For these important pieces of content measure Average Time To This Page.

That's almost three minutes from the time that someone entered the website to the time they found this page (say the one about swine flu).

On average people give a page two and half seconds before they click/leave. Consider how long three minutes is, and how many people might have given up in the process of finding this key information.

Unfortunately not too many tools, Google Analytics and other Paid Solutions included, provide this as a standard metric. I use ClickTracks and that this delightful metric as a standard offering.

I wish others would have it.

Remember: You can use this data to ensure that your best information is found by Visitors to your site quickly. Fix your top / left / right / bottom / whatever navigation you have on the site. Consider creating a prominent "box" on the top right where you "merchandize" these important links. More things like that.

Q5. Does the right information actually exist on the website? What major things might we be missing on our website?

I have consistently advocated my love for internal site search analysis. It is simply da bomb!

Like many other sites belgium.be has a internal site search engine. Typically Visitors who have a harder time with normal navigation (or limited data on a page) will make liberal use of this site search box.

Why not use that data?

Again, this is not their data :).

I recommend looking at the top typed search terms by the Visitor but then also looking at the metric: % Search Exits.

That's the bounce rate of your search results page. I.E. People come to belgium.be, search for the term hippo and the search results are so bad that 33.33% of the people exit from that page! They don't even bother to do anything. Just bail. Bounce. Kaput!

Now you know both 1. what information they were looking for, 2. what search results stink and 3. likely because you don't have the right or enough content about that keyword on your site.

Fix it!

I have one more idea to understand if you are missing information that your Visitors want on your government website.

Use Page Level surveys.

There are free page level surveys available or you can build your own (like the one above from a software vendor's website).

These can be an excellent way to understand what content is missing from your website. You can of course also use the open text voice of customer (VOC) from surveys like 4Q, look for Visits with Task Completion = No.

Remember: These surveys don't collect any personally identifiable information (PII) information, and that goes for your web analytics tools as well. Many government sites are extra concerned about privacy, as they should be. Do Please familiarize yourself with the privacy policies of the vendor.

Note: What was not tracked or emphasized. . .

Visits.

Unique Visitors.

Page Views.

Time on Site.

And so many more mundane and perhaps more "famous" web metrics.

I am sure most government or normal websites jump to that first. And why not, they are all staring you in the face when you crack open any analytics tool.

The problem is that these aggregate metrics barely contain any insight. If you focus on them, you'll be left holding a empty bucket / cry a lot / get fired / not get your government pension / dread meetings with your boss.

I hope the above ideas inspire you to do more, go beyond the obvious and less than useful.

One last quick example. . .

You can use the same strategy for other sites. Though remember the job the site is trying to do and the desired outcomes will decide which key performance indicators you end up using.

For example for www.recovery.gov in addition to some of the metrics above I would probably also measure Visitor Loyalty and Recency. That's because the government wants the data provided to be so sticky, and it is updated frequently, that it wants you to come and check it again and again.

In this case perhaps more than downloads I would also measure # of customized graphs created. When I measure content consumed (#3 above) I'll probably focus on understanding which departments get looked at more on the site (are they the ones most spending money?).

You can also bet I am going to be totally on top of reporting how many complaints we have received on the site for Fraud, Waste & Abuse! Getting a ton of those would be a key performance indicator! : )

Makes sense?

Don't despair just because you have a government site. Ignore the obvious. Focus on the site's jobs. Identify key outcomes. Do productive analysis.

Good luck.

Ok now it's your turn.

Are you responsible for a government website? What are your key performance indicators? What web metrics are important to you? Do you use any of the above strategies? If not, why not? Have you looked at www.belgium.be? What would you have recommend that I did not?

Please share your valuable advice / insights / feedback / critique.

Thanks.

PS: Like this post? Perhaps you'll consider ordering my * new * book: Web Analytics 2.0.

PPS:
Couple other related posts you might find interesting:

• Brand Measurement: Analytics & Metrics for Brand Websites
• Measuring Success of Non Ecommerce Websites
• Measuring Web Analytics Success for Small Business Websites
• Outcomes, baby!! Measure Macro AND Micro Conversions
• Six Recommendations For Measuring Success of Your Blog

Web Analytics Success Measurement For Government Websites is a post from: Occam's Razor by Avinash Kaushik

This post covers four commonly used measurement techniques that 9 times out of 10 work against the evolution of Reporting Squirrels into Analysis Ninjas.

I'll also admit that most of the times when I encounter them I might think slightly less of you (especially if you present the aggregate version to me rather, presenting the segmented view atleast gets you time to explain :)).

If I am being slightly tough minded here it is only because I am hugely upset by the fact that analytics on the web is deeply under leveraged, though the good lord knows we try and pump out KPI's by the minute.

One root cause of this under leveraging it our dashboards that are crammed full of metrics that use these four measurement techniques. The end results: Data pukeing and not insights revelation.

So who are the four amigos?

Each a technique that when used "as normal" actively hinder your ability to communicate effectively the insights that your data contains.

Only one caveat: I am not saying these techniques are evil. What I am saying is don't be "default" when using them, be smart (or don't).

Before we get going here's my definition of what a Key Performance Indicator is:

Measures that help you understand how you are doing against your objectives.

Note the stress on Measures. And Objectives. It it doesn't meet Both criteria its not a KPI.

With that out of the way lets understand why Averages, Percentages, Ratios and Compound Metrics are four usually disappointing measurement techniques.

#1. Averages

Raise your hand if you are average? Ok just Ray? No one else?

Raise your had if your visit on any website reflects an average visit? Just you Kristen?

No one is "average" and no user experience is "average". But Averages are everywhere because: 1) well they are everywhere, which feeds the cycle and 2) they are an easy way to aggregate (roll up) information so that others can see it more easily.

Sadly seeing it more easily does not mean we actually understand and can identify insights.

Take a look at the number above.

51 seconds.

Ok you know something.

Now what?

Are you any wiser? Do you know any better what to do next? Any brilliant insights?

No.

It is likely that the Average Time on Site number for your website has been essentially unchanged for a year (and yet, yes sirrie bob, it is still on your "Global Senior Website Management Health Dashboard"!).

Averages have an astonishing capacity to give your "average" data, they have a great capacity to lie, and they hinder decision making. [You are going to disagree, quite ok, please share feedback via comments.]

What can you do?

I have two recommendations for you to consider.

Uno. Segment the data.

Identify your most important / interesting segments for your business and report those along with the Overall averages.

You have more context. Social Media boo! Paid Search booer! Organic yea! Email yea! Etc Etc Etc. : )

While this is not the most optimal outcome, it will at the very minimum give your Decision Makers context within which to ask questions, to think more clearly (and mostly wonderfully ignore the overall average number).

So on your dashboards and email reports make sure that the Key Performance Indicators that use Averages as the measurement technique are shown segmented. It will prod questions. A good thing, as Martha would say.

Dos. Distributions baby, distributions!

If averages often (*not always*) stink then distributions rock.

They are a wonderful way to dissect what makes up the average and look at the numbers in a much more manageable way.

Here's how I like looking at time on site. . . .

So delightful.

I can understand the short visits (most!) and decide what to do (ignore 'em, focus hard core, etc).

I can see there is deep loyalty, about 30%, I can decide what these people like, what they don't like, where they come from, what else I can do. [Would you have imagined from the Average Time on Site that you have fanatics on your site who are spending more than 10% on each visit!!]

I can try to take care of the midriff, what is up with that any way.

See what I mean? The difference between the two: Reporting Squirrel vs. Analysis Ninja!

#2. Percentages.

Nothing, really nothing, is perhaps more ubiquitous in our world of Web Analytics than percentages.

You can't take a step without bumping into one.

Some percentages are ok, but very very rarely are they good at answering the "so what" or the "now what" questions.

The problem with percentages is that they gloss over what's really important and also tend to oversell or under sell the opportunity.

Let's compare two pictures. In the first one we just report conversion rates, see what you can understand in terms of insights fro this one. . . . .

Now try to answer the question: So What?

Any answers?

Yes some conversions are lower and others are higher? Anything else? Nope?

Ok try this one. . . .

Better right?

You get context. The raw numbers give you key context around performance.

[Update: I use this plugin to get raw conversion rate numbers into Google Analytics: Better Google Analytics Firefox Extension. I highly recommend it, you get the above and a bunch more really cool stuff. Must have for GA users.]

Also notice another thing, I'll touch on this in a bit as well. If you only report overall conversion rate (as we all do in our dashboards) your use of a percentage KPI is much worse. You get nothing.

By showing the various "segments" of conversions I am actually telling the story much better to the Sr. Management. What's working, what needs work.

Here's another constant problem with conversion rates. . . .

I am looking at a table of data (in any tool really) and it looks like there's something here.

Ok well I want to fix things. I want to know where I can improve bounce rates, so I sort. . . .

Data yes. Totally useless. I can't possibly waste my time with things that bring one visit.

So I re sort to see if I can find where its totally working for me. . . .

Strike three, again not very useful, just take a peek at the Visits column.

What I really want is not where the percents are high or low. I want to take action.

What I really really want is some way of identifying statistically significant data, where bounce rates are "meaningfully up" or "meaningfully down" so that I can take action confidently.

I can't do that in Google Analytics. Quite sad.

Some other tools like Coremetrics (by default) and WebTrends (in some places by default or with a external "plugin" you can buy from external consultants) will compute a %delta (difference between two numbers) and color it red or green.

That's not what I am taking about.

That is equally useless because that percentage difference make you take action where there is no significance in the two numbers. Don't fall for that.

It is truly a crying shame that the Google Analytics does not have something like the Google Website Optimizer does. . . .

. . . . a trigger for me to know when results are statistically significant, and by how much I should jump for joy or how many hairs I need to pull out of my hair in frustration. See those sweet colored bars in the middle? See the second column after that? Minorly orgasmic right?

Isn't it amazing that after 15 years web analytics tools are still not smart, even though they have so much data and computations. Ironic if you think about it.

What can you do?

I have three recommendations for you to consider.

Uno. Segment the data.

Wait, did I not say that already? : )

Do it.

Useless. . . .

Useful. . . .

Show opportunities, show failures, let the questions comes.

Dos. Always show raw numbers.

Often conversion rates mask the opportunity available.

Conversion rate from Live is 15% and conversion rates for Yahoo! are 3%.

Misleading.

We all know that Yahoo! has significantly more inventory than Live and even if you had all the money in the world you can't make use of that 15% conversion rate from Live.

Show raw Visits. It will look something like this:

See what difference that would make on a dashboard? No false alarms.

You overcome the limitation of just showing the percentage.

In the example above I am using Visits, because I want to show the HiPPO's where the constraints are (without them having to think, thus earning my Ninja credentials!). But I am most fond of using Outcomes when I pair up raw numbers (Orders, Average Order Value, Distribution of Time, Task Completion Rates, etc etc) because HiPPO's love Outcomes.

Tres. Don't use % delta! User Statistical Significance et al.

When you use percentages it is often very hard to discern what is important, what is attention worthy, what is noise and what is completely insignificant.

Be very aware of it and use sophisticated analysis to identify for your Sr. Management (and yourself!) what is worthy.

Use Statistical Significance, it truly is your BFF!

Use Statistical Control Limits, they help you identify when you should jump and when you should stay still (so vital!).

This is all truly sexy cool fun, trust me.

#3. Ratios.

Can I be honest with you?
[Ok so I can hear your sarcastic voice saying: "Why stop now?" ;)]

Ratios have a incredible capacity to make you look silly (or even "dumb").

I say that with love.

What's a ratio?

"The relative magnitudes of two quantities (usually expressed as a quotient)." (Wordnetweb, Princeton.)

That was easy. : )

In real life you have see ratio's expressed as 1.4 or as 4:2 or other such variations.

You are comparing two numbers with the desire to provide insights.

So let's say the ratio between new and returning visitors. Or the ratio of friend requests sent on Facebook to friend request received. Or the ratio of articles submitted on a tech support websites to the articles read. Or… make your own.

They abound in our life. But they come with challenges.

The first challenge to be careful of is that the two underlying numbers could shift dramatically without any impact on your ratio (then you my friend are in a, shall we say, pickle). . . .

I have put my "brilliant" excel skills to demonstrate that point. In your dashboard you'll how the ratio (all "green" for four months). Yet the fundamentals, which is really what your Sr. Management is trying to get at, have changed dramatically, perhaps worth an investigation, yet they'll get overlooked.

I hear you protesting all the way from Spain, "aw come one, you have got to be kidding me!". I kid you not.

Think of all the effort you have put into automating the dashboard and cramming all the data into it. Ahh… you've stuffed it with percentages and ratios to make it fit. And you've automated it to boot.

Casualty? Insights. Actionability.

The second problem with ratios is a nuance on the above. It is perhaps more insidious. It occurs when you compare two campaigns or sources or people or other such uniquely valuable things.

I see it manifested by a HiPPO / Consultant / Vendor Serviceman foisting upon you that 1.2 is a "good ratio".

Then you start measuring and people start gaming the system. Because you see 12/10 gets you that ratio as does 12,000/10,000. Yet they both get "rated" the same and that as you'll agree is dumb.

What can you do?

I have two recommendations for you to consider.

Uno. Resist just showing the ratio.

Throw in a raw number, throw in some other type of context and you are on your way to sharing something that will highlight a important facet, prod good questions.

Enough said.

Dos. Resist the temptation to set "golden" rules of thumb.

This is very hard to pull off, we all want to take the easy way out.

But doing this will mean you'll incent the wrong behavior, hinder any thought about what's actually good or bad.

You can a ratio as a KPI, but incent the underlying thing of value. For example Reach and not the ratio of Visits to Subscribers (!!).

#4. Compound Metrics (aka Calculated Metrics).

Here's a visual for you:

Its a compound metric. : )

A unrecognizable paste you produce after mixing a bunch of, perhaps perfectly good, things.

All kidding aside compound metrics are all around us. Most Government data tends to be compound metrics (is it a wonder that we understand nothing that the government does?).

A compound metric is a metric whose sub components are other metrics (or it is defined in terms of other computations).

Here's an example:

(% of New Visits) times (Average Page Views per Visit) equals, making something up here, Visit Depth Index.

What?

Yes what indeed.

The environments where compound metrics thrive are ones where things are really really hard to measure (so we react by adding and multiplying lots of things) or when confidence in our ability to drive action overtakes reality.

Honestly no matter what the outcome is here (or how much of a "god's gift to humanity" it is) how can you possibly do anything with this:

Website Awesomeness= (RT*G)+(T/Q)+((z^x)-(a/k)*100)

(If you don't know what those alphabets stand for just make something up.)

Compound metrics might be important, after all the Government users them, but they have two corrosive problems:

1) When you spit a number out, say 9 or 58 or 1346, no one,except you has any idea what it means (so a huge anti actionability bias) and worse

2) You have no way of knowing if it is good or bad or if you should do something. You can easily see how a raise in some numbers and fall in others could cause nothing to happen. Or all hell could break loose and yet you still get 9. Or 58. Or 1346.

What can you do?

I have three recommendations for you to consider.

Uno. Take them with a grain of salt (or a truck full of salt).

Really.

Regardless of if it comes from me or President Obama or [insert the name of your favorite religious deity here].

Stress test how you'll overcome the two challenges above. If your compound metric passes those tests you are all set.

Dos. Degrade to key "critical few" components.

Grinding RT and G and T and Q and z and x and a and k into a mush is the problem. Not RT or G or T or Q or z or x or a or k themselves.

Spend some time with your HiPPO's and Marketers and people who pay your salary. Try to understand what is the business really trying to solve for. Put the nose to the grind stone and so some hard work.

At the end of this process, as you decompose the individual components, what you'll realize is that all you need is RT and Q and G. Report them.

No not as a weird married "couple". As individuals.

Everyone will know what you are doing, you help the business and your dashboard focus, drive action.

Tres. Revisit and revalidate.

If you must use compound metrics please revisit them from time to time to see if they are adding value. Also check that they are adding value in all the applicable scenarios

If you are using weights, as many compound metrics tend to do, then please please stress test to ensure the weights are relevant to you. Also revalidate the weights over time to ensure you don't have to compensate for seasonality or other important business nuances.

End of story.

I'll close There are two schools of thought about Analtyics.

One is that math is easy so let's go add, subtract, multiply and divide because calculators, computers and data are easily available.

This is the "Reporting Squirrel" mental model, data above all else.

The other is that your entire existence is geared towards driving action. So think, stress test, be smart about the math you do. Computers and calculators are cheap but it does not excuse doing the things outlined above.

This is the "Analysis Ninja" mental model, insights above all else.

Good luck!

Ok now its your turn. What do You think of these four measurement techniques? Agree with my point of view? Why? Why not? Care to share your own bruises from the wonderful world of Web Analytics Key Performance Indicators? Got questions?

Please share your feedback. Thank you.

PS:
Couple other related posts you might find interesting:

• Eight Rules for Choosing Web Analytics Key Performance Indicators
• Be Awesome At Comparing KPI Trends Over Time
• Make Web Analytics Actionable: Focus On “What’s Changed”
• How To Excite People About Web Analytics: Five Tips.
• Consultants, Analysts: Present Impactful Analysis, Insightful Reports

Actively Avoid Insights: 4 Useful KPI Measurement Techniques is a post from: Occam's Razor by Avinash Kaushik

They had some biting, but fair, observations about short comings of Google Analytics. I appreciated that very much.

But the most valuable part for me was section 6.7.2. It was a listing of 11 barriers to an effective online measurement strategy. 11 painful reasons why extracting value from web analytics is still worse than attempting to climb Mt. Everest for some of the top companies.

Curious?

Here they are:

1. Lack of budget/resources (45%)
2. Lack of strategy (31%)
3. Siloed organization (29%)
4. Lack of understanding (25%)
5. Too much data (18%)
6. Lack of senior management buy-in (18%)
7. Difficulty reconciling data (17%)
8. IT blockages (17%)
9. Lack of trust in analytics (16%)
10. Finding staff (12%)
11. Poor technology (9%)

Makes for a slightly depressing read does it not?

Many, if not all, of these challenges are really hard and often the solutions are unique to each company. In as much it would be impossible to write a here is how you fix it all blog post.

Rather I am going to try and share some thoughts / ideas on that will atleast help you take step one. I very much encourage you to share your wisdom with us through comments.

First: A Brilliant Insight / Borderline Rant. : )

Before we go on nothing something absolutely astonishing…. we live in a culture where every Analyst, Blogger and Consultant is writing / posting / talking / presenting comparisons on web analytics tools.

We can't seem to take one step without stepping into one more pile of opinions about why this tool is great and what one is bad.

Yet the top ten barriers have absolutely no connection to features, and barely have any connection to tools. Its #11. An afterthought.

I wonder why we are not writing / posting / talking / presenting on how to solve these non-tool problems, things that actually matter to companies and practitioners in the real world.

Just because we are programmed to publish reports comparing tools?

Tools can provide a marginal advantage to a company of any size. But given where we are in our evolutionary stage we have much bigger fish to fry.

I hope its out with lets drop our clothes and compare sizes and in with adding real value to practitioners by focusing on issues like the above ten.

Do I hear a amen?

Now on with the show. . . . .

#1: Lack of budget/resources:

In some sense this problem never goes away. It bedevils you when you are small and just want to buy a web analytics tool or just start testing. It will still be a issue even after you have been successful with Site Catalyst and now want to plunk down a million and half dollars to buy the behavior targeting platform.

How do you overcome this challenge?

Start for free and earn your right to ask for budget.

No matter what tool you want it is now available for free. Web Analytics. Multivariate Testing. Behavior Targeting. Whatever.

Why are you asking for a tool budget? I know some worry that they don't want to have a tool for a year or two and then switch. Look, no one knows what the world looks like in 18 months, why are your planning for five years?

Implement Yahoo! Web Analytics, Google Website Optimizer, BT Buckets and have at it.

When you run into limits you'll have a proven track record of success which will make it easier to ask for budget for WebTrends, SiteSpect and Kefta.

The only reason you'll get turned down is if you showed no value to the company. Then you might not ask for budget. :)

Other tips for getting more budget:

• Don't focus on the value of the tool. Quantify the value of the outcome you will deliver. "I want an Analyst for our tech support site because I can reduce calls to our phone center and reduce costs by $1.6 mil and increase satisfaction by 5 points."
• Enroll your Customers and Competitors to help you. More here: Lack Management Support or Buy-in? Embarrass Them!

#2 Lack of strategy:

I am a bit flummoxed. I am not sure what to make of this.

If the barrier to an effective web measurement strategy is that your business has no web strategy then I think you should look for another job. [You can start looking now, ride out the recession, and then bail at the first opportunity.]

Someone up there, the HiPPO's, truly needs to get it and create a web strategy. Once they create even a rough cut of it you can help them. Without a rough cut this is a lost cause.

If you are a Director or a VP perhaps you can try and help plant the seeds for a strategy. Especially if you notice Measurement/Analytics is owned by IT (usually a kiss of death – with sincerest apologies to all my IT friends). Get it moved to a business function.

Other tips to try and create some strategy:

• If you are at a large company with many divisions etc and no consensus, then try to pick one division/country and make them a hero. Don't try to get everyone to agree on a set of metrics.

• If you think your boss wants to create a strategy, but needs a final push the check out this post for tips: Six Rules For Creating A Data Driven Boss.

#3. Siloed organization.

It is important to realize that silo's can break if you are able to show value. Everyone wants a bonus and they want to get promoted. Oh and they also want to help the company.

Exploit that fact.

Start small. Show some value. Go bigger (cover another business unit or now cover Marketing or Paid Search in addition to Email Marketing). Show value. Go bigger still.

That was my strategy.

When I started it was me and one Analyst in the "center" and no one would listen to us. Not the business units. Not the business functions (IT, eCommerce owners etc). But we executed the above strategy and over time I proved how data can be valuable to one silo in the company. They wanted move.

My reply: "We have got to break down the IT and Analytics silo, we can't be filling out tickets all day to make minor changes."

Ok, some painful business gyrations, I got a couple technical folks transferred.

Next spent more time doing things faster better with help of the team, proved more value, earned credibility.

Question: "How can we move to the next level?" Answer: "We need to move into doing more qualitative analysis, but that function is in that other silo." Feedback: "Well then lets go fix that."

It got fixed. One more silo broken, a more cohesive team, significantly improved execution.

Maybe you don't want to break silos around data execution. I doubt your execution model would be different. Make the best of what you have today. Work hard all day. Strike oil. Go back to work harder next day.

Most people don't want to work this hard. Most people don't have the kind of patience required. Then it is easy to complain and wait for someone to fix the silos. Won't happen. Sorry.

#4: Lack of understanding.

It is unclear from the report exactly what falls into this bucket, it seems to be this general complaint: "No one understands me, no one appreciates me (except my mom!), no one will help me."

If there is a lack of understanding of the value of data then have someone from your management team to attend one of the Vendor webinars (Omniture does a bunch of these). These webinars present one client's experience (usually extremely rosy) and perhaps that can get your boss to appreciate value of data.

If there is a lack of understanding of what analytics can do, get Google Analytics and slap it on a micro site if you have to and improve organic search to show how you can improve the number of visitors from search. Notice everything in this sentence is free except your time.

If there is a lack of understanding of what technologies exist in the market, do a quick Google search, identify the main vendors, get them to come do a dog and pony show for you (online or in person). Sure there will be some showmanship, but you and your boss will also learn a bunch.

Other tips for creating an understanding:

• If you are low or mid level employee then realize that you can't do this. Find a Sugar Mommy (or Sugar Daddy) who will help you.
• I'll repeat this again: Any understanding can be created by doing rather than by simply talking.

#5: Too much data.

Finally a web analytics problem!

I am astounded only 18% complained about this. Just goes to show how many people are still executing Web Analytics 1.0, clickstream only, strategies. If they were truly doing Web Analytics 2.0 strategies more people would say this, and it would be a good thing.

Can I be blunt?

This is a problem we, Practitioners, create. We are simply so eager to impress others about how much data we have and how we are so fantastic that we have 28,205 metrics we can reports on day one.

Who cares!

Two words: Critical Few.

You must not send a single report out, no not even a number via email, until you have identified what your critical few metrics are. That process starts with the question: What the heck are we solving for with our website?

Ok so maybe a bit more polite than that.

But honestly identify the one Macro Conversion (big goal) for your website and, up to, three Micro Conversions.

Now focus on just the few metrics help you measure success of these four things, with the highest priority being the Macro Conversion. [Blog post with ideas: Measure Macro AND Micro Conversions.]

Do nothing else. Ok do nothing else until you have mastered these. Don't irritate your companies with lots of reports with lots of metrics.

Other tips for reducing amount of data:

• If you can't get your management to identify goals for you, update your resume and apply for other jobs. While you are waiting focus really hard on only reporting metrics that will help 1. increase revenue 2. reduce cost and 3. increase satisfaction. Can't go wrong with those three.
• If that does not work report these six metrics, just six, and you'll be adding value: Six Web Metrics / KPI’s To Die For.

#6: Lack of senior management buy-in.

For me this is the same as #1. They don't want to give you budget or resources because there is no buy in. Perhaps because you have reported too much data which leads to a lack of understanding resulting in a lack of strategy.

Focus on the things we have discussed before and you'll have the thing you crave from your management:

For more here are three posts that fall in this space:

Six Rules For Creating A Data Driven Boss

Management Support or Buy-in? Embarrass Them!

How To Excite People About Web Analytics: Five Tips

Other tips for getting senior management buy-in:

• Realize that there is a difference between reporting and analysis. Be a Analysis Ninja, not a Reporting Squirrel.

#7: Difficulty reconciling data.

Let's tackle this at two levels.

At a macro level you should know that it will be impossible to reconcile data and that it is ok. I believe that one the web we must execute a Multiplicity strategy.

That means different tools, with different data sources, different metrics. Again, this is ok. There is exponentially more value in using these data sources than the alternative.

In this case invest in educating your manage met team why the numbers differ. They won't accept it entirely. Start with making small decisions based on this data, show value, earn trust, move to bigger things.

At a micro level this refers to reconciling numbers between Google Analytics and NedStat. Or between HitWise and Compete. Other such cases.

If this is your case then please use the Ultimate Web Analytics Data Reconciliation Checklist to identify where the issues might be and fix them. You'll never them them to match 100% but if you get say close to 10% delta, you are pretty much there. Move on to other problems.

#8: IT blockages.

If you think you have a chance to take direct ownership of large (or all) parts of the IT work required (tagging primarily) then follow the strategy I have outlined in point #3 above (breaking silos).

It is likely that your company will simply not allow you to touch the site. In this case both to do web analytics work (tags etc) and online marketing work (update pages, fix urls for SEO etc) you are stuck with IT (an organization that tends to be ultra conservative).

Use the sparkling power of data to unclog the blockages.

My friends Shane Atchison and Jason Burby from Zaaz have long advocated creation of models that identify the cost of delays. Here's one of their earlier models. . .

Current conversion rate and the value is identified at the top. In the first column are subsequent improvements in conversion rates, based on the goodness that Zaaz is going to bring to the table. The columns indicate incremental orders and value and, sweetness, the impact of a launching the changes in a month of in four months.

The last column shows the cost of the delay.

The worst improvement in conversion will cost the company $342,930 in real revenue. The expected improvement is to 9.25%, which would mean a three month delay will result in $1.5 million in lost revenue!

Do you think you can get IT to get unclogged? You betcha.

Here's another one where they have monetized lost opportunity from delays in improvement for conversions that happen offline from online leads. . . .

Intelligent decisions can be made if the project should be delayed three months or six months. :)

Don't pick political battles with IT. Use data.

#9: Lack of trust in analytics.

I think there is overlap here with #7. [Linus perhaps we need some consolidation? Also between #6 and #1?]

If the problem is that they don't trust the data, then use the techniques described above in #7.

If the problem is they don't trust web data then use these techniques:

More details here: Slay The Analytics Data Quality Dragon & Win Your HiPPO’s Love!

#10: Finding staff.

I have advocated the 10/90 rule for magnificent web analytics success for almost five years now. People are the key (and in some sense I am disappointed they come our #10 here).

Finding staff is certainly not easy, but I don't believe that it is all that hard. I think we tend to look too narrowly.

We look for people with ten years of Omniture experience. Or with experience in WebTrends, Optimost, iPerceptions and making coffee.

These are very hard to find, and narrow the pool of potential candidates waay too much. We are a young industry and that means it is hard to find people with deep experience in one specific tool.

The ironic thing is that so much has changed about Omniture (or GA or whatever) and the web so much in the last five years that any tricks you knew from five years ago are irrelevant now.

When you look for Analysts look for people in the Finance function. Look for people who are doing traditional Business Intelligenc work. Look for fresh college graduates who have the web in their blood (unlike us old folks) and teach them what buttons to press in Xiti.

Don't close your eyes to other possibilities. A peer of mine just hired someone who was in the HR team doing People Analytics (what a great analyst!).

Other tips for hiring people:

• Should you higher a bright freshly minted college grad or someone who has been around for a while? Answer here: Fresh blood or old hands? Experience or Novicity?
• How to ensure the Analysis Ninja you want to hire does not turn out to be a reporting squirrel? Answer here: Interviewing Tip: Stress Test Critical Thinking. Please.

#11: Poor technology.

I almost don't want to dignify this with a comment. After all we live in a world where the freest of the free tool can still help you make a ton of progress for months.

Let me say this. . . . if you are looking for technology to do traditional web analytics or web analytics 2.0 then you are in luck. Lots of magnificently powerful technology exists all the way from free to you'll have take off all your clothes and even your underwear and send them with your chq expensive.

The only places I find "poor" (hate that word, I prefer early evolution) technology is on the bleeding edge. If you are unable to collect real rock solid data for mobile analytics or social media analytics or truly distributed content analytics then you have my sympathy.

As a human race we have not really figured out what these things are, and they are changing with every passing day. It will take us some time to figure our optimal data collection and associated technologies.

But other than that don't give, or accept, the excuse "poor technology".

End of story.

My highest expectation from this post is that it will give you possible starting points as you start to tackle some of these tough challenges. If you got three ideas you can take back and action, you have just made my day.

Ok your turn now.

What do you think of the 11 findings in the Econsultancy study? Do they reflect your reality? Would you prioritize them the same way they did?

Have you dealt with and conquered some of these challenges in your life? Even if it was one of them, would you please share your experience and lessons with us? It would be greatly helpful.

Thank you.

Barriers To An Effective Web Measurement Strategy [+ Solutions!] is a post from: Occam's Razor by Avinash Kaushik

My hope in this post is to share some simple tips with you that might make your reports and analysis speak to you a bit more. Suggestions that might increase the probability that you'll bump into things that might be insightful, and communicate data more effectively.

None of them are very hard to do, but I think they make a world of difference.

Excited? Here we go. . .

#1: Go as deep as you can. Then, a little bit more.

Far too often in our daily lives we let our job titles limit how deep we go in our analysis.

For example let's say I work at a delightful car / health / spaceship insurance company. Naturally all of my analysis is focused on the efficiency of the website in moving the Visitors quickly from the landing page to click on that delightful Submit Quote button.

I am focused on what the site does because that is what my job title says: Web Analyst

I am analyzing campaigns (which ones convert better and which worse), I am looking a little bit at the bounce rates, and of course I am totally obsessing about my seven step quote submission funnel (and how to reduce abandonment).

Bottom-line: Quote, quotes, quotes.

And that is fine.

The data is easily available in the web analytics tool so why not.

Here's my advice: You should kick things up a notch. Don't focus just on the quote (the part the site does), include the final conversion to a paying customer (even if that data is offline).

The picture you get from stopping at Quotes might be very different from stopping at Policies Purchased.

Here's what you are focusing on (and it is good):

All my experience in these things suggests that it is dangerous to think that the Conversions column is representative of the final outcome.

Here is what it probably looks like (and this is going from good to great):

See how the ranking changed?

You would make different recommendations right? Would it save your company money? Would it make you refocus your efforts on where improvements are needed?

You betcha!

For straight ecommerce websites the first picture is what you use every day. But for most other types of businesses the final success does not exist in web analytics tool. So what? Get the data out of the crm / erp / "backend" system. . . dump it into excel. . . write a simple formula!

Usually you don't need a complicated multi year data warehousing effort with expensive business intelligence tools to buy. At least for this scenario you just need a column and a short movie data with your online IT person and a longish coffee break with your "backend" IT person to get the right primary keys set up. Then you can bring your sexy back!

Go deep.

You are paid to find real bottom-line impacting insights (remember line of sight to net income?). Do that.

If you are a purely ecommerce business then you can go a bit deeper too. Consider doing quarterly analysis that focuses on calculating customer lifetime value. Up a notch.

If today you are a content site that is only focused on measuring content consumed try to go deeper to understanding CPA of the ads or Visitor Loyalty. Once again going one step deeper, up a notch.

And so on and so forth.

Make it a point to pause every Friday at 0900 hrs. Look at your most important work / report / dashboard. Then ask yourself this: "How can I take my view of the data one step deeper?"

Now figure out how to do that. That'll impress me, your boss and your mom.

#2: Join the PALM club. [PALM: People Against Lonely Metrics]

This rule actually comes from my second book, Web Analytics 2.0. [Page 318, Principles for Becoming an Analysis Ninja, if you have the book already.]

The rationale for this rule, joining the PALM club, is quite simple.

You need a someone in your life. I need someone. Everyone needs someone else. A boy friend. A girl friend. A cat. A "you complete me" person.

So why not your metrics?

We do reports / dashboards like this one all the time:

Ok great.

I know the top referrers sending traffic to my site in a month. Maybe I can appreciate more the power of Twitter or google.co.in or whatever.

You might even impress me next month with a updated version of this where some of these might have shifted a bit up or a bit down.

I might not do anything with the data… but you surely hypnotized me for a few seconds.

This is the problem with lonely metrics.

They don't have any context. They fail to communicate if 841 visits from Twitter were any good. In fact is any of the above good or bad? How do you know?

Why not find a BFF for your lonely metric and present something like this. . . .

Much better right?

I found a "you complete me" for my Visits metric, Bounce Rate.

Now in an instant I can not only see which referrers are big or small, I can see which ones are "good" or "bad".

I could have picked conversion rate as the bff. I could have picked % new visits. I could have picked connection speed or mobile platform or underwear size.

Whatever makes most sense for my business. But putting two minutes of thought into my metric would help make my report a little bit more useful.

Kick it up a notch. Right?

Never ever never never never ever present any metric all by itself.

If you want a cop out then at least trend it over time. If you actually want love then join PALM and don't let your metric be lonely.

Let me close with one of my favorite examples of this rule, this one's to inspire you if you have a pure content (non-ecommerce) website. . . .

Good to know what content's being consumed. Column: Pageviews.

Much much much better to know what the $ index value is for each.

See that crazy blue line that's literally off the chart? You would want to know that about the 1,414 pageviews right?

Now go find your dashboards, your reports, your data pukes (sorry!) and make sure that for every dimension you are not reporting success or failure using just one metric. Join PALM!

[Tip: Not that you are trying to but if you want to impress me but if you are then make sure the second metric you pick is as close to an outcome metric as possible. Or an actual outcome metric. I. Love. Outcomes.]

#3: Measure complete site success. Measure everyone's success.

One of my greatest passions when doing analysis is to look at the complete view of things. Rather than just the obvious.

An application of that passion is to look at all the jobs the website is doing, representing all the work that is being done by people in your company who touch the site.

Ecommerce is too easy an example of this so let me use a non profit example.

San Francisco Aids Foundation is a charity I support. It does incredible work to prevent new HIV infections.

The only way SFAF stays in business is if you and I make donations. As an Analyst I would focus all my energies on trying to figure out how many donations we are getting and where those people come from and what they are doing on the site etc.

But donations is just one measure of success ("macro conversion"). There are other jobs that the site is trying to do, and people who work on those jobs. So why not measure those?

For example. . . .

Measure the above four micro conversions, in addition to the macro conversion of donation, helps give a complete view of success. And what to do better.

Maybe Google is really good at Volunteers and not optimal for attracting people who donate. If you focus only on donations you'll devalue Google. Or maybe facebook is the best source for sharing information (downloads). And more such things.

Not only are you measuring all that matters. . . . you are validating the jobs of people who put together all that content.

Most of the time we don't do this. We, web analysts, just focus on one thing and then we wonder why we don't have the impact we want to, or why everyone does not pay attention to us.

Broaden your view!

If I were analyzing Amazon I would measure sales AND affiliate signups, signups for amazon prime, credit cards, wish lists, "like's" on reviews, self publish inquiries, free downloads….

If I were analyzing L'Oreal Paris it would be sales AND reviews, coupons downloaded, successful completion of "Profile My Skin", videos watched, sign ups for mobile alerts….

In both cases a complete view of the website and success of every person who works on the site.

Ninjas do that. You should too.

[UPDATE: A key next step, post micro conversions identification, is to identify the Economic Value. See this post for specific ideas about how to do that: Excellent Analytics Tips #19: Identify Website Goal Values & Win!]

#4: Be smart about using time. Move beyond MoM.

This is one of the most common view of data presented in web analysis…

The picture illustrates the performance of a metric over two consecutive months.

This is of course better than just showing data for June.

The problem occurs when you proceed to look at six such graphs on your dashboard and then proceed to use the trends to deliver insights. You are reading too much into the ups and downs, you are inferring things that might not even exist.

Two months do not a trend make. Important lesson.

Not even for the world's most flat line no seasonality business.

So here is a best practice. . . . at least add three months. . . . if the data looks like below you'll think one thing (and every different from above pic)…

But if the data looks like the image below. . . . you'll think something else. . . .

Worry in one case. Jubilation for the temporary awesomeness for May in the other.

The more time you put into this graph (and if you have as much space as above you can easily add at least six months and it will still look pretty) the better.

But if I can only have three I love using current, prior, same month last year.

Better context right? Will take you off on a completely different line of inquiry, all from adding June 2009 to look at June 2010.

If June is the last month of your quarter and you have a cyclical business then maybe you want to compare Apr, May, June 2010 and have the first column be March 2010 because you want to see how the last month of this quarter did vs last month of the last quarter (because Apr and May don't really show if the trend ended as high or low as it should have ended).

So on and so forth.

Remember:

Sweet momma that is awesome!

Kick it up a notch, ok?

#5: Present data better, make insights obvious.

There are so many ways to present data that a small section of a blog post is insufficient. And of course there are so many people who are better at this than I am.

Let me just say that the way you present data matters, a lot. I'm not saying you should make it pretty. I could not care less if it is pretty or not. I'm saying present it in a way that the insights you think exist in the data become more obvious.

Here is a "data element", from an actual dashboard, that I really like. It might not be sexy but it is extremely functional and it is super awesome at communicating the smarts of the Analyst.

Three month trend for one very important business metric…

First note that rather than just showing one column for the performance of this metric it shows four. One for each key segment of the customer that the company has.

This would require you to know the business (good thing), know its customers (great thing) and track the segmented data. IE have your act together.

Second note that the data is for three months. You could show more but in this case you don't want to overwhelm the Executive. If you go more months, shrink the segments.

Third, really important, note that the overall goal is clearly indicated in the picture. 80. And to get that number you would have to talk to Finance and Marketing and HiPPO's and get an agreement up front. This is absolutely magnificent, key to you building relationships and finding insights.

The nice thing about our picture above is that the overall metric would get averaged out and show a trend similar those we showed in tip #4 above.

But would it be insightful enough? A single metric trend would hide insights.

In this case it is pretty clear that Blue, Red, Green segments are doing fine. In fact the one that is absolutely most important, Green, we are doing ok.

The stink bomb in the pile is Purple. It has been dragging the overall metric down (and you know that even if the overall metric is not even shown!).

And you know how much gap you need to overcome for each segment, and were to prioritize your work (PURPLE!!).

This is just one tiny, I call it "functional", way of presenting data.

The presentation is ok, could be made more pretty.

What's precious is the process that went into creating the element – talking to leaders, meeting with Finance and Marketing, identifying the key metrics, finalizing customer segments, and establishing goals.

We often don't do all the above work (the things that are mandatory for data driven organizations) and even if we do it we don't show it because we show lame single line graphs.

Don't do that.

Kick it up a notch. You are working very hard at your job, make sure your work shows up and helps identify better insights.

Those were the five simple things you can do every day to make the most of your daily data analysis. They are not very hard to do, and they'll pay outsized dividends.

I am not someone who leaves the good enough alone. No sirree bob!

With love and affection here are 4 more bonus tips on improving your analysis and truly kicking things up a few notches:

#6: Leverage segmentation, daily.

All said and done the number one way to move from being a Reporting Squirrel to an Analysis Ninja is to leverage segmentation. Every tool has on the fly current and historical segmentation feature set. Use it.

I'll honestly not respect anyone is not applying at least some primitive segmentation to their data.

Learn how to:

~ Web Analytics Segmentation: Do Or Die, There Is No Try!
~ Google Analytics Releases Advanced Segmentation: Now Be A Ninja!

#7: Move beyond the top ten rows of data, seriously.

The "head" of your data will sustain finding insights for a month or two. You might even action something. The real gold lies in your ability to analyze tens of thousands of rows of data at one time. It is harder to do, and hence the rewards are bigger and your competitors will eat your dust more.

Learn how to:

~ Analysis Ninjas: Move Beyond The Top Ten. Find Love (/Insights)
~ Make Web Analytics Actionable: Focus On "What's Changed"

#8: Perform "pan-session" analysis, and win big.

One of the absolute criminal behaviors in web analytics (and indeed online marketing) is that we are so obsessed about Visits, and visits based analysis.

Few people sleep with you on the first date. So why is that your mental model?

Every true Analysis Ninja focuses on measuring customer behavior of one person (or in our case Unique Visitor) over the entire span of that person's interaction one our website.

Hence my devotion to measuring Days and Visits to Purchase. Truly analyzing how people buy. Or analyzing Visitor Recency and Visitor Loyalty to understand now just the first Visit (and conversion) but rather subsequent Visits (and conversions).

I tell you this is honestly kicking your web analysis up five notches, not just one.

Learn how to:

~ Measure Latent Conversions & Visitor Behavior
~ Measure Days & Visits to Purchase

#9: Evolve to multichannel analytics, achieve analytics nirvana.

There is perhaps nothing harder and nothing more impactful than getting good at multi-channel analytics.

Measuring the offline impact of your online activities gives your business a view of success that is truly orgasmic. If you get good at measuring the impact on your website of your offline activities (television, catalogs, billboards etc) then you have truly accomplished the rarest of the rate.

Learn how to: Multichannel Analytics:

~ Tracking Online Impact Of Offline Campaigns
~ Tracking Offline Conversions. 7 Best Practices.

Feeling like an Analysis Ninja already?

Of course not, you have to go do all these things! :)

Remember that tips 1 through 5 you should be able to do quite easily, just need to remember them and remember to use them. Tips 6 through 9 take time, they take a lifetime. Remember them, practice when you have time and slowly evolve over time.

Ok?

Good luck.

As usual it's your turn now.

What are your favorite tips for data analysis? When you present data what is the "trick" that you use most often to be awesome? Have you used any of the tips above? Got any favorites? What do you think it takes to morph from a Reporting Squirrel into an Analysis Ninja?

Please share your feedback / critique / tips and all via comments.

Thanks.

5 + 4 Actionable Tips To Kick Web Data Analysis Up A Notch, Or Two is a post from: Occam's Razor by Avinash Kaushik

We all agree that reporting is not analysis. We all agree that great analysts are hard to come by and few and far between (yet it is interesting that people disagree with the 10/90 rule and keep insisting on spending money on tools). So what makes a great analyst? Do you think you are a “great” Analyst?

Here is my personal point of view, a check list if you will, on what makes a great Web Insights Analyst (it is important to caveat that this is not me, I only wish I were this good, this is something I aspire to be) :

# 10 You have used more than one Web Analytics tool extensively.
While each tool is the same in our field, each tool is really different. The way Omniture computes Unique Visitors is very different from ClickTracks, or how either one of them handles sessions. Using different tools gives you a broad perspective on how the same thing can be counted ten different ways and at the same time a rich understanding of why some tools are great and some sub optimal. The interesting outcome of a diverse experience is that a great Analyst can work with any tool and yet find meaningful insights.

You don’t have to be limited to what you have at work. If you do a View Source you’ll see that this blog is measured using MapSurface, Google Analytics, ClickTracks and AnalogX (so tagging and web logs and real time data and a paid and free tool, great for learning).

# 9 You have not only heard of the Yahoo! Web Analytics group but 20 mins of each day is spent reading all the posts.
The Yahoo! Web Analytics Group is the most awesome collection of smart people in our industry who share their wisdom on every topic under the sun that touches our world. I personally read all the posts every day and I learn about challenges others are facing, innovative ways to solve those challenges, general trends in the industry, pointers to the latest and coolest happenings that impact us and on and on. There are& repeat questions, the interesting thing is that even those get different answers all the time.

# 8 Before doing any important analysis you visit your website and “look” at the web pages (site experience).
This one probably sounds stupid. But it is amazing how many times, how many of us, simply look at tools and numbers and data but often have no idea what the website looks like. It is impossible to analyse the data without a solid understanding of the customer experience on the site, what the pages look like, where the buttons are, what new "great" navigation change went live yesterday. A great Analyst stays in touch with the website and the changes constantly being made the the designers and marketers on the website.

For example: Great Checkout Abandonment rate analysis is powered by actually going through the site, adding to cart, starting checkout (using all options available), going through checkout all the way and getting a order confirmation email. Then you will look at numbers in a new and more meaningful way, I assure you that you will then not have to torture them for insights rather they will sing to you.

# 7 Your core life approach is Customer Centric (and not Company Centric).
In the morass of data quality and TV and UV and cookie values and ab test id’s and sessions and shopper_ids we look at massive amounts of data and forget that real people are using our websites. Great Analysts have a customer centric view that makes their mind a lot more amiable to think like customers, all 1,000 segments of them, and you are aware of their personas and challenges (this is awesome by the way for data segmentation). This keeps you grounded in realityand will help you apply Occam’s Razor (because data trends and patterns without a "customer mindset" will always complicate thinking).

A great Analyst is capable of descending to the Customer level from the "analytical heights" and help her/him to move forward (because customers can't fly).

# 6 You understand the technical differences between page tagging, log files, packet sniffing & beacons.
This is specific to Web Analysts. How data is captured is perhaps the most critical part your ability to “process” the data and find insights. Each data capture methodology comes has its benefits and dangerous negatives. You understand hard core the technical differences between each data capture methodology and then appropriately adjust the kind of analysis you do and the value you extract from whatever your company uses.

# 5 You are comfortable in the quantitative and qualitative worlds.
Clickstream, on its best day, should be the source of 35% of your data. Rest comes for site Outcomes or Qualitative data (the Why, see post on qualitative data). Great analysts are just as comfortable in the world of parsing numbers as the “open ended / ambiguous / soft” world of observing customers, reading their words, inferring their unspoken intentions, sitting in a lab usability study to glean insights etc.

You have a inherent ability to hear people and their problems and all the while in your brain you are thinking of 10 interesting ways in which you can slice the Site Overlay or other clickstream metrics to validate. Great analysts follow a slide on core clickstream / outcomes KPI’s with a slide on Segmented VOC Pareto Analysis.

# 4 You are a avid “explorer”.
Reporting is straight forward. There are inputs, outputs, KPI’s, tables and rows. Analysis is not, it has no predefined paths to take, it has no preset questions to answers. It requires having a open mind, a high level of inquisitiveness and after hearing a ambiguous business questions a deep desire to find new and better ways to use data to answer those ambiguous questions. You don’t worry about the if and how it will work, you save that for later. You seek out possibilities and the non-obvious.

When faced with “incomplete / dirty” rather than think of all the reasons why you can’t analyse data you make reasonable assumptions and can find a nugget of gold in a coal factory. A vast majority of us fail at this, we face bad or incomplete data and we get paralysed. Framed another way you are really really good at separating Signal from Noise (be it using data segmentation, using statistics, using common sense, understanding your customer segments, or other methods).

# 3 You are a “smooth talker”.
In our world Analysts rarely have the power to action things or implement recommendations. Great analysts are great communicators, they can present their message in a very compelling easy to understand manner, and  be a passionate and persuasive advocate of company customers / website users. The 15 hours of complex multivariate statistical regression model analysis is hidden, they keep ego aside, and tell the “simple minded” decision maker that the changing product content presentation will have the highest correlated impact on revenue. They are just as comfortable talking to technical folks as presenting to the VP of xxx or yyy and selling either one of them a boat that they don’t need.

# 2 You are “street smart”.
Great analysts are not “theory sprouting making things complicated and much harder than can be in the real world types.” Think Occam’s Razor. They have oodles and oodles of common sense and a inherent ability to degrade a complex situation to its simplest level and and look at logical possibilities. This does not mean they can’t look at complex situations, on the contrary they have a awesome ability to absorb complexity but they are also scrappy enough to look through the complexity rather than end up in rat holes.  They know how & when to keep things simple.

(The original version of this was: You are Business Savvy. I think that is a incredibly hard quality to find, even harder to judge in a standard interview. Yet it is perhaps the one thing that separates a “report writer” from a “analyst”.  The ability to see the big picture, the ability to understand and solve for strategic objectives. But in my own experience I have found that people who are “street smart” inherently have this ability and hence the framing of #2 as you see above.)

# 1 You play “Offence” and not just “Defence.”
Most of us in this field play “Defence”: we supply data or we provide reports or we at times provide dashboards. Mostly we react. But we don’t play “Offence”:  we don’t get in front of the business and say this is what you should measure, we don’t reply to the question “show me what the tool provides” with “tell me your strategic objectives and I’ll tell you what insights I can provide with the data I have”.

Great analysts spend 30% of their time looking at all the available data just to look for trends and insights, time they don’t have and doing things that no one asked them to do. But that 30% of the time that allows them to play Offence, to provide insights that no one thought to ask for, insights that drive truly impactful actions. You do it because you realize that you are smarter about the site and data than anyone else out there and you do it because it is a lot of fun. :~)

This was supposed to be a Top Ten but here is a bonus:

# 0 You are a “Survivor.”
The reality of the world of our web decision makers is that most of them just want to measure HITS (KD Paine's definition of HITS: How Idiots Track Success). The other day someone asked me to give them a "Site Counter" to put on the website for measurement, I am sure you have not heard the words Site Counter to measure anything in the last few years.

A key skill of being a great analyst is the ability to have patience, survive and stay motivated in a world where people might ask for sub optimal things. Of course you know better but transforming perceptions is a very hard job and take a long time. But you are a survivor, except the part about a million dollars in the end! ; )

This is how hard it is to be a great analyst:

• If you meet five of the above criteria you are a good analyst and you are on your way to greatness.
• If you meet eight you are a great analyst. Congratulations (please send me your resume!).
• If you meet all ten (or 11) criteria then you my friend are a Purple Squirrel and I bow in front of you (oh and most surely send me your resume!!!).

Agree? Disagree? Would you have not included something above? Ranked something differently? Did I miss something all together that you value?

Please share your feedback and your own submissions via the Comments form below. If I get enough different ones I’ll create a new list and publish that (with due credit to you).

(Tip of the hat to Michelle, Oleg, John and Steven. You guys rock!!)

Top Ten: Signs You Are A Great Analyst is a post from: Occam's Razor by Avinash Kaushik

It is my distinct pleasure to share with you all 26 brand spanking new metrics that have been published by the WAA standards committee today. They have been a long time in the making (you can imagine what happens when practitioners, consultants, vendors, industry thought leaders all come together!!), and were announced at the Search Engine Strategies (SES) show in San Jose.

You can download a copy of this wonderful document: WAA Aug 2007 Standards.

I am incredibly proud of the work Angie, Jason as well as the committee volunteers have put in over the last so many months. Their hard work has resulted in standard definitions for some of the most foundational metrics in our world. Everyone will benefit from this additional clarity and all of us now have a benchmark to compare our own web analytics vendor's metrics.

Here are the new terms that have been defined:

Building Block Terms:
        Page, Page Views, Visits, Unique Visitors, New Visitor, Repeat Visitor, Repeat Visitor & Returning Visitor

Visit Characterization:
        Entry Page, Landing Page, Exit Page, Visit Duration, Referrer, Internal Referrer, External Referrer, Search Referrer, Visit Referrer, Original Referrer, Click-through, Click-through Rate/Ratio, Page Views per Visit

Content Characterization:
        Page Exit Ratio, Single-Page Visits, Single Page View Visits (Bounces), Bounce Rate

Conversion Metrics:
        Event, Conversion

Here very briefly are the definitions (the real gold is in the comments that you see in the document for each definition, make sure you download it and read it carefully):

Page: A page is an analyst definable unit of content.

Page Views: The number of times a page (an analyst-definable unit of content) was viewed.

Visits/Sessions: A visit is an interaction, by an individual, with a website consisting of one or more requests for an analyst-definable unit of content (i.e. "page view"). If an individual has not taken another action (typically additional page views) on the site within a specified time period, the visit session will terminate.

Unique Visitors: The number of inferred individual people (filtered for spiders and robots), within a designated reporting timeframe, with activity consisting of one or more visits to a site. Each individual is counted only once in the unique visitor measure for the reporting period.

New Visitor: The number of Unique Visitors with activity including a first-ever Visit to a site during a reporting period.

Repeat Visitor: The number of Unique Visitors with activity consisting of two or more Visits to a site during a reporting period.

Return Visitor: The number of Unique Visitors with activity consisting of a Visit to a site during a reporting period and where the Unique Visitor also Visited the site prior to the reporting period.

Entry Page: The first page of a visit.

Landing Page: A page intended to identify the beginning of the user experience resulting from a defined marketing effort.

Exit Page: The last page on a site accessed during a visit, signifying the end of a visit/session.

Visit Duration: The length of time in a session. Calculation is typically the timestamp of the last activity in the session minus the timestamp of the first activity of the session.

Referrer: The referrer is the page URL that originally generated the request for the current page view or object.

Internal Referrer: The internal referrer is a page URL that is internal to the website or a web-property within the website as defined by the user.

External Referrer: The external referrer is a page URL where the traffic is external or outside of the website or a web-property defined by the user.

Search Referrer: The search referrer is an internal or external referrer for which the URL has been generated by a search function.

Visit Referrer: The visit referrer is the first referrer in a session, whether internal, external or null.

Original Referrer: The original referrer is the first referrer in a visitor's first session, whether internal, external or null.

Click-through: Number of times a link was clicked by a visitor.

Click-through Rate/Ratio: The number of click-throughs for a specific link divided by the number of times that link was viewed.

Page Views per Visit: The number of page views in a reporting period divided by number of visits in the same reporting period.

Page Exit Ratio: Number of exits from a page divided by total number of page views of that page.

Single-Page Visits: Visits that consist of one page regardless of the number of times the page was viewed.

Single Page View Visits (Bounces): Visits that consist of one page-view.

Bounce Rate: Single page view visits divided by entry pages.

Event: Any logged or recorded action that has a specific date and time assigned to it by either the browser or server.

Conversion: A visitor completing a target action.

There is a lot more value added content in the document, it lays out key context that will help you think through and understand these definitions. Please download the WAA standards document.

In closing I would like to once again thank the co-chairs and the Volunteers for their hard work (with a special big warm hug for Angie!!).

This is fun stuff. Perhaps you'll consider joining the WAA, if you are not a member already?

Angie, Jason, the Volunteers and I welcome your feedback and critique of the definitions, please share your thougths with us.

Web Analytics Standards: 26 New Metrics Definitions is a post from: Occam's Razor by Avinash Kaushik

An Analysis Ninja, let's call him Philip Walford, asked a delightful question. Philip wanted to know if the impact of a faith based initiative in his company, product demo videos, could actually be measured using data.

Hurray!

Faith is good. Data is better. : )

[And before you flame me: know that I love my religion more than you love yours. Wait. That did not come out right. Let me rephrase that.]

In this thanksgiving week 2008 post I'll share Philip's question about how to identify value of video product demos on an ecommerce site, and my answer about involving customers.

Here's Philip. . . .

We are a large retailer with a lot of product on our site. In the past we have invested lots of dollars and time producing things like demo videos for our products, or adding other features and tools to our website to provide more information about a product. Our goal is to inspire customer confidence in their purchase (by giving them as much information is possible).

The question is, what are the KPIs of things like a demo video.

My recommendation was to measure conversion rate for the segment that views the video. If conversion is higher then the videos are bringing value. Others in my company have presented the hypothesis only customers that are a lot more invested in buying the product are likely to click on the video link and hence "pre qualified", hence that segment would have had a higher conversion rate regardless.

I understand their perspective but I feel they are reading too much into the situation but I don't know how to argue this point. There are several directions we could go with this but I wanted to see if you could share some guidance on this issue.

My answer to Philip. . . .

This is a complex problem, more than might be apparent on the surface.

It is also an example where it can be easy to jump into bed with your web analytics tool to get satisfaction but you wake up in the morning feeling. . . . well. . . . less than satisfied.

But before we go there I have to give a ton of credit to Philip and his crew for being skeptical of reading too much into their own opinions or biases.

I firmly believe that people who work for a company rarely (never!) represent customers. They are too close to the company and too different.

Just because I work for Microsoft and use a Zune (yes I do!) does not mean I can be a effective customer representative of Microsoft Zune customers. Company employee opinions rarely reflect those of customers. Do please be aware of that.

So when looking to make decisions, look for data (quant or qual).

I'll present Philip with three solutions / options as he battles the challenge of figuring out if the investment of muchos dineros in creating product videos is worth it (besides the fact that these videos ooze sexiness!).

1) Use ClickTracks (Compute Contextual Influence)

There are two challenges with using clickstream data and the "typical" measure of conversion rate to determine success.

A] You might be looking at a "biased" segment (as challengers to Philip's recommendation mentioned). I.E. Only the highly motivated people.

B] By comparing all people who converted and viewed the video with those that converted and did not see the video you are not comparing fair segments. You are also lumping all other "convince our visitors to buy" tools into one large bucket. Tools like Comparison Charts and Product Screenshots and Product information and Customer Reviews and more.

It is quite possible that those other tools might be getting people to convert at a much higher rate and by dumping them all together you are not being fair.

And of course you'll get a wrong read on conversion impact of the videos.

So even if you use your web analytics tools (your Google Analytics or Omniture or WebTrends or CoreMetrics or whatever) try to compute "contextual influence" (value of each feature in context of the others).

It is actually very hard (damn near impossible) to do this in all those tools (even for the Paid solutions, even after you plunk down half a million dollars for the mandatory Data Warehouse "add on").

ClickTracks is the only tool I know of that can do this out of the box, using its terribly named "funnel report". No data warehouse. No extra tags or variables or sprops or wt_&*#$. In fact not even much IT, I just need admin access to my tool (not site, web analytics tool).

Its easy to use. Create a hierarchy of your website. Add individual or groups of pages into each stage (notice I did not say step because you can jump steps here). Add an outcome (in my case say "Thanks for placing your order" page). Click Calculate.

Boom!

[You are not supposed to be able to read the analysis, sorry, privacy dictates that.]

What I want you to note is two things.

This is a site where each stage means a view of the site (and like a traditional funnel how many people get in, get out, move on etc).

Secondly note that each box (which represents a page/'s or a tool – videos, comparisons, reviews etc) has a different stage of blue.

What this lovely report does for you is compute "the influence"of each of those pages/tools in driving the ultimate outcome – purchase here. The darker the blue the more "influential" that piece of content. [Influence is defined by the existence of that piece of content in the visitor session, regardless of what path the visitor took, regardless of when the content was seen.]

Ain't that super sweet?

The analysis you see above is for a real ecommerce website. What it proved to us, delightfully, was that the product videos, we had created at a cost of over one hundred thousand dollars, yellow star above, was the least influential tool we had on our site.

The most influential, sexy pink star above, was a tool that had cost us $8 to produce – it was a page that compared different versions of the product (information that was handily available in the company).

We used actual customer behavior. We analyzed contextual segments. Ultimately it allowed us to put our precious few resources in the right area.

Of course it is quite likely that everyone who came to the site and did not buy (convert) might have loved the videos and rushed to stores to buy our products (one HiPPO actually said that!). There is no way to prove that using just the web analytics data.

What we did is proved impact on online buyers.

As to the HiPPO. . . . read on. . . .

2) Use Surveys (Actively Collect VOC)

When in doubt (or confronted by a HiPPO, remember don't run) what better way to go then gather some Voice of Customer. Dare I say the voice of god? :")

Two things I have tried (of many!) that work a lot of the times. Each covers one unique bucket of visitors to your website.

A] Consider sending a simple post purchase email survey to customers who have purchased on your site and ask them for the key influencers of their purchase.

You could share with them the various tools you have on your site (product information, comparison tools, images, videos, customer reviews etc etc) and simply ask them to rank order them in order of importance.

Don't ask them to tell you how much they like them, or choose ones they like, they tend to pick all. :) Just ask them to rank order. Or use a tactic similar to that.

This tells you want works for those who buy.

For the 98% that will never convert on your website. . . .

B] Consider a onsite survey like 4Q (though 4Q can only be customized so much so perhaps you want to use either your own or one of the big daddy paid survey tools).

This will go to a small random sample of people who are on your site (who may or may not buy). You'll ask them three or four questions about why they were there (primary purpose) and then what tools/features of your website they liked (rank ordered if at all your survey company can do that).

That will give you what you want.

Since this can also be thought of as a page level problem, you can also use something passive, a page level survey / poll, like Kampyle on your product pages and ask people to quickly rate the various features. There is a Site Content feedback topic in Kampyle which you can customize.

Now you have the most important piece of data you need, your customer's. Few website owners / marketers / hippo's can argue with this. Leverage this advantage.

Finally one last option for you. . . . hopefully one you'll use before you write a chq for a hundred grand to create your videos. . . .

3) Use… wait for it….. Testing! (Measure Actual Customer Behavior)

I am sure this does not surprise you. Run a A/B or Multivariate Test and let your customers help inform you of the value of these features.

For 30% or 40% or whatever %, don't show the product demo videos and for the rest show the product demo videos and see the impact on the data. Boom (!) you have your answer, without any biased opinions.

It is certainly going to take you a small amount of effort, get the Website Optimizer, talk to your IT folks, create version of the page with no product tour link etc.

But you are making a very expensive decision for your company are you not?

And here is the additional benefit of testing. You are free to use any kind of "conversion".

You can measure success as conversions (submit order).

You can measure success (of the test) as number of people abandoning from the product page.

You can measure success as the time people spend on the product page. [There is a very cool javascript code that does this with the Google Website Optimizer, it is especially helpful for rich media / flash sites. Without a doubt other vendors can do this as well, just ask.]

You can measure success through your survey tool if it is integrated (this is some extra work sadly, but for big bets I recommend it).

You can integrate your analytics tool with your testing tool (say Google Analytics with Website Optimizer) and use other metrics to measure success such as bounce rate or electric shocks etc :).

[For GA and GWO ROI has integration instructions .]

The bottomline is that you can define success and then let the customers tell you.

That's my answer to Philip.

Sounds exciting?

Am I the only one who thinks when you do this kind of analysis you are in a nearly orgasmic state?

Yes these methods are some small amount of work. But nothing in life worth having is easy. The tools might be free, but that does not eliminate your need to investing your time and effort! :)

And on the positive side with a recession looming people who involve customers in making decisions, rather than their opinions, will win big. The "guessers" will not win big. They might even win small. Or fail.

Plus if you do this you'll be a Analysis Ninja, not a Reporting Squirrel.

Ok now your turn.

Have you tried to analyze the features like Video Demo's on your website? Or perhaps other complex features you have launched? What works for you? What totally failed? In my recommendation to Philip, what did I overlook?

Please share your feedback, critique and hurray's.

Excellent Analytics Tip #14: Measuring Value of Ecommerce Sales Tools is a post from: Occam's Razor by Avinash Kaushik

Life has been a bit rushed for me recently and with all that I forgot to answer one very interesting question that Alan had asked. Since I have a few minutes now I thought that I'll answer the question now.

Alan asked for my thoughts on how a full-time web analytics person should spend their working week in order to maximize their opportunities for massive success both for themselves and their company.

My answer is below, preceding that is some context about the 80/20 rule for reporting (that I had first mentioned at the emetrics summit a couple years back).  The 80/20 rule for Analysts is a perfect lead into to the next question, as you’ll see…….

ALAN: You've also suggested a 80/20 rule for reporting: analysts should spend 80% of their time doing analysis and only 20% reporting. How much data should a good web analytics dashboard be? How many key metrics should an online retailer track to understand their site? And which ones?

ALAN: Let's imagine a hypothetical full-time web analytics person working for a major online retailer. Best case scenario, in a typical work week, what activities would he or she be doing, and roughly how much time would he or she be spending on each?

If it is a major online retailer the first thing I would recommend is that they also hire a young person, perhaps a intern, to take over the reporting responsibilities (or pay someone at your Vendor / Consultant to do it). It is important to realize that this is the first step.

In any large organization you can’t get away from reporting demands from lots of folks (as in “our department needs these three reports every two days” or “we have to publish these standard metrics company wide” and so on). Your chances of massive success are low with just reporting and so please don’t expect your Analyst to do reporting and for that to in turn to bottom-line impacting results.

A fresh person out of school should cost you a lot less than your Analyst, and they, Mr. or Ms. Fresh, will work enthusiastically on reporting (and all the while get great training on becoming a true Analyst).

So if you are a Analyst then here is what I recommend you start with (customize for your organization as you see fit):

• 20% Reporting – Sorry you can’t escape this, you are still going to do reporting. But on the bright side it is a great way to keep in touch with reality.

• 20% Analyze Acquisition Strategies – What is your company doing to attract traffic to your website? SEM? Affiliate Marketing? Banners? Email Marketing? What else?

There is no faster way to win the hearts and souls of your company stake holders than helping them understand how their efforts are performing.

Focus your analysis on Outcomes (revenue?) that your company desires and you’ll do fine.

• 20% Understanding On-site Customer Experience – Using a mix of ClickStream and qualitative methodologies analyze what the customer experience is on your website. Really. Not what you think it is, not what your company wants it to be. But what it is in reality.

There is no better way to make money then this (and you’ll get happy customers as a bonus).

• 20% Staying Plugged into the Context – Most Analysts suffer because of a lack of context. They are put away in a corner with Omniture or WebTrends or HBX or Google Analytics and expected to produce earth shattering insights.

We need to have our Analyst use 20% of their time simply to stay plugged into what else is going on, in the marketing organization, on the websites in terms of operational changes, with senior management (anyone higher then their boss) to know what their strategic pain points are (then imagine how web data can solve them), with the phone or retail (big box) channels, etc.

Web analysis is not a silo and the analysts needs to be plugged into the context so that they can look at the right data, better and provide relevant insights (that they can’t possibly provide when locked in their ivory tower (!!).

• 10% Explore New Strategic Options – I don’t know where your company is but you always want to move the ball forward, this chunk of time should be spent in experimenting with new and different ways to move your program forward.

Think testing, competitive intelligence analysis, multi channel integration, usability etc. If you really are a one man band this is really hard to do (especially with 10% of your time) but think of tiny ways in which you can show that web analytics is not just about ClickStream, it is about creating better customer experiences and it is about creating a strategic advantage using data.

• 10% Bathroom breaks , oh and lunch! I am generous aren’t I. : )

I realize that each company is unique and each Analyst is unique, but I hope that the above picture provides a semblance of universal guidance for anyone how has the word Analyst in their title. Do a quick back of the envelope of how you spend your day/week and compare it to the picture above. What do you find?

Bonus reading material: Top Ten: Signs You Are A Great Analyst.

What do you all think? Am I being reasonable with how Web Analysts should spend their time? Is 20% too much time for staying plugged into gleaning the context? Not enough? What about the 80/20 rule for reporting success? Please share your thoughts and critique via comments.

[Like this post? For more posts like this please click here.]

How Should Web Analysts Spend Their Day? is a post from: Occam's Razor by Avinash Kaushik

Two of his questions put me in a bit of a quandary (aren't all good questions supposed to do that?). Question three because it would have me postulate on what I would like to see in Microsoft Gatineau, a tool that is pretty much a done deal from what I know. Question four because well you’ll see.

Here is, if I had a four wishes, my wish-list of what I would love to see in Microsoft’s upcoming web analytics application: Gatineau:

3. What tools, features, and reports would you like to see in the upcoming Microsoft Gatineau product?

Hmm…. I don’t think my friend Ian Thomas has quite the luxury to build whatever I want, but let’s assume he does.

# 1: I hope that with Gatineau Microsoft figures out exactly who their target audience is and then delivers a tool exactly and specifically just for that audience. Being all things to all people means being nothing to anyone.

I guess I am saying I hope their tool does not have a billion standard reports out of the box, just the six that their target audience needs. Atleast initially.

# 2: Efficient segmentation. In four clicks (see I am generous!) anyone should be able to segment out traffic from the search engines or from a top referring url /’s or visitors who see x number of pages or come on a particular campaign (whose id is in the url or cookie).

It is very hard to dumb down the ability to do intelligent segmentation, yet that is the key to finding actionable insights.

# 3: Useful reporting for Search Engine Optimization. I love free traffic and with all the changes (especially at Google, such as increased “personalization”) the PPC gravy train is going to pause. SEO will become more effective at getting the right kinds of traffic yet today most tools pay lip service to the measurement of the results of SEO efforts, all you can do is measure organic traffic and if it goes up (that is hardly a measure of SEO).

I hope Gatineau can atleast tap into the MSN data and providing effective reporting for atleast MSN SEO efforts, it will definitely get a leg up on others if it does even this little bit.

# 4: Ok maybe I will ask for a reporting feature. I hope that all the reports will show one extra time period by default.

For example show eight days in a “weekly” trend and thirteen months in a “yearly” trend. Seems like a small thing but most web analytics tools are not great at giving context, and context is king. If you look at a eight day trend you could compare this Monday to last Monday and get a feeling for if you are doing better or worse this Monday, with most tools you don’t see last Monday. Ditto for this month vs. same month last year.

It gives context to your past performance and is a “internal benchmark” that can frame current performance. Might not scream answers at you but will get you to ask the right “why” and “what” questions.

There is nothing uniquely Microsoft Gatineau about the above three requests, though if they are really starting with a open mind it might be easier for them to consider requests from random bloggers such as myself.

4. Do you foresee a decline in the major players in the web analytics field such as Omniture and WebSideStory based upon free web analytics packages, or do you believe that they fulfill a niche and will remain?

For the answer to this question please read Jeff’s interview, you’ll find a interesting answer and learn a new term: YATR (Yet Another Tsunami of Reports). Curious? Click here.