A Great Analyst's Best Friends: Skepticism & Wisdom!

Six DropsHere's something important I've observed in my experience in working with data, and changing organizations with ideas: Great Analysts are always skeptical. Deeply so.

This was always true, of course. But, it has become mission critical over the last few years as the depth, breadth, quantity and every other dimension you could apply to data has simply exploded. There is too much data. There are too many tables/charts/"insights" being rammed down your throat. There has been an explosion of "experts."

If you are not skeptical, you are going to die (from a professional perspective).

And, yet… You can't be paralyzed by skepticism. At some point, you have to jump. Or, you are dead (again, professionally).

Let's do this post in two pieces.

First, a plea to be skeptical, of everything and everybody, illustrated using an example from one of the most respected sources of data out there. Followed, by advice on getting to a decision rather than what happens to poor analysts: paralysis.

Second, as we are on the topic of great analysts, I want to share how to recognize that you might be one, from a macro perspective, and, if you are, or are not, what's your value to your company.

Surely, you are intrigued!

#1A: Skepticism is your BFF.

I saw these two numbers presented the other day: 42% of online shoppers use video for pre-purchase research. 64% use YouTube to find products.

As soon as I heard them, I knew they were horse-manure.

The source of skepticism was simple, neither number is true for me – and I'm in a place, with people, who are the most connected people on the planet with more devices to do this type of research if it was true. I stood up. Did two things. I asked the 100 or so people in the room if either of these two numbers was even close to reflecting their reality, one person raised their hand. Then, I asked for the source of data. A 2014 AOL report and an online survey with n=600.

It was horse-manure.

Yet, they were being presented as facts on a tablet handed to Moses.

You might not yet have the experience to know if a number is true or not, perhaps you are evolving. But, if you actively invest in your education, awareness, being hungry to always want to dig just a little deeper, you'll get there in no time.

For example, you might read NADA say this: "85% of customers made up their mind to purchase a vehicle before they left their house." Your skepticism radar should go beep, Beep, BEep, BEEp, BEEP, and you should stop and listen to it. It does not matter how big NADA is and how many analysts they have – because accepting imprecise information will cause you to make career-limiting recommendations.

Here's a great example of hopping on the skepticism train right away.

[Update: As Mauro Avello points out in his comment, this could be an April Fool's joke by the team at The Economist. I did notice the date when I wrote this post, and read the comments and follow ups, and did not see anything that indicated to me that it was a joke. Hence it was used below. The lessons you'll learn still stand, but please do be open to the possibility that this was just a joke.]

The ever wonderful data viz team at The Economist had an irresistible link: Ice Cream and IQ.

Hard not to click on that, right?

It is a short article containing a line chart plotting ice cream consumption on the x-axis and the mean score on PISA reading scale…

economist ice cream-PISA scores

THE DATA TEAM (that's who the article is credited to) go on very seriously to share that more ice cream eating might be the solution to poor student score. They dutifully compare the Aussies and the Finns, commend the Canadians and crap on the Peruvians.

So. You are the smart Analyst.

Your first skepticism flag should be: The title of the article says IQ, do PISA scores measure IQ? Quick Google search. They do not.

Red flag.

Your second skepticism flag should be: Look for things in the data-set that disprove the summary statement. Notice Hong Kong, Singapore and it's neighbors have very high PISA scores, yet very low ice cream consumption.

Red flag.

Your third skepticism flag (for a smart Analyst, usually this is the first one) should be the perennial favorite: Correlation does not imply causation!

You pore over the article for signs that this simple rule is not broken. Is there anything that shows they looked into causation? No.

Giant red flag.

And at this point, a tiny part of you also died because you do so love THE DATA TEAM at the Economist.

To normal people (non-Analysts), this graph and article looks legit. After all this is a reputable site and it is a reputable team. Oh, and look there is a red line, what looks like a believable distribution, and a R-squared! Most normal people will take this as truth (and at least 67 of them will proceed to comment on the article and have fun).

You should not.

The thing that should go through your head is… Causation. What could cause this?

Here's one hypothesis: People who really care about their kids educational accomplishments come from families that tend to have parents who are a little bit well off – middle class -, they can focus on the kids. These families usually reward educational accomplishments. The reward of choice tends to be ice cream!

Remember, it's an hypothesis. We can go look for data. If it turns out to be true… It is not ice cream consumption that is the reason for the performance scores, it is the fact that families tend to have a certain income. Or, that they tend to have structured work time, which gives parents free time to focus on how their kids are doing vis-à-vis education.

There could be a number of other things. Weather. Number of women in the country. Longitude. Number of child workers. Crime. Anything honestly.

Look for causation. No causation means… data crime against humanity.

Let's bring this baby home, one more example, this one a bit more fun.

There is an extremely tight correlation between the amount of US spending on science and suicides by hanging… r-squared of 0.997…

hanging suicides-US science spending

If you are with me thus far, you are screaming that there is no causal connection between the two!

And, you would be right. Spending more money on science (please let's spend more!) will not result in more suicides. Though the two are as tightly correlated as any two things can be.

[The above graph is from Tyler Vigen. His website – and book – Spurious Correlations is wonderful. You can checkout many more correlations, and laugh and cry and laugh and cry. Start with the one about Nicolas Cage movies causing people to drown!]

Let's look at another example, because it is fresh off the presses.

All of us travel and it is of immense interest to us as to which airline has high "quality ratings" when it comes to performance. Here are ratings that came out today…

airline quality ratings 2015

Most press reports you'll read about these performance ratings will talk breathlessly about the ranking and the movement up or down of a particular airline. What none of them do is talk about how this data is calculated. But, you the smart Analyst, will have your skepticism radar up and you'll dig!

Your starting point can be the press release from the source. There is more drill down data there.

Then you'll poke around to figure out how on time arrivals and departures are actually calculated, who sets the standards/formulas, what kind of control airlines have when it comes to setting their schedules, who decides what data gets reported and who audits it, and more lines of inquiry before you buy any of this.

When you dig in, you'll earn that there is actually no standard for what the word delay means. Airlines are in full control of setting the duration of a flight. For example, my flights to JFK on United routinely arrive "on time," even though they depart 20 mins late, because United adds a half hour to the "scheduled flight time." That padding then controls what is reported (crap basically). Across airlines there is no standardization of how duration between two destinations should be set – so basically the ranking above is comparing apples and monkeys and asteroids. The airlines, like Frontier, that set the tightest durations, most accuracy, are being penalized in a way for good behavior.

And, this is just the start of the problem.

Very quickly you'll realize, the rankings and the data above is basically garbage. If, at the end of the day the purpose of this data is to help you make a smarter decision, it is miserably failing to do that because of the above issues.

Skepticism. A good thing in an Analyst because you know what to use as a source of decisions, and what not to.

Most data you see in the real world won't be as obviously wrong as you'll see in the gems shared by Bad Fox Graphics. The example's you'll see will be more subtle, they will look like they make sense, they will come from sources you trust, from tools you use and even implemented yourself, etc. That's when you need to be most vigilant of all to be a great Analyst.

Here are some techniques you can use:

1. Look to see if the conclusion ("insight") expressed has anything to do with the data in front of you. This, honestly will only take you a couple minutes.

2. Here's a great question: Where did the data come from? Tools, countries, people, devices, etc. Known gaps of what's unknown (particularly relevant in digital data).

3. Another one that you'll love: What types of bias might exist in the data? Sample bias? Sampling bias? What could cause it to be incomplete?

4. What principles you've learned already that might be broken by the analysis presented? Correlation/causation is the one we covered above.

5. Always, always, always ask this question: What assumptions were made in doing this analysis?

6. Your experience. You have a ton of it. Don't let it go to waste.

7. (Added via Richard Hren's comment below) Who gains or loses from this analysis? As with many things, follow the money, power, politics.

Numbers 8 through 12 were contributed by Ian Frantz…

8. Was the data collected with a measurement plan?

9. Was this data intentionally designed or is it the by-product of another activity?

10. Have we accounted for the attenuation of data as it was collected via a specific medium?

11. Did you create statistically wise boundaries when you choose these arbitrary categories?

12. If you handed all the data, order of operations, software over to me; could I reproduce the results?

Numbers 13 through 16 were contributed by Rod Jacka…

13. Clarify the reason for the claim

14. Test alternative explanations that can be concluded from the data or evidence

15. Challenge the implications and consequences of the claim. Would it follow that … will occur because of ….

16. Above all else, question the question itself :)

Numbers 17 through 24 were contributed by John Brazier…

17. If the data is given as a proportion, what were the raw values? (a 50% rise in shark attacks could be an increase from two to three cases)

18. If the data is a raw value, what’s the proportion? (a station I pass through every day warns that 19 escalator incidents have taken place in the last year. Since it has around 90 million entrances/exits per year, I think this number is pretty irrelevant)

19. Could the sample have been run multiple times? (If you get 10 batches of 20 women to test a hair-care product, you’ll likely find that one of the batches will produce a high enough satisfaction figure to advertise)

20. Is the law of large numbers a problem? (A UK website compiles statistics about response times for Members of the UK Parliament. On average, the Green Party is the fastest by a long way; they have 1 MP, compared to parties with more than 50).

21. Is a proportion penalising you unfairly? (Your site’s treasured 60% conversion rate might be damaged if you were picked up by a major news network and given 100,000 low quality hits. That doesn’t make it bad thing!)

22. Are you making a fair comparison? (‘Views’ on Facebook aren’t the same as ‘Views’ on YouTube)

23. What aren’t you being told? (Any news story about a charitable event that doesn’t mention the volume of funds raised is likely hiding that they didn’t make very much)

24. Does the person writing the statistic understand it? (Second-hand statistics are often missing crucial caveats)

I'm sure there are others. Would you please help me expand this list by adding techniques you've learned to help bring your healthy skepticism forward by adding a comment below?

When you see a piece of data, from inside your company or from the outside, be skeptical in general. It is a good trait to have as an Analyst.

#1B: Skepticism should not paralyze you.

You are going to feel I'm going to run all of the above under a bus now. Please stick with me.

The real world is not perfect, and you are paid to help your company (non-profit or for-profit) make smarter decisions every day (hopefully). One important thing at play here? A decision has to be made.

Novice Analysts get so caught up in the skepticism that they become paralyzed because if you even lift the covers under digital analytics a tiny bit, the way data is collected with scare the bejesus out of you. Oh, and offline analytics? A million times worse. And, tiny data samples to boot!

Great analysts get good at one of the most critical elements of our jobs: Timeliness. The ability to deliver an insight, a specific recommendation, in a duration that it will have an impact on the business.

An educated mistake is better than no action at all.

Our job is to be skeptical, to dig and understand and poke and prod and to reject the outrageously wrong and if it is not outrageously wrong then to figure out how right it might be so that you can make an educated recommendation.

This post is from 2006: Data Quality Sucks, Let's Just Get Over It You'll learn the six step process you can use to overcome the paralysis.

Here's a simple way to think about using your skepticism, but still making a decision.

If you were 100% certain about the data, you would immediately recommend to your company that they should start making plans to build a colony on the moon.

If you were 80% certain about the data, you could recommend that they shift the strategy with the International Space Station to start sending short visits to the moon.

If you were 40% certain about the data, you could still recommend a tripling of the investment in entities on earth that would study how to live on the moon.

If you were 20% certain about the data, you would go back to your team and figure out what strategies you all should put in place to get to at least 40% certainty.

That's what I mean by being skeptical – your quest is to get a more concrete feel for where that certainty lies. By not being paralyzed by perfection, I mean making a decision that reflects that certainty because the business needs timely decision making.

We are on the topic of being great Analysts. So. Here's a detour, and yet on theme extension of that idea.

#2: The Difference Between Knowledge-Insight-Wisdom.

As some of you know, I'm writing a twice-a-week short newsletter called The Marketing – Analytics Intersect. You can (should!) sign-up for it.

I often find that most people who have the title Analyst are essentially data collectors and data sharers with most of the value being added by them in the process is a tablefied or chartfied summary.

My newsletter on 29th March, shared a fantastic cartoon that exquisitely captured the difference between data – information – knowledge – insight – wisdom. It also added a layer, dare I say, wisdom by outlining the value of the job, salary and how quickly you can be replaced in the job.

Here's that TMAI, in it's entirety, I'll pick the story up again on the other side…


TMAI #12

Of all the cartoons related to data, and analysis, this one is my all time favorite…

data information knowledge insight wisdom

[Cartoon by David Somerville, based on a two pane version by Hugh McLeod.]

Isn't it incredible, it captures so much about the work we do in so little.

I love this cartoon because there are so many insights, :), to draw from it. Let me focus on one, how valuable you are to your company.

Data: You are a javascript jock, the slayer of ETL challenges. You are the data hunter and gatherer. Value: Low. Salary: Low. Replacement: Easy.

Information: You are a report creator, you fix code in emergencies. Value: Low. Salary: Lowish. Replacement: Easy.

Knowledge: You run a team of data pukers, you help meet divisional data needs, your team merges data sources. Value: Medium. Salary: Medium. Replacement: Takes two months.

Insight: You hold the Analyst title, most of the time you avoid being see as a data provider, you get invited to director-level business meetings. Value: High. Salary: High. Replacement: Hard, six to nine months.

Wisdom: You are an Analyst, but sit in a business team, d3js.org is your second home, you meet with the CMO every other week. Value: Priceless. Salary: High times 5. Replacement: Impossible'ish.

So, what job are you doing at your company? Information? Insight?

Is there anyone in your company in the analyst team, or the marketing team, whose explicit job it is to deliver wisdom?

Yes, you want to be in the Wisdom business. But, realize how hard it is to do. You have to be on a constant quest for self-improvement, and the most powerful skills you'll bring to bear are your business savvy and not data-crunching prowess. Ironic, no?


Great Analysts solve for Wisdom. And, above and beyond what you read in the newsletter, you can see how both being skeptical and not being paralyzed helps you get to Wisdom faster.

One last bonus before we close… If you would like to get a sense for specific salaries, four key choices you have to make to have a fabulous analytics career AND how to get there… Here's a post you'll find to be of value: Web Analytics Career Guide: From Zero To Hero In Five Steps!

As always, it is your turn now.

Do you agree we are not skeptical enough about data floating around in our companies or the interwebs? What are strategies you use to fuel your skepticism? How do you torture the data you see/get? Is there something that works for you particularly well when it comes to solving for timeliness? Are you solving for Wisdom in your current job? Insights? What caused your career to leap from Data to Information to Knowledge faster?

Please share your wisdom, :), experiences, tips, tricks and lessons from the front lines via comments below.

Thank you.

PS: Topics covered in my last few newsletters: Should you analyze individuals? NO!, The Best Ecommerce Experience In The World, Formulate Your Life, The Very Best Metric: Email Marketing. You should sign-up!

Comments

  1. 1
    Darren James says

    I love this post Avinash.

    As demands on our jobs increase and we are asked to do more things the activity that falls by the wayside is truly understanding how data is collected. Your post is a handy reminder that regardless of the source we should strive to understand what we are looking at.

    Both the Airline and Economist examples are helpful in how we could evolve our approach.

    Thank you.

  2. 2
    Matt Fleming says

    A timely reminder on a lesson we learn early, yet forget quickly as well.

    In our case we are stuck in Knowledge. For some client we are in Insight. The reason we cannot get to Wisdom is two fold. As an agency we are too far outside the business to be able to have all the context and integration required to be able to connect the Wisdom dots. Clients also structure contracts to, using your language Avinash, be reporting squirrels which leaves us in Knowledge in the best cases.

    Have you seen cases were Agencies are truly in the Wisdom bucket?

    Thank you for giving us a way to think about this problem. The cartoon is printed and stuck on my office wall now. :)

    • 3

      Matt: Really great points.

      The only time I've seen external Agencies get into the Wisdom bucket, is when they are essentially treated as a part of the team. This sometimes is manifested by having them be inhouse even, a part of the day to day operating mechanism of the business team. Internal, even if it is out-sourced folks.

      -Avinash.

  3. 4
    Mauro Avello says

    Hi Avinash,

    As always great analysis and your points are well taken. However, and regarding The Economist data analysis team, I just wanted to mention that the day it was published is VERY meaningful in this case…

    • 5

      Mauro: I did notice that when I bumped into the post (on April 3rd). But, as I did not see an update, or anything in the comments, I decided to use it as an example.

      You make a great point though, and I've added an update in the post itself. Merci.

      I hope the principles still apply though to any analysis.

      Avinash.

  4. 6

    Some insights from a philosopher largely considered as one of the fathers of skepticism: David Hume.

    As you wrote "correlation does not imply causation", but it is also true that there are only correlations, you'll never find something like a cause in the real world neither in your mind, so don't waste your time looking for necessary causes.

    There are only objects (that we firstly perceive and then retain in our minds as ideas) and there are exactly correlations between objects, built by our minds "by means of custom".
    You will only find objects that are very often connected together and you will infer (I mean, you will believe), with a limited degree of probability, that the first is the cause and the second is the consequence, but you have to stay aware that it is only a belief, the only reality is that you experienced a frequent correlation between two objects.

    So, back to your article, and trying to survive to the radical skepticism (which is also the attempt of Hume, I think, similarly to your suggestion to face the always lurking skeptic paralysis), the fundamental red flag is not "Look for causation" (a pointless attempt), but it is exactly your:

    "Your experience. You have a ton of it. Don't let it go to waste."

    Only our experience can help us evaluate if a correlation is meaningful and can somehow work in a specific context or if it is just a great mistake, a sort of a non-sense (similarly as philosophers that confusedly claim to demonstrate the necessary existence of God by stating that everything has a cause so the whole Being need a first cause as well).

    • 7

      Matteo: I love your comment and I'm grateful to you for bringing a philosophical element to our discussion. Anytime we are connecting an analytics post to the existence of God discussion, we are getting somewhere. :) Thank you!

      I live in the world of "explain things", and usually, unlike in nature, things are fairly controlled, or can often be more clearly defined. Hence, in these cases, often, we can find causality. Sometimes it takes a bit of effort.

      But, I totally get your point. I appreciate it very much.

      Avinash.

  5. 8

    technique 7: "was it posted on April Fool's Day"? (re. Economist, ice cream and your faith in their data team)
    :)

  6. 9
    Richard Hren says

    Another great article.

    The material in this discussion is exactly what's needed as part of a general education foundation. The ability of understand data, ask questions, and retain a healthy skepticism of what is being offered as a solution to any given problem. Don't teach coding, or statistics per se; teach the fundamentals of numbers and experimental design..

    One additional skeptic questions should always be: Who gains or loses from this analysis? As with many things, follow the money, power, politics.

    And my favorite part of the piece…. "Oh, and look there is a red line". Priceless

    Thanks

    Richard Hren

  7. 10

    When I first scrolled pasted this economist publication on FB, I thought this for sure would have been an April Fool´s joke (just have a look at the publication date). So I dismissed it right there (being a skeptical analyst and all).

    I see a couple of commentators that are also on to them, but I don´t see the Economist admitting it actually is a joke. But let´s hope it was!

  8. 11

    Skeptical Questions:

    Q: Was the data collected with a measurement plan?
    Q: Was this data intentionally designed or is it the bi-product of another activity?
    Q: Have we accounted for the attenuation of data as it was collected via a specific medium?
    Q: Did you create statistically wise boundaries when you choose these arbitrary categories?
    Q: If you handed all the data, order of operations, software over to me; could I reproduce the results?

  9. 12

    Hey Avinash,

    I've been thinking a lot about data to wisdom over the past few years and believe that most of the time analysts don't actually get to see the data.

    When I look in GA, I don't see data, I see information. It's been processed, categorized and deduplicated in some form or the other. When I take idea a step further I believe many analysts rely on our tools to get to the knowledge stage, with tools like Contribution Analysis from Adobe or Adometry from Google, the work in creating a connection between disparate data points is mostly done and packaged nicely.

    The real burden is Wisdom, which is the unicorn in analysis. The gap between knowledge and wisdom is so great I think there may actually be a step in between, haven't figured it out yet but I think experience, skepticism and context are factors to it.

    Cheers,
    Eric

    • 13

      Eric: : )

      I do appreciate your point, but I'll disagree.

      Smaller disagreement… I believe that our tools get us to Data and not Knowledge. Maybe with Contribution and Adometry, we are at Information. Sadly, they have some ways to go before they get to Knowledge (maybe the automated Intelligence Alerts we are at Knowledge?).

      Slightly bigger than smaller disagreement.. When I log into Adobe/GA/IBMI see processed data as well, but data that is ready for analysis. Somebody's got the clay all ready for me to be shaped, molded into a story. I don't need to go out and dig my own mud, check the composition, process it, water it, get the right consistency etc. etc. I take over when it is ready for value added work.

      As you correctly emphasize, it does not excuse the Analyst from knowing deeply how the mud was created, but they don't need to start from scratch.

      Thank you for making me think harder about this, about your comment on Wisdom. Much appreciated!

      Avinash.

      • 14

        It is hard to disagree with you because your analogy is solid (pun intended). I believe there is then an opportunity to redefine what data is because I don't believe that it takes the same skill set to analyze data that comes from pure database extracts vs data found in analytics platforms. For me the thing that we see in an analytics platform is a photoshopped image not the raw image, that means someone else (the person who photoshopped or the people who create the stored procedures to aggregate the data) have applied their analysis to the data.
        This may all just be semantics but I do think there is a fundamental difference between where we start our analysis now vs where we started with just raw log files.

        Coming back to your analogy, if the clay we are provided is poor quality then the value add is not possible. I'm not just talking about a bad implementation because that is a whole other post (or 10) but we rely on Adobe/Google/IBM to aggregate correctly which to me is process of data transformation.

  10. 15

    Hi Avinash,

    Some techniques to add (well perhaps not just techniques)

    Question why was the data, research, whatever created in the first place?

    Use the classic Socratic Questioning process

    – Clarify the reason for the claim
    – Identify and understand the assumptions used (This is covered already)
    – Assess the evidence used to support the claim (also covered)
    – Test alternative explanations that can be concluded from the data or evidence
    – Challenge the implications and consequences of the claim. Would it follow that … will occur because of ….
    – Above all else, question the question itself :)

  11. 16

    Nothing to say.

    As always an awesome post.

  12. 17
    Quentin says

    Great article as always.

    The Ice-cream chart is a great example of the Texas sharpshooter fallacy:

    "You cherry-picked a data cluster to suit your argument, or found a pattern to fit a presumption."

    https://yourlogicalfallacyis.com/the-texas-sharpshooter

  13. 18

    20 years as an consultant in the IT field and it took years to make the business understand data should be processed to make it informative and today with Big data, every data server is brimming and the question still remains if we are equipped enough to make any sense out of it to make way to better customer experience and engagement.

    Nice insight and gathered wisdom reading your article.

    Best
    Katie

  14. 19

    Amazing Post Avinash, thanks for sharing!

    I'm sure that will open the eyes of many people.

  15. 20
    Nico Hesser says

    If it is a chart, does the starting point of the axis make sense, or is it inflating or understating a trend/difference in data points?

  16. 21
    Gurmeet Singh says

    Thanks for sharing this blog post with us.

    Most blog posts only touch on the surface level, this post is very detailed. The list of skepticism techniques are priceless.

  17. 22
    Claire Gras says

    Hi Avinash, this article is excellent, thanks for sharing!

    How many times i have seen people take bad decisions because they look only for correlation and/or they derive "insights" from a very particular segment of customers whose behaviour is very different from a standard customer. ("We should show this content to all our visitors because customers who have seen it are converting far better" yes of course but maybe reason is that these customers were a lot more interested in buying in the first place and took pains to see this content just because of this and behaviour has nothing to do with the actual content…)

    Also how many analysts have no context at all about the data they are working on (reporting squirrels) and despite that CEOs are taking their conclusion as ultimate truth.

    And what to say when people conclude "customers just love this option", when this option was actually ticked by default…

    Thanks for enhancing that analysis is not just nice excel charts and that context & skepticism is the key!

    • 23

      Claire: OMG, I love the "show content to everyone because it is working for these handful" example. So many dead bodies on that path. :)

      I want to underline your emphasize on getting the business context. In interviews, this happened around seven or so years ago, I started to turn down candidates that did not demonstrate business savvy. They would have tons of data chops, but without business savvy, for reasons you point out, the outcome of those chops was a lot less impactful.

      Thanks!

      Avinash.

  18. 24
    Kayleigh says

    Hiya,
    I enjoyed reading that, thank you.

    I'm still new to the wisdom part of that…I think I've been wise a few times now but would like to be more consistent! haha! I find looking at our measurement plan and thinking about the client's business from a top level perspective always helps me. Taking a step back, going for a walk, making a cup of tea…just away from the screen always seems to help me think.

    As for skepticism -that seems to come very naturally to me. I liked the "Data Quality Sucks" article of yours. Whenever I find something funky going on with the data, I always weigh up looking into it against client priorities, how much it could affect insight and whether the time investment for the fix will change the insight/strengthen it. I didn't use to do that, and I think I'd pursue things that only I cared about that didn't really change anything. I really enjoy looking into that and will ask an apprentice to practice that but also realise that time is finite on a project/retainer and we should focus on what matters most.

  19. 25
    Shivani says

    Some of the key skills to get the successful analytic career, high sense of intellectual curiosity, mathematical oriented, big picture vision, and it make the visibility to differentiate between the tools

  20. 26

    Reminds me of the old adage, Figures don't lie, but liers figure.

  21. 27
    Dennis says

    Thanks for sharing, I enjoyed reading it, thank you.

  22. 28
    Niroshan says

    I felt the Skepticism when you share an article about High Expenditure in the Wedding vs Divorce rate. It means couples who are ready to reduce wedding budget may live together for long. There were no explanation about the causation.

    My hypothesis was divorce rate is higher for wealthy couples not the expenditure in the wedding :)

  23. 29
    Debbie Kay says

    I would add:

    – "On averages, averages are stupid" – Seth Godin. – Check for seasonality or other cyclical patterns. Remove outliers. Check if the data presented is averaging out trends you should be looking for (I've seen this so many times! Weekdays are not the same as Weekends and Sat and Sun cannot always be lumped together! My example is utilisation but am sure the same analogy can be made for digital analytics).

    – "The difference between try and triumph is a little umph." Sometimes, there are just no shortcuts – looking at a large new database of unknown data, you need sheer tenacity and just have to keep cutting the data, visualising in different ways (turn x/y axis around, change type of chart), drilling deeper, connecting the dots, asking why, until you see the light.

    – "The price of light is less than the cost of darkness." – Arthur C. Nielsen. I've seen analysts stress over assumptions that don't move the needle (esp those building VC models and I can imagine those allocating goal conversion values). Probabilistic decision making in the face of uncertainty! Just know where the largest drivers of change are and change your analysis / model / insights as and when you have do! Doesn't mean you stop analysing the data or building your models the more complex the data gets or the more unknowns there are. Build scenarios. Triangulate.

    – "Not everything that can be counted counts, and not everything that counts can be counted."
    Albert Einstein. Understand what drives value (i.e. profits) and focus! I.e. allocate a disproportionate amount of resources to it. As you've said on your blog and I've seen offline many times 20% of (__ fill in the blank _ customers / products / pages / keywords) drive 80% of outcomes (profits). Surprising how true this holds in almost every industry I've worked in. Segment your customers! Needs/behavioural segmentation, not by demographics! Same with products! At Bain we used to say sustained value creators were companies that "focused and profit from their core". But to calculate product / page / keyword profitability you need to you have to attribute cost correctly.

    And might I add suggest adding to your Return on Analytics model – a tab on Loyalty Economics. Especially since digital marketing has become so much more about the customer experience, the ability to identify who your loyal customers who go out of their way to promote your product are (as your detractors), quantifying the value and allocating the appropriate resources to convert detractors / promoters can can reduce your marketing costs by a disproportionate amount (via Net Promoter Surveys or other tools – just quantify the value appropriately and accurately for your business). LTV or CLV analysis often understate the value of a loyal promoter because it does not factor in the lower CAC, the value of word of mouth, higher retention rates, higher spend and lower cost to serve of a promoter. Hard to estimate for sure but worth it in all the companies I've seen do it right. Here's some data by industry from Australia about the lifetime value of promoters across different industries and businesses. http://www.bain.com/publications/articles/the-powerful-economics-of-customer-loyalty-in-Australia.aspx and how NPS calculates loyalty economics http://www.netpromotersystem.com/resources/toolkit/customer-lifetime-value.aspx)

  24. 30
    Adrian Bennett says

    Good to reflect on this.

    I can't help thinking that wisdom is about knowing the purpose for having the data in the first place, i.e. knowing what it will be used for.

    I don't know if I agree with the logical flow from left to right. I think the box labelled wisdom is actually insight. I think wisdom is separate because it doesn't necessarily mean that it is conferred on someone who has all/sufficient knowledge, but is being able to discern what information or knowledge is required and being able to find it.

  25. 31

    Great article, esp, the part about how the value we provide is directly proportional to how "dispensable" we are!

  26. 32
    Yann Pigeaire says

    Completely agree Avinash.

    As a twist on what you said about Hong-Kong vs higher countries, I would simply add that averages are meaningless if the std deviation is high. Look for an indication of variance/deviation.

    Nitpick for next time: you "pore" over data (not pour) and by-product (not bi-)

    Cheers,

    • 33

      I always like to go to the basics.

      1.) Take the data and try and report the exact opposite. This will tell you if you total 100% or more. It will also tell you if there is bias, immediately.
      2.) Even if your numbers add up to 100%, if you can some up with just 1 single other point or stat, you immediately know that you are measuring a subset and not the whole picture.

      I do this at each level and I quickly develop gaps and more accurate data.

Trackbacks

  1. […]
    In his blog post from April 4, “A Great Analysts’ Best Friends: Skepticism & Wisdom”, Kaushik talks about this surge in data and the resulting “explosion of experts” in the field. His post is a plea to be better analysts by virtue of skepticism and wisdom, as the post title would imply. Kaushik’s posts are typically lengthy, and this one is no exception, so I’ve summarized his key points for you below.
    […]

  2. […]
    Original article: https://www.kaushik.net/avinash/great-analyst-skills-skepticism-wisdom/
    […]

  3. […]
    » A Great Analyst’s Best Friends: Skepticism & Wisdom! Straight to the point as always, Avinash makes a great point in this post on what makes a great (web) analyst – the ability to be skeptical and always question data in front of you, while knowing when to stop and make a decision based on your (analysed) data.
    […]

  4. […] Kaushik] Encontrei a mesma imagem no post A Great Analyst's Best Friends: Skepticism & Wisdom! de Avinash Kaushik… Ele diz que foi feito por David Somerville, baseado num painel original […]

  5. […]
    TMAI is a monthly newsletter written by Avinash Kaushik. Kaushik is the Digital Marketing Evangelist at Google, co-founder and CEO of Market Motive, which offers online marketing education courses, as well as the author of two best selling books, Web Analytics 2.0 and Web Analytics: An Hour A Day. Some of Kaushik’s long-form emails get re-published on his blog, covering mistakes to avoid as an analyst and using your own skepticism to help make decisions.
    […]

  6. […]
    My friend Avinash Kaushik posted a wonderful article the other day about the importance of analysts to have a skeptical nature, and I absolutely agree with him. Skepticism, along with fact-checking, and a strong urge to take a step back to look at things from the larger perspective, is the key trait of anyone working with media strategies.
    […]

  7. […]
    Aqui é interessante cruzar com aquela famosa imagem “dados / informação / conhecimento / insight / sabedoria”, onde a coleta e o armazenamento seriam os dados, a classificação e categorização seriam a informação, a adição de informações seriam o conhecimento, a análise seria o insight e as recomendações seriam a sabedoria. É claro que não são exclusivos, mas dá para fazer uma associação simples para melhor compreensão de ambos os conteúdos.
    […]

  8. […]
    Avinash has a great post and links to the source of this cartoon. There are a bunch of broken links on here, but the content on this page is excellent.
    […]

  9. […]
    For beginners, it should be important to know that not everything you see on the internet is true and it is important to filter out this so-called “fluff” and make your experience as safe and enjoyable as possible. You should be quite skeptical of the information you observe online, whether it be from social media or anywhere in the web. As Avinash Kaushik said, professionally speaking; if you are not skeptical, you are going to die.
    […]

  10. […]
    Let’s break it down with Avinash Kaushik: Data: You are a javascript jock, the slayer of ETL challenges. You are the data hunter and gatherer. Value: Low. Salary: Low. Replacement: Easy. Information: You are a report creator, you fix code in emergencies. Value: Low. Salary: Lowish. Replacement: Easy.
    […]

  11. […]
    Sinceramente, deberías leerte este artículo del gran Avinash Kaushik para comprender un poco los peligros de confundir a estas dos ces. (¡Pero no ahora! Ande vas cosica, tú sigue leyendo y ya te vas luego vale?). ¿Comer más helado mejora el coeficiente intelectual? Otro claro ejemplo del grave problema de confundir la relación entre dos métricas o indicadores y pensar que uno causa el otro cuando, en realidad, simplemente es una coincidencia o correlación.
    […]

  12. […]
    One of the greatest traps when it comes to describing observations in data correlation is the use of inflammatory or sensationalist descriptions to prove a tenuous relationship and lead toward causation. Avoid this at all costs. Your executives will see straight through it. Instead, Digital Marketing Evangelist for Google, Avinash Kaushik, implores us to be skeptical and pose further questions to demonstrate our rigour. Ask yourself key questions:
    […]

Add your Perspective

*