In almost eighteen months of blogging this is a first:
a comparison of two tools.
I have not done comparison of various tools thus far, by design, mostly because this blog is less about tools and more about mindsets (Web Analytics 2.0 for example) and life lessons. And of course the 10/90 rule!
But last week Diana asked a great question about comparing Coradiant and Tealeaf (the former along with Maxamine was my recommendation for your Multiplicity Web Analytics 2.0 strategy). The answer to Diana's question means a temporary breaking of the rules, but I hope that you'll find it interesting.
At the very minimum this post is a peek into how I think when I evaluate tools and what I place more or less importance on. For what its worth, food for thought.
[If you are here for Web Analytics only, here's a post: Web Analytics Tools Comparison. Also please see the update at the end of this post.]
Diana asked me to elaborate a little more on my recommendation of Coradiant as a foundational tool, and not Tealeaf given that they are quite similar.
It is important to stress that my perspective on Coradiant is for the unique application that I am recommending it for:
Proactively identifying issues / trends of your users at a atomic user level to understand why your website technologically stinks at meeting the needs of your customers.
Why your conversion rate might be sub optimal, why your customer segments are not happy, why you are leaving money on the table (ecommerce or not)?
In my mind there are three main differences as you compare Tealeaf and Coradiant:
# 1 Time To Use:
With Tealeaf you'll buy software and consulting services and put them on hardware you'll buy. You will work with your IT folks for a bit of time on multiple threads simply to get is all going.
With Coradiant you'll get a box, it has two holes, you plug in power and network and you are good to go. You need IT but in a substantially reduced role and with less of their weapons aimed at you.
The Coradiant solution has a much faster time to use.
# 2 Time to Usefulness:
Tealeaf will provide you with completely unstructured data that you can slice and dice any which way to find anything you want. So will Coradiant. (There is one important difference: Tealeaf will allow you to replay the Visitor session. Coradiant won't, it will only give you all the data and reports.)
I am very impressed with the analytical tools that Coradiant will give you on top of the unstructured data. The AIM app will not only give you a report, it will actually analyze the data and proactively identify trends / problems that need your attention and, this you'll like, what changed before and after.
The biggest problem with saying "here is all the data in the world" is that I don't know where to even start. I like the mindset that "here is all the data in the world, start looking at these five things because there is something "funny" about them.
I also like that mindset because its you, the solution provider, not just being provider of a large amount of data.
I like any application (including Maxamine) that does not assume that there is a intelligent person on the other end. There usually isn't one. Then you have just wasted a quarter of a million dollars plus.
Coradiant has a much higher time to usefulness and hence, critically, action.
# 3 Cost – Benefit "Equality":
I grew up working in my dad's factory (mechanical engineering) and my upbringing taught me to always look for value, to make intelligent trade-offs to solve for the cost benefit equation to get the best bang for my buck.
There is a cost difference between Coradiant and Tealeaf, and for the applicability I am covering here there is also a benefit difference.
As I consider feature set I am willing to make tradeoffs, and with every extra dollar I am going to ask for five or ten or fifteen dollars of actionable insights in exchange. If I can't solve for then after a certain point I stop spending for the incremental 2% benefit.
Call each vendor, have them give you a specific written quote for your specific requirement.
Three simple things to consider when you buy anything to ensure that your company gets what it needs, at a price that is optimal with the shortest possible time to use or usefulness (unless of course you are on your way out then you can leave that for your successor to sort out!! :)).
Before I close some disclosures, my biases, to put all the chips on the table:
A] I have met senior executives from both companies and spent time with them (Tealeaf execs were kind enough to come meet me here in Mt. View and I met the Coradiant team in San Diego on a consulting engagement). I have talked to customers of both.
B] I do not fundamentally believe that video replay by itself is useful. Many people disagree with me, my feelings are crushed but I am ok with that.
On a website that has a million visitors a day, or a hundred thousand, how would you decide what sessions to watch? How would you know that you have watched enough to get a statistically significant sample so that you can take action? How many individual sessions can you actually watch?
I am deeply biased towards finding segments of problems with areas of the site, or pockets / segments of customers, not individuals. You can action both of those with confidence. Personalization has come to nothing, for a set of specific reason. Segmentation rocks.
C] I continue to believe that Privacy will define who we can identify and what we can track, and that most websites will essentially remain a anonymous experience for 95% plus visitors to the website. When I call you or walk into your store, you have no idea it is me (cookie id: 234A2D345EBC2342898F) or avinashk. :)
You might not agree with me, that is 100% ok. My biases could be leading me in the wrong direction. But now you know 'em all!
I'll close with stressing that Tealeaf is a solution that many companies will use where the three rules above meet client needs and in some cases where there are overriding factors. John at Tealeaf gave a great example were Tealeaf's cost was easily justified by a company wanting to see the session of a customer on the phone (this company did not use anonymous browsing and could locate the session). Banks are a obvious example. And do please remember that I am thinking either one of these tools for a specific purpose, as a foundational tool for Web Analytics 2.0.
You'll make your own decisions about what is right for you. My hope here is to simply provide food for thought and peek behind the sexiness and buzzwords.
Ok now it's your turn.
What did I get wrong? What else do you consider when you compare tools? Please share your perspectives, critique, additions, subtractions, bouquets and brickbats via comments. Thank you.
[Like this post? For more posts like this please click here, if it might be of interest please check out my book: Web Analytics: An Hour A Day.]
Update : This post was written in Nov 2007. On Feb 16 2008 I was honored to join the Board of Advisers for Coradiant, perhaps walking the talk on my part. (I left the board after the BMC acquisition in 2011.) It would not hurt my feelings at all if now you took this post with a grain of salt, or even consider it to be the finest quality communist propaganda. Ok the latter might hurt my feelings a smidgen, but you would be well within your rights.
As always here is a link to my Disclaimers & Disclosures page, which is always a handy reference. Thanks!
Update 2: In April 2011 Coradiant was purchased by BMC, in May 2012 Tealeaf was purchased by IBM. Both companies were evolving and surely after the purchase each is going take a path of its own. In the context of Web Analytics please use this post only for the broad themes of what to consider in evaluating a video tracking product. To learn more about what specifically each company is now up to, six years after this post, please visit their respective websites.
Via 
Hi Avinash,
On the video replay front (Tealeaf), would it not be fair to assume that this comes under the heading of qualitative analytics and not quant? On that basis one might have to forgo the statistical validity in the same way you would if you were conducting a usability study. I’m guessing that although you don’t believe in video replay as a standalone solution, it is acceptable in conjunction with more quantitative analysis to help act as a guide – multiplicity?
Very interesting post. Thank you for breaking your own rules.
Hi there, I just wanted to ask a bit more about the "video replay" technique. Fundamentally I agree with you because to me it looks like path analysis; a very weak signal to noise ratio. The issue is exactly as you describe – which session should I pick?
However is there value in picking out sessions that have a specific error and replaying them? Usually I find errors are known, can be replicated by IT folks but because the impact has not been measured there is little urgency to fix them.
It sounds like Coradiant can identify sessions with a specific error but not replay them – which seems to me to be enough to persuade people to fix it.
How about everybody else – does this ring true?
Many thanks
Matthew
Myself and some of my colleagues reviewed Tealeaf for my company. We were interested in analytics and they pointed out, rightly so, that the replay capability is a differentiator whereas they had only (this was a while ago) recently focused on analytics.
We concluded that the solution would assume both our client services folks who would make best use of the replay and our marketing folks who would make best use of the analytics had much more time to spend with the tool than they actually did. A solution with more pre-built segmentation, as you suggested, is more pragmatic and therefore is likely to have more positive impact in production.
I would be interested in an entire post on your thoughts regarding personalization vs. segmentation, though ;-)
To Matthew's point, if I remember correctly (from the good old days :-) SpeedTrap in the UK kind of falls between both camps in that it's possible to identify where the problems are, albeit using slightly more standard scenario analysis, but then can also replay (Tealeaf style) based on a pre-defined start and end point. Perhaps not exactly a combo of Tealeaf and Cordiant but an interesting comparison.
Thanks Avinash- for bending the rules and doing a great post on the two. I learnt some cool stuff about two tools I had very little idea about :-).
Avinash,
Great post.
Web monitoring is a great tool, but very resource intensive, specially when you are dealing with high traffic site.
One item that most decision makers overlook is the amount of resources needed to view the replays (this is a classic 10/90 rule candidate or probably a 5/95 candidate). If your site has high traffic, you might need a sizable team to look at the data, draw insights and drive change.
Some of the insights from replays can show you stuff which is very basic. If you have invested in site design, usability and most importantly written very good "test cases" for your applications and particularly forms, then you are in for some disappointment for the number of issues you can take to your IT team:–) Good news is you have done a good job at setting your site with the right capabilities in your ecosystem.
I agree with you on the Privacy concerns.
Thanks Avinash for the article.
-Rahul
Matthew : The IT team will probably react to a magnitude, not a instance.
I think we believe that if in a million unique visitors we find 50,000 errors and if only we can replay via video the error then it will get fixed.
My belief is that if your IT team can get a report of the error, the impact of the error (sales, "broken sessions" etc) then they might not be unreasonable enough to tell you to get lost.
Hence the stress on "proactive analytics" above, rather than giving unstructured data (go find anything you need), providing things like the AIM analytics will tell you what happened, where and impact.
Video replay is great for one scenario: "cover-your-assness".
Someone just wrote that to me. :) And I agree. If your site requires login to use and you have transactions you want to record to replay when you get sued or for government compliance then replay can be handy for "cover-your-assness". :)
Hugh : IMHO: At its core Tealeaf, SpeedTrap and Coradiant are deeply quantitative solutions that monitor traffic, bits and bytes in and out of servers.
They capture the What.
They do not capture the Why.
They don't explain it either.
Just like in Web Analytics Tools your interpretation of the clicks, sourced in your experience, is making up for the Why.
In a usability study the Why is sitting in front of you (or across the one way mirror), you are not interpreting, the customer is telling you.
It is tempting to believe video replay (just like Site Overlay) is usability, I am not in that camp. "Why" to me is me knowing it without my filters.
If you want "video sessions" AND the Why then go with something like Ethnio. Benefits:
1) You get to talk to your real customers who are really on your site, right now.
2) You can identify segments of customers you want to talk to, and then talk to them and watch what they do (while they are in their lacy red underwear!).
3) You can get Why from atleast 5,000 real customers (including voice and interactions) each year for the base price of Video replay. And you would be doing usability to boot.
4) In the end you also get a video! :)
In the end each solution has a need. Parse through the sexiness and buzzwords and literature to ensure that what we buy the best tool for the best purpose.
Push the limits of the thought to ensure you know Why your buy What. :)
Everyone : Matthew asked for your feedback and experience, I want to second that thought. Please add your voice, share your experience / recommendations.
Thank you.
-Avinash.
I have just started using tealeaf at rightmove which is one of the largest websites in the UK (10th). I guess the overall feeling is how do you get from "so what" it's nice to replay the session to identifying something important from all the noise. I can do a free text search within the session replay browser and then see how many times the particular problem I have searched for occurred eg number of sessions. The sessions appear as a list organised by date. In a very qualitative way, I have just started having weekly meetings where I go through sessions eg form abandoned sessions, with a couple of others to see if there are any "DUH" (Homer simpson) moments – gosh I can't believe we didn't see/realise this/that/the other, and then organise problems to be sorted in a priority list (but I'd love to find out how other people are using tealeaf).
But it is extremely hard to get a good aggregate view with tealeaf, of how important or unimportant a problem/usability issue actually is. And again, tealeaf is relying on me/analyst to identify what types of free text searches to search for – no top list of problems occurring in sessions automatically which would have been great – which is what I guess coradiant does.
I think that session replay is "a nice to have", but only if you already have a very powerful and sophisticated web analytics team internally to segment the data and come up with the actionable insights etc (and there aren't very many of those yet).
PS something like tealeaf is appealing for some big companies because of the privacy/security issue (banks) and because it uses packet sniffing – http requests/responses so you won't add to the page generated time and once you add your boxes it is very quickly deployed.
PPS I'll be presenting tomorrow in covent garden, do we need customer experience tools to get inside our customer's heads – more info on http://www.marianina.com
Timely post and hope you are well, Marianina
Our company is a major regional bank, and we are (in all likelihood) going to bring in TeaLeaf within the next several months. The replay ability is what sold us on TeaLeaf versus other passive-capture solutions. With an online banking application and an application for applying for an account, among others, there are myriad issues and errors than can occur but that cannot be duplicated by IT. We expect the tool to help us greatly, not for "lost sales" as it would for a typical retail site, but in allowing for bug fixes we would not otherwise have found. And TeaLeaf does have the ability to "segment," though perhaps not the segmentation you're speaking of – e.g., it can identify the number of instances/sessions where the word "error" appeared on a page and can detect other similarities between such sessions.
I would like to make the case for using Tealeaf as a foundational tool for Web Analytics 2.0 as we are using it for this purpose today. I am the Customer Experience and Business Activity Monitoring Lead for a Fortune 100 Financial Service Firm. I will walk you through how Tealeaf has evolved at my Company since we first bought it three years ago. The evolution has two stages: 1) Replay and Events and 2) Complex Events.
Replay and Events
————————–
In my experience, having the ability to visualize the customer experience allows one to better understand a site’s usability challenges from the customer’s eyes. Avinash makes the point that Tealeaf provides so much data, you don’t know where to start. There is a place to start and this is where events are powerful. The first thing you do when you install Tealeaf is create events. For example, the first event I setup is one that indicates the session contains a login and makes the LoginID searchable. Important events to setup are those that track your business process (e.g., customer starts an application, gets their credit results, completes the application, etc.) Being able to replay the business process makes setting up events very easy.
Once the business process events are setup, you will be able to determine you conversion rates and start looking at the segment of users that started an application, were credit approved, but never completed the application. Did they have a usability issue? Receive an error? Experience slow performance? If you do find instances of errors, you can setup events for them as well. Tealeaf traps HTTP 500 errors out of the box, but sometimes errors sneak in as regular HTTP 200 pages. Tealeaf provides very good search tools to slice and dice and they are much more powerful if you define events. For example, we had an issue where our Web site wasn’t storing the information necessary for a customer service rep to be paid their commission for helping book a loan. I was able to search for all sessions containing a completed loan event and export the loan confirmation number and customer service agents ID to Excel. That made our business area very happy and averted a customer service rep uprising.
Complex Events
————–
I gave a presentation at Gartner Group in September on how we are using Tealeaf to feed events into our Coral8 complex event processing (CEP) engine. A CEP engine allows you to slice and dice events that occur over time. For instance, for every user that receives an error, write a row to our database that contains their UserID, the page they were on when they got the error, the loan product they were applying for, their contact information, and how much they were applying for. A real-time report is then generated containing this information along with a link to replay the customer’s session. This report gives our business area the immediate ability to contact these borrowers to help them complete their loans and allow our support team to easily replay these sessions to troubleshoot the problem.
We have evolved in the CEP area as well and now using Tealeaf to feed real-time customer experience dashboards, detect fraud as it happens, and other value add applications. A good foundational layer allows you to do two things: 1) Build applications on top of it and 2) integrate with third party products. Regarding the latter, we are now integrating Tealeaf events into Omniture SiteCatalyst. Additionally, we are collecting customer lead information, alerting if we think a customer is in risk of leaving to go to a competitor, performing A/B split analysis, and modifying our sites to personalize based on what we know about the customer from past visits and based on what they are doing right now. To sum things up, Tealeaf provides a lot of value beyond replay and should be considered as a solid foundational layer.
Excellent post Avinash, though it is two years too late for us. We put in a 18 month effort just to install and get IT to do all the things we needed. Even after that effort and substantial cost we are no where close to realizing the value (your cost benefit equation).
In hindsight we had two problems you mention, the solution was too complex to implement in our Fortune 100 environment plus we needed very experienced SQL writers to find any data we needed. The business users were not able to use the application.
Thanks for the tips via your blog.
P.S: For video replay we are now trying Clicktale, Robotreplay and Tapefailure.
It's nice to see people talking about real-user web analytics. Too many solutions focus on marketing-based analytics or use-case analytics without thinking about the actual end-user experience. That's the number one thing people should care about.
Before I go on, in the interest of full disclosure I am a former Coradiant customer (I joined a startup and we just don't need this stuff…yet) and a big fan of what their product can do. I can vouch for all the points of Avinash's analysis.
Playback is to me, at best, just "interesting". You'll never truly see the end-user's rendering experience without some kind of screen capturing tool loaded onto the end-user's system.
If you're watching a playback of a session, when you see something funny happen it's a manifestation of the data flows between end-user PC and the web site, so you're going to dig down to try to correlate that data and analyze it anyway. It's nice if Tealeaf can do the correlation to get you to the data, but the data is there regardless, and Coradiant can find it for you as well.
In my not-so-humble opinion as a long-time (~20 years) Ops guy…playback is likely to remain nothing more than "interesting" compared to the real data analysis.
– K
We have been a Tealeaf customer for almost 2 years now. Over time, we have discovered countless errors on our e-commerce site. Our conversion rate doubled when we discovered an error with our online credit card acceptance. As mentioned, custom events relevant to your business needs will need to be created in order to fully leverage the software.
Tealeaf has two main features, a "Portal" and a "Viewer". The Viewer is for replay purposes. The Portal allows you to build custom charts that are defined by your events. At a glance, you could see the amount of traffic that made it through a series of steps but then received an error page. Or, you could quickly see how many 500 Errors have occurred today>which pages they occurred on>and then perform a playback to see what steps caused the error. Along with charting, Tealeaf allows alerts to be created from the events as well. One alert that we created involves orders that we receive that do not include an "order number". Generally when this occurs, it means that the order information did not transmit to the store location. In terms of customer expectations and service, this is huge to our business.
I used Tealeaf + Omniture daily for about 2 years. Our tealeaf install paid for itself in 2 months and now is absolutely essential for business operations
Replay is essential for investigative research.
1) The IT department actually does need single, explicit examples – especially if its an outsourced team. It enables them to see how to recreate the problem. This DRAMATICALLY increases solution turnaround for issues. Major painpoints were resolved within hours instead of days.
2) SEEING what the customer sees allows you to see patterns (such as other errors, types of data entered, etc) that allow you to properly quantify the impact of an event. This also helps with identifying new events (ie: what you should be tracking!)
3) Replay is HUGE for reporting up to upper management
4) Replay is ESSENTIAL for communicating issues to your designers and information architects, and especially for demonstrating where they may have made incorrect assumptions
5) Replay is often the ONLY way you can get your analysts/PM's/Managers/Developers to see and feel your customer's pain. It is also the BEST way to identify customer pain point that have to do with design (graphics or colors) or poor content (like wording of error messages).
Honestly… its just plain easier for me to SEE a problem like the customer sees it. Looking at a big screen of logged data doesn't give me the best insight. The best part about Tealeaf (and its replay abilities) is it allows you to fully 'grok' your customer experience.
Very interesting post. I haven't used either of these tools yet, but I had the same question that Matthew had about using video replay to look at sessions with errors. Path analysis has been a waste of time in my opinion as well, but the ability to show a decision maker what activities were going on before the error if the IT folks are being stubborn might help (humanizing all that data). Or maybe a better approach would be to try and quantify the cost of the errors as you mentioned in your response. Thanks for bringing these tools to my attention – it's not a normal post, yes, but I enjoyed the contrast!
Avinash,
I must say I have learnt a great deal reading through this post and the comments have been insightful. I agree with you that the emphasis should be on how we think about the huge data been churned out by various tools. As a new entrant into web analytics I'm discovering that an understanding of how the web works and how users behave on our websites will play a lot more role in using all the analytics / customer intelligence tools available in the market.
"Who you gonna call? :-)" – Was that a ghostbusters joke?^^
Better late than never…
From the perspective of an IT person, to much of the above discussion there are two major issues:
1. IT, like any other business support unit, has priorities set by external parties; internal resourcing issues and the like.
What we'd like to fix, and what we can and when we can, can be subject to all sorts of … pressures that we have little control over.
eg:
http://ars.userfriendly.org/cartoons/?id=19971117
http://ars.userfriendly.org/cartoons/?id=20071117
:-)
So generally, as Avinash points out, the greater the impact, the more likely the priority will rise compared to other priorities and hence resources can and will get assigned faster.
It should not be a case of "won't", it's more a case of "can't".
Do keep in mind I am grotesquely simplifying here.
2. Debugging. As Paul Ireland points out, being able to exactly reproduce a sequence of events taken with all data? Priceless. We can and do spend days trying to replicate obscure edge cases that the system owners create.
In improving developer efficiency, such a reply tool could pay itself off real fast. YMMV.
Having said that!!!
If architectured well, the same information could easily be stored in a database anyway – replay no longer necessary, as all the info required is available.
Although I did cynically laugh at Paul's #3. vis Upper Management. I've seen system availability tools which had 3D fly throughs of buildings etc. Wonderful for senior management to go Oooo Ahhhh over. Completely useless for those of us who need to actually monitor and fix systems.
Cheers!
– Steve
A really interesting technology that we're using in beta is ReplayDirector. This tool is similar to Tealeaf except that it actually replays code execution on our app servers. This means we can step through code in Eclipse and see what happened during the recording in code.
There is good info on their site.
I'm late to the conversation here but have some experience of where these tools might be useful.
Marianina has pointed this one out and I will second her – what about objects that are present but that are not interacted with?
For example, a page that sometimes shows a widget, button or icon next to a call to action button may influence the conversion. I found examples of this when using a home grown session recording tool (what a horrible piece of chicken wire and string that was ).
What it did tell me about was the presence of dynamic elements and how these interact with things like conversion rates. I also spotted a lot of interactions with forms that our analytics package missed.
Some of these insights really should have been picked up by our web analytics package but it wasn't configured to 'work that way'. For example, I dug into form submissions and error rates to see how these were losing us signups. The discovery was that form validation and error handling had a big impact on these.
Correcting the validation in key areas (for example, proactively correcting invalid postcodes with common errors) actually saved us a 7 figure sum in revenue each year.
I agree with *most* of what Avinash says but would say there *is* benefit from video replay, if you have another data source. Being able to look through a slice of our signups that bailed (and encountered a form error) gives me insight into the user flow surrounding the error. When each signup (per day) over the course of a year adds up, fixing small issues is worth a lot of money.
Context of the error and a video example (like a usability test clip) works extremely well in motivating people to fix the issue.
I can also see video replay being useful for investigation when a customer complains as often we may not have the precise error or issue logged in our systems or analytics tool.
Video replay is no substitute for usability testing and getting the product designed to work for the users first too!. I will be testing one of these tools soon and will post some feedback…
– Craig.
Would the replay function not be useful in replicating 'real world' style traffic in the load testing department?
My experience in load testing has shown that it is very difficult to simulate true production style user sessions on our guns because of several issues, one being the differing speeds and lag in user networks.
Craig : Thanks for adding to the conversation. You mention interesting cases were a video reply might be helpful but there is still a ton of chance involved or analytical skill and a thick layer of time available to hunt down the "issues".
That partly is my concern. Does this scale? And then there is the issue of ROI. Paid tools in this area can have a price of entry of three to four hundred thousand dollars, and then it goes up. That is just the software and hardware, layer in the other costs of people, process layers etc. Then it truly becomes a stretch.
There are a few comments on the post that show that there is ROI for some. So clearly my experience is lacking because I have yet to see any from the 15 clients I have spoken to. Almost always the actual implementation failed to deliver even half of what the sales pitch promised.
Perhaps ROI is a non issue now that there are so many free options. I just spent some time with a Israeli company that is doing some very interesting things with video replay.
My challenge to them was the same: "I don't care how cute it is to watch the sessions, it is cute. I care about how you can proactively help me find insights and I care about doing that at scale – being able to do two press releases / case studies from thousands who have installed the product is not enough and neither are one off blog posts. :)".
To their credit they are working on a few innovative things and I dearly look forward to those (and their video replay tool is free!).
Rob : I am sorry but I was unable to internalize your comment.
I spent quite a few years in IT so I very much understand load testing.
I am unsure of what you mean by "replicating real world style traffic" means as it applies to video replay.
If you wanted to know what a typical session is you can simply grab one of the hundreds of thousands of session on your website and use it.
Were you referring to hammering your site / servers to do "load testing" and then watching the replay of your load test?
Perhaps I am missing something. Would you be so kind as to share some more context?
Thanks so much to both of you for adding to the conversation.
-Avinash.
Helpful Link: My disclaimers and disclosures page (& also see the Update, in red, at the end of the above post).
Avinash,
LOL – I take your point about it being cute. I guess there is limited value for us based on the price of these types of products.
If we had a system which aggregated form handling issues and correlation with abandonment much better, this would be much better.
Keep us posted on the Israeli company you mention – sounds interesting.
OK,
I can get the argument that it is cool to look at user sessions but I come back to the points made by Avinash on this matter (see http://tinyurl.com/398x2f).
My question here is – what am I missing that I don’t get with a properly configured web analytics package?
Some of the things people have mentioned as being useful are:
• Error messages
• Broken stuff
• Form errors or validation issues
• Bugs / Rendering issues / Browser incompatibilities
• Process bugs (e.g. not accepting certain forms of payments)
• Correlating customer contact with particular web events (e.g. broken stuff causes contact)
But I can get all these from a web analytics solution, in aggregate format but without the video viewing!
The things I can’t get are:
• Watching a replay of the session to see exactly what someone did
• Debugging a complaint from a customer by getting developers to watch the session
• Providing proof of the customer ‘doing something’ from a legal or service perspective
• An idea of where the mouse moves
Are these things I can’t get worth the costs? Hmmm – the jury is still out on that one.
If, however, I have a badly implemented WA solution that hasn’t been set up to capture this kind of stuff, then replay tools with data aggregation facilities (like Tealeaf) can plug that gap. If my WA solution does all this, I don’t need another product.
The one cool thing mentioned about Tealeaf was the ability to collect WA data for ingest into a package (e.g. Webtrends) without having to do any page tagging. Lol – now that would make developers less chippy to start with .
Maybe tealeaf needs to work on me some more? Is there anyone out there that can refute the central plank of my argument here?
Craig.
Interesting discussion, I think the area of interest depends on "who" is using the tool and the business objective.
I've found that "performance" discussions with people who are on the performance and operations care to see incident management — identification of slowness by performance and breakdowns into what causes the slowness.
Marketing is interested in doing "replay" to understand the true customer experience. There are some compliance issues needing the ability to see a specific user replayed as well.
It also has depended upon whether the company saw performance as a "feature" or penalties for violating SLA's. That tended to drive operational real-user monitoring versus UI exploration which comes from a replay solution.
My two cents.
Well said, finally a good report on this stuff
In my experience (and I worked very closely with Tealeaf in a past life), I found that replay is great not only for troubleshooting specific problems in a website (it can be a VERY useful tool for call center personnel), but also for compliance and records keeping.
If you think Tealeaf costs too much for what they offer, there is a new session replay alternative available now called Pion. It lacks some of the bells and whistles but doesn't cost an arm and leg (maybe just a few fingers).
#1 Time to use – Similar to Coradiant, Pion is available as a network appliance. I agree that this approach makes it MUCH easier to get up and running quickly, compared to Tealeaf which requires provisioning and has lots of independent software bits that need to be integrated.
#2 Time to usefulness – Pion captures performance metrics, but it doesn't provide great reports out of the box like Coradiant. For session replay, however, Pion does start "working for you" right away without most of the messy configuration that Tealeaf requires.
#3 Cost – Benefit "Equality" – Pion can handle up to a gigabit of traffic with a single 1U box, while we've heard from Tealeaf customers that a similar configuration can require 20 or more servers! If replay is what you need, there are now much less expensive alternatives available than Tealeaf.
Disclaimer: I'm the obviously biased founder and CEO =)
I remember reading this post a few years back. Even though it is an old one now, I had to add to the conversation. I worked at Omniture for many years and recently joined Tealeaf. I worked with both the Coradiant guys and the Tealeaf guys while at Omniture. I was more attracted to Tealeaf because they collect everything. It is in their DNA to recreate the user's experience and that means dedicated resources and energy to data collection and identification of the data that matters. I agree that the replay won't have value in all circumstances, but it sure helps when you get stuck on an issue that you just can't flesh out. Since this post, they have added very insightful client side collection, flash/flex data collection and are moving heavily on mobile and other sides. It is the best blend of the client side and server side I have ever seen and I'm glad to be a part of their organization. Just wanted to put that out there.
I know this is an older discussion, but my Web Analytics professor recently used this article to start a class discussion. The following was part of my contribution:
I don’t think Dr. Kaushik should have taken such a broad-based position on the preference for Coradiant. This is a perfect example of the flexibility-usability tradeoff you find with many software applications. This is such a widely accepted design principle that it has its own Wikipedia entry ("Flexibility–usability tradeoff")
There will certainly be some companies, particularly the smaller ones, that want to be taken by the hand and guided as to the analytics to be performed. These companies might prefer to pay a little more for the software to avoid the need for in-house expertise, which could be much more expensive (although I realize Tealeaf was the more expensive alternative back in 2008 due to the session replay feature)
On the other hand, there will also be companies who want the flexibility of starting with the raw data and going in their own direction with the analytics. This will not only be the larger companies, but also any company that might not fit the “one size fits all” approach used by Coradiant. Companies come in all shapes and sizes, and the domain of web analytics is not a pure science. In combination, this creates a situation where a custom solution may be required in order to receive the most benefit. So, I imagine there will be plenty of companies who just want a reliable and efficient source for the raw data.
Gene: Thank you for sharing your perspective.
This post was kicked off by a specific question asked by Diana, hence the direct comparison. In doing so, what I stressed were less Coradiant x or Tealeaf y. It was a lot more: Think of these factors, make up your mind, here's how I made up my mind if I compare these two.
In your case and evaluation of those factors might yield a tool called Genemine (or any tool on the market). That is totally ok.
To your point on raw data, I can certainly imagine cases where companies just want raw data. I assure you from my experience that in 9.9/10 cases that is a recipe for failure to deliver business value in a digital analytics context. Why? Time to use, time to usefulness, cost-benefit equality. :)
In the 0.1 case, the raw data is classified data that is being used to train Machine Learning algorithms first, and then by the developed deep neural networks to make smarter decisions. If that's your use case, raw data is incredibly precious.
I hope this is of value. Please do thank your professor for using the article for the class discussion.
Avinash.