http://www.kaushik.net/avinash/2008/11/ultimate-web-analytics-data-reconciliation-checklist.html

where you mention one source for problems is people running with “javascript turned off (2-3% typical)”. I think it’s actually a bit more subtle than that.

With ad blockers it’s fairly easy to configure which parts of a complex page load, and to turn off not just the display ads, but also to refuse to load the Google Analytics tracking javascript. So your “turned off” reader might just be running silent, with most of the javascript working but the trackers gone.

Since all of that code runs in the user’s browser, it’s not beyond comprehension that someone suitably motivated could throw junk into the data stream as well, but I haven’t seen code to do that yet.

It has been fairly well established that one of Wal-Mart’s core strategic advantage is its IT system and the core essence of that IT system sits on a custom built database (not oracle or sybase or db2 etc). I was referring to that but clearly from your comment I should have been more expansive.

Thanks for the comment.

Avinash
PS: I did follow your instructions and did a search for Omniture on walmart.com, but this is what I get:

I am kidding of course, I know what you meant when you said “search for Omniture”! :)

Your statement about Wal-Mart building their own data collection solution is in-correct or alluded to they were using an inhouse solution which is incorrect. They do have their own Datawarehouse where they also do combined primary, secondary for brick and motar and tertiary analysis for websites but they do use a 3rd party vendor to do the primary website data collection, processing, and analysis.

Your Statement: “Often this is a easy choice to make of any company that considers its core competency to be to focus on its business and not developing web analytics solutions (though admittedly if you are Wal-Mart you can absolutely do that - for example they have invented their own database solution since nothing in the world can meet their size and scale).”

Wal-Mart uses Omniture to track and collect data for all their properties including their Windows Media Player music site. Do a search for “Omniture” on Wal-Mart.com and you will their tags.

I agree, filtering is needed when working with log files. But we already remove bot, css, xml and images from the logs, and it’s still 800Mb of data each day. I don’t say that javascript is better than log data mining, I just say that sometime log data mining is not worth the trouble. It all depend of the goal of the site.

One think log data mining will never be replace by javascript tag for, is: analysis load on server. But even this can be replace by product like Coradiant (www.coradiant.com)

These solutions are usually not any more expensive to personnel time or $$$ than the drop and go page tag solutions out there.

-Theresa

who said it isn’t 1Gb a day? :-)

Can I humbly suggest you’re approaching the problem in the wrong way, urm, twice. :-)
What percentage of that 1Gb is relevant to clickstream? What percentage is robots and other gumpf.
Do upfront filtering of both and you have probably reduced 1Gb to 50Mb. And if you can’t clickstream that….
Practically every dedicated loganalysis tool I’ve ever seen and/or used does a *really* poor job of prefiltering efficiently.
There are stock tools that do it far better. egrep, gawk and perl, for example, are far more efficient at pre-filtering. Use multi-cpu machines such that all are working on part of the problem. The pre-filter stages will also help with IO latency. Especially if you leave the logs compressed. IO latency is a killer for big log analysis.

eg. I do all my analysis on a HT 3Ghz dual Xeon IBMx235 server. I can process a full years worth of logs in about an hour or so. Multiple staged filtering, all 4 effective CPU’s working their butts off, disks ticking along nicely.
Worst case, do the filtering on multiple machines and stage it that way. Easy!
Hardware is cheap - your time is $$$$$.

Another trick - get an OS that can run in memory file systems. eg /dev/shm under linux or /tmp on Solaris. Stage your files into that. Thus you end up processing the files in memory. IO latency goes to zippo. Especially, if you build your filter chain/pipeline, such that the next log is there ready and waiting before anything needs it.

Which brings me to the other issue: Re-analysis is not as immediate, usually, as day to day analysis. So it doesn’t matter if it runs overnight. Or longer.
I don’t reanalyse 4+ years of data every day. Maybe 4-8 times a year. I did easily double that in two weeks at the start of 2006. It varies.

If your structure/layout changes, is clickstream even worth redoing? It’s no longer relevant any more. The value is not so much in the clickstream analysis anymore but elsewhere.
So that part, IMHO, becomes moot anyway.

Oh I agree the world is not black and white. It’s in colour! ;-)

But the key message I, probably badly, tried to put across:
Don’t ignore any source of data. Cross check, verify, recheck those assumptions. It’s too easy to get lulled into a false sense of security in this game.

Javascript Tagging, and here’s where I do agree with Avinash, is probably more than good enough for most people who genuinely care about how their site is used. But when *I* check my personal site (vs work’s), it’s next to useless. Misses huge swathes of folk. And that percentage changes too.

I’d have to ask him, it should be fairly obvious, but I’d betcha I’m invisible to any non-log analysis Avinash runs here. Why? I run the noscript & adblock plugin’s in Firefox. Got fed up with all the javascript rubbish that too many sites throw my way. And if a site totally breaks without javascript. Shrug, plenty of others that don’t. Not my loss. Very few indeed are the web sites that have content compelling enough for me to care.

Evil Grin: I know one site that switched on javascript tagging of the … conversion step you’d call it. To better track it and such. *BIG* site. 1Gb a day? Try per half hour if not more. ;-)
Well they hadn’t really looked at how their users actually used that conversion step and immediately alienated a huge chunk of their customers. Big Mistake. They reverted that change *very* quickly.
This was about 4-8 months ago from memory.

Todd,
makes some excellent points. At work I actually use about 6 additional logging subsystems as adjuncts to the base Apache logging. It varies. Some capture App specific stuff, others search engine (internal) and so on.
As they are all different, they assist with cross checks - bug hunting and so on. Impact is insignificant.
Pure Gold tho for getting different views on the system as a whole. And that’s the key here. It’s not a webserver, or even a webserver farm. It’s a system! that just so happens to serve webpages as it’s principle function.

Cheers!
- Steve

As regards to combo solutions, if there are resources and tools that make it easy for you to do it then that’s absolutely great. Usually though with finite resources (people and $$$) and a need for speed it might make sense to simplify and make a single choice.

You are right on downloads (Dr. Turner also rightly added robots and vendor independence as benefits for logs), they will only be in logs though there is very finite data you can get about download from a log file. Many companies are starting to use Akamai Download Manager type apps, in which case there is deep and rich information about the downloads that one can get from there. IMHO the information you’ll get there is actually business actionable (repeated attempts, tie to a user, aborts etc etc) which we simply can’t get from a log file.

Not every company will / can use a Download Manager in which case if downloads reporting is important log files are a good place to be.

Dr. Turner: Absolutely agree with the sentiment you have expressed in your comment. That’s it. No more comment from me! :)

Thanks,

Avinash.

When the size og your log file will be 1 gig by day, tell me how you will be able to calculate (in a finite time) clickstream from back to 2002?

Also, my web site structure happen to change over time, making clickstream data analysis difficult.

I still believe the world is made of shade of gray, not black or white.

1. Avinash Kaushik discusses the merits of javascript analytics over web log files. [...]

Regarding the comments you made on the pro’s and con’s of Javascript and the implied tagging implementation requirements and constraints, I’d like to point to a recent article by Bruce Tate, CTO of GoodWell, which he recently posted on the IBL community site.

In a nutshell, he describes how it is increasingly difficult to ignore or rule out Javascript in this Web 2.0 age.

Basically, if you’re going to track user behavior using Javascript, you might as well leverage that implementation to track your overall Web traffic ;)

Here is the link to the article

Cheers,

Julien