Your referrals are relevant and the visitor expectations are matching with what you offer. The quality of your landing page has to be questioned then.

There are several methods to study the landing page quality, A/B testing and multi-variable testing are for example. You can also get support of heatmap tools such as crazyegg or site overlay option at google analytics for the same study.
[...]

What is the difference between A/B testing and what you are calling Experience testing? It sounds like for the experience test you are creating a mega-variable, lets call it ‘site’, which is a bundle of attributes (pages, images, etc.). I don’t see how the actual testing is different than A/B.
For the multivariate testing we are looking at attributes that sit in several dimensions (think of the corner points in a hypercube). What we are concerned about is that there will be interaction effects over the variables we want to test - so that we need to know all of the values of each variable when determining the results.
One unmentioned ‘con’ of this approach is that it is very costly from a data perspective – we need lots of observations to ‘fill up’ that hypercube to make robust estimates. One way around this is to work with fractional factorial design testing – where we make assumptions about how many variables we need to include at any one time – this has the effect of collapsing or aggregating some of the corners of our cube so that we need fewer estimates.

Thanks again

Matt Gershoff

For creating experiments for Conversion Rate Optimization I am facing problems even for a very small test to execute.

Previously I have created around 5-6 experiments for our website using Website Optimizer, but from last one week I am not getting required combinations on setting up experiments.

I do add the scripts to the test and conversion page and also making variations, but at last step when I preview it I don’t find desired combinations.

Please help me as I feel that before 1 week I was successfully setting up experiments but from these last 1 week I am doing same actions that I used to but not getting desired combinations.

Thanks in advance.

Regards,

Shashank

You should talk about the limitations of Javascript and how only an installed solution like the one that Memetrics offers can test more than a few attributes on a simple page. Why not test Paid Search, Direct Mail or anything with a dependent variable.

Although I’m a big fan of using web analytic data to uncover a vast range of improvement possibilities with site content and conversions, for this post the most useful think I can do is direct you to the following pages on the excellent blog of Avinash Kaushik dealing with all things analytics:

* Experimentation and Testing: A Primer
* Excellent Analytics Tip#5: Conversion Rate Basics & Best Practices
* Excellent Analytics Tip #8: Measure the Real Conversion Rate & “Opportunity Pie”
[...]

Wonderful post. Thank you for your insights. The tools for analysis and observation have improved greatly over the last several years. What is available now does not compare to the rotators, Javascript trackers and numerous spreadsheets that I was using years ago.

In doing this conversion optimization work, we found that the segmentation that you speak of is a subset of “noise reduction” in terms of test design.

As the tester observes and refines the “conversion conversation” the natural extension is to create a user defined channel that serves the visitors concerns and needs. When that match is closely correlated the users actions become more predictable.

Your term - “Experience Testing” is a perfect articulation of the what I termed the “conversion conversation”

I am enjoying your Blog and thinking.

David

What do we want to answer? What are the Media Group’s reporting needs?……

….

I believe this. It’s in my bones. Sure, customers are not going to invent new, breakthrough stuff. They don’t know they need an iPod until everybody else has one. But what about the 99.999% of the rest of it? They do know how they like to buy. They do know how they like to shop. They know how they like to compare products and how they like to return products.

Avinash Kaushik is one of the most insightful and intelligent web analysts I’ve ever met. On his excellent blog, Occams Razor Avinash said it best. “80% of the time you/we are wrong about what a customer wants / expects from our site experience.”

Avinash describes his work at Intuit as dealing with website experience, behavior and outcomes. Outcomes are the goals the company sets - selling software. Behavior is all about the clicks. But, says Avinash, if he only had one of the three to work with, it would have to be the customers’ direct feedback and customer satisfaction.

This is from his post Overview & Importance of Qualitative Metrics:

….

Some of the Science Fair experiments effectively demonstrated outcomes that were non-intuitive; in the same way, the analysis of user behavior can highlight for us the sometimes unexpected and non-intuitive impact of design choices.

A good scientist assumes nothing and tests everything! Should a good web designer do any less?

—————-

Update: One of my favorite blogs on this topic, Occam’s Razor by Avinash Kaushik, has a wonderful post on the topic of web analytics and testing: Experimentation and Testing: A Primer . I highly recommend reading this post if you’re interested in this area.

Phoenix.edu Currently Tracking Weekly visitor counts (including users passing through to eCampus) Daily phoenix.edu lead counts Daily total web lead counts Daily share (phoenix…….

Dave

Thanks you for vastly enriching the value of my original post by adding your comments. I have learned more as I am sure have other readers.

I have personally and actively observed the Hawthorne Effect and hence I am biased towards testing and specifically Experience Testing (which is not so much personalization as putting randomly assigned people into two or more “controlled and different” experiences on the website and seeing which experience performs better against preset goals).

It is not perfect but we can not only measure conversion and revenue type stuff but also qualatitive like Task Completion and Satisfaction for these experiences. It is not perfect but in my mind a great tradeoff between bringing someone in and giving them $250 to run through a site vs doing it without the participant knowing.

By no means is Lab Usability over, not by any stretch of the imagination, but as you can see I am giddy at the possibilities of Experience Testing and the learnings that can come from that.

As you’ve asked for some feedback, here goes:

Experimentation and testing in the long run will replace most traditional ways of collecting qualitative data on our site experiences such as Lab Usability. Usability (in a lab or in a home or remotely) is great but if our customers like to surf our websites in their underwear then would it not be great if we could do usability on them when they are in their underwear?

Both lab testing (i.e. for usability) and transparent web testing (A/B, multivariate, etc.) have their places. Lab testing has many benefits, among which is the ability to reveal UI/usability issues that are difficult to quantity with web analytics. Likewise, A/B and multivariate testing are very strong quantitative tools that reveal user preferences that simply cannot be measured in a testing lab.

For example, asking a user in a lab test to decide which promotion is more appealing is unreliable for many reasons (Hawthorne Effect, for one). But ask users “in the wild” by presenting offers in an A/B testing where they don’t know they’re being tested, and let them vote with their wallet. This is the crux what marketing experiments reveal that qualitative usability research cannot.

It is difficult to control all the external factors (campaigns, search traffic, press releases, seasonality) and so you won’t be 100% confident of the results (put 70% confidence in the results and make decisions).

There’s nothing inherently more or less difficult to control with an A/B test vs. multivariate test. The key thing is to randomly assign visitors to the A group vs. the B group.

In A/B above you had to create three pages. Now imagine “modularizing” your page (break it up into chunks) and being able to just have one page but change dynamically what modules show up on the page, where they show up and to which traffic.

Another way to look at this: if A/B testing focuses on one site element (i.e. a product image), then multivariate testing focuses on multiple elements (product image plus headline). And a quick terminology note… in the realm of experimental design, these are commonly called multifactor tests. So when you see “multivariate” or “multivariable”, or “multifactor” in the context of web testing vendors, it’s useful to note that they essentially all describe the same process.

You don’t have to rely on your IT/Development team. All they have to do is put a few lines of javascript on the page and they are done. This is a awesome benefit because most of the times that is a huge hurdle.

This has nothing to do with a test being multivariate, and everything to do with the innovations provided by the vendors you’ve mentioned. More to the point: the hardest thing about A/B or multivariate testing is switching content - showing version A1B1 of a page to one user, while simultaneously showing version A1B2 of that same page to another user. The more factors you’re simultaneously testing (varying), the more crucial the content switching becomes.

You don’t have to rely on your IT/Development team. All they have to do is put a few lines of javascript on the page and they are done.

Well, yes and no. In some organizations, instrumenting pages with javascript each time you want to test a new area requires the IT/development issue. [Note that I'll freely admit my bias here: my company's product is the only one that truly takes IT out of the equation because it doesn't require javascript tagging.]

Finally, regarding Experience Testing, what you are articulating sounds alot like personalization with an experimental component thrown in. Besides ATG Dynamo, Microsoft Commerce Server and IBM WebLogic also support simplistic forms of this.

Thanks again Avinash, and welcome to blog-o-sphere! :)

Dave @ SiteSpect

In Web Analytics, if a site shows a different page (one created for the customer on the fly) based on who they are - how will the Analytics represent those variations?

Many different analytics tool can now handle this quite easily. The challenge is that we should have the foresight to know what we want to track. Typically you can set either cookie values or url parameters that a analytics tool will automatically pick up and then you can analyze.

For example you can come to http://www.kaushik.net/avinash and based on the keyword you came on (or campaign) you could see:
http://www.kaushik.net/avinash?key=marshall
http://www.kaushik.net/avinash?key=marshall-is-great

Now you might see different content on the fly depending on what “key” you came on and my analytics tool can measure every kpi imaginable now that it has captured the key value.