Something to consider the next time you want to test ‘that’ over ‘which’….

“Damn your a priori principles. Look!”
- Galileo, to skeptics regarding the solar system revolving around the sun (heliocentrism)
[...]

just google for “sample size calculator” for some free web-based tools

I did and for benefit of our readers a couple that show up high are:

• UCLA Dept of Statistics: Sample size calculator
• Creative Research Systems: Sample size calculator

But I don’t think it’s a one-way linear procession from simple to complex… as in “once you’ve gone MVT you’ll never go back” ;)
……
I do think web analysts need to consider testing things that they would not ordinarily think of, and perhaps this is your gist of “big” tests.

I completely agree with you, once you go MVT (I would say testing overall) you’ll never go back.

For reasons that are complex people start in the MVT world, try some tests with images and content and then just stay there. Or repeat those kinds of tests on other pages etc.

The thrust of my recommendation was, think bigger and think different and think crazy. Only then will we/you get disproportionately high impact on customer experience and company bottom line.

Thanks again so much Dave for sharing your wisdom.

A great post all around. There’s been alot written about the mechanics of testing and “things to try”, etc., but this is the first solid article I’ve seen about the cultural side. So kudos to you! :) And, of course, the feedback:

Decide what the success metrics for the test are before you launch and don’t forget to create a goal for those metrics. So you are launching a test to improve conversion rate. Great. By how much do you think you’ll improve the conversion rate?

This also provides an opportunity for estimating the sample sizes required to reach a statistically significant result. Why do you care? Because it’ll guide you as to what tests may be feasible to run within a time constraint vs. those that can’t.

For example, if your goal is to go from 2.0% to 2.5% CTR from a landing page (a 25% lift) you’ll need a sample of ~30,000 visitors. If you run that test to 30k visitors and don’t reach the goal, stop the test and move on.

But perhaps counter-intuitively, the higher the goal (change in response rate), the smaller the required sample. So if our goal is instead to go from 2% to 3% (a 50% lift) we actually only need ~8,000 visitors! Hmm. So we can validate that hypothesis in about 1/4 of the time.

So just ask yourself - what is the smallest change that would have a material impact on our business goals? Is it 25%? 50%? more? Answering these questions will help you decide which tests are feasible given the available time — and when to “stop” a test because the success criteria wasn’t met.

[n.b. a sample size calculator ought to be part of everyone's testing toolkit - it your testing solution doesn't haveone, just google for "sample size calculator" for some free web-based tools.]

Almost all testing is “single goal based”, especially the current swath of multivariate testing companies …. Life and customer experiences are significantly more complex …. If you only solve for conversion rate you might be majorly and negatively impacting your customers.

Absolutely critical! It’s too easy to get caught up with “one step at a time” improvement. Behavior needs to be measured across multiple response metrics, across multiple visits (where applicable.)

Here’s an example… a user of SiteSpect recently ran a test where he improved landing page click-throughs by 91%. Not bad! But what did visitors do after the click when they saw the actual registration page? Turns out that the “best” landing page actually yieled only minor improvements in registration. Instead, a worse-performing landing page (worse for CTR) yielded the highest lift in the registration form.

So what’s this mean? :) Track multiple behavior and measure multiple processes/goals. Learn how each site element contributes to each goal or process (the multivariate piece). But know that some changes may improve certain response behavior while hindering others.

your end goal should to have the mindset shift proceed towards the direction of doing complicated “big” tests, ones that put a lot bigger things on this line and not just play with the hero image on the home page.

I agree that testing is an incremental learning process. Start simple and increase your sophistication with with each test. But I don’t think it’s a one-way linear procession from simple to complex… as in “once you’ve gone MVT you’ll never go back” ;)

Surely people’s sites change over time, requiring certain areas to be revisited and retested. My own opinion is that sometimes an A/B test is fine (like when you need to choose between several promotions), and you don’t need to redefine the problem (making it more complicated) just because you have the ability to run a multivariate test.

I do think web analysts need to consider testing things that they wouldn’t ordinarily think of, and perhaps this is your gist of “big” tests. I’m talking not just the usual testing fodder of headlines, images, copy, etc. Think outside the box and challenge deeply-embedded site elements. Page layout, navigational structure, style elements (CSS), etc. The sky’s the limit.

# 2 Create a fun environment:

Great! :)

cheers
Dave

Yes, yes, yes, yes, yes, yes, and yes. Great thoughts. We must wonder why the web metrics “companies” and web research “leaders” only understand #2 and #3…

I sense it is because the blogosphere is full of “Type II” evangelists motivated by their web 1.0 scars and emboldened by their Public Relations degrees!