I think this is a deeper subject and borders along the lines of more traditional qualitative marketing research.

Some of the steps mentioned above such as recruitment of candidates can be done internally without the need for an outside recruiting source–I suppose this just depends on you availibility and resources to select.

I recommend checking out Handbook of Online Marketing Research as well as there is a website just for UI studies–I’ll find it and post it soon!

I was recently at a presentation by Amy Africa at ECMOD in London where she described a test in which the people were given a budget and given a set period in which to spend it.

She described how some candidates would do things like send text messages to their friends about being in ‘this cool test’ and end up failing to make a purchase. One of the points in this case being that people do not use web sites in isolation. We are all distracted. More so on the web where we have the on-line distractions on top of all the off-line ones.

Conducting usability tests in the relevant environment, whether the home or office, can be frightening.

I have also heard of lab tests as an answer to everything from the Research team so the framework you provide around how to use testing and when is very helpful.

- It is essential to have key decision makers and stakeholders observing the users. Unless they witness it themselves they will have a hard time validating the actionable insights suggested from user behavior.

- The greatest benefit I have found is uncovering the latent needs of users. things like “I wish there was a button I could just click on right here.” When the majority of your group feels something is missing that can help them achieve their goal that is the great benefit that ONLY this kind of research can provide.

- Behavior can vary by segment. Younger folks use the web/browsers/etc. in ways older folks can’t imagine. Also some segments are more “wired” than others.” things like screen resolution, broadband all impact behavior and need to be taken into account in the lab set-up to align with your customer base. There can be variance by geography as well (though less likely) as well.

- You need to act on the data. I can’t tell you how many wireframes I created after labs that rotted on someone’s hard drive. There can be no fear of change if you want to reap the rewards of listening to your customers.

- Facial expressions sometimes tell more than anything the user says.

- My experience is that the less task based you make the session the more natural experience you will get from the user. Thus your insights will be more in line with actual real-world behavior. People always have the natural usrge to please when you ask them to do something in this type of environment.

I could go on and on on this topic….thanks for bringing this qualitative tool attention in the quantitative world.

Jonathan