Menu

UIEtips Article: Cue: A Usability Testing Bake-Off

by Jared Spool

UIEtips 4/17/07: Cue: A Usability Testing Bake-Off

What’s the difference between science and art? Is usability testing more of a science, that produces predictable, repeatable results? Or is it more of an art form, where the skill and talents of the people involved will have tremendous influence on the outcome? These are critically important questions, as we try to bring techniques like usability testing into the mainstream.

In 1999, Rolf Molich challenged the usability world with the CUE-2 study. In this landmark experiment, he had 9 separate usability teams each evaluate the same interface: Microsoft’s Hotmail.

If usability testing is a science, we would expect that every team would produce essentially the same results by finding the same problems and reporting them essentially the same way. Certainly, as with any scientific process, we could expect that the teams would generally agree on the most serious usability problems and rate problems with similar scales.

But that isn’t what Rolf found. Instead, each team essentially had their own methods and, surprisingly, found their own set of problems. 75% of the problems reported were only found by a single team — many of them very serious problems with the interface.

When everybody does something differently, it means we’re closer to art than to science. And being close to art, it means that the people involved have more impact than the process or methods chosen.

Is this bad? Not really. However, I do think there is something good about moving our craft to be more scientific. And it’s research like the CUE studies that will help us do just that.

In this week’s UIEtips, we’re re-printing an article I wrote back in 2005 about how Rolf’s work can help us learn about usability testing best practices and hone our own capabilities.

As always, I’m interested in your views on this. How do you learn to do your work better? Have you come up with ways to learn new techniques and tricks? I’m always interested to hear what you’re doing to improve your results. Leave your thoughts and join the discussion below.

Read today’s UIEtips article.

[If you want the opportunity to compare your work to practitioners taking part in Rolf’s CUE projects, then you should seriously consider attending his full-day seminar at UI12, Advanced Methods for Usability Testing. Under Rolf’s careful direction, you’ll walk through all aspects of conducting a usability test, from test scenario creation to reporting, to see what some of the best practitioners have done. You can read more about Rolf’s session here: Advanced Methods for Usability Testing]