Task Success Rate – Is that the right way to judge a usability test?

Over at the Boxes and Arrows LinkedIn discussion group, Carrie asked:

What is a good success rate for a usability test task?
We just conducted user testing on a site map. So we have success rate percentages for each task. They range from 9% – 51% success (in up to 3 tries). Obviously there are problems. (And no, we didn’t create the site map, which makes me feel good.) But what would be considered a “good” success rate? I want to say over 70% for this test. It is only site map, no content, which will limit the success anyway. Maybe I’m aiming too high?

Thinking in terms of % of completion may not be the right approach. (In fact, I’m hard pressed to come up with a time when it is the right approach.)

You haven’t said anything about who the users are or what the site map information contains. But let’s pretend the users are doctors and nurses and the site map contains the necessary information for them to administer drugs safely. If one of those doctors or nurses doesn’t find the information they need, they could improperly administer a treatment which could kill their patient. What would be an acceptable failure rate under these conditions? I’d say 0% — the system needs to ensure success of every user.

Why is your system any less important? Why would you be willing to tolerate any failures?

The real question isn’t “what is an acceptable level of failures?” The question I think you want is “What’s preventing people from succeeding?”

Instead of looking at how many people succeed versus how many fail, what if you were to analyze the failures themselves. Can you rank and categorize all the things that prevent your users from succeeding? Can you assign a classification that helps you understand whether the problems are life and death (as in the example of doctors and nurses I used above), problems that will lose customers, problems that will cost support money, and problems that are annoying without painful side effects?

This will also help you look at the participants you’re recruiting for your study. How similar are they to real users? How realistic are the tasks you’re asking them to complete? How well does the system, if they make a mistake at the site map, help them still succeed by having guidance for common errors on the content pages themselves? (Such as “If you’re looking for x, click here.” type lateral navigation.)

In the end, you really want to understand the problems real users will encounter. That’s the purpose for the studies. Then you want to explore solutions that resolve those problems. In an ideal world, it’s not that you get 100% task completion, it’s that you have addressed and solved all the problems.

The closer you can get your studies to map true in-the-wild user behavior, the more you’ll understand about the problems you’re uncovering and the solutions that will help. Focus on the problems and their resolution and you’ll get the design to where you’d like it to be.