Menu

Watch Out For Observer Stamina

by Jared Spool

The longest usability test session we ever ran: 20 hours. Not in a row. Spread across five days.

That’s right. We conducted a usability test where the same participant worked on the same interface for five 4-hour sessions. In fact, we did it 8 times over a four week period, with 2 participants each day.

We were testing application development tools in an integrated environment. We wanted to see a developer take a project from concept all the way to completion. The project involved building a database, creating screens, and writing code. No way we could do that in a 60-minute session.

The sessions were amazing. We learned a ton about how developers create applications. We saw the programmer make mistakes, then use the debugging facilities to discover them. We saw them read the manuals and the online help (or make guesses instead of reading them). The tests gave us tremendous insight into the software development process.

Recently, I saw an email discussion concluding that you shouldn’t run usability tests for more than 60 minutes because the participants get tired and stop giving useful information. However, we had no problems with our users during the 20-hour test. They never tired out.

Why? Because they were doing what they normally do, 40+ hours every week. They were doing their jobs and, for them, that isn’t tiring. Of course, they took breaks and made frequent pauses to answer our questions. But, at the end of the week, they seemed very pleased with themselves and very anxious to keep answering any questions we had.

Not every test we run is 20 hours. Most only last 2 or 3 hours. Occasionally, we do one as short as an hour.

We believe that getting a user into the lab is a very costly and difficult process. Once we get them there, we want to have as much time with them as possible. That’s why we tend to go for the long tests. And, since we’ve carefully screened users in the recruitment process, we rarely run into the problem of them tiring out and reducing the quality of their feedback.

In fact, we have more problems with observer stamina than we do with user stamina. Observing a 3-hour session can be exhausting. As a facilitator, I’m more concerned about my observers losing focus than my users losing focus.

When observers lose focus, they find it hard to pay attention. They start to drift off or talk amongst themselves. Or worse, they get silly. Since we don’t typically use a real usability lab, unfocused observers can distract the user and create an additional burden for the facilitator.

To ensure observers stay with us, we take frequent breaks, even if the user doesn’t feel they need it. These are planned into our session.

A client recently told me that they scheduled six 1-hour tests in a single day, back-to-back, with only a 45-minute break for lunch. At the end of the day, the team was so loopy that they couldn’t concentrate on anything the last participant did.

We try not to do more than an hour without a 10-minute break. When we have more than one session in a row, we give ourselves a minimum of 30 minutes to rest.

When we conducted our 20-hour tests, we left 90-minutes between the morning and afternoon tests to get lunch, go for a walk, check up on email and generally have “away time,” in addition to the 10-minutes breaks we took every hour with the participant. The days were long, but the breaks were very helpful.

Even when we do field studies, we make sure we have time between visits to regroup and get a sense of what we learned. We found that if rush from one site visit to the next, without a break, we aren’t as effective as we are if we take breaks between each one.

So, in our work, we don’t worry about tiring out the participants. We worry about tiring out the observers. If we take good care of them, everything seems to go really well.