More on Usability Tests with 30 Observers
We’ve had some great comments on my post, Usability Tests with 30 Observers. Here are my responses:
Daniel asked:
Would the result be the same if the participant sat in a different room from the 30 observers?
Are there benefits by having them in the same room?
On initial reads this seems scary for the participant (especially when walking in to see 30 people)
In my experience, the result is different for both the observers and the participant when they are in different rooms.
For the observers, when there’s a window or a video link separating them from the participant, that distance sends a message. I’ve found observers are less likely to pay attention and more likely to make jokes and derogatory comments when they are isolated.
Participants aren’t dumb and know someone is watching. They can see the mirrors and the cameras and know they are being observed. But, because they can’t see their observers, they can only imagine what those observers are doing. I’ve noticed participants tend to assume the worst and the mirror/video-link unnerves them more than having the observers right in the room.
Another benefit comes from direct interaction. Having the observers in the same room as the participants means they can interact. I’ve found developers and stakeholders come away from sessions they attend this way with a solid connection to the users they are trying to design for.
Chris said:
I would be terrified to have my back to a room of 30 stakeholders and experts.
Chris, I’m assuming your fear may be justified because of your organization’s culture. In my experience, most of the time, this fear is completely unfulfilled and everyone behaves nicely, getting a lot out of the study. However, if you don’t want to have the stakeholders behind you, there are two options:
- Rotate the moderator and participant station 90 degrees, so the stakeholders are not behind you, but on one side of you. (You’ll be positioned between the participant and the observers.) This way you can keep an eye on both. I’ve done this plenty of times and it works well.
- Get a buddy to help you with “crowd control”. Either you or your colleague sit with the observers and, through careful passing of notes, help them understand what they are seeing. Don’t underestimate the power of a well-timed hand-written note.
Dr. Pete asked:
Do you find that doing observation on an actual subject has more value than just taking the 30 observers through a simulated session? Do you throw that subject’s data out?
Dr. Pete, it sounds like you think I’m doing this only for demonstration purposes. In fact, these are real tests that the observers are participating in. No data is thrown out. The entire series of tests use the same setup.
The last time I did this was for a top-10 e-commerce site (who averages approximately $1.2 billion in sales each year). We observed 24 users this way, collecting the data from each one. Each session was filled with 30 folks (we had to put together an elaborate signup and waiting-list system to handle it), often with senior executives — directors, VPs, and even the CEO. Most observers came to more than one session and would cancel other meetings to attend.
As a result, changes were made that look like they’ll realize about $250,000,000 in additional revenue this year. I don’t think we could’ve had the impact if I didn’t have the 30 observers in each session.