Dana Chisnell – Gaining Design Insights from Your Research Recruiting Process
Podcast: Play in new window | Download (Duration: 25:03 — 12.7MB)
Subscribe: iTunes | Android | RSS
Getting great participants for usability studies can provide invaluable insights for your design process. But if you aren’t doing your own recruiting, you could be missing out on additional important information. Dana Chisnell has learned that the best way to find great participants is to think of recruiting as bonus user research.
Dana is the author of The Handbook of Usability Testing. In her virtual seminar, Gaining Design Insights from Your Research Recruiting Process, Dana shares her thoughts on recruiting, how to find the best participants, and what types of things predict behavior. The audience asked a bunch of great questions during the live seminar. Dana joins Adam Churchill to answer some of those questions in this podcast.
- How can you convince stakeholders not to use an agency for recruiting?
- How measurable is the difference in quality of the participants when self-recruiting?
- How can you get busy people to participate?
- What is the best way to approach recruiting users with disabilities?
- How do you recruit outside of your geographic location?
Recorded: December, 2013
[ Subscribe to our podcast via 
←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]
Full Transcript.
Adam Churchill: Welcome, everyone, to another edition of the SpoolCast. Earlier this fall, Dana Chisnell joined us to present her virtual seminar, Gaining to Design Insights from Your Research Recruiting Process.
Dana’s seminar, along with 115 others that teach us the tools and techniques you need to create great design is now part of the UIE user experience training library, soon to be unveiled as UIE’s All You Can Learn.
Real user research, the kind that doesn’t waste everybody’s time, means finding qualified people, whose feedback shapes your business, that involves a more strategic recruiting process, one that might be shorter and less budget intensive than you think.
Dana Chisnell knows what it takes to get the right people for a research study by using open-ended interviews with respondents instead of screening questionnaires.
In her seminar she shared her repeatable process for finding and interviewing real users to get great data.
For today’s podcast we’ve invited Dana back to discuss some of the issues and remaining questions from our October seminar.
Hey, Dana. Thanks for agreeing to spend a bit more time with us.
Dana Chisnell: Oh, it’s absolutely my pleasure.
Adam: For the folks that weren’t with us that day, can you give us an overview?
Dana: Yeah. I, over the years, have done a bazillion usability tests and user research projects.
In my early days I used market research companies to recruit participants for my studies. I became increasingly unhappy with the kinds of participants I was ending up with. They were very fine people but they didn’t really suit what I was trying to find out in my studies.
So over the years I’ve come to this conclusion that there’s a better way to do recruiting and it doesn’t involves a screener. In fact, the best way to think about recruiting participants for studies is as additional user research, In fact, kind of bonus user research.
I really think that the researcher should do the recruiting themselves. Those are the big messages from the virtual seminar.
And then I went through and talked about where to find participants and how useful it is to do snowball recruiting. We can talk about what that means. And how to do the open ended interview rather than using a screener and why screeners don’t work that well.
And then I talked a little bit about how important it is to focus on the behavior that you’re looking for because I have questions…
The thing that you want to observe in a usability study, especially, but probably also in a field study or some other kinds of user research that you’re doing. Rather than focusing on how old the person is, what their job title is, how much education they’ve had, and other kinds of demographics we’ve just learned that… I don’t mean to say that we’ve recently learned it, but we have figured out over time. Given this message that demographics don’t predict behavior.
What else did we talk about? We talked about setting expectations for the participants to make sure that they know what the session is going to be like when they get there and about the importance of scheduling and compensating participants for studies appropriately. Scheduling at the convenience of the participants rather than the convenience of the team and being as generous as you possibly can.
I think that those are the highlights.
Adam: The highlights are a lot there, sure.
Just to reiterate one of your points there, that aspect of onus user research was something that I know resonated with our audience and certainly with myself, as well. Never enough opportunities to talk to your users.
Dana: Yeah. I got a couple of emails afterwards saying, “I never thought about it that way before,” which made me very happy.
Adam: Let’s talk about some of those questions. Along those lines, maybe one of those people you were talking to, their boss says, “All right. We need to do this user research project. Go to the agency and have them get us some participants.” How do you help those people convince their team and maybe their boss or their client to use your approach instead of using a screener or an agency?
Dana: One of the things you can do to convince your team is to just say, “Hey, you know, I want to pilot test our criteria, our user profiles. So, we can spend a couple hours doing some of the recruiting myself. If anybody would like to sit in on the call with me that would be super awesome.”
Just taking it on yourself in maybe a little bit of a stealthy way, even if you end up using a screener you’re going to end up with a better screener for having tested out the questions that you’re going to be asking. A lot of it is just practice.
The other argument, of course, to management is that it’s going to be cheaper for you to do it than to go out to an agency. Now the costs are actually different. You’re just moving the cost from one pants pocket to the other because it’s not a cost that you’re going to write a check for to an agency but instead this is the time that you’re putting in to do it.
But the other way to look at that, too, is that you’re going to end up with much better participants because you have a better idea of who’s going to be in the room. You can be more selective.
You’re going to learn more about who the users are and the range of skills and personalities that they bring to your design. And you’re going to save costs because you’ll have fewer no shows, which means that you don’t have to recruit as many people. So you’re overall time for doing the recruiting is just going to be less.
Adam: There was a question that came in from our audience. Is there a measurable difference in the quality of participants based on the difference in these recruiting methods? And actually, this person makes a comment that they have about 1 in 12 participants that don’t fit the requirements when they use the screening method. I suspect there are a lot of folks out there that would say, “1 in 12 would be awesome. We’d love that.”
Dana: Yeah, really.
Adam: Measurable difference?
Dana: Yes. In the 10 years that we’ve been using this method our show rate is about 94 or 95 percent. That means that for every 100 participants that we recruit 95 actually show up. That’s not special to us. I don’t think that we have any particular superpowers and we are not threatening anybody if they don’t show up something bad will happen to them or anything. I just think that it’s a natural outcome of using this process.
One of the problems with using an agency and a screener in combination, especially, is that you, as the researcher, on your design and development team are way inside what that domain is and what the design is supposed to do. You know what the outcome is that you want.
An agency, a person at an agency with a screener is a couple of levels of abstraction away from your project. They don’t know what your business goals are. They don’t know what you know about the users.
And so, they are not going to ask the questions of you. They’re just going to go through the screener and if they’re lucky and they have people in the panel who are close to answering all the questions right in the screener then you’re going to get a pretty quick fill on all of the slots in your study.
But there’s not a good way to know unless you work really closely with your recruiting agency which you can do, and I know a lot of people who do this. Unless you work really closely with your agency to help them know where you’re trying to go, what your objectives for the study are. Sometimes the agencies don’t care about that. They’re in the business of filling seats.
But there are agencies that do care about doing a really great job for you. And there are agencies that are kind of looking to expand their business beyond finding people to taste test food that fortunately hardly ever makes it to market to doing more UX related recruiting.
And so, the most successful situations that I’ve had with agencies, and I hear this from other researchers, too, is when you work closely with an agency to educate them about your process, about your methods, about your business, technology, and user goals then they can do a better job of recruiting.
You can measure this in whether people show up, A, and B, whether they are actually motivated and in the market to do the behaviors that you want to observe in your study. Over time you’ll see a higher return and better data from your studies by using this method.
Adam: Again, specific to the recruiting piece of your user research, how do you get busy people to take part? I guess I’ll take it a little further. Any thoughts you have on actually who should be participating?
Dana: Getting busy people to participate as study participants, I assume, is what we’re talking about rather than team members who are avoiding being part of the research. There are a few ways to do it. One is just to schedule the sessions when they are available. We are inclined, generally as a practice, to doing stuff during business hours. Sometimes our people are just not available during business hours.
I had this situation where my recruiting collaborator, Sandy Olson and I, were recruiting for a company that does online merchant processing. That means handling payments and receiving payments and invoicing and things like that.
They had wanted to talk…They’d just build this new usability lab and spent a ton of money on it. They were really excited about it. They’d set up a specific time every week that they were going to hold the usability test sessions or interviews or whatever in this space. They had talked the executives into being there.
This is terrific and I congratulated them for all of these big achievements except that the participants they wanted were people who were a small office, home office, and small businesses.
They hadn’t thought about the fact that if they wanted these people to come into the lab for an hour, and this was in a place in Northern California that’s very urban and there’s a ton of traffic, that even within driving distance there’s no place you could come from to get to this particular lab that didn’t take at least 30 minutes to get there.
They spend an hour with you. They spend 30 minutes to get there. They spend 30 minutes or 40 minutes or maybe even an hour, depending on the timing of the study session. They’ve invested a solid two hours with you which means you have taken them out of business for those two hours. The cost of that is much higher than the $100 that you’re going to compensate them for.
Doing the sessions at times when they have more downtime or they have scheduled downtime or they have less going on, usually outside of business hours, is one way to get busy people to take part in your studies. The other thing is if you don’t have to be in the same room as them and you can do your session remotely, that can help, as well, because the time invested to getting places goes down quite a bit.
But this inevitably leads to a conversation about compensating participants. We look at it as compensation or honorarium and not incentive. Incentive is a word that’s taken from people who do surveys all the time. It’s basically manipulation and trickery. But instead, we look at this as, “Well, these people are really helping us learn things that will make the design better so we need to compensate them for their expertise.”
And so doing that as generously as possible and not just with a token thing. Mugs with the company logo are nice but I don’t know about you but we have a lot of those in our cupboard at home and I don’t really need another one. There are a few collector’s items in there, though.
It’s nice but cold, hard cash is one of the best ways to compensate people. And this goes for many levels of participants, many types of participants. One of my favorite examples of this was I was working with a biotech company in the days at the beginning of DNA sequencing, when the Human Genome Project was at its peak. There were five or six labs around in this particular geographic area that my client wanted to build a lab workflow system for.
All of the geneticists, all the lab people knew each other. It was a very incestuous thing. And so, we just decided to bring them in for focus groups and made them sign things that said they were not going to use any trade secrets that they learned from anybody else from these meetings. It actually worked out really well.
But these projects were funded by the market research group at this biotech company. They had this idea that the people that we were bringing in were not going to respond well to cash. That was crass and ethically it felt weird so they did things like they bought gift certificates for dinner for two at really nice restaurants or gift certificates from super high end groceries, sent them fruit baskets and things like that.
Of course, because people wanted new, good tools for doing the work that they were doing they were excited about taking part in the study. That was nice. We did a few rounds of these focus groups but there was some point when I said, “You know what? We can’t keep giving these people dinner for two at swanky restaurants because it’s not that interesting to them. They could do this on their own. They make enough money. So what else can we do? Can we actually give them cash and if we did that how much would we need to give them?”
We figured out that dinner for two at the swanky restaurant was worth about $400 or $500. And so, when we recruited these people for one round of these groups we gave them a choice between dinner for two, equivalent cash or a donation to their favorite charity. Damned if every one of them didn’t opt for the cash.
Adam: Cash is king.
Dana: [laughs] Everybody on our side was surprised but we went with it. Now, since then, there have been a number of accounting practices that corporations have been required to use, most notably Sarbanes Oxley requires a lot of accountability. So some accounting departments freak out about the idea that you’re giving cash to participants. There are a couple of ways to get around that.
One of the ways that I’ve seen to get around that is that the research group just has a line item in their budget that is basically a cash till. They document that they’re giving cash to people. They get a receipt from the participant so they can show that the money was spent this way and the accountant is happy with that. That is one way. There’s just this cash box in the research department.
Another way that groups have got to deal with this is giving cash equivalent gift cards.
Adam: And that gets around the accounting constraints. It’s not the same as cash?
Dana: Yeah.
Adam: That’s great.
Dana: I think this is weird because it’s still a gift so if the objection is that we’re giving people gifts and that’s not cool or they can’t accept gifts. There’s a lot of discussion to be had with the legal department and the accounting department about how to handle these things but gift cards can work well for some participants and for some companies. I find them a pain in the rear.
But the other thing is you can sometimes have…If you are working through an agency, again, sometimes pay the agency an additional fee on top of the per head cost, because that’s usually the business model, right.
It is that you ask for 12 participants and they are somewhere between $150 a head and $500 a head to add on to that cost that you’re paying the agency whatever you’re going to compensate the participants, and then they compensate the participants.
I don’t like that either, because that means that it’s not the best experience for the participant because they have to wait to get paid. Ideally you pay the participant, or give them their gift, or whatever it is that you’re giving them for taking part at the session. That’s the really respectful thing to do. It’s the right thing to do.
Adam: What advice do you have for recruiting users with disabilities?
Dana: The best advice I can give your for recruiting participants with disabilities is to connect with their community groups, and make friends with those people and help them see what the sessions are going to be like. Our best successes on that have been actually showing up at some place like Lighthouse for the Blind, or the National Federation for the Blind, or other groups like there are groups for practically every disability out there that you can observe in any way.
Showing up there or calling and having a meeting with the executive director or whoever is the membership director. There’s usually somebody who does community outreach, and explaining what you’re doing and how it works. Maybe even demonstrating with them what you do and how you do it, and making that person your partner in doing the recruiting. The other thing that we found that’s really helpful is to do a few sessions.
Get two or three participants to do the sessions and then ask them to kind of snowball it for you. Invite other people now that they’ve seen what this is going to be like, they can help set the expectations for the other people who we want to recruit. That kind of community connection is very helpful and good not only for one study, but for a continuing relationships.
We’ve just found that that works really well to have a personal connection there to accomplish two things. One is to help people know what the session is going to be like. It’s not scary, that they’re not being tested, [laughing] there aren’t going to be any electric probes or anything like that. But also to stay in touch with that community so you can come back another time and not only do testing but get advisors and work more closely with people.
So you understand better the experience they’re having with your designs.
Adam: Dana the next question is something I know you have a lot of experience is, especially with all the work you’ve done with voting. How do you recruit outside of your geographic location?
Dana: This is so much easier than it seems [laughs] . People get really intimidated by these kinds of boundaries, but it’s really not that hard. Unless what you’re trying to do is intercept people in front of the Walmart or something. There’s this thing called the Internet. People are on it from all over the world, and it’s sort of magical. One of the ways you can do recruiting outside of your geographic area is to practice what we call snowball recruiting.
That is in its simplest form, contacting people you know saying what you’re looking for, and describing the kind of people you want to have in your study, and saying “Do you know anyone like this?” Often, people will come through and say, “Yes! I do!” And they’re off in some far-reaching place, but there are a bunch of other ways you can do it, too. Professional associations if that’s appropriate can help you connect with people in lots of different places in the world.
For example, many of the studies that I’ve done over the years have been with say, network administrators, or system administrators, or other technical people, developers, engineers. So you can partner your local or the international IEEE or other society and say, “Hey! Can we buy a membership list, or can we show up at a member meeting and put the word out? Is there a newsletter that we could put this listing in?”
That often works really well. Those can be some good connections. But we’ve really found that the personal connections in our social media network and our personal networks really are a great way to start finding people abroad, finding people outside the neighborhood.
Adam: Very cool, Dana. Thanks for making time for us.
Dana: You’re most welcome. I hope that my answers were useful and will be helpful to people coming up.
Adam: I would be surprised if people didn’t agree that they were. To everyone listening in, thanks for joining us and for your support of the UIE virtual seminar program. Bye for now!