Christine Perfetti – Jumpstart Your UX Research Program
Podcast: Play in new window | Download (Duration: 29:18 — 15.0MB)
Subscribe: iTunes | Android | RSS
UX folks often have to sell the importance of the field to stakeholders. That’s also the case with user research. The costs and time associated with starting a research program, and actually interacting with users, are sources of a lot of friction. Organizations are now seeing the value in user research but it’s daunting to know where to begin. It’s also difficult to fit research into an already established process.
Christine Perfetti is no stranger to consulting with companies about developing a UX research program. She’s often run into circumstances where organizations believe they have too many internal constraints, be they budgetary or otherwise, to conduct their own research. Rather than running their research program in an ongoing manner, Christine shows them that research methods don’t have to be painful and laborious.
Ideally, you’ll engage in research early in the process, but avoid arguing with stakeholders when you’re brought in late. Rather, focus on the value of research and actually get product managers and other stakeholders involved with the studies. This will have an effect on the opinion of research moving forward.
Check out Christine’s daylong workshop from the User Interface 18 conference, now in our All You Can Learn Library.
Recorded: July, 2013
[ Subscribe to our podcast via 
←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]
Full Transcript.
Jared Spool: Welcome, everyone. I am very happy that you all have joined us today because we have a special treat for you. We have Ms. Christine Perfetti, who was, many years ago, a premier employee of the UIE empire, but she has moved on to great things. She’s now at Acquia. She’s been consulting for many years.
She’s basically one of the smartest people I know when it comes to getting your user research program started. She’s giving a workshop at the UI18 conference called “Jumpstart Your UX Research Program.”
If you are thinking about what it might take to get a UX research project and program going in your organization, then you definitely want to talk to her about this and possibly even come to that workshop. Today, that’s exactly what we’re going to do. Hi, Christine.
Christine Perfetti: Hi, Jared. How are you?
Jared: I’m fine.
Christine: Before we get started, I just wanted to say what a pleasure it was to be part of the UIE empire.
[laughter]
Jared: You don’t work with me anymore. You don’t have to kiss up any more.
Christine: [laughs] Just for your audience, I’ll kiss up a little bit. I don’t know if your audience realizes I expected my time at UIE would be a one- to two-year stint. I just couldn’t get away from you and the team for eight years.
Jared: Yes.
Christine: [laughs]
Jared: The armed guards at the door were helpful with that.
Christine: [laughs] I’m really excited to have a chance to talk to you. [laughs]
Jared: It’s our retention program. [laughs] That’s funny. Since you left, you went on this adventure where you went off and did consulting for many years. Then you started working for companies. You basically developed this reputation for coming into companies and helping them get their UX research program off and started.
The problems that these companies have when they call you and come in…It feels to me like this is this interesting turning point. I’ve seen this with other folks. I don’t know if this is the way it’s been for you.
These companies are often at this place where they realize they have to do something, but they have no idea what to do. They have all this history of doing things, for lack of a better way of saying it, the wrong way. At least in their mind, it’s the wrong way.
But they don’t know how to do it differently, so they keep doing it that way even though they’ve asked you to help them do it differently. Is that the experience you’ve had?
Christine: Yeah. To give you a little bit of history, as you were mentioning, I was working with you. Then I did go off on my own for consulting for a few years. For the last three years, I’ve been immersed in a couple of companies as the UX Director. The experience I’ve had is I’ve been running research now — I think I started back in 2000.
I’ve been conducting research and working with clients or working with internal folks for, now, almost 14 years. I’ve seen some interesting developments. The first thing is, back in the days when I was working with you and the researchers at UIE, I’d say the biggest challenge I would have going into an organization and talking about research is just getting people bought into the idea of conducting research with users, whether it was customers and prospects.
There was a lot of, as an external consultant, selling organizations on the importance of research.
The last five years, as I’ve been out on my own as well as coming into organizations, that’s not the challenge anymore. Anytime I’m working with an organization and with the stakeholders, whether they be engineers or product managers or business stakeholders, the organizations, typically, really see the value of research.
In fact, in many of these organizations, they’ve started jump-starting their own research programs. The challenge I’m having isn’t that people no longer see the value. It’s that they’re trying to figure out how to run robust research studies and really have high-quality research results that will inform their work.
Now, I’d say, in the last few years, what I’ve been seeing is that a lot of organizations are starting up their programs. They have buy-in from their organization, but what they’re struggling with is how do they really, really run quality research.
Jared: When these folks are at this point, what is it that you think is keeping them from just doing it themselves?
Christine: There are a couple of challenges there. When I’m talking with organizations who have never conducted their own research, as I mentioned, it’s not that they’re not bought into the process and not seeing that it would be valuable. Instead, they’re really thinking that it’s going to be a long, laborious process. [laughs]
It’s going to be too much work. They’ll have limited resources. They don’t have enough budget, internally, in-house, to run their own studies. Oftentimes, organizations were coming to me initially because they thought they had too many internal constraints, in-house, to run their own research.
What those clients eventually were surprised by was, when I was coming in as an external consultant, what my goal was, was not to, in an ongoing way, run research for these organizations. Instead, my goal was to educate the organization and say, “Look, your research practices are going to be far more effective if you’re running your own research studies and not bringing in someone like me from the outside to run the research for you.”
In terms of getting teams excited about running their own practices internally, it’s a lot about showing and demonstrating to the internal folks that research doesn’t have to be this arduous process. It’s not something that has to take weeks and sometimes months to plan and prepare. A lot of the work I did going into organizations is showing them how easy it is to run quick-and-dirty studies and start informing their products.
Jared: Is it the case that — because I hear a lot of folks say, “Our culture here just doesn’t understand research, so we just have to restart the culture.” It’s almost like they want to come in and reinstall the company’s operating system and then reboot with this brand-new version that understands how to do UX and how to do research. Does that ever work?
Christine: Quick answer? [laughs] In my experience, no. [laughs] This is based on a lot of experience, a lot of trial-and-error on my part.
I can’t tell you, over the last 10 years, when I’ve gone to community events — by community events, I mean either usability or UX events — how many talks I go to and how many panels I go to about “How do we start selling the importance of UX to our organization, and how do we make our organization a UX-centric culture?”
What I’ve learned over the years is that’s absolutely the wrong question to be asking. If you and an organization are starting to think, “How do we drastically shift a culture so everyone’s UX-centric or it’s a UX-focused culture?” you’re not really focusing on the right things.
Instead, what I have found…To be fair, seven or eight years ago, I was one of the people talking this way, like, “How do we start getting stakeholders within an organization excited about research?” What I’ve found is it’s not coming into an organization and trying to drastically shift the way the culture is set up at the time.
Instead, I mentioned the last few years, I’ve been immersed in a company. Coming into organizations as a UX director, you may think that I bulldoze through and come in and say, “Hey, stakeholders, we’re really going to be focused on being a UX-centric culture.” I’ve found that that’s the least effective and compelling approach to get your organization excited about research and design.
This is a surprise to folks when they ask me for advice about how to get buy-in. Instead of going in and really evangelizing UX best practices, I spend the first three to six months immersed in an organization talking with my stakeholders and, in essence, conducting a research project with my users within an organization.
Who are my users in the organization? It’s my stakeholders, the UX team’s stakeholders. That’s the product managers, the engineers within an organization, the business stakeholders, the sales folks. It’s everyone in the organization who has a say in product decisions.
In my last couple of jobs, including my most recent where I’m three months in, it’s been really having conversations with all of those team members and stakeholders to understand how the culture is working right now, what their specific problems and challenges are, and how my stakeholders think I can help them.
When I’m describing that the absolute wrong question to ask is how do you make your organization excited about UX, it’s because the organization and the folks within the organization are our users, and we should be figuring out not how to convince them that UX is the way to go. Our job, as researchers and designers, is to work to satisfy our stakeholders’ needs and really, really understand what they need for their jobs to produce best-in-class products.
I spend my first few months just having lots of conversations with people internally to hear what their pain points are.
Jared: What are some of the pain points that come out when you do that?
Christine: There are a lot of pain points. [laughs]
I mentioned that I spend a lot of time talking to folks internally when I come in. I usually start out with leading a UX team. I’ll start with the designers and researchers on the UX team to hear about their challenges, and then also talk with the product managers and engineers.
The biggest challenge I’m hearing from organizations now — I touched upon this earlier — it’s not that people aren’t bought into wanting to produce compelling, best in class user experiences.
All of the stakeholders in house want to do that. The problem is limited time. There’s pressure to release things on time. There’ a lot of disconnect in process. I’d say the number one thing I hear from designers and researchers that’s a huge challenge for them on a UX team is how to be immersed and seen as a partner to the engineers and the product manager.
An example of this, I’ve gone into a few organizations, both as a consultant and now, being one of the employees, where the biggest challenge the design team has is they’re not brought into projects early enough to really have a strategic say in product decisions that are made.
In many cases, the designers and the design are seen as more of an agency approach, where the product managers and engineers will come to them later on in the process, once they’ve already made a lot of decisions about what the product and what the design should be.
They’re getting very frustrated [laughs] that they’re not brought in early enough. That’s one big problem. The other big problem I’m seeing quite a big is the designers are often times very siloed off from the rest of the organization, that I like to refer to this as a waterfally [laughs] type problem, where the designers in many organizations are asked to deliver their designs very early on the process.
But then you know what happens as we pass along our designs, and then six, seven, eight months goes by without there being any iteration and collaboration with the engineers and product managers.
And it’s a problem that I see at many organizations, is how to immerse design in such a way with the product chain that they’re really seen as full partners and they’re communicating with teams on an ongoing, regular basis.
Jared: This thing of not being brought in early enough, that feels to me like a constant struggle. What happens in organizations early on that sort of creates that culture where design is this afterthought? Is this an evolutionary thing that people are like, “Just bring in the designer to do that designery thing” at the end? How does that happen?
Christine: What I’ve started seeing happen, and it’s something that now I’ve seen [laughs] many times. It’s come out from dozens of conversations with product managers and engineers. It’s that when I come into an organization, they’re trying to figure out just as I am how to immerse design into their process.
“If you involve us early with your research, you’re just going to produce much, much more effective results.” That’s not helpful to product managers and engineers.
Instead, what I’ve been doing over the last couple of engagements I’ve had is rather than spending a lot of time trying to convince my stakeholders that they should involve us early, instead, I’m thinking about getting involved when they ask for us to be involved and showing them, regardless of what stage in the process they are, what the value of research can be.
What I mean by that is there was a recent study that I was running with stakeholders, engineering and product managers, stakeholders, and the UX team, specifically the research team, was brought in pretty late in the process. But that’s OK.
Instead of arguing with our stakeholders that they should have brought us in early, what we did is we ran a very robust study where we brought in a number of different customers to interact with product. Very, very importantly, we involved the product managers and engineers in the research.
You know what happened? The more you start involving your stakeholders in the research process, the more they see what’s actually happening with customers when they’re interacting with the product.
I had a moment with one of our product managers where my first couple of months at the company, we kept talking about how important it will be to have upfront research with customers to help inform product strategy.
She got it in concept, and she told me this, but she said until she sat in on several of our research sessions, she didn’t really have the real sense of what we could bring to the conversation and bring to the product. Just after having her, she’s now attended six of our sessions, she’s already thinking about how she could bring us in earlier.
It’s not about telling people that they should bring us in earlier. It’s about showing them the value of the research we bring to the project. That turns out to be key.
Jared: Is this an extension of the thing that I’ve always heard when I’ve worked with teams and I bring something like usability testing or field visits to them for the first time, inevitably, someone who’s been at the company for a really long time will turn to someone else and say, “Dammit, we should have done this two years ago.”
Christine: That’s been happening a lot.
Jared: [laughs] Do you think this is an extension of that sort of symptom, of basically they see the value as soon as you show them the power of what it means to get that feedback right upfront? You don’t have to do a selling job. It’s just like yeah, OK.
Christine: Totally true. One thing that I’ve found is that the more the UX team is getting others in the organization selling the importance of research and design, the better.
This has happened now dozens of times where once one of our key stakeholders, whether it’s a product manager or a lead engineer, sits in on one research session and has this, I refer to it as, the aha moment, where something in the session just happens where they go I get it. I now understand how I’m gathering insights just from watching these users that I couldn’t get any other way.
What I’ve started finding is that now I’m not evangelizing the importance of UX research in my organization. What’s happening is that the lead engineers and the lead product managers are going around to their teams and encouraging them to go because they’ve seen so much value from the research.
Jared, I do a lot of training with teams who are trying to kick off their research programs, and I have to say, the most common question I get from people is, how do we sell to stakeholders the importance of research? The one thing I say, and it’s just a very quick answer, it’s not about selling them, and it’s not about telling them how important research is. It’s about getting them to go to one session. [laughs]
I’ve tried this in many ways. People will come to me and say, “Hey, can you come into our organization and sell our stakeholders on research?” I said, “Yeah, I could come in, but it’s not going to help you.”
Jared: I have the same thing, yeah. It’s like I’m not going to convince your people to do this. You’re the outsider, they say. They listen to outsiders.
Christine: It’s the difference between…what I’ve found super fascinating over the last few years is the difference between being an external consultant who comes in for a week or two to help teams jumpstart their research versus being immersed in a company. It’s just such a fascinating difference.
What’s so exciting for me, being immersed in a company is, if I would go into an organization for two weeks, everyone would be excited for the two weeks I was there, conducting whether it was usability testing or field research. They’d all be thrilled. They’d be bringing the stakeholders in.
But guess what I found out that was happening at least half of the time was that once I left and once that excitement was gone, the teams weren’t conducting any more research. [laughs] What’s really exciting now is that as I’m working within a team is to make it so that the team — by the team, I mean the product managers, the engineers, the design folks — are coming to our researchers in essence like begging for more research and begging to be involved.
It’s not something we’ll run research once a month or once every two to three months. It’s that in an ongoing way every week we’re sitting down with customers and prospects and seeing what they do. The requests now are not coming from our internal design team. The requests are coming throughout the organization.
The nice problem I have right now, and it’s a really nice problem, and it’s a challenge, is I now have too many stakeholders coming to the UX research team clamoring to come out into the field [laughs] and seeing what customers are doing. The problem we’re having is that there are about 20 people that want to go out into the field in a month from now, and a research team is a team of three.
Now what we’re focusing on is trying to train as many people in the organization to go out so they don’t need us, because over time, it’s really not going to scale to hold these stakeholders back and not get them out with customers just because of our limited resources.
Jared: How do you deal with this thing that I hear folks who are like serious user experienced folks who’ve been trained and they’ve doing it for years, and when you say to them, “Well, you need to train others in your organization to do this.”How do you deal with that sort of resistance that say, “Well, but they’re going to do it wrong?”
Christine: It comes back to…I’ve had a lot of discussions with those trained researchers. I’ve said to them it’s about talking with them about their specific challenges that they’re having. That’s typically the pushback is that my concern is that these folks will do it wrong, and they won’t come back with the robust data that we need.
But my philosophy over the years is that usability testing is not this perfect science, and anyone who thinks it is and that’s the purpose of usability testing within an organization it’s just not true. The purpose of usability testing when we’re working within an organization is not to be controlling for all the factors. It’s not to be conducting science. It’s to help inform our product teams to make good decisions.
What I start to talk to these experienced researchers about is specifically what they’re challenges are, and typically, the number one challenge I hear from these folks who say, “I don’t want to educate others because they may not do this well.” The number one thing I hear is that they’re not getting their teams immersed in the research or they’re conducting hundreds of studies every year, but no one’s actually listening to the result.
When I start hearing that, I say, “Look, you may be running the best, most scientific studies in the world, but if none of your stakeholders actually care or are engaged or using these results to inform decisions are you really successful?”
As a community of researchers, we really have to think about what our end goal is. Is our end goal to be running these perfect [laughs] research experiments, or is our end goal to really start helping product teams make informed decisions. The best way to do that is to focus not on perfection with our research studies, but giving people tool and techniques that they can use where they may not be as experienced as they are, but they’re going to be getting incredible data back.
It’s not about doing things perfectly. It’s about getting information that the product team needs to make decisions.
Jared: Yes, I’ve never understood this argument that, “Well, they’re going to do it wrong and they’ll make bad decisions, and they won’t need us anymore.” I think it comes back to, “They won’t need us anymore, and they’ll just do crappy work.” I don’t know. I don’t think that’s true.
Christine: I think they need us even more. I’ve been seeing as I look at organizations and their maturity with their research when I mentioned that when we’d go out at UIE 10 years ago, the number one request people had was, “Can you just show me how to run a usability test?” They had never run a usability test before and they wanted to see how to do it.
I’m not hearing that anymore from most organizations. When I’m teaching courses with the engineers and the product mangers, and I ask, “How many of you have either observed a usability test or moderated at least a test?” at least half of the people in the room raised their hand.
What I’m starting to see there are organizations have evolved to the point where most people understand what a usability test is, but here’s the thing. They want to start being immersed in this research because they realize that the more that they’re running the test, we can make quick and dirty changes to the product in a much more efficient way.
I’ve always gotten this sense, and I agree with you, Jared, that there’s a fear that if others in the organization understand what we do and can conduct the research that there’s not really a place for us. I think it’s just the opposite that what I’m finding in my work is that what ends up happening is the engineers and the product managers can conduct their basic research studies and conduct usability testing on a weekly basis, but what they come to the research team is for their expertise in terms of what technique might we most want to use?
We’re in an advisory role. That’s what I’m hoping to get to in my organization is that people know the basics of research, but they’ll come to us when they have a really, really tough problem to solve, and they want some help in terms of structuring their research studies.
Jared: It feels to me like this is a little bit like when someone goes to a restaurant, and they have this amazing meal. They think, “Oh, my god. I want to learn how to cook this,” and they go home, and they cook it themselves, and it comes out good, but it doesn’t come out just as good as the restaurant. It’s not like they don’t ever want to go back to that restaurant.
It’s almost like you appreciate it even more when you’ve tried it and you realize, “Wow. This didn’t quite come out as good as when the chef did it. I wonder…There’s obviously something I’m not getting here.”
Christine: Yes. That’s something that I’m seeing all the time. Typically…I’m not saying that teams always have the bandwidth where they’re running their own testing, but what I’m finding is that in the last couple of organizations I’ve been working at, we’ve been really focusing on quick and dirty and lean testing.
We’re not typically for a study recruiting in 8 to 10 participants over a week long period. Instead what we’re doing is in a very quick and dirty way bringing in two to three customers every week, typically same day every week, and just learning and reiterating over time. What I’m finding is that though internal folks can run these quick and dirty studies as they have bigger problems that they’re trying to solve…
For example, when we’re really, really trying to understand our customers and our prospects pain points and motivations and goals, they’re coming to me in the team, and they’ve saying, “Hey, we want to start going out into the field. We want to start learning more about the underlying pain points and problems users have.”
They don’t want to go out without us to a one- to two-day field visit. They want our expertise because we’ve done this for a long time, and if we’re really, really trying to run robust field studies, guess what? They don’t just want to wing it. They want to come to the experts, and that’s never going to change.
Jared: This has been really awesome, and I can’t wait until your workshop at UI 18. The techniques that you’re going to talk about are really what a lot of folks need, and I think people are going to come in. They’re going to figure out how to get their research program off the ground and running and people bought in and excited about it, and it’s just going to be a fabulous day of learning. So I’m looking forward it to.
Christine: I am, too. I can’t wait. It should be a fun session. In terms of learning, what’s most fun about this session is, as I mentioned [laughs] earlier, I’m coming from the external consultant background, but also being immersed being in the company I think what folks will find really fun about the discussion is that I’ve had a lot of wins along the way and things have worked really, really well.
But there are a lot of stories of pain and failure and learning iteration, and really over the years learning what plays well and even more importantly what I wasn’t doing well. If I can be sharing those war stories, I think it’s a really great opportunity for these folks that are just trying to jumpstart their research program to frankly learn a bit from some of my failures. [laughs]
Jared: I can’t wait to deep dive into your failures. That will be fun, but it’ll be also great to see all these techniques that you have for success. So for those of you who are interested you want to go to UIConf.com and sign up, and Christine’s session is going to be October 23rd.
The whole conference is October 21st to 23rd. There are eight great workshops and a whole bunch of fabulous presentations, and you can get the whole description of the conference at UIConf.com. Christine, thanks for talking with us today.
Christine: It was fun.
Jared: I want to thank our audience for spending the time with us, and if you listen to us on the iTunes, if you wouldn’t mind going and putting in your thoughts in the ratings and reviews, that would be awesome. We will get those, and we love to see what you’re saying.
That’s all for today. Thank you very much for encouraging our behavior. We’ll talk to you next time. Take care.