Cyd Harrell – The Challenges of Usability Testing Mobile Apps
Podcast: Play in new window | Download (Duration: 31:09 — 15.9MB)
Subscribe: iTunes | Android | RSS
As much as we may like to pretend, there is nothing natural about usability testing. There’s always a level of concentration involved that likely wouldn’t be present in a natural setting. This “unnaturalness” is magnified when testing mobile applications. Users have to focus on things like posture and how they’re holding the device while trying to interact with it realistically.
Cyd Harrell of Code for America, and formerly Bolt | Peters, has developed some clever hacks over the years. These techniques can be more effective, both in scope and cost as well as results, than a formal testing lab. Even something as simple as “hugging” a laptop with the screen angled away from you and using the built in camera can give fantastic insights into how a user will interact with a mobile device.
The aim of these techniques is to provide an environment that’s comfortable when the user engages engage with the app during the test. Cyd has demonstrated a technique allowing the user to sit in a comfortable chair while a camera documents their activity. Avoiding distraction for the user in testing by taking the burden of concentration away results in much more accurate and useful data.
Check out Cyd’s daylong workshop from the UX Immersion Mobile Conference, now in our All You Can Learn Library.
Recorded: December, 2013
[ Subscribe to our podcast via 
←This link will launch the iTunes application.]
[ Subscribe with other podcast applications.]
Full Transcript.
Jared Spool: Hello, everyone, and here we are again at another episode of the SpoolCast. I’m so glad you were able to join us today. I have with me someone else who I’m exceptionally glad was able to join us, the wonderful Cyd Harrell, who is UX evangelist for Code for America. She’s one of the smartest people I know, particularly on the topic of conducting usability research for mobile apps, which is, coincidentally, the name of her workshop for the UX Immersion Conference that’s going to happen April 7 till 9 in Denver.
Cyd, it is awesome to talk to you again.
Cyd Harrell: It’s fantastic to be here. Thanks for having me.
Jared: Thanks for giving us the time. Last year, you did this workshop, and I got to tell you — it rocked the house. People just loved this. I remember talking to folks afterwards, and one of the things that they loved so much about it was how much they were learning from other folks. Then I talked to you about it and you talked about how much you learned from other folks in the workshop. What actually happened? I thought you were supposed to go and teach people stuff?
Cyd: Well, I hope I did that. At the same time, the field is evolving so quickly. There are so many tools that people are out trying things in some cases, that I haven’t had the chance to try yet. I wanted to respect that in my workshop. So I said “Number one,” and I’ll do this again this year again I think, “This is going to be an open questions workshop. So people can ask questions at any time during the day. In fact, I’d also love to hear your stories of mobile research that you’ve tried, for those of you that have already done that.”
We had some really interesting ones. I had seen an article the week before the workshop, by a woman named Tonya Lang in Australia, who had figured out a way to do a scrolling paper prototype of a mobile application. As it turned out, Tonya was at the workshop and happened to have one of the paper frames that she constructed to make these with her. I asked her to stand up for a moment and show the audience from the source what this cool little paper prototyping technique was.
During the break, several people came up to me and talked about studies that they had successfully done. Where they had looked at people walking around stores. Other things, variations of things that I had done. I said I would hate to have this kept to just the break time. So at the beginning of each section, I asked a couple of people to share those stories.
I think we all really appreciated that. I certainly did because we’re learning this together. While you very kindly said that I am known for this particular area, I think that there are many people out there experimenting. The more of us that can share with each other, the more quickly we’ll understand what the best practices are, especially as they evolve so fast as the technology changes.
Jared: I think we’re in this stage where everyone is just getting used to the tools and figuring out what they can do and just playing around, and it’s a beautiful stage. I remember when we were just starting to do usability testing for what was then desktop computers, and in some cases for mainframes and minicomputers, back in the late ’70s, early ’80s, and we started to realize that we didn’t have to do this like we were testing rats in a maze, with a very strict protocol. We could start to play around in the sessions, and we started to experiment with different types of mechanisms for conducting user research, some of which were wildly successful and some of which were just dreadfully awful.
Cyd: [laughs]
Jared: At the time, we got a chance to really play with the idea of what is user research and how does it work, and I feel like we’re back in that mode again.
Cyd: I think we are, too. It’s interesting to see that they’re developing some solutions to some of the problems that we talked about last year. At the time that we were talking last year, people were really figuring out, how do I allow any kind of observation when we’re talking about a mobile screen? Do I need to restrict my participants to keeping their device on a square paper on the desk — which, of course, completely impedes their natural holding pattern in the sense of how their hand would normally hold a mobile device — in order to make the video look good to the people in the observation room? What are the trade-offs there?
There are a lot better tools and positions and so forth that have evolved to do that this year, but people are still really playing with, how do I research mobile device use in context? I might be able to do something, and I know I’ve personally participated in moving-car device research and store walk-around device research with good WiFi, in diorama-experience sampling research, where I send somebody a ping and they send me back, “This is what I’m doing,” or “This is a picture of where I am.”
I think there are some really important things that are coming out, partly tool-based and partly practice-based, to allow us to research the actual way that people use these in the context of mobile device use in their lives.
Jared: I think that it’s really an interesting, fun time. Some of this stuff gets really low-fi.
Cyd: You mean like scrolling paper prototypes on adding-machine tape.
Jared: Yeah, yeah, yeah, or just rubber-banding screen shots to an actual phone or…
Cyd: Drawing screens on Post-It notes. They’re a great form factor for mobile phones.
Jared: Yeah, yeah. I’ve seen people sketch something and then take a picture of it and then just use the photo app to simulate actual use patterns.
Cyd: You can do that, and then you’ve got POPAPP, which actually lets you put interactions into sketched prototypes like that.
Jared: Oh, I haven’t heard about this. What is POPAPP?
Cyd:This is cool, actually. I used it in the workshop, and it is exactly for taking a photograph of your sketches, and then you can put hot spots in it. You can basically create a clickable prototype for a mobile app in 5 or 10 minutes.
Jared: That’s very cool.
Cyd: There’s a ton of capability and a ton of tools and options. It’s funny. I went to review the tools that I had talked about last year, and just about everyone has at least significantly more competition than it had last year.
There are a bunch of options now if you would like to embed code in a native app and record the actual use patterns of people, with their clicks and their faces. There are several options for screen-sharing with a mobile device that didn’t exist this past spring. We’re still talking in December, and we’ll be talking to your audience in April, and I’m sure we’ll see a few more things develop even before then.
Jared: It’s just so crazy.
Cyd: It’s the Wild West.
Jared: It really is. It really is.
Cyd: That’s what I think is fun, honestly.
Jared: I think it is. I think it is. I think to do user research with mobile, it has to be in your nature to be flexible and a quick learner.
Cyd: [laughs] I think so, but I think even if you don’t think that’s your nature, it’s probably in you. I have certainly continued to need to do that in my practice here at Code for America, as we send these tiny little teams into cities to solve really big problems and build an app on a really short time line. I continue to evolve — “Oh, OK. You have two days to do UX research on a kind of app I haven’t heard of before? Let’s figure it out.”
I think that those are often what people need to do when they’re faced with the need to do mobile UX research. They don’t get months to plan it. It’s sometimes an afterthought or it’s hard to shoehorn in there. The reality is that the technical aspects of it can seem intimidating, but if you come at it with the right attitude — and your trusty roll of duct tape, as I like to say — you can just about always put together a method to solve the research question. Putting together your own methods is actually one of the most fun parts of doing any kind of design research, I think.
Jared: I’m intrigued by this. Tell folks what you’re doing at Code for America. I think a lot of folks probably haven’t heard too much about it. You’ve been doing some really particularly interesting stuff with mobile there.
Cyd: That’s right. I’m really proud of several of the things that came out of the projects this year.
Code for America is a nonprofit, and one of the major things we do is run a fellowship program. We bring in techies and designers, and we put them in small teams of three or four, together with an American city hall. The idea is for them to solve a significant problem that city hall has identified through building some technical solution and also teaching and embodying the practices that go along with that.
A couple of the interesting ones this year. In San Francisco, the problem that our fellows were tasked with was helping keep people’s food-stamp benefits current.
Jared: Hmm…
Cyd: Yeah. [laughs] Doesn’t sound all that technical, right? The practice that existed in the city was that they would send people letters, and people had to reconfirm, “Yes, I still have income below a certain level. Yes, these family members are still living with me.” They had to fill out a complicated form that would be delivered to them by snail-mail with not a lot of time to spare. Something like two or three percent of San Francisco’s beneficiaries were falling off the rolls every month, and they wouldn’t find out about it until they went to purchase groceries at a store.
Jared: Wow.
Cyd: Yeah. That’s a nasty moment, right? You’re there with your kids in a grocery line with that nice debit card that looks like everybody else’s debit card, and suddenly you get this, “I’m sorry, ma’am, your benefits are no longer active. You’ll have to put all that food back.”
Jared: Wow.
Cyd: Yeah. What our fellows realized, after observing at the sign-up center for a human-service agency in San Francisco, is that most of these adults who were in this program actually had text-capable smart phones — smart phones or feature phones, I should say. They were able to persuade the city of San Francisco to implement a policy change to collect cell phone numbers from beneficiaries, and set up an application by which, instead of sending a snail-mail letter, which, of course, is problematic because often people who are on public benefits have less stable addresses than people who are a little higher up the socioeconomic spectrum, but their cell phones often stay constant.
The HSA now is able to batch-send a text message to people saying, “We need to reconfirm these facts about you, and you can call this number and speak with one of our representatives.” They had close to 1,000 people sign up in the pilot and have prevented a whole bunch of people from losing their benefits, through the use of mobile phones with a clever service design.
Jared: That’s really cool. To build this out, your guys did a lot of different types of research, yes?
Cyd: Yes. They did what you would think of as fairly traditional ethnographic research, observing in a physical space. They met with clients and understood what types of phones they had and what their capabilities were.
They did some research around language and accessibility. There’s a double language issue there, right? You have a set of languages, some of which are English and some of which are not, that people are most comfortable with, and then you also have language limitations in terms of the space of an SMS message and language limitations in terms of what reading level you want to pitch those to. They did a tremendous amount of refining the language and the way that this would appear on some of the crummiest feature phones out there, to make sure that people would understand what was happening but not be scared and not over-interpret the message to think that it said, “You are losing your benefits right now,” but that it just said, “There’s something you need to take care of here to make sure that everything keeps going.”
They didn’t do anything that I would call traditional lab research. It was just about all in the field. They did a little bit of piloting each research project with some of their colleagues who were on other teams here that were associated with other cities, but by and large, they used techniques that we would recognize from most stages of user research, just re-jiggered to work with people who have feature phones, who have a harder time responding to a fancy survey like might come from Typeform or SurveyMonkey Mobile or one of those things that are actually great. They pared it back to the basics.
Jared: When you say “pared it back to the basics,” you mean…?
Cyd: I mean talking with individuals, tracking them down, if they didn’t respond to an initial request for feedback, and watching them use the actual prototype on their own, personal, actual phones rather than on test models.
Jared: Some of the surprises that they ran into, what were those?
Cyd: I think these guys were surprised by exactly how spare and clear and essential they had to make the copy. Almost all of it could be considered microcopy. How do you convey a complex idea like, “You need to complete a required form within the next X days in order to continue, in effect, these benefits, which have a really bureaucratic name but which you know as your food card,” while still satisfying the legal requirements related to the language?
There was a ton of iteration, and they were very surprised by how much of it was copywriting. Then, of course — and this was kind of interesting because I think this happens a lot — there’s a maintenance aspect on the back end if it isn’t on a mobile device. Working out the back-end system for this — which is called Promptly, by the way — so that it made it hard for government officials who were sending messages to make mistakes and send the wrong message or an inappropriate message to any of the actual end users of the app who were experiencing it on their mobile phone. That took a lot more effort than expected.
Jared: I was just talking to Karen McGrane, and we were talking about how the back ends get neglected way too often.
Cyd: I think it’s really true, and I think a big component of a mobile experience is, where is it coming from? [laughs]
Jared: Right.
Cyd: Often, it is coming from an experience in a different realm. In order to deliver an excellent mobile experience, they had to look at, also, a desktop, Web-based experience. That was part of how the government officials would create this experience over time for the actual beneficiaries.
Jared: Was that something that came to them from the very beginning of the project, or was that something they discovered along the way and said, “Oh my gosh, we have to think about that”?
Cyd: [laughs] They discovered it along the way. To their credit, they set it up and they thought about the end-user experience first, which I think is the lens that we always want to think from. Then, as they started to train the people who would be working with this system after they left town, they realized how incredibly important it was to have a great experience for them as well.
Jared: That makes perfect sense. All of that seems — again, this idea of just having this full toolbox at your beck and call and looking at the research you need to do at the moment and saying, “OK, what tool is best?” and reaching into that toolbox and grabbing something that could do the job pretty well and just going ahead with it.
Cyd: Yes. Like I say, my favorite thing in the toolbox is always duct tape, because sometimes a hammer’s what you got and it seems like a screwdriver would be better, but maybe you can make it work with a nail.
Jared: That’s very funny. Years ago — I’m talking 1980s — we did this project for a large construction-tool manufacturer. A large manufacturer of construction tools. The tools were actually not very large. Home construction, carpentry, that sort of thing. We would go out on these sites where there were all these master carpenters, and we would spend a lot of time watching them use it. One of the things we were focusing on were how they used drills.
Cyd: This sounds so fun, by the way.
Jared: Yes, yes. It was a blast. It was absolutely a blast. These were really talented, smart people we were working with, and they were really dedicated to what they were doing and they loved their job, and they loved their tools, and their tools were part of them.
We were paying attention to the electric drills. The electric drills, we actually spent time measuring what they were used for in the course of a project. They’re used for drilling holes very infrequently, it turns out.
Cyd: [laughs]
Jared: They do get used as hammers, and they have to be made rugged because sometimes the drill is easier to grab than the hammer. The number-one thing that they are used for on a construction site, particularly an outdoor construction site, is to hold the blueprints down.
Cyd: [laughs]
Jared: In fact, the manufacturer we were working with eventually put a paper clip. They built into the plastic shell a clip that you could use to clip papers to the drill, because so many people used it to hold essential papers down while it was on-site. Then, inevitably, they’d grab it real quick, and whatever papers it was holding down would go flying, and they were like, “OK, we can clip the papers to it so at least that doesn’t happen.”
[laughs]
Cyd: That’s amazing! [laughs] What a great story.
I do think we are in a space like that with the tools for mobile research, right? There are getting to be some great mobile survey tools, for example, and I think the people who make them are focusing on that as their use case. What someone like me is likely to do is say, “Hmm. OK, I want people to give me a screen shot of whatever’s on their mobile phone around commute hours every day, so I’d like to use one of the text-blasting systems to recruit a list of people and send them all an SMS at about 8:30 their time and ask them to take a screen shot, put it into one of these surveys.”
Maybe my survey will just be, “I’d like a caption for the picture that you’re taking, and I’d like you to answer two radio-button questions and then send it off.” I don’t use a lot of the features of the survey, in terms of complex branching logic and so forth, but I make it. I give it a shortened URL from one of those services. I use one of the text-blast services to send it out to people. I do some scheduling, probably with WhenIsGood or something like that, in order to figure out when to send things to people. Then I’m going to use something on the back end to collect all of the photos and responses that I get. Maybe that’s just Google Docs and Dropbox, right?
Jared: Right.
Cyd: All of a sudden, I’ve taped together an experience-sampling remote research tool for mobile users of commuting applications.
Jared: That sort of thing is really cool. Of course, hopefully the tool vendors are paying attention to all this and saying, “OK, we can get rid of the duct-tape parts of this and actually make this something useful.” It is neat that we have all these little components and we can start to plug them together and do something really cool with them.
Cyd: I agree. I think we have the basics, really. If there is any kind of research question that people want to solve about mobile phones, I think we have, absolutely, the technology available to solve it. It’s just perhaps going to require a little duct tape and a little ingenuity — and in some cases, a fair amount of time, partly because they’re some of the basic pieces that we don’t have in the way that we have data from 25 or 30 years of usage of desktop computers.
I saw a great study yesterday — it was actually from February — where someone stood on the streets of New York and observed in what position people were holding their mobile phones as they walked by.
Jared: Oh! Luke Wroblewski was talking about something similar, where some dude went out and looked at which hand they were holding it in and whether they were using their thumb to operate it or their forefinger.
Cyd: Yeah, I think it must be the same study. It was in UXmatters a while ago.
Jared: It was like 1,700 people they observed, some incredibly large sample size, and they got some really interesting breakdowns on how people held it. It was purely street-observation stuff.
Cyd: Yes. No consent, just public space.
Jared: Yeah. In New York, you can do that, because no one in New York wants to look at other people, so they all walk down the street looking at their phones.
Cyd: [laughs]
Jared: In fact, there was a video that somebody put out about people bumping into people walking down the street with their phones. Then somebody else was doing something about standing in front of people as they walk down the street looking at their phones, to see if they notice you before they collide with you.
Cyd: That’s too funny. [laughs]
Jared: It was all that same sort of thing. Yeah. The type of data we can collect and gather now is really fascinating.
Cyd: Then you’ve got really useful basic resources like the Pew Internet and American Life Project, which I hope everyone is tracking. Because that’s where I find out, most recently, in October, they said, “Well, we’ve crossed a line. A majority of American adults now have smart phones.” 90 percent of American adults now have cellular phones of one kind or another.
One of the things that that is is a great piece of data to take back to your management and say, “We need to be thinking about mobile.” Of course, one of the things that we talk about in terms of access for everyone is that there are getting to be more and more mobile-only Internet users in the United States as well as in a lot of countries around the world.
Jared: I keep hearing stories about people giving up their machines, or they’re just not near a desktop machine most of the time, and therefore mobile is the only way they’re interacting with the majority of websites. It’s not unusual — I see this on the train all the time. Someone has their laptop open, but they’re interacting on their phone at the exact same moment.
Cyd: [laughs] Yes. Those combinations, I think, are very interesting. Makes me feel good about the little hack that we’ve all been using for mobile remote research for a while, which is to turn a laptop around and hug it.
Jared: Right.
Cyd: So that the phone can be seen. Well, that’s not quite as unrealistic as maybe we thought at the time.
Jared: Explain that hack a little more, because I’ve seen it but I bet a whole bunch of folks haven’t seen this crazy thing that you guys do.
Cyd: [laughs] I actually have to credit it to the wonderful Jenn Downs from MailChimp. A couple folks at Mozilla started around the same time, but she was the first one to present it that I saw.
The idea is, you have a webcam on most laptops, right at the top there. If you angle the screen down just a little bit and turn the laptop around so that the camera is then pointing away from you, and then you give the laptop a cuddly little hug so that you hold your mobile phone underneath the overhanging screen section of the laptop, the webcam will be pointing right at your hand using the phone. If you enter a GoToMeeting or a Skype or Google Hangout or any of those things that makes use of the webcam, you suddenly have a remote research setup where you can see everything the user is doing on the phone.
Jared: That’s just such a clever technique. It’s with equipment we already have.
Cyd: Yes. Yes. GoToMeeting is now allowing you to broadcast your screen from a tablet, which is great, and they have GoToAssist as well, which will let Android screens be shared from a phone. Don’t ask me the tablet question, because I don’t know the specs on that — it hasn’t come up in any of my projects — but there’s, again, a thing where I’ll bet someone who will attend my workshop will know.
Jared: Right. When I came into your workshop last year, you had this comfortable chair in the front of the room, and you weren’t using it to torture anybody, I don’t think.
Cyd: [laughs] You’d have to ask the guy who was my guinea pig. I think his name was Greg, and he was very nice.
One of the things that I think is really important is to put the user, if you’re doing a lab study, in a somewhat-plausible context, and that can really be just a small thing.
What we did back when I was Bolt | Peters and what I’ve continued to do is, if I want to have a mobile research session and show the participant’s screen to observers in another room, instead of having the person put their phone on, say, a square paper on a table and then poke at it with their finger in a very unnatural way, I give them a comfortable armchair so that they can hold the phone in either their right or their left hand. Then I stick a Flip Cam or a GoPro or an IPEVO, or really any little camera that has a broadcast capability, on a microphone boom and adjust it so that the person can be using their phone one or two-handedly as they normally would, and the people in the observation room can get a feed from that little camera on the boom.
Jared: That’s really cool.
Cyd: Yet really simple.
Jared: Yeah. Yeah. That’s got to really put people at ease, being in that comfortable chair.
Cyd: It’s a really big difference. I think, given what we know from that study where the guy observed the 1,300 people — and I think his name was Steve Hoober, by the way.
Jared: OK. That sounds right.
Cyd: What we know from that and the way that people preferentially hold their phones, I think you’re going to get much more reliable data, not just because the person is comfortable but because the person is doing something in the way that they normally would.
Jared: I think that there’s nothing realistic about a usability test. We always delude ourselves when say, “We’re trying to make it realistic,” but there isn’t anything realistic about it. If people are focusing on their posture, they’re being distracted from everything else, and it does create more issues with what you’re trying to learn. Because participants tend to want to please you in a study, if you’re making them do something that really requires thought and concentration in a way that they normally wouldn’t do it, you’re now taking away from their natural reactions and things like that.
I think that’s where the problem comes from. Making it feel natural lets you focus on the things that are really what you came to focus on, and not can someone keep their posture in an awkward position for 20 minutes.
Cyd: Right. I think, if you got out your phone right now and set it on your desk and tried to open your email and read a couple, using only your pointer finger, it wouldn’t be an accurate representation of how quickly you’re able to use it.
Jared: No. I’m at half-speed. That’s crazy.
Cyd: [laughs]
Jared: It makes perfect sense to me that we’d want to do it this way.
Cyd: We’ll definitely demonstrate that, and there are a lot of variations of that that you could do, of course.
Jared: I’m really excited to see the workshop this year and to see all the new stuff that you’re going to be adding into it and how people are going to come out with this rich sense of all the different tools they can use to get there. Also, the Code for America stuff is so cool. Every time I hear about a new project that you guys are doing, I’m just blown away. I’m anxious to hear more about how you guys are doing that and particularly how you’re using all this mobile stuff, because I bet you are not letting your fellows get away without doing research.
Cyd: [laughs] With me on-site, they don’t have a lot of excuses. We had to actually do a voice UX project as well, which was another way to accommodate people using feature phones. It’s fascinating. I love my job, and I love the evolution that’s happening in mobile. I’m really excited, too, to find out what people have been up to as they’re trying to tape together mobile research on their own.
Jared: This is great. Thank you for spending this time explaining all this to us and helping me to understand what is going on in this world of mobile research and how it’s the wild west and we can use it to our advantage right now.
Cyd: It’s my pleasure. It’s so much fun, and I am always delighted to talk to you.
Jared: Fabulous, fabulous. I want to thank our audience for spending the time listening to us, and of course, thank you once again for encouraging our behavior.
If you’re listening to this on iTunes, it would be awesome if you went and gave us a rating. Whatever you think, just put it in there, because that’s how people discover this. If you think it’s worth other people discovering it, or if you think it’s worth warning people away from it, that’s what the ratings are for, so that would be awesome if you could take a moment and do that.
You can catch Cyd at the UX Immersion Conference, April 7 through 9, doing her full-day workshop, “Conducting Usability Research for Mobile Apps.” It’s a fabulous workshop, and you don’t want to miss it. Go to uxim.co and you will find all the workshops that are there, including Cyd’s, which will be on April 7. Again, that’s April 7 through 9, Denver, Colorado. Hope to see you there.
We’ll talk to you again next time on the SpoolCast. Thank you so much. Take care.