Chapter 3. Faster User Research
Learn how to get research results faster without sacrificing quality.
Find out when it’s safe to use remote or unmoderated testing.
Understand the right way to run surveys.
Feel superior to people who refuse to do research for several awful reasons.
Now you know some ways to do the right kinds of user research at the right time for your company. That will save you a huge amount of time right there, because you won’t be wasting time doing the wrong type of research.
But can we make it even faster? I think we can.
Regardless of the type of user research you’re doing—from observational studies to five-second landing-page tests—you can make your research far more efficient with some simple rules.
Iterate! Iterate! Iterate!
I used to run a lot of usability tests for clients. One time I was asked to participate in a particular test. “OK,” I said, “so we’ll be recruiting six to eight test participants, right? That way, if we get a couple of no-shows, we’ll still have plenty of people to get good data.”
That’s when they surprised me. The client wanted a “statistically significant” test. They wanted to talk to at least 35 people.
Now, let me be perfectly clear. I was a contractor. I was getting paid by the hour, and this client wanted me to sit through 35 hours of testing, rather than the five or six I would have recommended.
I begged not to do it. It was, I explained, an enormous waste of the client’s money. We did it anyway. It was an enormous waste of the client’s money.
Honestly, this is a composite story of many times when clients have asked me to do many, many sessions of testing in a row. Inevitably, what happens is the following:
We do a few tests.
A few really obvious problems crop up.
In every subsequent session, we learn the exact same things about the obvious problems and have a very hard time getting any other information because the big problems are sucking all the focus away from any smaller problems we might have found.
Imagine if you are testing your product. You have 10 people scheduled to come in, one right after the other. You realize in your first test that nobody can log in to your product because of a major UX failure. How useful are those other nine tests going to be? What could you possibly learn from them—that all 10 people can’t log into your product? Couldn’t you have learned that from the first person?
Here’s the key: Patterns start to emerge in usability research after the first few tests. After five, you’re really just hearing all the same stuff over and over again.
But here’s another important tip: Once you remove those major problems that you find in your first few tests, you can start to find all the other problems with your product that were blocked by the original big problems.
So, for maximum efficiency in any type of user research, you want to find the minimum number of people you can interview before you start to see a pattern. Then you want to interview that many people over and over, leaving plenty of time between sets to make changes to your product, mockup, prototype, discussion guide, or whatever else you’re testing.
There is one more important note! There are a couple of types of research where you might want to have larger numbers of participants in each round of iteration (although, you never want as many as 35...just never). For example, things like five-second tests can take 10 or 15 people before you start to see patterns. This is fine, since they’re incredibly cheap and fast to run.
How You Can Do It Right Now
When you set up your next research plan, whether it’s usability testing of a prototype or customer validation on a particular type of user, recruit a small number of participants in a fairly short amount of time—no more than a couple of days.
Run those sessions, and then stop and analyze your information. Look for patterns or problems. If you’re doing usability testing, try making some changes to your prototypes to fix the obvious usability flaws you have discovered. If you’re doing five-second testing, change your messaging or images to address any confusion you’re causing users. If you’re doing customer validation, think of some new types of questions you want answered based on the input you’ve received so far.
Then do it again.
In strict usability testing, you’re going to keep repeating this pattern until your test participants can get through all the tasks with relative ease and minimal confusion. For customer validation, you keep doing this until you think you’ve identified a serious problem for a specific market that you think you can solve. In landing-page tests, you keep going until people look at your landing page and actually understand what your product does.
Whatever type of research you’re doing, keeping it small and then iterating will always give you the best return in the least amount of time.
Stay in the Building
Getting out of the building doesn’t always require actually getting out of the building. In fact, sometimes staying in the building can be far more efficient. No, I haven’t suddenly lost my mind. This makes sense.
While visiting users’ homes or offices can yield wonderful information, many times remote research can give you what you need in far less time, and at a far lower cost. You just need to determine whether the type of research you’re doing really requires being in the same room with the subject.
For example, many types of prototype usability testing can be done remotely with screensharing tools like GoToMeeting, Skype, or Join.me (or a dozen others). Often customer development interviews can be done over the phone.
The only types of research that absolutely have to be done in person are the ones where you need to understand the environment in which a user will be accessing your product or when a test subject needs to be in the same room as the product—for example, many mobile applications or products that are meant to be used onsite, like in a doctor’s office or a factory.
Besides the cost and time savings, remote research has the added benefit of allowing you to get feedback from users all over the world. This is a huge advantage if you have global users.
How You Can Do It Right Now
Instead of scheduling a test participant to come to your office or making an appointment to go to her, send her a link to a screensharing session and give her a call.
Just make sure when you’re testing a prototype that it’s accessible to the other user, either through the screenshare or on a server somewhere. You need to have a setup that allows the test participant to manipulate the prototype. This isn’t a demo; it’s a test.
If you have a mobile app that can be downloaded, you can ask the subject to turn on her webcam so that you can watch her use the product. It’s not perfect, by any means, but it will allow you to test with people who aren’t in your immediate vicinity.
Of course, anytime you use extra technology, you should make sure you’ve run through the process once or twice before trying it with a real test participant. If you’re using something like GoToMeeting or WebEx, make sure you’ve started a session before and learned all the controls, so you don’t spend the first half of the test troubleshooting your testing tools.
Unmoderated Testing
The last few years have seen the creation of a whole host of new products designed to help you get feedback without ever having to interact with a user. Used correctly, unmoderated testing tools can help you get certain types of fantastic feedback very quickly and cheaply. Used incorrectly, they can waste your time and money.
Unmoderated testing is a way to automatically get a video of a real human using your product and attempting to perform various tasks that you assign. These are obviously aimed at web-based products, although at least one company has made it possible to test mobile apps this way, as well.
You just need to provide a link to your product, a few tasks for the user to perform, and a credit card. After a few hours, you will receive video screen captures of real people trying to perform those tasks while narrating what they’re trying to do.
Because there is no recruiting, scheduling, or moderating on your part, you can get usability feedback within hours rather than days.
Before you run out and start using unmoderated tests, let’s review what they are fantastic for:
Finding out if your product is easy enough to use that a person who has never seen your product before can come in and immediately perform an assigned task.
Now let’s review what they are terrible for:
Finding out if people will like your product.
Finding out if people will use your product.
Finding out if people, when left to their own devices and not given any instructions, can figure out what task they’re supposed to perform while using your product.
Finding out how real users of your product are using your product on a daily basis.
Finding out how to fix the usability problems you uncover.
Everything else.
Here’s an example of a time when I used UserTesting.com, one of the many testing services, to uncover and correct a serious design problem.
I was designing a flow to allow a user to sell something online. It’s the kind of thing that doesn’t seem like it would be tough to get right until you actually try selling something online and realize how horrible most marketplace sites are.
I designed the flow and tested it in prototypes, so I was fairly confident that it would go well. As soon as it was live on the site, we ordered up three UserTesting.com users and asked them to go to the site and try to list something for sale.
Interestingly, the flow for selling something went really well, just like in the prototype tests. The problem that we saw immediately, though, was that users were taking far too long to find where to start the selling flow in the first place.
Luckily, they were consistent about failing to find the feature. They all went to the same (wrong) place. Of course, what this meant was that they weren’t the ones who were wrong. We were!
We quickly made a change to allow users to start the selling process in the more obvious (to the users) place, ran three more users through unmoderated tests, and came up with a clean bill of health on the feature.
Of all the things that we did on that product, the one piece of feedback we consistently got was how incredibly easy it was to list things for sale. That almost certainly wouldn’t have been the case if users had never been able to find the feature in the first place.
How You Can Do It Right Now
First, pick the right sort of thing to test. Ideally, you want a few simple tasks that a new user might perform on a product that is available on the Web.
Your goal for this test is to understand whether somebody new to your product can quickly figure out how to accomplish a task. Remember, this is straight usability testing. It’s not going to tell you anything about whether anybody is going to like or use your product.
Also, if your product requires a specific type of user, this may not be ideal for you. You can use one of the companies, like UserTesting.com or OpenHallway, that allows you to recruit your own users, but doing that removes one of the benefits of this type of testing, which is that it doesn’t require you to do any recruiting.
Once you’ve got the right sort of tasks, find one of the dozens of blog posts that compare the different unmoderated usability testing options out there. I’ve used UserTesting.com, but there are lots of options, like Loop11 and TryMyUI, and they all have pros and cons. If you want to do this type of testing on mobile, there are fewer options, but keep checking. More companies are being started every day, but books don’t get updated often enough for me to give you a complete list.
Now go through the hopefully well-tested process for starting a test.
You’ll most likely be notified that the test is complete within an hour or two. The site should provide you with some sort of video that you can watch. So, you know, watch it. Even better, watch some of the videos with your whole team so that they can all experience firsthand exactly the problems that your users are having.
Then fix those problems and do it all over again until you can stop cringing at the pain your users are feeling.
When to Survey
Frequently when I ask entrepreneurs if they’re in touch with their users, they say something along the lines of: “Oh, very in touch. We do surveys all the time.” Then I count to 10 and try not to imagine murdering them.
Surveys do not count as being “in touch with your users.” They don’t. Let me explain why. In the vast majority of surveys, you are the one setting the answers. In other words, if you ask somebody “What’s your favorite color? Red, blue, or yellow?” you’re going to miss out on everybody whose favorite color is orange. You’re not even going to know that orange exists as an option.
“Aha!” the surveyors say, “We can combat that by just giving people an ‘other’ option!” Sure, except that you’re wildly underestimating how badly you’re biasing people’s answers by first presenting them with some standard answers. This is especially true if you’re asking something like “What do you hate about the site?” If you present them with a bunch of things they’re likely to hate, most people will simply choose a preselected answer.
“Fine,” the surveyors go on, irrationally undaunted, “but what if we just allow them to type in their answer in a text box instead of giving them preset answers?” You know what people hate doing? Writing. Look, if I see a long survey with nothing but a bunch of open text boxes, I will simply leave, and I am the kind of person who writes a book! Stop expecting your users to do all your work for you by typing lots of long answers into a survey. There are things they’ll be thrilled to tell you on the phone that they would never type into a web form.
Let’s be perfectly clear. As with so many other tools, surveys can be extremely useful, but they’re not a replacement for listening to customers talk about their needs or watching people use your product.
They are, however, a great way to quickly follow up on patterns you spot in the qualitative research you should be doing.
Here’s an example. A colleague and I were doing some preliminary research on the attitudes and behaviors of female angel investors. We wanted to learn more about their motivations and see if those motivations differed at all from those of male angel investors. To that end, we interviewed several female angels, male angels, and some wealthy women who could conceivably have made an angel investment but hadn’t.
But, of course, we didn’t interview all female angels. We didn’t even interview a statistically significant number of them. You see, this sort of qualitative research isn’t a statistically significant science experiment. It often doesn’t have to be. All it has to do is present you with some likely hypotheses you can later test in a more thorough manner.
As with the vast majority of this sort of research, we started seeing very early patterns. After speaking with around five people in each group, we had some interesting hypotheses. Based on this, we decided to run a survey to see if the patterns held true over a larger group or if we had somehow found a very biased group of test participants to interview.
We ran a survey asking a few simple follow-up questions about things like gender and whether they had ever been asked to make an angel investment. We asked some specific, factual questions that the participant could answer easily and a few slightly more complicated questions that asked people about their attitudes toward angel investing.
Most importantly, the goal of the survey was not to generate new hypotheses about why women did or did not make angel investments. The goal was to validate or invalidate the hypotheses that we had formed during our initial research.
Surveys are great at reaching a large number of people very quickly, so they are fantastic for following up on ideas and patterns that you spot initially in qualitative research. However, because of the structure of the questions and answers, they are often terrible for helping you to spot patterns or form hypotheses about important topics like how users feel or what new features users would like to see.
How You Can Do It Right Now
First you need to figure out what question you want answered. Is it something like, “What is the most confusing part of my product?” Do you want to know which feature to build next? Do you want to learn more about what other products your customers use on a daily basis?
Once you’ve determined your question, recruit around five of the right sort of person to answer that question. If it’s a question about power users, recruit power users. If it’s a question about your product’s first-time user experience, recruit people who are in your persona group but who have never seen your product before.
Then do your research. There are lots of books on how to run basic user ethnography or usability testing. This is not one of them, but the basic gist is to interview them about the questions you have. Watch them use the product. Talk to them about their likes and dislikes. Basically, ask a lot of open-ended questions and do a lot of observing.
Once you’ve noticed patterns or come up with some very specific questions that you want answered by a larger group of people, you turn those questions into a survey.
Don’t forget to include some screening questions to make sure you’re getting answers from the right sorts of people. For example, if you only care about how women feel about something, you should probably ask for the participant’s gender as one of your questions.
By using qualitative research to generate your hypotheses and realizing that surveys are only good at validating or invalidating those hypotheses, you can turn surveys into an incredibly powerful tool. Just don’t try to use them to come up with new ideas. They’re the wrong tool for that.
Loosely Related Rant: Stupid Reasons for Not Doing Research
Almost every company I talk to wants to test its products, get customer feedback, and iterate based on real user metrics, but all too often they have some excuse for why they just never get around to it. Despite people’s best intentions, products constantly get released with little to no customer feedback until it’s too late.
Whether you’re doing formal usability testing, contextual inquiries, surveys, A/B testing, or just calling up users to chat, you should be staying in contact with customers and potential customers throughout the entire design and development process.
To help get you to stop avoiding it, I’ve explored six of the most common stupid excuses for not testing your designs and getting feedback early.
Excuse 1: It’s a Design Standard
You can’t test every little change you make, right? Can’t you sometimes just rely on good design practices and standards? Maybe you moved a button or changed some text. But the problem is, sometimes design standards can get in the way of accomplishing your business goals.
For example, I read a fascinating blog post by a developer who had A/B tested the text on a link. One option read, “I’m now on Twitter.” The second read, “Follow me on Twitter.” The third read, “Click here to follow me on Twitter.”
Now, anybody familiar with “good design practices” will tell you that you should never, ever use the words “click here” to get somebody to click here. It’s so Web 1.0. But guess which link converted best in the A/B test? That’s right. “Click here” generated significantly more Twitter followers than the other two. If that was the business goal, the bad design principle won hands down.
Does this mean that you have to do a full-scale usability and A/B test every time you change link text? Of course not. Does it mean you have to use the dreaded words “click here” in all your links? Nope. What it does mean is that you should have some way to keep an eye on the metrics you care about for your site, and you should be testing how your design changes affect customer behavior, even when your changes adhere to all the best practices of good design. So to put it simply: Prioritize what you care about and then make sure you test your top priorities.
Excuse 2: Company X Does It This Way
I can’t tell you how many times I’ve heard, “Oh, we know that will work. Google/Facebook/Apple does it that way.” This is the worst kind of cargo cult mentality.
While it’s true that Google, Facebook, and Apple are all very successful companies, you aren’t solving exactly the same problem that those companies are, you don’t have exactly the same customers that they do, and you don’t know if they have tested their designs or even care about design in that particular area.
You are, hopefully, building an entirely different product, even if it may have some of the same features or a similar set of users.
Is it OK to get design ideas from successful companies? Of course it is. But you still need to make sure your solutions work for your customers.
I previously worked with a company that had a social networking product. Before I joined them, the company decided that, since other companies had had good luck with showing friend updates, they would implement a similar feature, alerting users when their friends updated their profiles or bought products.
Unfortunately, the company’s users weren’t very interested in the updates feature as it was implemented. When we finally asked them why they weren’t using the feature, the users told us that they would have been very interested in receiving an entirely different type of update. This was later backed up by metrics when we released the new kind of update. Of course, if the company had connected with users earlier in the process, it would have rolled the feature out with the right information and gotten a much more positive reaction on launch.
Another thing to remember is that just because a company is successful and has a particular feature doesn’t mean it’s that exact feature that makes it successful. Google has admitted that the “I’m Feeling Lucky” button loses it page views, but it keeps it because the company, and its customers, like the feature.
That doesn’t mean it’s a good business plan for your budding search engine startup to adopt a strategy of providing people with the equivalent of the “I’m Feeling Lucky” button. In fact, this is a great example of why you might need to employ multiple testing methods: qualitative testing (usability, contextual inquiry, surveys) to find out if users find the feature compelling, and quantitative testing (A/B, analytics) to make sure the feature doesn’t bankrupt you.
The bottom line is it doesn’t matter if something works for another company. If it’s a core interaction that might affect your business or customer behavior, you need to test it with your customers to make sure the design works for you.
Obviously, you also need to make sure that you’re not violating anybody’s IP, but that’s another book.
Excuse 3: We Don’t Have Time or Money
As I have pointed out before, you don’t have time not to test. As your development cycle gets farther along, major changes get more and more expensive to implement.
If you’re in an Agile development environment, you can make updates based on user feedback quickly after a release, but in a more traditional environment, it can be a long time before you can correct a major mistake, and that spells slippage, higher costs, and angry development teams.
I know you have a deadline. I know it’s probably slipped already. It’s still a bad excuse for not getting customer feedback during the development process. You’re just costing yourself time later.
Excuse 4: We’re New; We’ll Fix It Later
I hear this a lot from startups, especially Agile ones, that are rushing to get something shipped, and it’s related to the previous excuse. Believe me, I do understand the pressures of startups. I know that if you don’t ship something you could be out of business in a few months. Besides, look at how terrible some really popular sites looked when they first started! You have to cut something, right?
Great. Cut something else. Cut features or visual polish. Trust me, people will forgive ugly faster than they’ll forgive unusable. Whatever you decide to cut, don’t cut getting customer feedback during your development process. If you ship something that customers can’t use, you can go out of business almost as fast as if you hadn’t shipped anything at all.
Potential users have a lot of options for products these days. If they don’t understand very quickly all the wonderful things your product can do for them, they’re going to move on. Take a few hours to show your ideas to users informally, and you will save your future self many hours of rework.
Excuse 5: It’s My Vision; Users Will Just Screw It Up
The fact is, understanding what your users like and don’t like about your product doesn’t mean giving up on your vision. You don’t have to make every single change suggested by your users. You don’t have to sacrifice a coherent design to the whims of a vocal individual.
What you should do is connect with your users or potential users in various different ways—user tests, contextual inquiry, metrics gathering, etc.—to understand whether your product is solving the problem you think it is for the people you think are your customers. And, if it’s not, it’s a good idea to try to understand why that is and develop some ideas for how to fix it.
Besides, how many people do you think spent months creating their perfect vision, then shipped it and realized that nobody else was seeing the same thing they were?
Excuse 6: It’s Just a Prototype to Get Funding
This is an interesting one, since I think it’s a fundamental misunderstanding of the entire concept of customer research. When you’re building a prototype or proof of concept, you still need to talk to your customers. The thing is, you may have an entirely different set of customers than you thought you did.
Maybe you think the target market for your new networked WiFi lunchbox is 11- to 13-year-old girls, but they’re not going to pay you to build the first million units and get them into stores. Your first customers are the venture capitalists or the decision makers at your company or whoever is going to look at your product and decide whether or not to give you money.
Even if they’re not your eventual target market, it’s probably a good idea to spend some time talking with whomever you’re trying to get to fork over the cash. I’m not saying you should change your entire product concept based on this feedback. I mean, if you really want to start the company on your credit cards and a loan from your mom, don’t change a thing! The important takeaway here is that you may have different audiences at different points in your company’s life. And the best way to find out what they all want is to talk to them!
Out of Excuses?
Those are the most common excuses I hear, but I’m sure you can think of some clever ones. Then again, your time is probably better spent connecting with your users, understanding their problems, and thinking of ways to address them.
Go Do This Now!
Learn from a distance: Try running a remote or unmoderated test and compare your results with in-person research.
Run a (good) survey: Try creating a survey based on hypotheses generated by qualitative research.
Confront your own reasons for not doing research: Try to think of all the excuses you have for not talking to customers this week. Now do the research anyway.