The Wrong Mindset

As we learned in the introduction, product research doesn’t have to be complicated. It can be quick, cheap, collaborative and simple, but you do need the right approach. Much of this is about how you view product research in the first place. One of the reasons research projects fail is that the researchers are starting with the wrong mindset, being led by their own agenda instead of staying open to insights. The main factor in this is ego.

Ego can be the enemy of good product research. We think we know the right product to build because, based on our experiences and knowledge, we know the exact product, feature, or service our customers need. Don’t we?

Some years ago the online marketing company Constant Contact experienced an increase in call volume from its customers (mostly small businesses). More and more callers wanted answers to their marketing questions, and data showed an increase in visits to its support pages and forums from mobile devices. This led the VP of Customer Success to believe that Constant Contact customers weren’t getting answers to their marketing questions.

An executive had an idea to package all the company’s content into a mobile app called Marketing Smarts. The idea had backing within the company (partly due to the political ramifications of going against an executive) and it landed on the innovation team’s desk. But rather than jump in and spend more than $200,000 dollars to build the app, the team decided to run a concept test with customers and watch their reactions. What they found was that customers tended to be in front of a desktop when they had a question, so they didn’t need a mobile app at all. The prototype the innovation team built in a few hours quickly invalidated the need for the product and highlighted the real problem: that the customer support content and forums were disorganized and difficult to navigate, causing customers to pick up the phone. Constant Contact subsequently improved and relaunched its Help Center.

Constant Contact’s project started because someone thought they’d had a great idea. Their ego said that this was the solution to the current issue, and the rest of the company wanted to prove them right. It was only when confronted with the actual behavior of the customer that the real problem—and the solution—were found.

In the mid-1970s researchers at Stanford ran a study asking two groups of students to correctly identify real suicide notes from fake ones.¹ The first group only identified 10 out of 25 notes correctly. Remarkably, the second group correctly guessed the authenticity of 24 out of 25 notes. The study was of course a set-up - the scores the students were given were entirely fake. But then the researchers did something else: they came clean to the students and told them their scores had been made up. They then asked them how they thought they’d really performed in the task. The group that had been told they’d achieved 24 out of 25 said they thought they’d actually done very well and reported above-average scores, while the group that had been given a low score reported that they’d probably performed just as badly in reality. Despite being told that their scores were fake, the groups still conformed to their existing beliefs. Knowing the facts didn’t change their minds - it reinforced what they already thought.

What does this mean for product research? It means that if we find ourselves to be “right”even once, we’re going to think we’ll be “right” again in the future. Challenging this egocentric mindset is a key part of doing product research well.

So what if we switched things around? What if, instead of expecting to be right, we expected to be wrong? What if we sought to be wrong? Each time we disprove a theory or belief, we in fact discover more validation that we can be right, because we know the process we’re undertaking is real.

The challenge here is that while you’re reading this book, your boss, CEO or other executive in your organization might not. The whole team needs to adopt the right mindset for product research. We’ll address this later when we talk about how to share insights, but for now it’s enough to start challenging your own limiting beliefs.

Below we’re going to describe two particular mindsets that can cause product research to fail. But before we do, there’s one more thing to remember: You need to care. You can pay people to do lots of things they don’t want to do, but it’s really hard to pay someone to care. Product research requires the heart as well as the head, and as much as we all think everyone is rational and cerebral, people are not. People are emotional and messy. Having a mindset that’s open to that mess and how it may differ wildly from our initial beliefs will set you up for product research success. You’ll never discover something new unless you’re prepared to be wrong.

Transactional and Confirmatory Mindsets

C Toddm in his role as VP of Product at MachineMetrics, a data platform for manufacturers, visited a customer some time ago to resolve some specific issues they were having with the product. As he started to discuss a particular feature, he found himself asking,“Wouldn’t it be cool if you had…?” The details of the feature in this story don’t matter. What matters is that his question was all about confirming his initial idea. By prefacing it with “Wouldn’t it be cool…?” he was turning an opportunity to gather genuine insights into a leading question that would only confirm or deny his own agenda.

There are two approaches that push you into that confirmatory space: transactional and confirmatory approaches. Let’s see how these affect our mindset in product research.

A transactional approach is exactly what it seems. When you’re a hammer, everything looks like a nail. This approach is all about the transaction and limits itself to whether or not a customer would buy a product. You might find yourself asking: “Would you buy…?” A transactional approach doesn’t take into account the nuances of a customer journey² or the complex needs of the user. It explores the topic at a surface level, avoiding the depths at which the researcher may find evidence that they are wrong. This approach is often seen in marketing research that solely focuses on sales performance, and it is not useful.

A confirmatory approach is where we try to get the answers we want: If you beat up the data, it will tell you what you want to hear. This problem with this approach is that it’s about confirming the biases you already have rather than listening to what the customer has to say. You might find yourself asking leading questions like: “Wouldn’t it be cool if you had…?” or “What if you had…?” If this sounds more like a sales pitch than research to you, you are correct.

Neither of these approaches is useful in creating insights that would teach us something new about our customers and our products. The only thing that they create is a false comfort zone that make us feel better about ourselves.

The Problem-Finding Mindset

When people hear the term user research, many of them immediately think of usability tests, with the emphasis on the word test. They’ll treat a usability test like a driving test: There are right and wrong answers and you either pass or fail. The tester sees themself as a pro who knows how to drive and is knowledgeable enough to assess other people’s driving skills. They focus on a rubric of predefined criteria and what deviates from these standards. The goal is to see if someone is good enough to pass the test.

Similarly, many people think of interviewing as interrogation. They assume that the users are hiding a problem, which they are there to extract from the user. They think the users will give them a great insight if they keep hammering them with why, why, why, why! They are there to find out what the problem is, and they do not care about the experience of users as a whole.

What does this problem-finding mindset do to product research? It focuses the study on problems, and a product that has too many problems to pass the “test” might be abandoned along with its positive qualities. Conversely, a product that shows no problems and passes might be adopted despite being a poor market fit.

The Insight-Making Mindset

Good usability testing, like product research in general, is the opposite. Unlike the confirmatory mindset and the problem-finding mindset, it focuses on the positive and negative aspects of user experience. It does this with an open mind, trying its best to withhold judgment, and focuses on the research question at hand. It’s interested in generating insights, not confirming opinions.

This insight-making mindset allows you to keep a cool head when you discover serious issues with your product, which is especially challenging when those issues arise from features we built with our own hands. If you don’t focus all your attention on the product’s problems, you can see its strengths, so you know what to retain as you develop it further.

The insight-making mindset adopts a diagnostic approach, which is about getting to the real problem. It’s about trying to understand the needs of the customer with as little bias as possible and asking open questions such as: “Tell me about the last time you...” or “What happens when…?” Diagnostic questions are based on past events and decisions, not a hypothetical future.

A friend who runs a startup accelerator remarked that of all the startups he sees, the most successful take the diagnostic approach to understanding their customers. In our early years of product research, we admit that we often took the confirmatory approach. It took time and experience to realize that all we were doing was looking to be right - and that this was making our products more and more wrong.

Focusing on insights with an open mind not only gives you a better return on the time invested, it causes less finger pointing when the results are out.

Starting with an insight-making mindset and checking your ego is crucial to making product research efficient and actionable. That frame of mind will carry through to every aspect of your project.

The Wrong Questions

Another reason product research fails is that the team isn’t asking the right questions. Teams that are new to product research often fall into this trap. In their excitement to find out about how their product is being received, they do “a general check” with their users, posing questions about everything. The result is lots of information, but rarely any genuine insights. To conduct valuable product research, it’s essential to narrow your objectives to a single, crisp research question. By asking too much, you risk gathering data that can’t possibly be distilled into actionable results in a reasonable time.

You could also be asking questions that are just too vague. A popular research method is Net Promoter Score, or NPS. This is where customers are asked to rate how likely they are to recommend the product or service to a friend. To organizations, NPS surveys are pretty seductive - they give you a single number that tells you everything you need to know about your customer’s loyalty to your product. They’ve even been called “the most important customer metric”³. But do they actually tell us what customers think? We’ve learned the hard way. In evaluating the success of one product, C Todd’s NPS score was 90. That would suggest that the product was fabulous; unfortunately, growth data suggested otherwise. NPS scores are at best worthless. At worst, they can be dangerously misleading.

The other problem with questions like NPS surveys that ask users about what they’re likely to do is that we’re terrible at predicting our own future. A customer might say they’re likely to recommend a product, but will they actually do this? In context, there are many reasons they might or might not. For example, they might recommend a product because a friend works for the company, or because they were asked. They might say they’d recommend the product because they’ve had an incentive to respond, such a voucher or competition prize. That doesn’t indicate loyalty to the product; it’s just the way humans behave. Good research questions focus on why someone would give a particular score, not on the score itself. As convenient as it would be if we could represent customer experience with a number, there’s no single number that will do that job.

Starting your research with a clear question makes it easier to see what matters. A question that is well formulated for the purpose of the project helps everyone focus on the common problem, and no one gets distracted by the surrounding context. Of course, that doesn’t prevent you from capturing other issues you might observe as you go along, but it does help you to isolate them for subsequent rounds. This leaves you free to learn about the issue you set out to address.

So how do you know what to ask? How can you distill everything you want to know into a single research question? We say that before you can focus on what you want to know, you need to focus on what you know already.

Let’s say your bicycle won’t work properly. You notice when riding that it becomes difficult to slow down and stop. Here are the symptoms, questions and answers that might arise:

Symptom: Bike won’t slow down.

Question: What usually stops the bike?

Answer: The brakes.

A more specific question: What’s wrong with the brakes?

This more specific question, informed by things we already know (that brakes usually stop the bike) can be reframed as a good research question:

Practical problem: My bicycle is broken; the brakes don’t work.

Research problem: How can I find a way to fix it?

Research question: How might a mechanic fix bicycle brakes?

We narrow down the information based on what we already know and create a single question that will lead to real insights. Instead of asking about everything, we focus on what we really need to know.

But how do we know that our question is the right one? Just as important as focusing on a single question in product research is ensuring that we’re asking about the right thing. In this case, how you ask is as important as what you ask.

Suppose I asked you “How many ones add up to two?” The only answer you could give would be two. But what if I asked the question differently? What if I asked “What two numbers add up to two?” By framing the question differently, I’ve now opened it up to a broader set of answers. You could answer 1 + 1, but you could just as easily answer “2+0” or even “3 -1” and mathematically there are an unlimited number of answers.

This is why how you formulate your research question is so important. The answers you get depend on how you frame the question. The question you ask will also guide your research in a particular direction: if you begin with the wrong question, you may be led astray.

The Wrong Users

We’ve seen how forming the right question is key to avoiding long, expensive research projects. But just as important as finding the right question is finding the right people to put it to.

Imagine you’re building an aircraft maintenance system for airport workers. Would you test the prototype with a group of florists? While that could be a fun test to watch, you wouldn’t learn much from it and, aside from some possible GIF-worthy slapstick, you’d be wasting everyone’s time. Yet this is what we do when we ask random people to test a product: We seek information from people whose needs may be totally unrelated to those of our users. So how do you ensure you’re researching with the right users for your product?

Identifying your users is the first step to targeted product research. One team C Todd spoke with a few years ago was developing an app aimed at athletic coaches. When grilled about this, the team revealed that the app was more specifically aimed at coaches who worked for endurance athletes. Without this distinction, it would have been harder for that team to reach the specific users they were targeting. Narrowing your users down to the right niche will ensure you’re working with the people who’ll actually use your product.

In the case above, the app developers knew they wouldn’t find their target users at their local Starbucks, or even by posting an ad on Craigslist. So where did endurance coaches hang out? Was there a secret endurance-coach club they could visit? Unfortunately, no, but there were running and cycling races, as well as niche websites and forums where these coaches spent time. Think outside the box when it comes to finding your test candidates, go where they are, and make sure they’re the right people for your study.

The Wrong Method

Even research projects with a great question and a targeted user base go wrong, and that’s usually down to them employing the wrong method. Working out what type of study you need to do isn’t about identifying “good” and “bad” methods of product research (although there are some bad methods out there, like NPS surveys); it’s about matching the method with the insights you want to discover.

If a sibling or friend has symptoms of the flu, how would you find out how bad they feel? You’d ask them. You wouldn’t use an ear thermometer to measure their body temperature and then extrapolate how they must be feeling from the reading you get. On the other hand, if you suspect they might have a lung infection, it would be more appropriate to run some tests than to simply ask them how they feel.

How we go about conducting research depends on what we want to know. Using an ear thermometer to understand how someone feels is like running a usability test to understand how much future customers would pay for your product. And asking someone to judge if they have a lung infection based on how they feel is like sending out a survey to understand why people abandon their carts online. Different research questions require different methods and, just like in medicine, if you pick the wrong process you might end up missing something vital.

We’ll be showing you exactly how to choose the right type of study for your research project in Chapter 3. In chapters 5 and 6, we’ll show you how to use the right methods to get the most valuable results.

Closed Analysis

In some cases, it’s not the research project itself that’s flawed, but what happens afterward. Not so long ago, product research was done only by expert researchers. They were highly educated (think PhD level), and had been doing research for years. They knew their topic well, they recognized the nuances, and they had the benefit of years of experience. They would work with users in a series of studies, then sit down and - many months later - come up with a small crystal vial of insight.

This approach might have worked in the past with simpler technology products that could stand alone. But the products we use today are part of larger, more complex ecosystems. Analysis undertaken by a small, closed group of people won’t produce results that are actionable and dynamic. The world is changing rapidly, and product research needs to involve as many members of the team as possible. Successful product research projects are collaborative, not closed. The more people involved from the start, the more likely you are to get buy-in when your insights are put into practice.

Inaccessible Sharing

Collaboration also means communication. Done correctly, product research can provide incredible results. But if those results are buried in a long, impenetrable report, what chance has anyone in the team got of applying them to the product? If someone writes a research report and no one reads it, did their efforts produce insights?

One reason product research fails is that the people who could act on the insights can’t access them. In Chapter 8 we’ll show you how to share insights so you get maximum buy-in from the people who matter. We’ll also show why reports aren’t always the best way to do that, and give you an achievable framework for sharing insights that inspire.

Summing up

Product research doesn’t have to be long, expensive, and arduous to yield valuable results. If the approach is right, simpler research projects can yield more meaningful insights in a shorter period. Following a set of simple rules for product research will make it simple and effective, leaving you with valuable insights that hurt less to find.

Key Takeaways:

• The wrong mindset will lead to research being restricted by the researcher’s agenda.The right mindset will leave you open to genuine insights.

• The wrong question will lead your research astray and provide little actionable information. The right question is formed from what you already know.

• Asking the wrong users will not give you the results you need. Finding the right ones will ensure your findings are accurate.

• How you conduct your research is as important as what you ask. Matching the method to what you want to find out will ensure that you get the results you want.

• Restricting the analysis of the data to a small team of researchers will limit your insights. Letting everyone have a say will get more commitment from the people involved.

• The way you share your information matters. Making your research accessible will inspire people to action.