Chapter 15

Traditional Approaches: Why Not Just Ask People?

In This Chapter

Understanding why consumers often don’t know what they think, but believe they do

Asking questions consumers are more likely to answer truthfully and accurately

Identifying the pros and cons of the three workhorses of traditional market research

Appreciating why we need to measure both conscious and nonconscious consumer responses

In this chapter, we review traditional market research approaches, which generally share the same underlying idea: If you want to understand consumers’ attitudes, opinions, preferences, and product choices, just ask them.

Earlier in this book, we show how this self-evident idea is challenged in many ways by recent brain science, and how neuromarketing has risen, at least in part, as an alternative way to deal with the very counterintuitive discovery that human beings actually have very little awareness of why they do the things they do. So, when people are asked what brands they like or what they’ll buy in the future, their answers are often no better than guesses. For the most part, when people make these guesses, they aren’t lying or trying to deceive researchers — they’re literally unaware of the real causes and reasons for their opinions or actions.

In Chapter 2, we also introduce a further complication: People not only make up plausible explanations of their mental states, but they vastly overestimate the accuracy of their own reports. Scientists call this confabulation (the replacement of a gap in our memories with a made-up substitute that we truly believe to be true). Countering confabulation is probably the most important justification for the development and adoption of neuromarketing research techniques.

Given these challenges, you may think that neuromarketers would advise marketers never to ask questions, but this is not the case. People’s responses to researchers’ questions — whether in an interview, a focus group, or a survey — are still incredibly valuable data points. Learning what people think they think, even when you know their answers may be poor predictors of actual future behavior, is extremely useful because it tells you what aspects of their nonconscious mental processes have risen to the level of their conscious thoughts and expressions. The trick is to listen carefully to what people say, but not necessarily to believe them.

Understanding Why Asking Questions Is Risky Business

Market researchers have known for years that people sometimes fail to tell the truth in response to research questions. This is called response bias, because incorrect answers bias (or distort) the results of the research. Several types of response bias have been documented and studied by psychologists, including the following:

Agreeableness bias: People like to be agreeable. If they think they know what answer will please the researcher, they’re more likely to give it.

Social desirability bias: People like to express opinions that they believe are socially acceptable. If they have an opinion they think others don’t share, they’re less likely to reveal it.

Knowledge exposure bias: People don’t like to reveal what they don’t know. If they don’t have an opinion, they may just make one up so they don’t appear ignorant.

Misinformation bias: Sometimes people are just contrary, and give deliberately false answers because they don’t like being questioned. This has become a more frequent problem with people who are deluged by surveys, like voters in primary election states in the United States.

All these sources of bias share the feature that they involve deliberate deception on the part of the person being questioned. In other words, people know they aren’t giving truthful answers in each of these cases. Researchers have developed various techniques to minimize these types of inaccuracy, so they’re all to some degree controllable within the overall question-asking model.

More troublesome are sources of bias that people aren’t aware of. These are the guesses that people make — and believe are true — because they don’t have access to the nonconscious processes that underlie their conscious thinking. There are three major types of bias, and each one impacts an important kind of question that researchers ask all the time:

Memory bias: Researchers regularly ask people about their past experiences with a product or brand. But the answers that people give can be inaccurate because memories are constructed and subject to distortion based on many factors, including immediate needs and desires. This is not to say that memories are always or even regularly wrong, but inducing false memories is very easy (see Chapter 6). The bottom line for researchers is that memories can be reliably accessed, especially if they’re relatively recent in time and emotionally relevant, but they can also be influenced easily by questioning or priming if researchers aren’t careful.

Emotion (and preference) access bias: Researchers are always asking people what they like and don’t like. But our understanding of our emotional states, like our memories, is largely constructed. When asked a relatively simple question like “How much do you like broccoli?”, we don’t have direct access to the answer. Instead, we ask ourselves how much we like broccoli and, like detectives, we start assembling evidence from our experience. This is a highly selective and biased deliberative process, so the answer may not be accurate even though we believe at the moment it is. Questions about preferences suffer from a similar problem. Preferences are much less stable than we think they are (see Chapter 8), so we often construct them when we’re asked to do so.

Prediction bias: Researchers regularly ask people what they’ll do in the future. But predicting the future accesses both the memory and emotion constructive systems, and adds another. We try to imagine what we’ve done in the past, we try to estimate what our feelings will be in the future, and from that we try to make an educated guess as to what we may do in the future. Mountains of research have documented that these predictions are seldom accurate. This shouldn’t be surprising, because all the assumptions on which the predictions are based are faulty. Yet, we tend to have completely unjustified confidence in our predictions.

Given all these obstacles to getting accurate responses from consumers, it’s fair to wonder whether there are ways to ask questions that may minimize some of these obstacles. In fact, there are. Researchers interested in understanding how and when attitudes predict actual behavior — such as whether pro-environmental attitudes lead to behaviors like recycling — have developed guidelines for asking people about attitudes, called the Theory of Planned Behavior. These guidelines are useful for any type of methodology that aims to predict behavior from interviews or survey questions:

Ask about attitudes toward specific behaviors, not attitudes in general. When people are asked to recall how they actually behaved in the past, they’re more likely to accurately infer their attitudes. Instead of asking, “How much do you like broccoli?”, ask, “How often do you cook broccoli?”

Ask about the attitudes of relevant peer groups, not personal attitudes. Asking people what others think is a good way to get them to express their own beliefs without having to feel accountable for them. For example, ask, “How popular do you think broccoli is as a dinner vegetable for moms in your neighborhood?”

Ask about control factors that can help or impede behaviors. These questions provide a way to evaluate situational factors that may contribute to behavior. For example, ask, “How expensive is broccoli compared to other vegetables at the supermarket?” or “How many different ways can you cook broccoli?” A person may love broccoli, but if she thinks it’s too expensive compared to other alternatives, she’s less likely to buy it.

Some additional guidelines that can increase the honesty and accuracy of consumer responses to research questions have to do with when and where the questioning occurs:

Ask questions as close as possible to when and where people are actually engaged in the behavior you’re interested in. For example, ask your broccoli question in the produce aisle at the grocery store.

Ask questions when people are in the right mind-set and context. For example, ask about food preferences when the consumer is in a grocery store.

Ask questions to confirm or clarify what’s observed in behavior. For example, in a grocery store interview, ask, “I notice you have broccoli in your shopping cart. How often do you buy broccoli?”

And that’s the last time we talk about broccoli in this book. We promise.

Introducing the Three Workhorses of Market Research

For decades, market research has rested on three main data collection methods: consumer interviews, focus groups, and surveys. Each of these methods has undergone numerous changes and improvements over the years, but the basic idea they all share remains the same: If you want to know what people think, you ask them. The differences are in how you ask:

In-depth interviews: Good interviews tend to be relatively unstructured. The skillful interviewer follows the interviewee’s lead, identifying interesting areas to explore and new or unexpected directions in real time. Interviews often go deeper than other techniques. For example, one common technique is called “Five Why’s.” The interviewer doesn’t let a question go until he has asked “Why?” to an answer, and then has asked “Why?” to that answer, and so on for five answers. Sometimes interviewers simultaneously take notes, but more often the interview is recorded or videotaped and studied in detail later on.

Focus groups: Possibly the most well-known and maligned research technique, focus groups bring together a small number of consumers (usually six to ten) to discuss a marketing issue, concept, or idea, managed by a moderator. The exercise is usually performed in front of a large one-way mirror, behind which marketers observe the proceedings. The usefulness of focus groups is very dependent on the skills of the moderator, who must balance complex group dynamics to get a good reading of the views of the group as a whole.

Consumer surveys: The most scientific of the three workhorses, surveys consist of structured questionnaires that are administered to a representative sample of consumers, with statistical analyses exploring key issues. Surveys used to be conducted door-to-door, then by telephone, but today they’re conducted overwhelmingly online (although telephone surveys still play a large role, especially when responses need to be turned around quickly).

Until the rise of neuromarketing, these three approaches were the primary means by which consumer responses to products and marketing were collected. They still make up about 90 percent of the research that’s performed in the market research industry today.

In this section, we look at each of these methods in turn and describe how and when they’re most useful in today’s mix of available methodologies. We show that, in certain circumstances and for certain purposes, they continue to have an important role to play.

Conducting in-depth interviews

As the name suggests, in-depth interviews try to get below the surface of rationalizations, attempting to explore the drivers behind consumers’ attitudes, opinions, decisions, and actions. Assuming the interviewer is skilled and experienced in this type of interviewing and that the respondents (interviewees) are at least roughly representative of a target group of interest, we can see several uses for this approach, but also some risks and limitations.

When in-depth interviews make sense

In-depth interviews offer the luxury of studying a relatively small number of consumers at length, providing an intimacy of findings and insights that standardized interviews and surveys can’t match. However, this depth is gained at a cost. The sample size is usually quite small, and the lack of standardization across interviews makes it difficult to generalize with any statistical precision from the results. But often the insights obtained provide useful inputs into later marketing and product development initiatives, so these drawbacks aren’t seen as a serious hindrance.

In-depth interviews aren’t recommended for testing existing hypotheses, but they’re great for early-stage studies — for generating hypotheses and discovering and exploring new ideas. They provide an excellent way for marketers and product developers to learn from their consumers, enabling a rich exchange of ideas between companies and a subset of their customers.

Sometimes in-depth interviews are accompanied by metaphor elicitation techniques that probe for deeper connections and motivations, such as having consumers recount childhood experiences or asking them to assemble a collage of images representing how they feel about a product or brand. These methods are believed to uncover the deeper motivations that drive consumer attachments to brands and products, making them complementary to neuromarketing methods that test for nonconscious goals and motivations in consumer choice and behavior.

Risks and limitations of in-depth interviews

In-depth interviews are time consuming and expensive. The quality of the findings depends heavily on the expertise of the interviewer. After the interviews are completed, a significant amount of time is often required to analyze and interpret the results, so this approach isn’t suited to getting quick feedback from the market.

The biggest potential risk of in-depth interviews is that findings that have been discovered through interviews may be misidentified as findings that have been confirmed. Unfortunately, marketing professionals are just as susceptible to judgment biases as consumers are. It doesn’t take long for a newly discovered insight to begin to appear inevitable, thanks to the distorting effects of hindsight and accessibility biases. But in-depth interviews actually have relatively poor test-retest reliability, which is a fancy way of saying that if you interview a different group of people, you’ll probably come up with a different set of results (which will in retrospect also begin to appear inevitable). So, falling in love with your insight is a temptation to be resisted.

Use interviews to generate hypotheses, but use surveys or neuromarketing techniques to test the generalizability of those hypotheses.

Seeking the wisdom of focus groups

Focus groups typically bring together six to ten consumers, representing a particular target group, to discuss a particular issue or to comment on some concepts, ideas, or items. It’s a free-flowing discussion, managed by a moderator, who explains the rules, tries to keep dominant group members under control, encourages shy participants to contribute, and gets the discussion back on track when it deviates from the topic.

For years, commissioning focus groups was the standard first response by many marketers wanting to acquire consumer input about a marketing challenge, or simply wanting to confirm that they were on the right track. More recently, some large companies have begun to sour on focus groups, even going so far as to ban them from their research methodologies because focus groups are so poor at predicting actual marketplace behavior.

In retrospect, this result shouldn’t have been surprising. Focus groups, like in-depth interviews, are a qualitative methodology — that is, they capture how a particular group of people are reacting and evaluating right now, but they consist of too small a sample to generalize with confidence to the attitudes or behaviors of a larger population (such as all consumers who may buy a product). So, like in-depth interviews, focus groups are good for identifying hypotheses and issues that need to be verified, but they aren’t good for testing hypotheses.

When focus groups make sense

Focus groups can be a useful research methodology when marketers want to find out whether consumers understand their message and interpret it as intended. You can’t generalize from a few focus groups to the total market, but focus groups can provide a kind of early warning system that may indicate when a message or product is missing its mark.

Focus groups are also a good idea when the people being brought together share a decision-making process or product-related experience. For example, to understand holiday shopping, it makes sense to conduct a group discussion involving all members of a family. When friends typically go shopping for clothes together, it makes sense to invite friends to a group discussion, and so forth. The resulting discussions are likely to be more natural because the group composition reflects a real-world situation.

For marketers, the main value of a focus group is that it allows them to see their consumers close-up. They can observe and pick up subtle cues and clues that may only emerge in face-to-face interactions. It’s often the case that the best information marketers can glean from a focus group is not from what people say, but from what they do — small gestures, facial expressions, reactions to the statements of others, and so on.

These interpretations are often intuitive in nature. Experienced researchers can draw intuitive insights from how participants talk, how they express themselves in body language, how they change their tone of voice, whom they interact with, and so on. Such insights are not strictly logical and testable in a scientific sense, but they can, nevertheless, be extremely useful. Reading the memoirs of marketing legends, it’s surprising how many of their most successful insights appear to be extracted from such intuitive observations.

Don’t trust these observations and insights too much. They’re subject to the same biases and misattributions as any other human observation. Don’t allow them to begin to appear inevitable and necessarily true. At best, they’re hypotheses that need to be subjected to more rigorous, scientific testing before being used to inform or change decision making.

Risks and limitations of focus groups

Because focus groups deal with a group of consumers, many unique challenges come from the resulting group dynamics. People act differently in groups than they do in one-on-one situations. Challenges of group dynamics include

Dominance: The so-called alpha participant problem, in which one participant (or sometimes a pair of participants) dominates the discussion and overpowers the opinions of others in the group.

Groupthink: A psychological tendency for groups to agree with each other and coalesce prematurely into a consensus.

Motivated participation: Participants may feel pressure to contribute something even when they actually have no opinion. (This can be exacerbated by a feeling of obligation to participate because they were paid to do so.)

False leads: When consumers are asked to spend 90 minutes discussing something they would normally give little thought to, they’re likely to come up with a wide range of semi-random comments that marketers may take much more seriously than they should.

Although focus groups are good for seeing how a message or product idea comes across to a group of consumers, they shouldn’t be used for trying to determine what a message should be. The dynamics of focus groups tend to encourage conformity and inhibit creativity. Groups tend to value the known over the unknown and the familiar over the unfamiliar. When it comes to developing a strategic marketing message, focus groups are unlikely to make a creative contribution.

Sampling opinions in consumer surveys

Consumer surveys are the most used methods in market research. Especially since the introduction of online surveys and consumer panels devoted to every conceivable research topic, they’ve become a cheap and extremely fast-turnaround tool for probing the opinions and behaviors of consumers around the world. But they don’t come without some costs and risks.

When consumer surveys make sense

It makes sense to use surveys to ask people questions they can answer, such as what they normally do or have done, especially if referring to the recent past. It’s more problematic to ask people about their beliefs and feelings, because the true sources of their preferences and attitudes may be largely unknown to them because they reside in their nonconscious minds. It’s also risky to ask people about what they’ll do in the future, because people’s guesses about the future are subject to numerous uncertainties and biases that are unlikely to yield accurate predictions.

Researchers easily can be misled by answers to questions like these, because people sincerely believe they’re answering truthfully and accurately. But they are, in fact, providing rationalizations and guesses that can lead researchers astray.

Certain techniques help to limit such rationalizations, such as asking people to make rapid, simple binary choices rather than formulate and express considered opinions. This approach minimizes one of the main sources of distortion in survey responses: the fact that people are being asked to use System 2 deliberative processing to tell a plausible story about choices and actions that they often perform using System 1 automatic processes. (See Chapter 8 for more on System 1 and System 2.)

Theory of Planned Behavior techniques (see “Understanding Why Asking Questions Is Risky Business,” earlier in this chapter) also provide useful guidance for formulating survey questions that minimize self-presentation biases and improve the predictive accuracy of results. Building questionnaires around the three Theory of Planned Behavior guidelines — asking about behaviors, asking about behaviors of peer groups, and asking about situational controls that influence behaviors — can improve both the usefulness and predictive accuracy of consumer survey results.

Risks and limitations of consumer surveys

The primary limitation of surveys is that they assume that people can accurately report on their attitudes, beliefs, and feelings; describe their preferences; remember what and why they did something in the past; and predict what they’ll do in the future. Despite the finding of brain science and the warnings of neuromarketers that we describe in some detail, we suspect that these questions will continue to be asked in consumer surveys. As most market researchers would acknowledge, it’s better to know something that may be wrong, and treat it with appropriate caution, than to know nothing at all.

Some additional methodological issues can impact the validity and generalizability of survey results. We mention them here only in summary because they’re beyond the scope of this book, but some of them pop up again in our discussions of methodological issues relevant to neuromarketing studies in later chapters.

Designing the questionnaire: The biggest pitfall is assuming you know all the questions that need to be asked. All too often, surveys are limited to the questions that marketers believe are important, potentially missing more important questions that may be on the minds of consumers. Conducting in-depth interviews before formulating survey questions is a good way to avoid this pitfall.

Selecting the sample: The biggest challenge here is ensuring that the sample is representative of the target population of interest. This is especially tricky with online surveys, which draw from online panels of consumers that are nonrepresentative in various ways, known and unknown. A common mistake is to assume that this problem can be solved with larger sample sizes. A large biased sample is just as damaging to statistical inference as a small biased sample.

Analyzing the results: It’s important not to confuse statistical significance with substantive significance. Especially with online surveys, huge samples can be generated relatively inexpensively. Such samples will produce some highly significant associations because many statistics (such as the often-used correlation coefficient) are sensitive to sample size. Also, surveys should be analyzed using sophisticated multivariate statistics, not simple two-variable correlations, because most relationships of interest involve several variables, not just two.

Other Ways to Ask Consumers Questions

In this section, we outline three variations on the core methodologies outlined in the preceding section. These three variations are widely used in market research today.

Test marketing using experimental designs and targeted samples

Experimental designs are typically used to test variations in marketing offers. Most commonly, they’re used to fine-tune direct marketing or online offers. Because it’s inexpensive to test variations in offers or pricing with relatively small random samples, it’s possible to identify an optimal combination before rolling out the offer to a full market. Experimental designs are also useful for testing new products in test markets before a broader rollout. Online forced-choice testing can be used to refine offers with very large-scale samples in a very short amount of time.

A risk in test marketing is that it can alert competitors to the imminent launch and the form it will take. Also, test marketing can be expensive and can take a long time to gain conclusive results. However, properly conducted test-market experimental designs are highly reliable because they’re based on real-world consumer behavior.

Consumer panels

Consumer panels consist of volunteer members who allow researchers to keep records of their behavior for a period of time. For example, participants in a media panel may be asked to record whenever they listen to the radio or watch television, while participants in a grocery shopping panel may be asked to record all their grocery purchases. Today, much of this data collection can be done automatically and much more accurately, with set-top boxes for media usage and cash register and point-of-sale purchase card data for shopping. Panels provide a rich source of information for understanding how advertising and marketing impact shopping decisions for real consumers.

The main challenge with panels is that members may become overly aware of the fact that they’re being monitored, which, in turn, may impact what they do. Validation studies have shown that consumers do exhibit some tendency to adjust their behavior when they’re part of a panel. The interesting point is that they typically don’t lie about their behavior, but rather start to behave in a way that mirrors how they want to be seen as behaving.

Observational studies

Observational studies, also called anthropological or ethnographic studies, rely on the intuitive power of a skilled observer. Instead of asking consumers questions, observational researchers spend time with consumers in their natural environments and activities — at home, shopping, socializing, snacking, searching for information, and so on. The researcher draws intuitive conclusions from the natural behavior observed and the context within which the behavior unfolds. The key principle of the observational researcher is to observe but not interact. The researcher tries to become the figurative fly on the wall.

Observational studies are mostly qualitative in nature and, like in-depth interviews, typically cover only a small sample, so they don’t generate results that can be generalized. But also like in-depth interviews, they provide insights that can be uncovered in no other way.

For example, observational studies can be used to find out whether consumers purchase a brand habitually. If the observer sees that most consumers pick the brand off the supermarket shelf without hesitation and without comparing prices or checking labels, she can reasonably infer that she’s observing habitual buyers in action. Of course, this study won’t explain why or how consumers became habitual buyers, but it will help the brand’s marketers understand the behaviors of an important segment of their market.

Mixing and Matching Traditional and Neuromarketing Approaches

Marketers, like all human beings, can be sorely tempted to believe results that seem to confirm what they want to believe. To eliminate this potential bias, marketers need to be willing to scientifically test any insights they derive from qualitative methods like interviews and focus groups. Scientific surveys can sometimes play this role, but even the most statistically rigorous survey result should also be subjected to complementary checks before being translated into marketing and business decisions that can commit millions of dollars to a new product or marketing campaign.

We believe that traditional approaches and neuromarketing methodologies largely complement each other. Together, they provide a more complete picture than either set of methodologies can deliver on its own.

Traditional approaches should be used for exploration and hypothesis generation (in-depth interviews, focus groups) and for documenting recent behaviors across large samples of consumers (surveys). Consumers’ self-reports of memories, attitudes, feelings, and future plans are important because they represent what people consciously believe to be true, but they should be interpreted by researchers from the perspective of conscious–nonconscious interaction. They only represent the tip of the iceberg of what’s going on in people’s minds.

Neuromarketing approaches, on the other hand, are designed to identify and measure those nonconscious workings of the mind. They rely on rigorous experimental methods and statistical inference and may or may not be expressed in conscious statements and behaviors.

For the modern market researcher, the interaction of conscious and nonconscious consumer responses should be what matters. What happens below the level of consciousness deeply influences conscious responses, but conscious responses also deeply influence nonconscious processes. Through the mechanism of expectation formation, our conscious beliefs and attitudes shape our subsequent nonconscious impressions and evaluations. So, if researchers want to understand not only what goes on in consumers’ minds, but also how consumers change their minds and behaviors based on experience, they need to understand both conscious and nonconscious consumer responses.