CHAPTER 1

Introduction

Research starts with a question. Sometimes these are why questions. Why do some people vote Democrat and others vote Republican? Why do some people purchase health insurance and others do not? Why do some people buy a particular product and others buy different products? Why do some people favor same-sex marriage and others oppose it? Why do some people go to college and others do not? Other times they are how questions. If you are a campaign manager, how can you get people to vote for your candidate? How could we get more people to purchase health insurance? How could you get customers to buy your product? How could we convince more people to go to college? But regardless, research starts with a question.

Have you thought about how we go about answering questions in everyday life? Sometimes we rely on what people in authority tell us. Other times we rely on tradition. Sometimes we use what we think is our common sense. And still other times we rely on what our gut tells us. But another way we try to answer questions is to use the scientific approach.

Duane Monette et al. suggest that one of the characteristics of the scientific approach is that science relies on systematic observations.¹ We often call these observations data and say that science is empirical. That means it is data based. However, the scientific approach doesn’t help you answer every question. For example, you might ask whether there is a God, or you might ask whether the death penalty is right or wrong. These types of questions can’t be answered empirically. But if you want to know why some people vote Democrat and others vote Republican, the scientific method is clearly the best approach. Relying on what people in authority tell you or what tradition tells you or your gut won’t work.

Research Design

Your research design is your plan of action. It’s how you plan to answer your research questions. Ben Jann and Thomas Hinz recognize the importance of questions when they say that “surveys can generally be used to study various types of research questions in the social sciences.”² The research design consists of four main parts—measurement, sampling, data collection, and data analysis. Measurement is how you will measure each of the variables in your study. Sampling refers to how you will select the cases for your study. Data collection is how you plan to collect the information that you will need to answer the research questions. And data analysis is how you plan to analyze the data. You need to be careful to decide on your research design before you collect your data.

In this book, we’re going to focus on data collection and specifically on surveys. The book is organized in two volumes. In the first volume, we’ll focus on the basics of doing surveys, and we’ll talk about sampling, survey error, factors to consider when planning a survey, and the different types of surveys you might use. In the second volume, we’ll focus on carrying out the survey, and we’ll talk about writing good questions, the actual carrying out of surveys, the impacts of current technology on survey research and survey reporting.

Observation and Questioning

Irwin Deutscher succinctly summarizes the different ways we collect data: “(1) we can observe it in process; (2) we can view the records men [and women] leave behind . . .; and (3) we can ask questions and listen to answers.”³ In this chapter, we’re interested in two of these approaches—observation and questioning.

Matilda White Riley makes the following comment about observation and questioning, noting that one method is not inherently superior to the other but that observation and questioning focus on different aspects of the social setting we are studying.⁴

Researchers sometimes feel—mistakenly, we believe—that they can obtain a true picture of a social phenomenon only if they observe it with their own eyes. To be sure observation and questioning often give different results; but this occurs, not because one method is more valid than the other, but because the two focus . . . on different sets of social system properties.

Observation and questioning give us different information about what is going on in the world. Observation gives us information about what people do. Questioning gives us information about what people say and the context to help interpret their observations.ⁱ This suggests that we often need both observation and questioning to give us a complete picture of what is happening and why it happens.

Elliot Liebow, in his book Tally’s Corner, provides a clear example of these two different approaches to data collection.⁵ Liebow studied a group of men who hung out on street corners in Washington, DC. He notes that “men and women talk of themselves and others as cynical, self-serving marauders, ceaselessly exploiting one another as use objects or objects of income.”⁶ The men in Liebow’s study “are eager to present themselves as exploiters to women as well as to men.”⁷ In other words, this is what they say. He goes on to say that “in practice, in their real relationships with real women, the men frequently gave the lie to their own words.”⁸ This is what the men do. So how does Liebow explain this apparent contradiction between what men say and what they do? He suggests that there are two opposing impulses at work. “The impulse to use women as objects of economic or sexual exploitation is deflected by countervailing impulses and goals, especially the desire to build personal, intimate relationships based on mutual liking and love.”⁹ The apparent contradiction between what the men say and what they do is explained by the “interplay of these opposing impulses.”¹⁰

Let’s consider another example. You’re doing a market research survey for a company that manufactures condoms. You want to know whether people purchase condoms and the particular brands they buy. It’s easy to imagine a discrepancy between what people say and what they do. Some people might be embarrassed to give you this information, and others might feel that it’s none of your business. What people say might not accurately reflect what they do.

Even though we see that observation and questioning give us different information about the world, we are still surprised when there is a lack of consistency between what we learn from observation and from questioning. Deutscher, in his book What We Say/What We Do, describes an early study by Richard LaPiere.¹¹ In the early 1930s, LaPiere traveled across the United States with a Chinese couple. They ate together in restaurants and stayed at hotels and autocamps and were refused service only once, and this was during a time in the United States when there was considerable prejudice toward Chinese. Six months later, LaPiere sent a questionnaire to these same hotels and restaurants asking the following question: “Will you accept members of the Chinese race as guests in your establishment?”¹² He describes the results of his survey as follows:

With persistence, completed replies were obtained from 128 of the establishments we had visited; 81 restaurants and cafes and 47 hotels, autocamps, and “Tourist Homes.” In response to the relevant question, 92 percent of the former and 91 percent of the latter replied “No.” The remainder replied “Uncertain, depends upon circumstances.”¹³

So what are we to make of this? Is this an example of the inconsistency between what people say and what they do? Or does it simply reflect that observation and questioning are telling us different things about the world? LaPiere’s classic study sparked a great deal of interest and follow-up studies. Howard Schuman, in his book Method and Meaning in Polls and Surveys, describes a study that he and Robert Brannon carried out in 1969.¹⁴ He refers to this as “an attitude–behavior field study.”¹⁵ In a survey, respondents were asked their opinion of open-housing laws. Here’s the question they were asked and the percentage of respondents giving each answer.¹⁶ (DK stands for don’t know and NA for no answer.)

Three months after the survey, the same respondents were asked to sign a petition. One of the petitions supported the first law and the second petition supported the other law. Those who said they would sign the petition were then asked if they would be willing to have their name appear in the newspaper as a supporter of that petition. Schuman summarizes the overall consistency between what people said and what they were willing to do for those opposed to the open-housing law: “85 percent were consistent in signing the Owner’s Rights petition, and 78 percent were consistent in refusing to sign the Open Housing petition which gives an overall average of 82 percent consistency.”¹⁷ The same type of consistency was also found for those who supported open housing. Schuman concludes that in this study “attitudes can predict behavior to a reasonable extent, though of course not perfectly.”¹⁸

In a more recent study, Eleanor Singer et al. studied the “impact of privacy and confidentiality concerns on participation in the 2000 Census.”¹⁹ This is another example of what Schuman referred to as the attitude–behavior question. Their analysis found that attitudes toward confidentiality and privacy were significantly related to behavior (i.e., returning the census form). It’s interesting that they also report that other researchers found that “many more people . . . say they would not provide their SSN [Social Security number] to the Census Bureau than actually fail to provide it when it is asked for on their census form.”²⁰

There are many more examples of the attitude–behavior issue, but these are sufficient to show that sometimes people behave in a way that is consistent with what they say and that other times what they say is different from what they do. As Liebow pointed out, there are various factors affecting both what people say and what they do, and it is the interplay of these factors that eventually determines the outcome. For our purposes, it is important to keep in mind that observation and questioning provide us with different information. Questioning tells us how people feel, and observation provides us with information about how people behave. It’s not surprising that sometimes these two types of information are consistent with each other and other times they are not. The focus of this book is on questioning and how we carry out surveys. But we should never lose sight of the fact that this is not the same thing as observing how people behave and interact with each other.

Triangulation

Triangulation refers to the use of data from different sources and methods of data collection. All data suffer from different types of error, and error is inevitable. We’ll have more to say about that in Chapter 3 (Volume I), “Total Survey Error.” It follows, then, that using data from different sources and methods of data collection is a powerful research strategy. Eugene Webb et al., in their book Nonreactive Measures in the Social Sciences, put it this way: “If a proposition can survive the onslaught of a series of imperfect measures, with all their irrelevant error, confidence should be placed in it.”²¹ In other words, if data from different sources and methods of data collection lead to the same conclusion, then we can have more confidence in that finding.

An informative example of triangulation is William Foote Whyte’s work on community conflict and cooperation in Peru.²² Whyte studied 12 communities using both surveys and observation in the 1960s. In 1964 and 1969, the questions used to measure conflict were slightly different. In 1964, the survey asked, “Is there much conflict or division among the people of this village?” In 1969 the question was “Is there much conflict in this village between the people who want to change things and the people who want to keep to the old ways?” The question used to measure cooperation was the same in both years: “When it comes to cooperating on some project for the community, how well do the people cooperate?”²³

Whyte’s analysis of the survey data focused on change over the 5-year period. He found that four communities shifted from high conflict and low cooperation to low conflict and low cooperation. Only one community, Huayopampa, shifted from low conflict and high cooperation to high conflict and low cooperation. This single piece of data would probably have been dismissed were it not for the fact that Whyte also had observational data from graduate students who observed behavior in these communities. The data from the observers corroborated the survey findings. This led Whyte to reject the commonly held belief that conflict and cooperation were different ends of a single continuum and to assert that it was possible for communities to be high or low in both conflict and cooperation, an important theoretical breakthrough. Whyte concludes that this “theoretical breakthrough . . . would have been impossible without the combination of research methods used in our program.”²⁴

This example of triangulation focuses on conflict and cooperation at the community level. But it is easy to see how this could be relevant for other studies on conflict and cooperation at the organizational level. Imagine that you were studying hospitals or multinational business corporations and you wanted to study conflict and cooperation among staff. You might employ the same strategy of using both survey and observational data to achieve triangulation.

Questioning (Interviewing) as a Social Process

Interviewing is in some ways similar to the types of conversations we engage in daily, but in other ways it’s very different.ⁱⁱ For example, the interviewer takes the lead in asking the questions, and the respondent has little opportunity to ask the interviewer questions. Once the respondent has consented to be interviewed, the interviewer has more control over the process than does the respondent. However, it is the respondent who has control over the consent process, and it is the respondent who determines whether and when to terminate the interview. We’ll discuss nonresponse in Chapter 3 (Volume I), “Total Survey Error,” and in Chapter 3 (Volume II), “Carrying Out the Survey.”

Raymond Gorden has provided a useful framework for viewing the interview as a social process involving communication. Gordon says that this communication process depends on three factors: “the interviewer, the respondent, and the questions asked.”²⁵ For example, the race and gender of the interviewer relative to that of the respondent can influence what people tell us, and we know that the wording and order of questions can also influence what people tell us. We’ll discuss these considerations in Chapter 3 (Volume I), “Total Survey Error.”

Gorden goes on to suggest that the interaction of interviewer, respondent, and questions exists within the context of the interview situation.²⁶ For example, are we interviewing people one-on-one or in a group setting? Many job interviews occur in a one-on-one setting, but one of the authors recalls a time when he was among several job applicants who were interviewed in a group setting involving other applicants. Rest assured that this affected him and the other applicants. Are we interviewing people in their homes or in another setting? Think of what happens in court when witnesses are questioned in a courtroom setting. That clearly affects their comfort level and what they say.

Gorden notes that the interview and the interview situation exist within the context of the culture, the society, and the community.²⁷ There may be certain topics such as religion and sexual behavior that are difficult to talk about in certain cultures. Norms of reciprocity may vary from culture to culture. Occupational subcultures, for example, the subcultures of computer programmers and lawyers, often have their own language.

Norman Bradburn views “the interview as a microsocial system consisting of two roles, that of the interviewer and that of the respondent. The actors engage in an interaction around a common task, that of completing an interview.”²⁸ He goes on to suggest that “there are general social norms that govern interactions between strangers.”²⁹ Two of those norms are mutual respect (including privacy) and truthfulness.

It’s helpful to keep in mind that the interview can be viewed as a social setting that is affected by other factors, just as any social setting is influenced by many factors. In this book, we will be looking at many of the factors that affect the interview. We’ll look at the research that has been done and how we can use this research to better conduct our interviews.

Brief History of Surveys

Don Dillman et al. have an excellent and brief history of surveys in their book Internet, Mail and Mixed-Mode Surveys—The Tailored Design Method. Dillman says that “during the first two thirds of the 20th century, there existed only one generally accepted mode for conducting surveys: the in-person interview.”³⁰ Edith DeLeeuw reports that the “first scientific face-to-face survey” was a “study of working-class conditions in five British cities” in 1912³¹ and “the first documented mail survey” was in 1788 sent to “ministers of all parishes of the Church of Scotland.”³² By 1980, Dillman says, the mail survey was commonly used, and by the early 1980s, the phone survey became a dominant mode. By the mid-1990s, another form of surveying had emerged—the web survey. However, by 2014, Dillman says, “it is no longer practical to talk about a dominant mode of surveying, as in-person interviews were described in the middle of the 20th century and telephone was referred to from about 1980 to the late 1900s.”³³

One of the factors that influenced surveys was technological change.³⁴ By 1970 almost 90 percent of households had a landline phone.³⁵ By the early 2000s, the cell phone was commonly used, particularly by young males. Caller ID made it easier for people to screen their calls and not take unwanted calls. E-mail and the web created a new medium of communication. By the mid- to late 2000s, web surveys were common, and in the past few years there has been a rapid transition to mobile devices. Today, there has clearly been a shift to what are called mixed-mode surveys, which rely on a combination of face-to-face, mail, phone, and web-based surveys along with new technologies that have appeared, such as the interactive voice response survey, where respondents use their touch-tone phone to record their answers, and audio-computer-assisted self-interviews, which are respondent-administered surveys on computers and mobile devices.

During the last 50 years, there were shifts in the way people used these technologies. Response rates to phone surveys began to drop because of the difficulty in contacting respondents and survey refusal. Roger Tourangeau and Thomas Plewes conducted an extensive review that looked at nonresponse in a number of large surveys. They conclude that “the experience of this illustrative set of surveys provide evidence that nonresponse rates continue to increase in all types of cross-sectional surveys, with little to suggest that the trend has plateaued.”³⁶ They also point out that the increase in nonresponse rates for phone surveys has been particularly large.

Two landmark events are worth noting in the history of surveys. In 1936, The Literary Digest conducted a mailed survey of potential voters in the presidential election, which pitted the Democrat incumbent President Franklin Delano Roosevelt against the Republican governor Alf Landon.³⁷ They sampled phone numbers in telephone directories and names in state automobile registration lists. Their sample was extremely large, totaling over two million potential voters. However, it did not include those without phones and automobiles and clearly did not adequately represent the population. The Literary Digest survey predicted a victory by Landon, but Roosevelt won in a landslide. This clearly demonstrated the danger of using lists that were biased in terms of variables such as education and income.

Another significant event occurred in the 1948 presidential contest between Democrat incumbent President Harry Truman and the Republican governor Tom Dewey.³⁸ The major polls such as Gallup and Roper predicted that Dewey would defeat Truman, but Truman prevailed in the election. This brought about changes in surveys such as the decline of quota sampling and the practice of continuing polling right up until Election Day.

The history of surveys demonstrates the influence of societal changes on surveying. The dramatic increase in landlines in households by 1970 and the rapid increase in the use of cell phones in the 2000s along with the development of a significant segment of individuals who use only cell phones have changed surveys drastically. It is likely that this will continue in the future. Address-based sampling is another development that has become common, where residential addresses are sampled from the U.S. Postal Service’s Computerized Data Sequence File and made available through third-party providers.

Overview of the Book

Here is a brief summary of what we will cover in Volumes I and II of the book.

Volume I

• Chapter 2—Sampling—What are samples and why are they used? In this chapter, we discuss why we use sampling in survey research and why probability sampling is so important.

  Common types of samples are discussed along with information on choosing the correct sample size.

• Chapter 3—Total Survey Error—Error is inevitable in every scientific study. We discuss the four types of survey error—sampling, coverage, nonresponse, and measurement error, focusing on how we can best minimize it.
• Chapter 4—Factors to Consider When Thinking about Surveys—In this chapter some of the fundamental considerations about surveys are presented: the stakeholders and their roles in the survey process; ethical issues that impact surveys; factors that determine the scope of the survey; and how the scope, in turn, impacts the time, effort, and cost of doing a survey.
• Chapter 5—Modes of Survey Delivery—There are four basic modes of survey delivery—face-to-face, mailed, telephone, and web delivery. We focus on the critical differences among these different modes of delivery and the relative advantages and disadvantages of each. We also discuss mixed-mode surveys, which combine two or more of these delivery modes.

Volume II

• Chapter 1—Introduction to Doing Surveys—The introduction focuses on the importance of research questions and the research design. Surveys involve asking questions, and it’s important to think about the social process of the interview.
• Chapter 2—Writing Good Questions—Here we look at survey questions from the perspective of the researchers and the survey participants. We focus on the fundamentals of the design, formatting, and wording of open- and closed-ended questions and discuss some of the most commonly used formats in survey instruments.
• Chapter 3—Carrying Out the Survey—Every survey goes through different stages, including developing the survey, pretesting the survey, administering the survey, processing and analyzing the data, reporting the results, and making the data available to others. Surveys administered by an interviewer must also pay particular attention to interviewer training.
• Chapter 4—Changing Technology and Survey Research—The chapter focuses on the impacts technology has had on survey research. As computers became widely available to the general public, survey platforms adapted to self-administered formats. Likewise, as cell phone technology replaced landline telephones, survey researchers had to adapt to new issues in sampling methodology. Currently, rapid advances in mobile technology drive both opportunities and challenges to those conducting surveys.
• Chapter 5—Presenting Survey Results—In this chapter we talk about the last step in the survey process: presenting the survey findings. Three major areas, the audience, content, and expression (how we present the survey), which shape the style and format of the presentation, are each discussed along with their importance in the creation of the presentation. The chapter concludes with a discussion on how to structure different types of presentations such as reports, executive summaries, and PowerPoints and how to effectively present survey data and results.

Annotated Bibliography

Research Design

• Matilda White Riley’s Sociological Research I: A Case Approach is an early but excellent discussion of research design.³⁹ Her paradigm of the 12 decisions that must be made in constructing a research design includes the alternative methods of collecting data—observation, questioning, and the combined use of observation and questioning.
• Earl Babbie’s The Practice of Social Research is a more recent introduction to the process of constructing a research design.⁴⁰
• Delbert Miller and Neil Salkind’s Handbook of Research Design & Social Measurement provides many examples of the components of the research design.⁴¹

Attitudes and Behavior

• Irwin Deutscher’s What We Say/What We Do is an excellent discussion of the age-old question of the consistency between what people say and what people do.⁴²
• Richard LaPiere’s Attitudes vs. Actions is the classic example of the inconsistency between attitudes and behavior.⁴³

Triangulation

• William Foote Whyte is the author of Street Corner Society, which is one of the classic examples of participant observation that relies not only on observation but also on questioning.⁴⁴ His study of 12 rural communities in Peru is an excellent example of how observation and questioning can be combined in the research design.⁴⁵

Questioning (Interviewing) as a Social Process

• Raymond Gorden’s Interviewing: Strategy, Techniques, and Tactics is one of the clearest discussions of the communication process and the factors that affect this process.⁴⁶
• Norman Bradburn’s article “Surveys as Social Interactions” is an excellent discussion of the interactions that occur in interviews.⁴⁷

Archival or Available Data

• There are different sources of data—observation, questioning, and archival or available data. Although archival or available data is not something that we’re going to discuss in this book, if you want to learn more about it, the classic introduction is found in the book by Eugene Webb et al., Nonreactive Measures in the Social Sciences.⁴⁸

ⁱSomeone might point out that we can ask people questions about their behavior. For example, we might ask what products they bought or whether they voted in the last presidential election. Certainly, we are getting information about how people report their behavior. But we know that people overreport how often they vote and underreport other things such as illegal or deviant behavior. We’re still getting their subjective reports of their own behavior.

ⁱⁱFrom now on we will refer to questioning as interviewing.