CHAPTER 4

Factors to Consider When Thinking about Surveys: The Groundwork

Do You Really Need a Survey?

Alicia was sitting at her desk early one morning, when her boss, Bob, came rushing into her cubicle. “Mornin’ Alicia,” Bob said, slightly out of breath and his face flushed. “We’ve got a problem.” Oh my, not the dreaded “we’ve got a problem” problem, Alicia thought to herself. “Candice (Bob’s boss) called me into her office the first thing this morning,” Bob continued. “Turns out the company plans to launch a new product line aimed at the 18 to 25 demographic this fall in select east coast cities. We know about our customer base in that age range, but we don’t really know about the demographics of the six major cities that we’re targeting for the launch. You’re going to have to get some surveys done, so we can see how those places stack up against our customer base. We’ll need that to tweak our marketing. Unfortunately, I’m going to need you really pushing on this because we must have results in 8 weeks at the outside. So give it some thought and we’ll talk more tomorrow.”

Bob rushed out as fast as he had entered, leaving Alicia’s head spinning. How would she get six surveys developed, organized, and into the field and back in 8 weeks? Not to mention the time it would take to analyze the information and prepare a report for the VPs. It was going to be a brutal couple of months. With only four researchers in her unit, Alicia wasn’t even sure she had enough staff to pull off that kind of effort.

Alreck and Settle make the following observation about why surveys are conducted:

Surveys are often conducted simply because it’s the only way to get the information needed. Even when the information is available through other means, survey research may be an easier, quicker, less expensive, or more accurate way to get the required information.¹

We might suggest that Alreck and Settle’s observation can be turned around to serve as a good litmus test as to whether conducting a survey is the best way to answer a research question. Specifically, the sponsor and researcher should ask two questions before undertaking a survey: (1) Is a survey the only way to get the information needed? and (2) Is survey research an easier, quicker, less expensive, or more accurate way to get the information? In our previous example, before Alicia gets too focused on the procedural aspects of doing the surveys, it would be wise for her to consider these questions about the proposed surveys. Of course, to answer these questions, the most fundamental issue not only to survey research but also to research in general must be answered, namely: What is the research question? In our earlier example, Alicia needs to get a clear indication from her managers as to what overarching question they are trying to answer. In doing so, they will need to tell her what demographics they are looking for, if they are interested in the demographics of just the targeted age range or of the target cities in general, and so forth. Only then can she truly answer the two questions about the necessity and practicality of implementing the fairly extensive survey effort which was presented to her. If, for example, Alicia’s company managers are looking for demographics (including economic data) to compare with the demographics of their customer base, there is a wealth of such data readily available through the U.S. Census Bureau, which currently conducts more than 100 well-designed surveys each year.² It is a distinct possibility that the information they seek is already available without having to go through the expensive process of conducting surveys.

The Stakeholders

Assuming the question of whether a survey is needed has been answered in the affirmative, it is then important to give thought to who will be involved and why. At this stage and throughout the survey process, it is important to keep in mind that a survey is actually a social interaction in the same way that social media provides a platform for social exchange. There are multiple stakeholder groups involved in the survey process. The first group of these stakeholders are those involved in the development, production, and execution of the survey. This group would primarily include: (1) the sponsors, (2) researchers, and (3) the survey participants. A second group of stakeholders would be the consumers of the survey results. This group would include those who are informed of the results, whether directly by the researchers or indirectly by the media. This group could include the public and other scholars and researchers who use the results of a survey in their own surveys or analyses. The third stakeholder group may occur in settings where individuals are assigned to approve or appraise surveys such as human subjects committees or journal editors when results are formally published.ⁱ

Because our focus in this book centers mainly on the development, production, and execution of surveys, our discussion here will be confined to the stakeholders most directly engaged in these processes: the sponsors, researchers, and participants. Alreck and Settle³ discuss the roles of two of these stakeholder groups: the sponsors and researchers.

In certain respects, the sponsor is the owner of the survey, as the sponsor is the one who provides the financial support and, in the case of proprietary surveys, owns the products derived from it. We would note that while the sponsor generally provides the funding to conduct the study, in some cases a third party will provide the resources while the sponsor’s role is taken over by an organization that needs and asks that the survey be conducted. This latter situation is seen most often with nonprofit organizations, such as charities, which receive funding for the survey from a corporate sponsor, or a university group, which receives a foundation grant or government funding to engage in a survey.ⁱⁱ

The first stakeholder group contains the sponsor who identifies the purpose of the survey, the population of interest, the timeline, and the approximate resources available. The sponsor should also indicate what the expected deliverables are, such as a report, presentation, or data. While not generally writing the specific questions for the survey, the sponsor should provide the researchers guidance as to the areas in which information is needed. It is also important for the sponsor to provide a background context and any technical or specialized background information the researchers will need to construct a relevant survey instrument.

The second stakeholder group, the researchers, is comprised of the individuals who have the content and technical expertise to conduct the survey. Their role typically includes the development, design, implementation, analysis, and reporting of the results. Large companies, organizations, universities, and governmental agencies may employ in-house researchers with considerable expertise in designing and conducting survey research. However, even for large organizations, it may not be cost effective to maintain an in-house survey research staff if the organization rarely has a need to do a survey. Similarly, smaller companies and organizations, even those who do have a need to do frequent surveys, may not have enough resources to maintain an in-house research capacity. Fortunately, there is a wide assortment of external help available through for-profit companies and individual consultants, university researchers, nonprofit organizations, and governmental units. Unfortunately, however, there is a large variation in the quality of the expertise available. If you type search terms such as survey expertise or help with surveys into your web browser, you will see a staggering array of websites offering everything from customized one-stop survey design and execution to those whose services do little more than step you through a fill-in-the-blank form, supposedly creating an online questionnaire.ⁱⁱⁱ It should also be noted that some companies are merely brokers. They have little real in-house capacity but instead actually subcontract needed survey services, such as conducting the survey; data entry; or the expertise required to design questionnaires, develop sampling designs, or run the analyses.

Well-established private companies that have established track records of providing survey expertise are characterized by being able to provide a documented history of work they have done, by not overstating what they are capable of doing, and by being transparent regarding their operations and cost structure. Colleges and universities also typically have individuals well versed in conducting survey research and statisticians who can help with design and analysis issues. University faculty also can often bring disciplinary expertise to your survey effort, such as understanding educational survey issues or human resource or marketing concerns. Some universities even have survey research centers, such as the ones the authors headed. In short, there is a wealth of information from texts, online services, and professional researchers to help with your survey should you lack in-house staff or expertise. Again, however, it is probably worth repeating that the old adage “any offer that seems too good to be true probably is” certainly applies here.

To the previously mentioned sponsors and researchers, we add participants as the third group of stakeholders in the survey process. The participant group is made up of the individuals who will be taking the survey. Such individuals might be patients in a clinical trial, coworkers in a company, clients, customers, potential customers, and so forth. During the survey process, they are commonly referred to as respondents. If you accept the argument that we made in earlier chapters regarding the value of probability sampling, then participants should be selected in a random fashion and this group should be reflective of the larger population in which you have an interest. Currently, this is a growing issue because of the extensive use of online survey research seen today, which has given rise to a corresponding extensive use of online panels. Such online panels can be broadly characterized as either probability-based panels or opt-in or access panels.⁴ According to Revilla,⁵ in probability-based panels, a random sample is drawn from the population and the selected units are contacted and invited to participate in the panel. Individuals who do not have Internet access are usually provided with it. Thus, such online panels maintain the integrity of the probability sample process. In contrast, with opt-in or access panels, individuals volunteer to participate. If they do not have Internet access, they cannot be part of the panel. This raises the issues of how representative for the target population different online panels truly are. In Chapter 4 (Volume II), we discuss the growth of survey panels vis-à-vis the advances in mobile technology.

Ethical Considerations

As we mentioned earlier in this discussion, there is a social interaction during the survey process. Irrespective of how brief the contact, during the survey process a relationship exists between the three major stakeholder groups—sponsors, researchers, and participants. And like relationships in general, survey relationships work better when those involved approach the other parties with openness, honesty, and respect. This is important not just because it’s the ethical thing to do but because the future of survey research depends on it. Talking about market research, Ian Brace notes,

The ability of the market research industry to continue to use sample surveys as sources of primary data depends upon the willingness of members of the public to give their time and cooperation to answer our questions . . . . To be able to continue, market research needs to maintain this goodwill.⁶

We believe that Brace’s conclusion is applicable to all types of survey research.

The end game, when it comes to survey research, is maintaining ethical relationships among the three stakeholder groups. To achieve this result, we must begin with the relationship between the researchers and sponsors. The sponsor can consciously or unconsciously put pressure on the researchers to produce favorable results. This latter situation frequently occurs when the individuals within the sponsor group carry the attitude that they already know how the respondents think or feel, but simply want to have a survey done to verify this. When both sponsors and researchers are in the same organization, the researchers may consciously or unconsciously feel a need to give the sponsors, who may be managers or executives in the company, the results the organization is hoping for or would like to see. Even with external researchers, there may be a tendency to produce results that please the sponsor so as to secure future business. Even more egregious is the situation when supposed survey research becomes direct marketing or database building. Research by Ian Brace et al.⁷ indicates that in today’s environment, where surveys have become commonplace with almost any kind of customer transaction, those being surveyed oftentimes cannot distinguish between surveys and marketing ploys. The use of such ploys to direct market services or products or collect data from consumers usually requires some type of complicity between sponsors and researchers and thrives in organizations that turn a blind eye to such survey manipulation. Unfortunately, this subterfuge diminishes the value of legitimate surveys and quality of the information that organizations should be able to obtain from surveys.

Professional associations recognize this issue and have developed a code of ethics for their members. The Insights Association in the United States, for example, provides a comprehensive code of standards and ethics for survey research organizations that specifies the responsibilities that survey organizations have to respondents, clients, and outside contractors and in reporting study results.⁸ The American Association of Public Opinion Researchers (AAPOR) similarly lays out principles of ethical research which emphasize ethical issues such as:

• The responsibility of minimizing risk or harm to the participants;
• The protection of the privacy and confidentiality of information for participants and clients;
• Eliminating misrepresentation of research when conducting other activities (such as sales, fundraising, or political campaigning) under the guise of conducting survey and public opinion research;
• Adherence to best practices in survey design; and
• Conducting and reporting of surveys.⁹

Scope of the Survey

What resources are needed to do a survey? Like the response to most very general questions, the answer is, “It depends.” Every year on college campuses around the country, hordes of students in research methodology classes descend on their fellow students, professors, and the world beyond, armed with clipboards, armloads of paper surveys, and coveted e-mail and other electronic contact lists to conduct a survey. They have been schooled in the basics of survey design, questionnaire construction, methodology, and data collection. Their efforts are modest, generally requiring a small number of hours, perhaps a few supplies like paper forms, and a notebook or laptop computer. Yet the essentials of these students’ survey efforts are the same as those of the vast survey efforts that require enormous investments of time, such as the Center for Disease Control and Prevention’s Behavioral Risk Factor Surveillance System (BRFSS), which was established in 1984 with 15 states. BRFSS now collects data in all 50 states as well as the District of Columbia and three U.S. territories. This survey, which currently relies principally on Computer-Assisted Telephone Interviewing (CATI), has a total sample size of more than 100,000 participants completing more than 400,000 adult interviews each year, making it the largest continuously conducted health survey system in the world.^iv In essence, however, the real difference between small student-directed exercises and this huge, comprehensive, and complex one might best be characterized as one of scope. What then determines the scope of a survey?

Snijkers et al. argue that we should conceptualize survey research as a project with “a clear purpose (to collect survey data) [and] a clear beginning and end.”¹⁰ When thought of as a project, surveys can be broken down into sequential stages: (1) the initiation, development, and planning stage; (2) the data collection stage; and (3) the data analysis and report production stage. In the language of survey researchers, these phases are often called: (1) the prefield stage, (2) the field stage, and the (3) postfield stage.^v Because these stages are sequential, the third stage is highly dependent on the successful completion of the second. Similarly, for the second and third stages to be undertaken successfully, the initiation, development, and planning stage must be well done. Thus, while surveys differ across the three stages, it is the first stage where the scope for the study is set. This includes the identification of the survey purpose and objectives, determination of time and resource parameters, and the design and construction of the data collection and analysis methodologies, which are assembled into the research plan.¹¹

Complexity and Number of Responding Participants

There are certain fixed costs required to conduct a survey. These items remain relatively static as the scope of the survey changes around them. For example, if an organization decides to equip a CATI lab to do phone surveys, there will be costs for purchasing computers, software, work stations, headphones, and so forth, and these costs will remain the same whether the organization does interviews with 5,000 or 50,000 participants. There are a number of these fixed costs, such as office space, salaries and benefits, supplies, and so forth, which will remain nearly static even if the scope of the survey changes. One of the decision points for an organization, then, is whether it engages in enough survey work to warrant the outlay of fixed-cost expenditures to create and maintain survey capacity. The alternative is for the organization to contract out all or part of its survey work. Doing so, however, doesn’t mean these costs disappear; they are simply overhead (frequently called indirect costs in academic or governmental research) that will be embedded within the pricing structure of the vendor. The difference is that they are only incurred if and when survey work is actually done. For the company or organization, there is a trade-off, then, between the costs to maintain in-house survey capacity and the loss of some control over survey process, data, and cost structure. Once base or fixed costs have been factored in, the scope of a survey determines the actual resources required by the project.

Because there are so many variables that may be included in survey cost, it is best to find a way to standardize costs to allow oranges-to-oranges, apples-to-apples contrasts. This standardization can be accomplished by determining cost per response, which is the total cost of completing one survey response, whether the information is collected with a mail-out paper survey, a telephone or personal interview, or electronic questionnaire. The cost per survey, then, is driven by two major factors: survey complexity and the number of responding participants. These two factors differentially impact different phases of the survey. In this regard, Alreck and Settle offer an insightful guideline:

Survey research strategy tends to lean in one of two directions: Obtain a large amount of data from a small sample or obtain a small amount of data from a large sample. Usually the resources are limited while the information needs are insatiable.¹²

First, the complexity of a survey is based on what information is needed, who we need to get that information from, and how we’re going to get that information. As Kennedy et al. point out, “Planning and development efforts rise with survey complexity.”¹³

The more complex the survey, the more up-front time and cost go into developing a blueprint for it; designing our survey instruments, such as questionnaires or interview forms; creating sampling and data collection methodologies; and developing the methods of analyzing the information we receive from the survey’s respondents. Similarly, the greater the complexity, the more back-end time and cost are required for activities once the data collection has been completed, such as setting up the survey database structure, developing and running the analyses programs, writing up the findings, and presenting the report.

The second factor, number of responding participants, impacts cost and time resources most during the data collection phase of the survey. Please note here that we place emphasis on responding participants rather than just participants, although the two are correlated. If you recall from the discussion in Chapter 2 (Volume I), required sample size is based on the responding participants. Thus, the rate of response becomes an important variable when considering survey cost and time. A survey in which 1 out of 5 participants who are contacted respond results in a much more efficient survey than one in which 1 out of 10 participants respond. There are many issues that affect this rate of response on a survey including, for example, the length and design of a survey instrument such as a questionnaire, the accuracy of the participants’ contact information, and the use of incentives.¹⁴

Finally, although web surveys are a cost- and time-efficient alternative compared to the traditional modes of data collection,¹⁵ the introduction of web surveys has not solved the problem of declining response rates. On the contrary, several meta-analyses show that traditional survey data collection modes outperform web surveys in terms of their response rate.¹⁶

To summarize, while the size of our sample could remain fairly stable, the resource utilization would increase as the complexity increased. Conversely, we could have a well-designed survey, whose resource needs would change if we required a bigger survey sample. It is because up-front and back-end tasks and costs remain fairly constant even as the sample size grows that researchers seek to maximize the precision of their results by increasing the sample size when possible. It also provides a solid rationale for repeating surveys using the same designs, instruments, and so forth since much of the cost of such efforts has already been accounted for in the first administration.

Now that we’ve reviewed some of the fundamental considerations in conducting surveys, we will turn our attention to the delivery modes used for surveys.

Summary

• Do you need to do a survey?
  • Can a survey answer your research question(s)?
  • Is a survey the only way to get the information?
  • Is the survey easier, quicker, a less expensive, or a more accurate way to get the information needed?
• There are three distinct stakeholder groups involved in a survey effort.
  • Sponsors are essentially the owners of the survey. They create the basic research question(s), help define survey objectives, provide the financial support to conduct the survey, and own the survey products (in proprietary surveys).
  • Researchers have the content and technical expertise to conduct the survey. Researchers are responsible for the survey’s development, design, data collection, data analysis, and reporting of the results.
    • In large or mid-sized organizations that do a lot of surveys, researchers may be in-house employees. Caution must be exercised to keep surveys independent and objective when using in-house research staff.
    • For larger or smaller organizations that only occasionally need to do surveys, or for smaller organizations without resources to maintain full-time researchers, there are a variety of outside consultants, outside research companies, and university units and faculty who are able to provide expertise and capacity.
  • Participants or respondents are the individuals who will actually be taking the survey.
    • Participants should be randomly selected from the population to ensure they accurately represent the larger population of interest.
    • Individuals take part in surveys for a variety of reasons. Some participate because they feel they are doing the right thing, others because they want to make their opinions known or want to influence future events, and yet others take part simply because they are compensated.
    • Recently, online survey panels have gained in popularity as a way of identifying participants for marketing and consumer surveys. Online survey panels are comprised of individuals who self-select or opt in to take part in surveys and therefore randomly selected.
• Ethical considerations
  • The survey process involves a relationship between sponsors, researchers, and participants and works best when there is openness, honesty, and respect among the parties.
  • Sponsors should be aware of the problem of applying pressure, consciously or unconsciously, on the researchers to produce favorable results.
  • Researchers should be equally aware of a tendency to provide results that the sponsor is looking for.
  • Using surveys as a pretext for marketing or advertising is not only unethical, but it creates a sense of mistrust for participants and can ultimately impact the willingness of individuals to take part in survey research.
  • Protection of participants from risk or harm should be the foremost consideration for both sponsors and researchers. Confidentiality and privacy concerns are two of the major risk areas that need to be addressed.
• Scope of the survey
  • Surveys, whether small, simple, low-cost efforts or huge, complex, resource-intensive projects, contain the same fundamental elements and are comprised of sequential stages:
    1. The initiation stage,
    2. The data collection stage, and
    3. The data analysis and report production stage.
  • There are certain fixed costs, such as equipment, office space, personal, supplies, and so forth involved in conducting surveys, which are relatively static even when the scope of the survey changes.
    • Once the fixed-cost expenses are figured, the scope of the survey determines the actual resources required by the project. Survey costs can be standardized to cost-per-survey response, which permits apples-to-apples, oranges-to-oranges-type comparisons.
    • Cost-per-survey response is driven by two factors: survey complexity and number of responding participants.
      • More complex surveys require more up-front time and cost for development and design and more back-end time and cost for cleaning and coding data, setting up survey databases, analysis, and preparing reports.
      • The number of respondents impacts resources most during the data collection phase of the survey.
      • The rate of response, or ratio of participants contacted to the number who return surveys, is a critical factor and affects the efficiency of the survey.

Annotated Bibliography

Need for Surveys

• There are many books on survey research which take different disciplinary perspectives as to where survey research fits in the larger frame of data collection and its usefulness to address research questions. In Chapter 6 of their book on marketing research, Hague et al. talk about “desk research” and the amazing amount of information available from different sources.¹⁷
• Ger Snijkers et al. present survey research with a business focus. In Chapter 1 of their book, Designing and Conducting Business Surveys, they provide a nice overview of the data available for business decision making and how surveys fit into this larger realm.¹⁸

Survey Stakeholders and Roles

• A nice introduction to survey research stakeholders is provided by Alreck and Settle.¹⁹
• Another perspective is provided by Ian Brace,²⁰ who discusses stakeholder roles within the context of the questionnaire design.

Ethical Considerations

• Robert Oldendick²¹ provides an in-depth review of ethical issues in survey research. The protection of confidential information in the electronic data collection age is becoming a major focal area in research.
• Hundepool et al.²² provide a comprehensive look at what is termed as statistical disclosure control.
• A good source of information on the protection of human subjects and the requirements of institutional review boards (IRBs) is provided by AAPOR, which just announced the release of “updated and expanded resources for researchers who conduct surveys subject to institutional review board (IRB) review.”²³

ⁱIt is important to note here that members of a particular stakeholder group may also be included within a different category of stakeholders. For example, sponsors of the survey could well be involved in the development of the survey but may also be consumers of the survey’s results.

ⁱⁱEthically, if the funder is different from the sponsor, this fact should be disclosed to the researchers and participants and included in any reports that are released to the public.

ⁱⁱⁱCompanies selling survey services and products often provide tutorials on their websites covering topics such as designing a questionnaire, obtaining a sample, analyzing results, and so forth. However, as a note of caution, many of these tutorials only really provide instruction on using the tools that the vendor is selling.

^ivTo learn more about the CDC BRFSS survey, go to http://www.cdc.gov/brfss/about/index.htm

^vTraditionally surveys are said to be in the field during the time when data collection occurs.