Stage 1: Initial Recruitment

It’s tempting to jump right in and begin writing survey questions, and today’s software packages almost entice you to do so. But as we learned with the Sirens, adventurers who dashed off on their journey without appropriate preparation ultimately met with tragedy. Consequently, it’s essential to first consider the questions you want to ask and how you intend to use the answers to the questions. This is not simply a matter of taking your intuitions and translating them into survey question form. We’ve found, for example, that the phrasing of the question itself is highly contextually sensitive and that the survey-takers often do not have the same interpretation of questions as the survey-makers.

One way to mitigate this risk is to create a very short survey and cast a wide net to recruit participants with familiarity on the topic. For example, one survey on data science practices randomly sampled employees and asked them three simple questions: (a) “Do you perform data science activities?”; (b) “What’s your biggest frustration when performing data science?”; and (c) “May we follow-up with you?”

Leave the survey open for at least a week to give participants adequate time to respond to the short survey. As responses come in, you’ll want to recruit diverse participants to get a variety of stories from different backgrounds. For example, to learn how professionals in an organization use log and telemetry data, we recruited not only developers, but also project managers, operations engineers, data scientists, and even content developers. Next, follow up with several of these participants and conduct interviews with them.

Stage 2: Interviews

The interview is one process through which we can gain information about software engineering. We’ve found a successful approach is to ask participants to share specific stories that relate to a particular topic. Often, participants will tell these in the form of “war stories,” which describe particularly challenging situations and how they overcame them.

Think of these interviews as incredible sources of knowledge. Like the songs of the Sirens, these 30–45 min interview sessions provide us with rich stories, told directly through the voices of those who experienced them.

Nonetheless, these isolated anecdotes are difficult to generalize because they are highly contextualized. Also, because interviews are expensive and time-consuming to conduct, only a limited number of stories are practical to obtain. But by translating these stories into survey response questions, we can both generalize and scale these stories.

Stage 3: Survey Design

It’s now time to design the survey. Just as Odysseus realized early on that he would not be able obtain the Sirens’ knowledge through force, a similar principle holds for survey design. Since online surveys are self-administered, be careful about wording. Instead of inventing terms, use language that you encountered in the interviews conducted earlier. Otherwise, your questions might not connect with the survey participants.

There are two broad categories of questions you can ask: closed and open response. Closed response, such as multiple choice or Likert scales, are easy to statistically analyze, but fix the set of possible responses. Open response, such as short answer text fields, can provide new and unexpected insights. However, it’s laborious to analyze these unstructured, free-text responses. Our preference is to use open responses sparsely and instead rely on the interview content for these types of questions.

The key to successful survey design, then, is to ensure that the majority of your questions are grounded in and derived from the experiences of the interviews. As an example, in our survey about game development and traditional software development, we asked a set of Likert statements on coding, process, and team. The statements were of the form “My software has high technical debt.” (Coding), “My team uses a waterfall process, rather than an agile process.” (Process), and “Creativity is highly valued on the team” (Team).

How did we know which statements to ask? Our statements came directly from the stories told by participants during our semistructured interviews. By doing so, we can triangulate stories we heard in the interviews and generalize those experiences quantitatively.

At the same time, your interviewees told you a lot of stories, so you’ll have to be selective about what you ask survey participants about. Time permitting, you may also want to refine your interview statements through a review of existing literature. If a topic is already well-understood, it’s probably not worth wasting the participants’ time asking about it. Consider your goals carefully and only ask survey questions that help you reach those goals.

Stage 4: Survey Piloting

Don’t count on your initial survey design to be perfect. In an extreme case, a collaborator spent nearly a year in preparation while his team argued about question wording. By the time they “finished,” the survey had become obsolete and ultimately proved to be useless.

Instead, consider a more agile approach by piloting the survey: ask a few friends to take the survey and then analyze the results the way you normally would. With as few as five participants, you can often catch the most egregious mistakes. Piloting with people who are representative of your target population will help you find more mistakes.

During piloting, a useful perspective is to test your survey like you’d test software. Survey bugs can appear in the most unlikely of places. For example, one “bug” we found through this process was in our consent form. A participant could choose “I don’t consent to participate,” but still filled out the survey! This would have been disastrous, because we wouldn’t know who consented and who didn’t. We’d have to throw out all the data.

Here are some observations during piloting:

• Length matters, and you should pay attention to how long it takes to complete the survey. Based on our experiences as survey designers, we recommend that the survey not take more than 15 min.

• Cut, cut, cut! This is a good opportunity to identify questions that are redundant, confusingly worded, or overly difficult.

• Identify drop points where participants may give up on the survey.

• Pay attention to open response questions and don’t expect rich answers. In general, people don’t like typing stuff—that’s what interviews are for.

Finally, providing a strong initial recruitment letter is perhaps just as important as the survey, but it is often overlooked or done as an afterthought. Participants are busy people, and it’s easy for them to ignore your letter if they don’t see why it’s relevant to them. To improve participate rates, you should respect their time and explain how the participant benefits from the survey. Why should they take it? What do they get out of it? They always want to know, “Why’d you pick me?”

Though Odysseus had only one chance to face the Sirens, piloting offers you multiple opportunities to quickly practice your survey before embarking on the real journey.

Stage 5: Survey Deployment

It’s time to send out the survey! If you’ve done everything right so far, this is where you, like Odysseus, will sail past the Sirens and obtain insight and knowledge from the journey. During deployment, there are still a few items to keep in mind:

• Have a finite close date. We usually give our participants two weeks to complete the survey.

• Responses come in a bathtub model. You get a lot of responses early, then nothing or a trickle, followed by a final burst. Send a reminder to those that didn’t respond the first time around, but you should consider preliminary analysis after the first set of responses so that you’re not blocked on survey completion.

• Be respectful and avoid oversampling the same participants or communities. Aim for diversity. Certain open source communities, such as Eclipse, Mozilla, and Apache, tend to be swamped with surveys from researchers all over the world.

There’s no hard and fast rule for the number of survey responses to target, but for quantitative data, power analysis is a useful technique for estimating the number of responses you’ll need.

Stage 6: Survey Analysis and Write-Up

You’ve made it past the Sirens, but there’s still some work to do. Expect to do a certain amount of data cleaning, since you’ve generated the survey in a way that benefits the survey taker, not you. For example, Likert responses are easier to analyze when converted to an ordinal 1–5 scale. Other questions, such as “number of hours worked on a task,” are easier to ask in terms of hours and minutes, but simpler to analyze as fractions of hours (1 h 30 min to 1.5 h).

Even short surveys tend to generate large amounts of data for potential analysis, but not all of this information will be useful in your eventual write-up. That’s okay, but consider it a learning experience—was it worth participants’ time in asking a question (or asking it in a particular way) if you’re not going to analyze the results?

During this process, try to triangulate and form a coherent narrative about the work using the interviews and the survey responses. Do the survey responses match the experiences of the interviews? Are there cases where they don’t? Connecting your survey results back to the interviews enables you to generalize your interview results. This approach, for example, allowed us to say with some statistical certainty that “Creativity is valued more in game development teams,” a statement that we heard in our interviews on games development and software engineering.