We conducted a direct observation case study and analyzed the data using grounded theory. Direct observation involves one of the researchers sitting in the same office as the participant, watching everything that the participant does and writing it down in a log. We often prefer direct observation to interviews or surveys because, as nonparticipant observers, we can be impartial and record every detail of the action, which gives us an exact view of events as they occurred. This contrasts with an interview or a survey, where subjects suffer from several biases. Typical are generalization bias, where they tell us generally how their day goes instead of giving us specific events, and memory bias, where they remember more recent events better than older ones. In qualitative studies like ours, the goal is not to eliminate all bias (which is impossible), but to choose the biases that will have the least influence on the results we uncover.
The case study aspect of our approach was to choose a small number of participants to observe. Participants were selected to be as different from one another as possible, so that we might observe a diverse set of activities, behaviors, and phenomena. Direct observation is an expensive way to gather data due to the time commitment of following someone around for many hours of the day. This limits the number of people we can include in the study, so we try to make sure that we see as much phenomena as we can from a small number of participants.
To analyze our data, we used grounded theory [Strauss 1987]. Grounded theory is a qualitative research methodology that enables researchers to create theories about what they see solely from their data, and not from any prior assumptions about how people behave. This methodology is useful when we are not sure what kind of data we will get and do not yet know which aspects will be most important. Grounded theories analyze their data in three phases:
- 1. Open coding
In this phase, every artifact in the log is tagged with a label (or multiple labels) that describes what it is about.
- 2. Axial coding
In this phase, the labels created in open coding are related to one another, to show cause and effect, moderators to effects, or actions that take place because of some behavior.
- 3. Selective coding
In this phase, all of the labels and their relationships are organized into a coherent storyline. This often forms the basis of a theory about the data that was gathered.
In this section, we describe our observation and analysis techniques, and present several examples of our raw data.
We selected eight developers newly hired by Microsoft between one and seven months before the start of the study. We identified 25 available subjects (based on manager approval and schedule consideration) and selected 8 (seven men and one woman), balancing years of schooling and division within the company. Four had BS degrees, one had an MS, and three had PhDs, all in computer science or software engineering. Of the BS recipients, two were educated in the U.S., two in China, one in Mexico, one in Pakistan, one in Kuwait, and one in Australia. All PhDs were earned in U.S. universities. We also selected for the least amount of previous software development experience (none outside of limited internships), with the exception of the subject from Australia, who had two years of development experience outside of Microsoft. Subjects were compensated weekly (USD $50) for their participation.
Each developer was observed for 6–11 hours over 2 two-week periods, with a one-month break in between. We observed the developers in their own work environment without interrupting them, followed them to meetings, and watched them as they interacted with their colleagues and friends. We recorded our observations in time-stamped logs, which were the basis for our subsequent analyses. Similar to what you might see on reality TV, developers were asked to record a video diary entry at the end of each day that they were not observed. They were asked to talk about the most interesting thing that happened that day, followed by a question asking them to reflect on some aspect of their college education or new hire experience.
In the open coding phase of our grounded theory analysis, we identified the tasks that novices perform by reviewing the observation log entries of what subjects were doing, and tagged each with one or two task and subtask types. The task types we coded are arranged in the taxonomy presented in Table 26-1. For example, an entry might be tagged Coding/Searching and also Communication/Asking Questions if the developer asked a colleague where an API call was defined in the code.
Table 26-1. Tasks performed by novices, listed in the order of their frequency
|
Programming |
Reading, Writing, Commenting, Proofreading, Code Review |
|
Working on Bugs |
Reproduction, Reporting, Triage, Debugging |
|
Testing |
Writing, Running |
|
Project Management |
Check in, Check out, Revert |
|
Documentation |
Reading, Writing, Search |
|
Specifications |
Reading, Writing |
|
Tools |
Discovering, Finding, Installing, Using, Building |
|
Communication |
Asking Questions, Persuasion, Coordination, Email, Meetings, Meeting Prep, Finding People, Interacting with Managers, Teaching, Learning, Mentoring |
In this section we give a flavor of the observation of one of our participants along with a section of its associated activity log (shown in Table 26-2). Timothy was assigned to fix a bug. After reading the five bug reproduction steps, he attempted but failed to successfully execute the first step. He spent 45 minutes trying to debug the problem by swapping various software libraries on his computer, including swapping computers, to no avail. He went across the hall to a colleague, Abdul, and asked him for help. Timothy explained what he had tried, but Abdul disagreed with his assessment of the problem. Abdul returned with him to the office, noticed that Timothy had copied incorrect libraries to his computer, told him where to find the proper binaries, and then went back to his office.
The incorrect binaries were a consequence of Timothy’s debugging strategy, however, and not the original problem. Timothy tried to reproduce the bug for another 12 minutes before going back to Abdul to tell him that things were still not working. Abdul explained more about the libraries he copied, causing Timothy to realize that he had the wrong mental model of the libraries and the directories in which they were meant to be placed. Abdul taught him the proper model, recounting stories of his own debugging prowess from years back, and sent Timothy on his way with another few generic debugging strategies, both of which Timothy had already tried. Timothy did not press his colleague for more help until he could prove that he had tried these strategies with the corrected mental model, though he knew they would not work. After another 25 minutes of debugging and testing on his own, Timothy had still not successfully executed the first reproduction step, and appeared to have made no progress at all.
On each day that we did not observe them, the subjects in the study recorded a 3–5 minute video diary entry using a webcam we gave to them to use. We created 40 questions, of which most subjects recorded answers to the first 20. One made it to 35. The numbers vary due to absence, lack of free time, and number of observations we made. We listened to the videos and transcribed the answers to 13 of the questions that related to learning and the college experience.
Empirical qualitative studies like ours see all the complexities of real life. Naturally, with so many multifaceted experiences spread over just a few subjects, it is difficult to ascertain with 100% certainty what caused the effects we saw. Some aspects of our study and its participants may have caused biased results.
Our study subjects came from a range of educational backgrounds, and were at different stages of personal and professional development at the start of the study. In addition, each changed and learned a different amount over the course of the two months we watched them. Although this makes it hard to compare participants with one another, this learning effect is the focus of the study.
An unavoidable bias in our study is the Hawthorne effect, where the observer’s mere presence influences the subject to change his behavior. We believe our continuous and lengthy observations helped put them more at ease. At the end of the study, we asked the subjects if they noticed any change in their own behavior, but most remarked only that they took fewer breaks while we were around. To avoid a hierarchy bias, we made it clear to the newcomers and their managers that all data collected would be private and would never be available to anyone, including management. In addition, we took steps to ensure that our subjects were not coerced to participate by their managers. At any point, even after the study had finished, subjects could withdraw and erase all their data. None did.
When reading our findings, one must keep in mind that a developer’s expression of a need for information does not tell the entire story about what developers really need to do their jobs. In particular, due to the nonintrusive, observational approach of our study, it was possible to observe that novices do not always recognize that they need some information. That is, they may flail, stop making progress, task switch, or do any number of things when, in the eyes of the observer, seeking out some information could allow them to make progress. In particular, we observed many cases in which a novice did not recognize that they should seek information from a colleague in order to make progress on the current task. Our data indicate what novices actually do, but not, perhaps, what they should be doing.
Table 26-2. An activity log from Timothy shown with tagged task types and subtypes