13

Lessons Learned from a Research Saga

An Ambitious Content Analysis of Television Form

Matthew Lombard

This chapter describes lessons I learned from a long research saga: designing, conducting, writing up, presenting, and publishing an ambitious content analysis of television form and its unexpected offshoot projects. The origins of the research project are described first. This is followed by a discussion of some key issues and challenges the research group faced regarding the design. These include issues with sampling and material collection, the development of measures, coder selection and training, intercoder reliability, data entry and analysis, and writing, authorship, and presentation of the original study. The chapter also discusses the project's unexpected afterlife. It concludes with advice based on lessons I learned about conducting not just content analyses but any research project.

Origins: Developing Research Questions

The saga began in early 1995, when I was a junior professor in the Radio–Television–Film department at Temple University. Although research questions and hypotheses can come from personal observations, “future research” sections of published articles, and a myriad of other places, the questions in this case arose naturally from my previous work. My dissertation project at Stanford University, and a subsequent series of three experiments conducted with a group of doctoral and master's students at Temple, had explored “direct responses” to mediated stimuli – what I would later identify as presence (short for telepresence). The idea was that, at least in some cases and to some degree, people respond to images and sounds on television (and in other media) not as representations of people, objects, and events but as the people, objects, and events themselves (i.e., directly). The first experiments used created and then found footage of individual people speaking to a camera. They manipulated screen size, shot length, and viewing distance (independent variables) to replicate the dynamics of “personal space,” demonstrating that when people are made to feel physically close to people on the screen, their responses (dependent variables) mimic those in nonmediated settings (see Lombard, 1995). The Temple studies (Grabe, Lombard, Reich, Bracken, & Ditton, 1999; Lombard, Ditton, Grabe, & Reich, 1997; Lombard, Reich, Grabe, Bracken, & Ditton, 2000) expanded the types of content tested and established that the formal production feature we called quick, forward point-of-view movement – as when a camera is attached to the front of a roller coaster or race car – produces a greater sense of enjoyable physical self-motion (measured via questionnaires) and physiological arousal (measured via finger sensors that capture the skin's electrical conductivity) when the filmed movement is viewed on a larger screen, an indication that viewers felt more “present” in the scenes.

These studies demonstrated that we could produce presence responses in viewers in an experimental setting, but they led us to wonder how often media consumers in the “real world” actually have these responses. We wanted to know how often, when, on which kinds of programs and channels/networks, and especially in which ways the presentation characteristics of television were most likely to evoke a sense of presence in viewers. We realized this would be an ambitious, time-consuming research project, and wanting to be efficient, we brainstormed about other ways we could use the data we would collect. Since most of the variables we had measured would concern the form rather than the content of television, and since there was already an extensive literature in media psychology regarding how viewers respond to many of these “formal features” (see Lombard, Bracken, Linder, Snyder, Ditton, Kaynak, Pemrick, & Steward, 1996), we would be able to assess and report on the “state of the medium” at the time, allowing researchers to extrapolate about viewers' responses to everyday viewing. We and others had noted the apparent increase in TV clutter – more, shorter commercials and promotional messages in more and longer program interruptions, and more complicated and potentially distracting graphics in news programs – so another part of our research could provide empirical evidence regarding these contributors to clutter. And we dreamed of future studies that would examine television in different eras as well as other media, expanding our Structural Features (SF) projects, conveniently abbreviated SF-Presence, SF-State, and SF-Clutter, into a near endless series of future projects.

Design

While it is possible – and some might say common – to choose a method first and to adjust one's research questions to fit it, the best approach is to identify the questions first and select the most appropriate method(s) to address them. In this case, content analysis – “a research technique for the objective, systematic, and quantitative description of manifest content of communications” (Berelson, 1952, p. 74) – was the obvious choice. Although I had never conducted one, I had learned about the method and read many reports of content analyses. I knew the coding itself, which involves categorizing the content into meaningful categories, would be monotonous (in graduate school we were told every researcher should do the coding on one project and then have other people do that work in future projects!). But after conducting experiments and surveys I particularly liked the idea that a content analysis would not require us to recruit and deal with the quirks of human participants or even to obtain approval from an institutional review board (IRB) to conduct the study. So we set about designing the ambitious content analysis, which meant making careful decisions about what material to collect and code, what characteristics (variables) to measure, how many coders we would need and how to train them, how to evaluate the consistency or agreement of the coders (intercoder reliability), and how to record, analyze, and summarize the results.

Sampling and Material Collection

Because we were setting out to study not a specific genre (e.g., news), content feature (e.g., violence), day part (e.g., prime time), type of channel (e.g., major broadcast networks) or formal feature but all of television, we chose to sample the 10 broadcast stations in Philadelphia (the fourth largest media market in the United States then and at the time of writing) and the 15 basic cable networks available in most households in the United States (according to Cable TV, April 30, 1995). The selection of most available rather than most watched cable networks was particularly strategic because we wanted to include both the frenetic features of popular MTV and the sedate features of the less popular C-SPAN.

The standard and most available video recording medium at the time was VHS videotape – unfortunately not even as high in quality as the then already disappearing Betamax format. While the format allowed for six hours of recording at the slowest speed (EP), the analog video editing equipment we knew we would need to use for the coding could only display tapes recorded at the standard two-hours-per-tape speed (SP). We knew that the ideal sample would be compiled over a whole year of the traditional fall to spring TV season since there are likely seasonal variations in the form of television. To be practical and still capture some of this variation, we selected a four-week period that included weeks both during and outside of the spring ratings “sweeps” period, when stations select programming to maximize their audience for the more detailed ratings taken by ACNielsen.

We wanted enough material to compare channels and times of day, so we settled on recording six tapes for each of the 25 channels for a total of 150 tapes. We investigated the possibility of buying tapes from a bulk seller but ended up finding good deals at local electronic and department stores. We used the Minitab software to generate several different series of random numbers to select channels (1–25), weeks (1–4), hours (1–12), and a.m. or p.m. (1–12). We used the random numbers to create the list of dates, start times, and channels for the recordings, and refined the list by applying a “rule” that the selected times be three hours apart to prevent overlap and to insure that all day parts were represented. Then we divided the selected recording obligations among the six team members with access to VCRs, avoiding assignments that required the same person to record different channels at the same time. What we had not considered is that in some cases people would forget to program their VCR or misprogram it, that some machines and tapes would fail, and that during some of the selected recording times the channel would be off the air (although all but unheard of today, this was common then and a particular problem with a small local PBS station in our sample).

We dealt with these problems by using the random numbers to select additional recording details and ended up extending the sampling period to a fifth week (and seven rather than six recordings per channel). Between August 13 and October13, 1995, 175 two-hour blocks of television broadcasts/cablecasts were recorded; 160 of the tapes (320 hours) of television programming were viable for coding. Even these many years later I have boxes in my office filled with the VHS tapes we created.

Measures

No one had conducted a study like this before. Most of the structural features of the television medium either had not been examined in research or, when studied (usually in regard to their effects), had been defined inconsistently or imprecisely. So we began with variables and definitions in a well-known production handbook (Zettl, 1984), studies that examined particular structural features in any context, and our own knowledge of television production techniques. We quickly realized we could not even have a useful conversation without developing a lexicon of basic terms; for example, a frame is an individual photograph on a videotape, the screen is the physical border that encloses the image on a television set, an image refers to the contents of the entire viewing screen, and a picture is a self-contained, bordered representation of objects and entities (there may be more than one picture in a given image).

To create our coding instrument we identified as many formal features related to presence, psychological processing research, and various types of video clutter as we could find or create. The variables represented characteristics of television form at several distinct levels including a single frame (e.g., camera shot length), a continuous shot (e.g., shot duration), a transition between shots (e.g., type of transition – cut, dissolve, etc.), a 10-second interval (e.g., artificial special visual effects), a program segment (e.g., genre), and an entire program (e.g., duration of opening and closing credits). In many long discussions we eliminated variables that were similar to each other or relatively unimportant, or that we decided we could not code reliably. Then we assembled two rough and incomplete coding manuals – one for complete programs and one for variables linked to single “time points” – that contained each variable and its corresponding response options, detailed definitions of terms that we thought might not be understood by different people in the same way, and examples to illustrate what we considered proper coding of many of the variables. We used extra tapes we had recorded to discuss each of the variables and how we might refine our collective understanding and the corresponding written descriptions of each one. Along with detailed instructions for coders, the final version of the “Coding Manual for Program Variables” was 23 pages long and contained 11 variables (in addition to several that identified the details of the recording and coder). The “Manual” for time point and 10 second interval variables was 39 pages long and contained 27 variables.¹

The 320 hours of recordings contained 322 complete programs. We used Minitab again to randomly select seven time points (hours, minutes, seconds) for each codable two-hour recording, resulting in what was eventually a total of 965 codable time points. The time points were restricted to being five minutes apart to reduce the likelihood of repeated coding of the same parts of the recording. These time points determined the units to be coded for a single frame (the image at the time point), a shot (in progress including the time point), a transition (the first that followed the time point), a 10 second interval (beginning with the time point) and a program segment (in progress at the time point).

Coder Selection and Training

Because our sample of television recordings was so large and our coding scheme so extensive, we decided we needed to hire additional coders to share the work. Even with optimistic estimates of the time required to code a single program (15 minutes per program, 25 minutes per time point), and with the need for additional coding to establish that we could code reliably, we had set a huge task for ourselves. While some members of the research group were students being compensated for their time with assistantship awards, course credit, research experience, and/or authorship (more on that below), we needed more hands. Using a small research grant I had received from Temple University, we hired three people – one doctoral student and two undergraduates who answered an open call for those who might be interested – and signed contracts outlining an agreement to pay them each $1,000 in return for their completing training and then performing both reliability assessment coding and regular coding. Given the unpredictability regarding complications and time commitments along the way, we made the payment contingent on completion rather than based on an hourly rate. Extended conversations with each of the paid coders confirmed their basic understanding of the coding process and the required focus on detail, sense of commitment and responsibility, and overall reasonableness, an especially important criterion.

We began the coder training process by having everyone read the detailed coding manuals we had created and report any questions or problems. The coders then coded just one and then a series of programs and time points from a set of recordings that was not part of the study sample but made at the same time. At each step, we responded to issues that came up and in many cases altered the coding instrument and manual accordingly. For example, several variables were dropped because of concerns about reliability (as will be discussed shortly). The process took several sessions over a period of weeks.

Intercoder Reliability

I knew from graduate school and from reading reports of content analysis that an essential component of the method is the calculation of agreement among coders, which establishes that the patterns reported are not those perceived idiosyncratically by a single researcher but are matched by those of other independent observers. However, I naively assumed that researchers had long ago reached consensus on the best index of this inter-judge agreement, made available software tools to conveniently calculate reliability values, and established standard criteria for evaluating the values. We soon realized none of this was true and began searching for authoritative guidance on how to proceed.

We learned that merely calculating the percentage of times coders did agree out of all the opportunities they had to do so – though intuitive, easy, and commonly reported (Pasadeos, Huhman, Standley, & Wilson, 1995) – overestimates coder reliability. For example, two coders randomly selecting the values of a two-category variable would agree 50% of the time (see Seun & Lee, 1985). Of the other indices, Cohen's kappa and Scott's pi seemed to be the most common and recommended, but the former was also criticized as inappropriate and both had significant limitations given our data set (as many as seven coders, all levels of measurement from nominal (e.g., yes/no) to ratio (e.g., durations)). Krippendorff's alpha seemed generally accepted and flexible, but extremely complex to calculate especially given our large number of variables. Thankfully, Professor Krippendorff worked across town at the University of Pennsylvania and generously allowed us to use a beta version of a software program he had commissioned. It was a bit primitive and required (as most reliability software does) a very specific arrangement of the input data, but it was a lot more practical than the alternatives – “by hand” calculations or custom-designed programs known as macros we created for use with SPSS analysis software.

Because we set out to code the manifest rather than latent or hidden characteristics of television, we expected it to be relatively easy to get coders to consistently make the same choices among the response options for each variable as they watched the recordings we had made. But our first test of coder reliability was a miserable failure. We quickly identified a primary reason: We hadn't realized that in the era of analog recordings on magnetic tape, which stretches and slips with each play, there is little chance of two people finding the exact same frame (time point) on a tape. So before we could demonstrate similar decision processes, we had to insure we were making decisions about the same things. After some brainstorming, we solved this problem with something we called an “anchor frame.” The first person to code a recording moved the tape to the previously selected random time point frame, set the time counter to 00:00:00:00, wrote down a short description of the image at that frame, moved the tape forward to the first frame that followed the first cut (not dissolve or other gradual transition) after the time point frame, and wrote down a description of the image at this “anchor frame” and how many frames it was beyond the time point frame. Each subsequent coder then used the descriptions of the time point and anchor frames and the distance between them to insure they were about to complete the coding sheet based on the identical starting time point.

With the mundane mechanical obstacles to reliability removed, we were surprised to find that the coders were still making different judgments regarding many of the variables. The reasons were predictable – vague definitions and descriptions, unforeseen characteristics in the recordings – but frustrating. It turned out that television form is much more complicated and subtle and less manifestly obvious than we had initially thought. One coder who had rarely watched television helped the rest of us identify large and small assumptions we had made about how to understand and categorize what we were coding.

One example of the kinds of refinements to the coding instrument and manual that these reliability problems led to stands out, and also illustrates the type of information we had to provide for the coders to insure reliable results. In the early versions of the coding manual a variable meant to assess a type of presence was “Apparent broadcast type” at the beginning of each 10-second interval to be coded. It appeared in the manual this way:

During practice coding and initial reliability testing it became clear that the variable was drawing coders to apply “facts not in evidence” (e.g., the timing and production attributes of an initial broadcast) and the only way to reach an acceptable level of reliability was to reframe the question in more direct and obvious terms. The entry in the coding manual was revised.

A more vexing problem regarding reliability concerned variables with little variation. For example, one variable required coders to indicate whether the images in a 10 second interval were color and/or black and white. Black and white images were exceedingly rare, in reality and in the coding results, but Krippendorff's alpha for the variable was near 0. If a single coder missed one of the rare instances of black and white, the formula determined that we could not reliably identify such occurrences and penalized us accordingly. Because we knew from our coding experience (and collective years of television viewing) that black and white content was so rare, we decided not to create and retest reliability with a new artificial sample of recordings that contained many black and white segments and instead established a rule that if Krippendorff's alpha was below a certain threshold (.7), percent agreement must be exceedingly high (.9). While reviewers might criticize this approach, we believed we could defend it as appropriate in the context of our variables.

Data Entry and Analysis

In an era of analog videotape, few laptop computers, and no iPads or other tablets, we had to make many printed copies of the coding sheets for the coders to complete with pen or pencil as they watched each recording. Coders then entered the data from the completed sheet by working in pairs, one person reading responses out loud as the other typed the information into a word-processing file on a desktop computer. Each coding sheet page generated a separate line of numbers in the data file. To identify the many possible errors from this data entry process, we printed each record entered and held the paper up in front of a bright light against another paper, a template of correctly entered data, to identify missing and extra characters and lines of data. Then all of the word-processing formatted files had to be converted to unformatted text files and combined as input data files for the then typically used mainframe version of SPSS, in which specific “jobs” had to be submitted and printed results picked up from numbered bins at a computer center on campus. A detailed data cleaning process followed, in which the frequencies of each response for every variable were calculated, unlikely or impossible values in the results checked, and errors in the data files fixed.

After all the very monotonous and time-consuming data entry and cleaning (first for reliability data and then the full program and time point data sets), the calculation of frequencies and cross-tabulations of frequencies for all of the variables was relatively simple and quick. While this is not the place for detailed reporting of the results, we were pleased that findings for approximately 70% of the variables related to presence suggested that viewing television encourages rather than discourages presence responses. For example, most television in the summer of 1995 was mostly color, live-action, with convergent sound and picture except for background music, leisurely paced with simple transitions, and very rarely included explicit on-screen indications that the material had been created earlier.

Writing, Authorship, and Presentation

Over the next four years (1996–1999) we reported the results of our research in peer-reviewed papers at the annual conferences of the International Communication Association (ICA) in Chicago, Montreal, Jerusalem, and San Francisco. Ironically the first two papers were focused on The State of the Medium (Lombard et al., 1996) and The Cluttering of Television (Lombard, Snyder, Bracken, Kaynak, Pemrick, Linder, & Ditton, 1997), with the third paper finally focusing on our original proposed study, Presence and the Structural Features of Television (Ditton, Lombard, Bracken, Snyder, Pemrick, Linder, & Kaynak, 1998). The fourth paper (Lombard, Snyder-Duch, Bracken, Ditton, Kaynak, Linder-Radosh, & Pemrick, 1999) reviewed the major findings in the entire project. With the exception of a student who moved away and left the project, the authors were the same for each paper but the authorship order was rotated according to earlier arrangements – fortunately, I had learned that such issues need to be negotiated early in the process to prevent disagreements and bad feelings later on.

While everyone including me had learned a lot, after five years and more time-consuming challenging obstacles than any of us had expected, the doctoral students on the research team became very busy in their new jobs as assistant professors, and in one case in industry, and we all gravitated to new projects. By the time we were able to revise the State of the Medium paper for submission to a journal in 2002, the recordings we had made were seven years old and perceived as out of date. It takes a team to bring a project to the final publication stage, but the team leader is ultimately responsible and I very much regret not doing more to follow it through. A respected senior scholar at one of the ICA conferences praised the project and suggested that pursuing it might become the primary focus of my academic career, with regular follow-up studies and explorations of new structural features in new media, but that was not to be. However, the project was far from over.

Unexpected Afterlife

Because we felt we had learned so much while designing, conducting, and reporting the study, the three people who had been most involved with it wrote and presented another conference paper that relayed some of what we had learned as a “guidebook” for those conducting similar large-scale content analyses (Snyder-Duch, Lombard, & Bracken, 1999).

The biggest unexpected challenges we had faced concerned properly assessing and reporting intercoder reliability. As we read other reports of content analysis studies in communication, we realized that many of them failed to adequately address – and in some cases failed even to mention – this critical aspect of the method which is arguably most identified with our field. So, based on the annoyance generated by the fact that we had struggled to “do it right” while others had not, and as a service to others who might want to but did not know how, we began another project: a content analysis of intercoder reliability in communication content analyses – we called it CAR for “content analysis reliability.” The much simpler coding scheme, applied to a sample of 200 studies published between 1994 and 1998 and identified as content analyses in Communication Abstracts, confirmed that authors often (about 30% of the time) failed to report reliability. When they did report it they often (apparently, since the reports are typically vague) relied on the inadequate percent agreement index. Based on a review of concepts, indices, and tools, and our results, we provided a set of concrete guidelines to help other researchers to more easily and completely assess and report reliability.

Coincidentally, following our presentation of the CAR paper at ICA's 2001 conference in Washington, DC (Lombard, Snyder-Duch, & Bracken, 2001), I saw a call for papers for a special issue of Human Communication Research on “Statistical and Methodological Issues in Communication Research.” We submitted the paper and it was quickly accepted. When the article (Lombard, Snyder-Duch, & Bracken, 2002) appeared, Professor Krippendorff contacted us to point out a handful of errors in a table that none of the three of us, or any of the reviewers or editors along the way, had noticed – a disconcerting commentary on the peer review process and a reminder to always review others' and one's own work carefully. Professor Krippendorff also lobbied us, in an increasingly unpleasant series of exchanges, to modify our suggested guidelines in a jointly authored correction, an option we eventually rejected to maintain our intended role as independent reviewers of the literature and its ongoing controversies. The journal eventually published a separate correction and further explanation (Lombard, Snyder-Duch, & Bracken, 2004).

The impacts of the SF and CAR studies continue. The original article referred to an online resource (Lombard, Snyder-Duch, & Bracken, 2003) that we established to summarize the article and provide updates as new evaluations of indices and tools became available. Nearly a decade later, I still receive email messages thanking us for the resource and asking questions about reliability. I was invited to give a talk at the Conference on Optimal Coding of Open-Ended Survey Data at the University of Michigan (Lombard, 2008). The materials, papers, and stories from both projects have been very useful in the classes I teach. Even this chapter represents an entirely unexpected outgrowth of the original structural features of television project. And it is possible that I will return to one of our original goals – comparing the television form from the now long ago 1995 with the present, a task that would be much easier for two reasons. The first is the amazing evolution since the mid-1990s of digital technologies for capturing and replaying images. Recordings could be created with home DVRs and transferred to a computer or, better yet, directly captured and recorded on a computer's hard drive, and then analyzed with frame-by-frame accuracy using electronic time code in Final Cut Pro or other digital editing software. Meta-tags could even be added to individual frames and found later via searches to speed the coding process. The second reason the recording and coding process would be easier today is that we have learned many lessons through the process of developing the original project.

Lessons Learned

So what were those lessons learned during the project? I think the most important ones were the following.

Be ambitious but realistic. We were right to start out designing the most creative and ambitious project possible, one that would answer all of our research questions and generate multiple research reports. Start with the ideal study and for the most part ignore constraints on your time, money, and other resources, and then be realistic as you reduce the scope and modify the design details to match those constraints. This is something that, in retrospect, we did not go far enough in doing. Researchers can get too attached to specific content analysis variables but, as with experiment variables, survey questions, and in all other forms of research, it's essential to prioritize.

Expect things to take longer. At nearly every stage of the structural features research unexpected issues arose to delay us. In fact, these delays were so substantial that three team members got married, four babies were born, and several people graduated and/or moved away during the course of the project. Underestimating the time needed to complete complex tasks is typical – it has an official status as Douglas Hofstadter's law (see Hofstadter, 1999). Even though it is difficult, however, researchers should always keep this problem in mind. One good strategy is to make a good faith estimate of how long major steps in the project will take and then add another 50% or 100%.

Do research together. This project would have been impossible for any single person to complete, and not just because content analyses require multiple coders. I would argue that, with important exceptions, a research group produces better ideas, accomplishes more, and provides a richer experience than single researchers working independently. Although group dynamics can be complicated and everyone's needs and preferences have to be thoughtfully considered, the increased efficiency and the ability to brainstorm, celebrate, and commiserate together is worth the effort.

Do research with students. At various times six doctoral, two master's, and three undergraduate students worked on the structural features project. Aside from the many benefits their participation contributed at every step, they – or more accurately all of us – were able to learn about research by actually doing it, fulfilling another important mission of a university. Project leaders also need to delegate specific sets of tasks to student team members so they learn as much as possible and gain a sense of “ownership” over the research. Besides, the leader simply cannot do everything.

Be organized. This project required an extra level of attention to organization and detail, with so many researchers; variables; sampling units; videotapes; coders; coding sheets and manuals; data storage and analysis software programs, formats, and files; and articles and presentations. But organization is an undervalued aspect of every research project. Keep detailed electronic records of everything (and printed copies as needed). Make backup copies, ideally via a real-time automated system such as the software provided with most external hard drives. Use shared online calendars, spreadsheets, and word-processing files (e.g., via Google Docs) to coordinate information accessible to and updatable by team members (this wasn't available in the mid-1990s). Devise naming conventions for electronic and hard copy files and folders so that you and others can find specific material when you need it. Follow these conventions even when you're in a hurry to get to other tasks – for example, don't put files in a generic folder to be sorted properly later. Document all procedures and instructions and make them easily accessible. Keep each major (and even minor) draft of most documents so you can refer to them later if necessary. Communicate constantly with the other researchers: send updates, ask questions, hold informational and brainstorming sessions, both online and especially in person (over coffee, snacks, lunch, or dinner – food always improves communication and creativity!).

Be flexible and creative. When we realized videotape stretches, we designed the anchor frame mechanism; when reliability was unexpectedly low for some variables, we modified the measures or figured out how indices could be combined to better represent the underlying situation; when we realized how much we had learned, we found new projects and venues for both publications and service to the field. Although it is not easy, researchers need to be flexible and creative, open to “outside the box” solutions and new opportunities that take the original plan in new directions.

Follow through. Although we persevered through many challenges for many years and the structural features project generated several conference papers and then indirectly more papers and publications, we (mostly I) failed to follow through quickly enough to produce publications of the original results. So my advice to others, which I continue to struggle to follow myself, is to not give up on or to let projects get cast aside in favor of new distractions. Most challenges, including negative peer reviews – there is always a critic or three out there you will not be able to satisfy – can and should be overcome.

Recognize trade-offs. Our initial belief that content analysis would be easier than other social science methods because it does not involve human respondents or subjects was naive. Every research choice represents a trade-off – time versus money, validity versus reliability, depth versus breadth, problems with participants versus problems with coder reliability. There's no “free lunch”!

Enjoy it. Despite the obstacles and frustrations along the way, the research I've described here was extremely rewarding and a great adventure. Given the chance, I would do several things differently but I would definitely do it again. My most important advice to other researchers is to follow your interests and passions in selecting topics and conducting research – to enjoy it. Life is too short to do anything else.

ACKNOWLEDGMENTS

Thank you to friends and colleagues Cheryl Campanella Bracken, Theresa B. Ditton, Tim Hicks, Selcan Kaynak, Jodi Linder-Radosh, Naila Mattison, Cynthia Mugo, Janine Pemrick, Woong Ki Park, Nandini Sen, Jennifer Snyder-Duch, and Gina Steward.

NOTE

1 See http://matthewlombard.com/research/sfp_ab.html (accessed August 9, 2013) for the complete documents.

REFERENCES

Berelson, B. (1952). Content analysis in communication research. New York, NY: Free Press.

Ditton, T. B., Lombard, M., Bracken, C. C., Snyder, J., Pemrick, J., Linder, J. M., & Kaynak, S. (1998, July). Presence and the structural features of television: The illusion of nonmediation and formal features of television. Paper presented to the Information Systems division at the annual conference of the International Communication Association in Jerusalem, Israel.

Grabe, M. E., Lombard, M., Reich, R. D., Bracken, C. C., & Ditton, T. B. (1999). The role of screen size in viewer experiences of media content. Visual Communication Quarterly, 6(2), 4–9.

Hofstadter, D. (1999). Gödel, Escher, Bach: An eternal golden braid. New York, NY: Basic Books.

Lombard, M. (1995). Direct responses to people on the screen: Television and personal space. Communication Research, 22(3), 288–324.

Lombard, M. (2008, December). Standardization in assessment and reporting of intercoder reliability in content analyses. Paper presented at the Conference on Optimal Coding of Open-Ended Survey Data at the University of Michigan, Ann Arbor, MI.

Lombard, M., Bracken, C. C., Linder, J., Snyder, J., Ditton, T. B., Kaynak, S., Pemrick, J., & Steward, G. (1996, May). The state of the medium: A content analysis of television form. Paper presented to the Information Systems division at the annual conference of the International Communication Association in Chicago, IL.

Lombard, M., Ditton, T. B., Grabe, M. E., & Reich, R. D. (1997). The role of screen size in viewer responses to television fare. Communication Reports, 10(1), 95–106.

Lombard, M., Reich, R. D., Grabe, M. E., Bracken, C. C., & Ditton, T. B. (2000). Presence and television: The role of screen size. Human Communication Research, 26(1), 75–98.

Lombard, M., Snyder, J., Bracken, C. C., Kaynak, S., Pemrick, J., Linder, J. M., & Ditton, T. B. (1997, May). The cluttering of television. Paper presented to the Mass Communication division at the annual conference of the International Communication Association in Montreal, Canada.

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2001, May). Content analyses in communication: Assessment and reporting of intercoder reliability. Paper presented to the Information Systems division at the annual conference of the International Communication Association in Washington, DC.

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2002). Content analysis in communication: Assessment and reporting of intercoder reliability. Human Communication Research, 28, 587–604.

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2003). Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Retrieved August 9, 2013, from http://matthewlombard.com/reliability/

Lombard, M., Snyder-Duch, J., & Bracken, C. C. (2004). A call for standardization in content analysis reliability. Human Communication Research, 30, 434–437.

Lombard, M., Snyder-Duch, J., Bracken, C., Ditton, T. B., Kaynak, S., Linder-Radosh, J., & Pemrick, J. (1999, May). Structural features of US television: Primary results of a large-scale content analysis. Paper presented to the Mass Communication division at the annual conference of the International Communication Association in San Francisco, CA.

Pasadeos, Y., Huhman, B., Standley, T., & Wilson, G. (1995, May). Applications of content analysis in news research: A critical examination. Paper presented to the Theory and Methodology division at the annual conference of the Association for Education in Journalism and Mass Communication (AEJMC) in Washington, DC.

Seun, H. K., & Lee, P. S. C. (1985). Effects of the use of percentage agreement on behavioral observation reliabilities: A reassessment. Journal of Psychopathology and Behavioral Assessmentt, 7, 221–234.

Snyder-Duch, J., Lombard, M., & Bracken, C. (1999, May). Learning from our experience: A guidebook for large-scale television content analysis projects. Paper presented to the Mass Communication division at the annual conference of the International Communication Association in San Francisco, CA.

Zettl, H. (1984). Television Production handbook. Belmont, CA: Wadsworth.