4

Beyond the Qualitative/Quantitative “Divide”

Reflections on the Utility and Challenges of Q Methodology for Media Researchers

Carolyn Michelle and Charles H. Davis

Introduction

This chapter aims to introduce readers to a relatively little-known methodology within media studies, one we believe has much to offer those interested in understanding the subjective dimensions shaping media production, reception, consumption, and use. Q methodology can be described as a methodological hybrid that is neither qualitative nor quantitative, but draws on some of the strengths of both traditions. It thus combines the mathematical rigor of quantitative methods with a strong interpretive component more typical of qualitative approaches to human research (Stainton Rogers, 1995; Stenner, Watts, & Worrel, 2008; Watts & Stenner, 2012). Q methodology is primarily concerned with revealing similarities and differences in the viewpoints, attitudes, beliefs, or experiences of participants, and does so in a way that is relatively systematic, reliable, and replicable. Furthermore, by deploying Q methodology's unique capacity to reveal shared subjectivity in conjunction with a more traditional questionnaire seeking details of respondents' sociodemographic memberships (gender, age, ethnicity, occupation, and so on), discursive affiliations, and political interests, it is possible to explore whether and in what ways shared viewpoints might be related to aspects of identity and subject positioning. For these and other reasons, we believe Q methodology has much to offer critical media scholars located within a range of disciplines.

In this chapter, we briefly outline the history of Q methodology and explain some of its key principles and processes. Following this, we present a brief case study of our recent experience using Q methodology in an online survey of cross-national responses to James Cameron's 2009 feature film Avatar. We take readers behind the scenes to offer our personal reflections on some of the issues and challenges researchers may confront when using this method. We also highlight some of the unique insights that Q methodology offers, which we hope will provide useful guidance for emerging researchers interested in adopting this approach. Finally, we discuss some of the main strengths and limitations of this method, and outline why we believe Q methodology may have unique potential to shed light on some of the key questions in media studies today.

What Is Q Methodology? A Brief History and Overview

Q methodology was created by psychologist-physicist William Stephenson in the 1930s as a structured research tool designed to analyze the nature and diversity of people's attitudes, perspectives, or subjective experiences relating to a given topic (Stephenson, 1953). He conceived it as an alternative to existing methods used to measure attitudes within psychology, which typically relied on the researcher's a priori categorizations and other normative considerations (Brown, 2006). An exploratory empirical research method, Q methodology can elicit insight into human subjectivities in the rich and holistic way more typically associated with in-depth interviews and focus groups, while also providing clear structure, capacity for replication, and robust measurement-based analysis. In Q research, participants are given tools with which to express their own priorities and understandings, from their own particular standpoint. Rather than imagining infinite variability among people's perspectives, however, Q methodology presumes a finite degree of diversity in viewpoints (Stainton Rogers, 1995). It aims primarily to identify and chart the nature and range of distinct subjective perspectives on a given topic that are shared by groups of individuals, reflecting an objective substructure or “defined set of interpretations” (Brown, 1977, p. 572) that occur naturally, rather than being the result of the researcher's prior categorizations.

While Q methodology has grown in prominence within disciplines such as psychology, tourism and environmental studies, political science, communication studies, policy studies, marketing, and health sciences, it remains on the margins of media studies scholarship, and is frequently neglected in methodology handbooks within this discipline. But, as we hope to illustrate in this chapter, Q methodology has considerable potential to offer insight into many aspects of media production, reception, consumption, and use. We believe it constitutes a very useful addition to the media researcher's methodological toolkit, while also offering a new way of approaching some of the lingering questions in the field.

How Does Q Methodology Work?

In brief, participants are first invited to “map” their own viewpoint on the topic by responding to a diverse selection of relevant ideas and opinions. These “maps” are then collectively analyzed to identify factors, which reveal that particular groups of individuals share a similar perspective on the topic in a more holistic sense, in that they tend to generally agree and disagree about the same things. Of course, this process is a little more complex in practice.

In a typical Q study, a research question is first formulated. Q methodology particularly lends itself to research questions “that have many, potentially complex and often socially contested answers” (Stenner et al., 2008, p. 219). The kinds of questions that Q methodology may assist in answering are among the key ones for contemporary media researchers. For instance: what are the subjective influences that shape the creation, production, and consumption of media content? How do the perspectives, beliefs, and motivations of the producers of media texts inform their professional and creative practices? How do people make sense of and use different forms of media? How do media representations inform people's understandings and evaluations of themselves and of the wider world? What is the place of media in people's everyday lives? What kinds of consumption patterns and cultures of use are evident in relation to traditional media, compared to new digital media forms? How do people experience and respond to growing degrees of media convergence? What motivates people to become creators of media in their own right, and what are the conditions that stimulate such “produsage” (Bruns, 2008)?

Once a question is determined, the researcher generates a list of items to sort (the Q sample), which is a broadly representative sample of self-referential and subjective statements drawn from the wider discursive field that surrounds the issue or topic under investigation, referred to as the concourse (Stephenson, 1985). The concourse “consists of all that can be thought and said about a situation, event, or phenomenon” (Durning & Brown, 2007), and includes diverse statements of subjective opinion that constitute a part of the “oral public culture” relating to the topic at hand (Fiske, 1987, and Bondejerg, 1988, cited in Stephenson, 1995–1996). In Q methodology, individuals are assumed to formulate their own perspectives with reference to the ideas and perspectives that are circulating within the wider field of discourse, with which they will be familiar but not necessarily agree (Stephenson, 1995–1996).

Usually, the Q sample items comprise short statements – just a sentence or two expressing one coherent idea or perspective – derived from in-depth interviews or questionnaires with a wide range of individuals, or perhaps from published sources such as academic criticism or letters to the newspaper, or even online sources such as reviews, news items, or discussion forum comments. The researcher can seek suitable Q items from wherever the concourse around their topic is thought to exist, and in whatever form it is most often expressed. Depending on the topic, images, objects, or other stimuli such as sounds can also be sorted. The most complex case reported in the literature on communication, media, and audiences is Grosswiler's (1992) multimedia Q sample used to investigate the theory, articulated by Marshall McLuhan and others, that the emergence of new media forms generates related shifts in human values and consciousness. Grosswiler's Q sample included items from newspapers, magazines, literature, art, and poetry, as well as audio and video recordings. In a subsequent study, Grosswiler's respondents sorted slides of magazine and newspaper front pages, videotapes of television program excerpts, and audiotapes of radio program excerpts in a way that reflected their preferences for certain kinds of media content (Grosswiler, 1997; see also Rhoads, 2009). Whatever their form, all the items used are generally carefully selected and considered by the researcher in relation to each other in order to attain a Q sample that represents, as much as possible, a very wide spectrum of suitable stimuli, most often in the form of subjective responses to the given topic, from multiple different angles.

Since respondents must compare each item with all other items and the number of possible pairwise comparisons is vast, the researcher usually selects a smaller number of items from an initial survey of the wider concourse for the final Q sample. Items may be selected in a structured way, perhaps to represent categories associated with an existing theory, or more commonly in an unstructured, representative way (Dziopa & Ahern, 2011; McKeown & Thomas, 1988). The chosen items are presented individually on cards, or represented on virtual cards if conducting the survey online.

Next, the researcher selects the respondents (called the P set). Nonrandom sampling is used to select a population of individuals who are likely to have quite different views about the topic, or who represent theoretically relevant subpopulations. For instance, a researcher may endeavor to select a wide range of individuals who are diverse in terms of their gender, age, education, political or religious orientation, or areas of expertise (McKeown & Thomas, 1988, pp. 38–39). Unlike conventional quantitative survey methods, it is not necessary to include a large number of respondents, and most Q studies have only a few dozen respondents (Danielson, 2009). Stainton Rogers (1995) suggests between 40 and 60 participants is optimal in terms of maximizing the range and diversity of viewpoints.

Once selected, each respondent is asked to create a map or model of his or her viewpoint using the items the researcher has selected from the concourse. Respondents rank-order the numbered items in the Q sample according to a condition of instruction, such as from “most strongly agree” to “most strongly disagree.” The convention in Q methodology is that the items be sorted into a quasi-normal distribution – roughly in the shape of a bell curve – which can have several degrees of agreement with the neutral or indifference point in the middle, as in the example of a 32 item sort shown in Figure 4.1.

Respondents are asked to first sort the items into three piles – disagree, neutral, or agree – before selecting the few items with which they most strongly agree and most strongly disagree, placing these at either end of the set distribution. They then continue to place all the items, gradually working their way toward the middle (see Brown, 1986, 2008; McKeown & Thomas, 1988; Watts & Stenner, 2012, for a fuller description). Having hopefully provided the same kinds of discursive resources available within the wider concourse, the researcher steps back to allow participants to determine for themselves the value, meaning, and significance of the different items from their own unique point of view, and to express this through their ranking and placement of statements on the grid. The final configuration is taken to represent a holistic expression of their subjective experience or perspective, at least in as far as it is possible to express this, given the items made available to them – this is why careful selection of the items in the Q sample is crucial. The positioning of each item on the grid is then recorded.

Figure 4.1 Example of a 32 item Q sort grid

Individual Q sorting is generally administered in person, using physical cards containing the Q items and a large grid on which these can be arranged. However, online Q research using dedicated software such as FlashQ is increasingly practiced. This software visually replicates the card sorting and ranking process on screen through a user-friendly interface, guiding respondents through each stage of the process as they finalize the arrangement of the various statements on the virtual grid. This software also makes it possible to include a tailored questionnaire for respondents to complete, which can collect relevant sociodemographic data and/or information about personal attributes the researcher suspects might be relevant. The software then captures all the data for each individual and records it in a format that can easily be imported into an Excel spreadsheet, which makes it very easy for the researcher to “eyeball” the data.

The process of sorting the items is normally followed by asking respondents to explain in further detail their reasons for selecting the “most strongly agreed” and “most strongly disagreed” items. In an online Q survey, respondents are usually invited to write some additional comments on why they chose each of those items in open text boxes. This is an important step that aids the researcher in interpreting the results, and often elicits rich qualitative information about unique aspects of the subjective perspective of each respondent (Davis & Michelle, 2011; Schrøder & Kobbernagel, 2010). However, eliciting rich insight is somewhat more problematic when conducting Q studies online, since it relies entirely on respondents' willingness to offer further comments, and some may prefer not to do so – although in our experience the vast majority do make additional remarks. The online Q researcher is also less able to gauge respondents' reactions to particular items and to the process of ranking the statements while the research is in process. But as we shall discuss further below, online Q sorting also has some distinct advantages, and future technological enhancements may offer ways of overcoming these kinds of problems.

The full set of individual Q sorts is then analyzed to identify associations between respondents on the basis of their rank-ordering of the Q items. A matrix of correlations among the Q sorts is created which is then factor-analyzed – by person and with reference to the entirety of their arrangement of the Q items, not by discrete items – to identify those respondents whose Q sorts are significantly similar, in that they independently ranked and arranged the same items in much the same way on their own grid. In this way, it becomes possible to systematically identify and describe the range of distinct viewpoints, perspectives, or orientations that are shared by different groups within the sample (see Brown, 1994; Dziopa & Ahern, 2011; Stainton Rogers, 1995; Watts & Stenner, 2012). While it may sound complicated to the uninitiated, there are now dedicated Q methodology software programs available that are relatively user-friendly and handle much of the serious number crunching. PQMethod is available for free, while PCQ, a commercial software package, is also widely used (see the Q method website at www.qmethod.org for hyperlinked resources). PCQ and PQMethod handle the statistical computations that occur during factor analysis, and provide various factor solutions while enabling the researcher to perform additional tests on the data. Further information about these analytical procedures can be found in McKeown and Thomas (1988) and Brown (2006).

The researcher then interprets these factors by synthesizing all of the data to “tell the story” of how those individuals who loaded significantly on each factor commonly responded to the condition of instruction. This interpretation considers areas of strong agreement, disagreement, and neutrality, and notes points of similarity and difference between the factors. The quality of the interpretation thus largely depends on the researcher drawing on his or her accumulated knowledge and expertise to consider the significance of how respondents have ranked the different statements in relation to each other, and in this they are guided by the respondents' own explanations of why they most strongly agreed or disagreed with particular items. A written narrative encapsulating the defining features of each perspective is produced for each factor, which usually incorporates examples of participants' explanatory comments as additional supporting evidence for the validity of the interpretation, and to add depth and texture to the analysis. Such statements help restore some of the “discursive meat” to the “analytical skeleton supplied by the factor analysis” (Schrøder & Kobbernagel, 2010, p. 126). What emerges from this process is a more holistic representation of a finite range of complex, multifaceted, yet still coherent viewpoints on the topic under investigation. This is one of Q methodology's key strengths, and we believe it makes the Q method particularly suitable for research investigating the subjective perspectives, beliefs, experiences, and practices of media producers, audiences, consumers, and (prod)users.

Utilizing Q Methodology in Media Research

Indeed, although Q methodology remains on the margins of mainstream media studies, a growing number of scholars are using this method in studies of media production, consumption, interpretation, and use (see Davis & Michelle, 2011). Significantly, Stephenson himself explicitly applied Q to reception analysis in advertising and screen audience studies (Stephenson, 1976, 1978, 1995–1996). To give readers a taste of the kind of work being done, consider Robinson's (1975) Q study of the attitudes expressed by upper middle-class professionals toward television and film, a group that had previously been considered unenthusiastic and critical viewers of screen media. Robinson found that, while there were some shared attitudes among the respondents, a broad-brush demographic distinction based on socioeconomic class obscured important differences in tastes and media preferences which tentatively appeared to be more specifically shaped by occupational memberships. Six distinct types of viewers were identified, including scientifically inclined “information absorbers” and more legally oriented “community leaders,” both of whom sought escape and tended not to actively interpret information from television. Other groups included “analytical artists” and “film freaks” which were highly selective and analytical in their viewing; primary loaders in these categories were university professors.

Carlson and Trichtinger (2001) used Q methodology in a study of viewers' subjective reactions to the characters and themes portrayed in an episode of the US television series Law and Order, which was a fictionalized account based on a real-life racial incident. They discovered seven distinct “readings” of the episode that appeared to engage in quite different ways with the message and thematic content of the text, reflecting the interaction between limited textual polysemy and the capacity of viewers to determine “what aspects of the conversation are heard, assimilated, and made part of a subjective political reality” (Carlson & Trichtinger, 2001, p. 270). The authors also identified suggestive links between the viewpoints and the racial and ethnic identities of participants who positively or negatively loaded on those factors.

More recently, Schrøder and Kobbernagel's (2010) study of cross-media news consumption by 35 Danish consumer-citizens identified seven types of users and found tentative correlations with age, education level, gender, and whether the respondent lived in an urban center or a provincial town. Predominant “types” included more traditional, versatile news consumers, popular culture-oriented digital news consumers, light newspaper readers, and news update addicts. The authors argue that the typology generated using Q methodology offers a more complex picture of the subtle distinctions between consumer perceptions and consumption of news media than has been suggested by other large-scale quantitative surveys.

A case study of Q methodology in action may offer readers further insight into the Q research process. In what follows, we focus on our recent research exploring cross-cultural receptions of James Cameron's 2009 movie blockbuster Avatar, taking the reader behind the scenes to offer more specific insight into this method and its capacity to illuminate the nature and content of human subjective responses, along with its ability to generate unexpected findings.

Receptions of Avatar: A Case Study in Using Q Methodology for Media Research

In this study, we were interested in understanding the nature of people's subjective responses to a film that, while generally well received and certainly extremely popular internationally, had also been critiqued on a variety of grounds. While people clearly had many specific opinions about the film, what was their overall response, in a more general and holistic sense? Did viewers in different countries have basically similar kinds of responses, or were there major cross-national distinctions? Were different responses in some way linked to aspects of viewers' location and sociodemographic attributes, or did other more relevant discursive affiliations perhaps cut across those broad-brush classifications? How well might existing theoretical understandings about the nature of audience reception predict and “capture” naturally occurring demarcations in the responses of our participants?

In the initial stages, we conducted an intensive “cultural trawl” of the wider concourse around Avatar by drawing on online professional and lay film reviews, fan website discussion boards, Facebook site message boards, international news coverage, and media commentary to gain a very broad and inclusive impression of the diverse subjective responses to Avatar within the public domain. Throughout this process, recurring patterns in the content and tone of the views expressed were noted and archetypal statements relating to the film's explicit and implicit messages, dominant and marginal themes, aesthetic quality, production values and imperatives, intellectual analysis, and emotional reactions were cut and pasted into an Excel file.

From an initial pool of 260 “raw” statements articulating recurring ideas, opinions, or reactions expressed in various different sites, it was necessary to make further selections for the Q sample. Rather than adopt a random selection process, this study used a structured sampling method grounded in a particular theory. Other Q audience studies have used different theoretical frameworks to guide the selection process, such as Holbrook's (1999) framework of consumer value (Davis & Vladica, 2010); generally the choice of model reflects the specific focus of the study. In this research, because our interest lay with charting the nature of any distinctions in the subjective responses of Avatar viewers, we drew on Michelle's (2007) composite multidimensional model of audience reception. The captured statements were thus reviewed and loosely categorized according to this particular schema, which synthesizes a range of existing conceptual frameworks and research findings, identifying broad distinctions in the subjective responses of different individuals and groups to a range of screen media texts and genres viewed in various cultural contexts. Such distinctions are reflected, for instance, in Hall's (1980) model of encoding/decoding which identifies the potential for dominant, negotiated, and oppositional readings; Liebes and Katz's (1990) analytical distinction between critical and referential readings; and recent work in media effects research on narrative transportation (see Michelle, 2007, 2009; Michelle, Davis, & Vladica, 2012).

Taking care to preserve as much of the original diversity of responses as possible, an initial cull systematically reduced the number of Q statements to 48, eliminating as much repetition of themes and content as possible. In the course of a face-to-face pilot study conducted with 24 respondents in the city of Hamilton, New Zealand (pop. 131,000), these statements were progressively re-evaluated and revised to insure each expressed relatively unambiguous ideas or themes and to eliminate further redundancy among the statements. Gradually, the statements were whittled down to the final Q sample. For each of the four modes theorized in the composite model (transparent, referential, mediated, and discursive) eight items were finally selected. The final 32 item Q sample was thus constituted with a dual purpose: (1) to reflect as much as possible the diverse range of subjective perspectives circulating around this mainstream entertainment text, and (2) to operationalize a theoretical model of the different modes of audience reception developed by one of the authors. This renders our study somewhat unusual, but also offers insight into the kind of potential Q methodology may have for exploring and refining media theory.

Participants performed their Q sorts online with the free software application Flash Q, and before submitting their responses were invited to complete a “personal details” questionnaire seeking information about their sociodemographic memberships (gender, age, nationality, ethnic identity, education level, and occupation), acquired media competencies, fandom, political affiliations, religious beliefs, and military experience. We sought this information because we were curious whether these kinds of factors might be linked to different modes of response to the film, given some of the reactions expressed in the wider concourse. We were also interested in exploring the potential uses of Q for linking modes of reception to the intersection between relevant aspects of viewers' social location.

The Avatar Q sort went live online on May 18, 2010, and was open to English-speaking participants with Internet access anywhere in the world. Immediately before carrying out their Q sort, participants read a set of instructions on screen which stated that the researchers wished to know how they responded to and felt about Avatar when they first saw it. As the condition of instruction, they were asked to “Please recall the thoughts and feelings you had immediately after watching Avatar while you rank the following statements.” After first reviewing and sorting all the statements into three piles – disagree, neutral, or agree – they were instructed to arrange the 32 statements on a continuum from “most strongly agree” on the right to “most strongly disagree” on the left in a forced normal distribution. Respondents were also invited to provide a written explanation for why they selected the four statements they placed at each end of the scale. These qualitative comments were crucial in guiding our interpretation of the subjective viewpoints that emerged, and we incorporated typical comments into our descriptions of the factors.

While conducting a Q study online has its advantages, there is some debate in the Q community about the suitability of a technologically mediated approach to studying human subjective responses – an issue to which we shall return below. In the case of our study, we specifically sought to reach an international population of respondents in order to glean some insight into the cross-cultural applicability of a particular theory, but were also constrained in terms of available resources; hence an online survey open to participants from any country seemed desirable, but it did present some challenges. One of the key difficulties we faced was accessing a suitably diverse range of participants for the P set, since presumably those most highly motivated to participate would be more likely to be Avatar fans rather than critics. Online surveys naturally attract more highly motivated respondents and are likely to over-represent the views of those who have a keen interest in the subject matter, potentially amplifying one particular perspective. To counter this tendency and obtain a more diverse P set, we actively encouraged participation from a wider range of viewers by posting invitations to participate on the “Avatar Sucks!” Facebook group message board, and on general film discussion boards such as Rotten Tomatoes. Special interest groups were also targeted via active Facebook groups such as Survival International, the Indigenous Environmental Network, Military.com, and Conservative Americans United. Invitations were also posted on key fan sites such as Avatar-forums.com, and on Facebook fan group sites such as the Official Avatar Movie group.

Despite our efforts, fans rather than critics were clearly more motivated to participate: over half of the 89 significant sorts reflected the same very positive viewpoint, with many of these participants being self-declared fans of the film. Nonetheless, we clearly did secure a number of participants who were harsh critics. Because an online survey solicits participants who are self-selecting, there is a case to be made for using somewhat larger online P sets, from which a smaller number of more representative sorts may be obtained for the factor analysis, with selections being based on particular sociodemographic characteristics, or perhaps more usefully, the stated discursive affiliations and interests of respondents in relation to the topic at hand, which is somewhat akin to the practices normally used to select a nonrandom P set for face-to-face sorting. This information can be easily obtained through the personal details questionnaire that accompanies the Q sort in programs such as FlashQ.

Since respondents were primarily recruited through online communities, our sample consisted of people who are active on the Internet. Recent estimates suggest that this now accounts for 75–90% of people in most Western developed nations (Internet World Stats, 2009). Nonetheless, online research of this nature is likely to over-represent the views of what Hannerz (1990) called Internet savvy “‘cosmopolitans’: people whose life orientation revolves around global interconnectedness rather than their local communities” (Kuipers & de Kloet, 2009, p. 104) – and these are likely also to be relatively economically empowered individuals, especially in developing nations. This bias appears to be a significant limitation of online research generally, and was evident in our Avatar sample. Aside from a large number of university students, relatively few respondents reported earning low incomes or having blue-collar occupations.

One perhaps more significant problem with conducting an anonymous Q study online relates to the problems posed for interpreting the results, given that the researcher is not present to observe participants' responses during the sorting and ranking process, and is not able to question them about any strong reactions to the items as they occur, which would normally happen in face-to-face sorting. This issue is currently hotly debated in the Q research community. Information gained from face-to-face Q sorting can be very useful – seeing when and why a participant seems unsure or has trouble deciding, or reacts strongly to particular items, can aid in interpreting the particular configuration of items for that person. However, in our study, we were more concerned with identifying the range of viewpoints within the international audience for Avatar, and unfortunately this precluded being able to observe the sorting process directly, although we agree that it is desirable to do so where possible. Such observations were, however, possible during the face-to-face pilot study, where the ability to observe and inquire about participants' verbal and nonverbal reactions proved very helpful in identifying statements that were ambiguous, unclear, or insufficiently distinct from others.

While there are related concerns within the Q community that the technological mediation of online Q sorting potentially changes and degrades the process of accessing and understanding human subjectivity in its “original” untainted state, we believe, perhaps controversially, that it may be possible to at least partially mitigate such objections by more fully utilizing modern technological developments, and particularly the now widespread availability of inbuilt web cameras in many computers and laptops. Potentially, the researcher could observe the process and even interact with the respondent live, as now frequently occurs with the use of Skype and other video conferencing tools and applications. In other words, the technology now exists to make the researcher virtually “present” in the online sorting process, if not physically so. For those studies where it is particularly important that each respondent be observed, there is the capacity to do so regardless of the participant's physical location.

Obviously, recording the sorting process in this way will likely impact participation rates, particularly when respondents are self-selected. There may also be some topics for which the anonymity and independence offered by anonymous online Q sorting might be quite advantageous, as it may lessen the incentive for participants to engage in impression management, and may also reduce any perceived embarrassment about expressing their true opinions on controversial or deeply personal topics. In other words, depending on the topic, it may be appropriate to adopt different research strategies that offer greater or lesser degrees of anonymity, and in which the researcher is more or less “present” in the process, depending on how important direct observation is deemed to be and its possible impact on respondents.

It is also important to acknowledge here one other possible effect of technologically mediated applications of Q, one that relates more to the limitations of visually presenting Q items on screen. In face-to-face Q sorting on a table, the respondent can easily see the entire array and make the necessary holistic comparisons visually. But with online Q sorting, the smaller the screen, the smaller the Q item cards, and the less text is visible on each of them; the readability of the cards is potentially compromised. This may well affect the respondent's ability to make the many-to-many holistic comparisons which are the foundation of Q methodology. This is a technological pitfall that the researcher should be aware of. Ideally, Q sorts should be performed on a large computer screen where possible. Given recent trends toward screen miniaturization it seems that in the future, online Q sorting software may need to be modified or even completely redesigned to suit different formats, while still adhering to Q sorting conventions – a potentially significant challenge.

Despite these contentious methodological issues and other difficulties encountered along the way, our results provided a fascinating insight into the clear distinctions between Avatar audiences and even between fans themselves. Factor analysis revealed four distinct subjective orientations characterized by differing degrees of emotional and cognitive involvement, as well as different foci in terms of the most salient issues and concerns for particular groups of viewers. These four viewpoints accounted for 74% of our 120 participants, and there were a number of others whose Q sorts were mixed in the sense that they bore similarities to more than one viewpoint. What was more exciting from our perspective was that these factors appeared to exhibit notable similarities to the four modes of reception theorized in the composite model of reception. Whereas Factor 1 was marked by suspension of disbelief, feelings of being transported to the amazingly realistic world of Pandora, emotional engagement with the Na'vi's plight, and agreement with preferred messages, all key aspects of the theorized transparent mode, Factor 2 exhibited estrangement and emotional detachment, with these viewers rejecting Avatar as an overcommercialized Hollywood entertainment product while also engaging critically with the film's ideological or message content – a defining feature of the discursive mode. While Factor 3 focused attention on the film's similarity and relevance to past and present struggles occurring in the real world against Western imperialism, militarism, and capitalist exploitation of natural resources, and thus appeared to be framed by a referential mode, Factor 4 related primarily to Avatar as a constructed entertainment media product which was aesthetically pleasing and technologically remarkable, but had significant shortcomings in terms of script and storyline – a response that is fairly typical of those adopting a mediated mode of reception. Each viewpoint appeared to reflect a very different subjective approach to making sense of the same film, and focused on different levels of denotative and connotative meaning, as theorized in the composite model of reception.

Two of the viewpoints had inverted or bipolar versions, in that a few participants reacted to the same kinds of ideas and themes in diametrically opposed ways to most others. For instance, while most of those who loaded significantly on Factor 3 saw the film as essentially quite realistic and true to life in the sense of mirroring processes of colonial and military oppression and the exploitation of indigenous peoples or the environment here on earth, a few reacted most strongly against this aspect of the film, and considered its depiction to be historically inaccurate and highly unrealistic, even a distortion of real-world events. In theoretical terms, both can be considered variations of the same referential mode of response, since both viewpoints primarily rely on the viewer making assessments of textual realism with reference to their knowledge and understanding of the real world outside the text.

Another key finding was that those who had a very positive view of Avatar did not all have the same response to the film. While some were “swept away” to another world and felt deeply emotionally engaged with the plight of the Na'vi, others responded positively to the film as a realistic reflection of real-world events, and still others were most impressed by the technical and aesthetic features of the film, especially the computer-animated visual effects, but remained more emotionally detached. This finding is important as it offers insight into the complex nature of audiencing, which is not necessarily premised on the same kinds of experiences and emotional, aesthetic, and intellectual responses, but rather can take diverse forms and may focus on radically different aspects of the same media text (see Michelle et al., 2012, for full details).

We were also able to discern some suggestive associations between particular viewpoints and shared sociodemographic characteristics such as gender and ethnicity, as well as other suggestive influences grounded in life experiences and discursive affiliations, including level of education, religious belief, political activism, and military service (see Michelle et al., 2012). Namely, those who read the film in referential or discursive terms were more likely to be politically active professionals and to identify themselves as a member of an ethnic minority, while those who adopted a mediated mode were more likely to be male, not an ethnic minority, and politically inactive. These particular variables appeared to cut across any broader cultural or national differences. Nevertheless, these associations did not meet the threshold for statistical validity normally required of quantitative research. Q methodology can pose significant limitations in this regard, particularly when such studies are conducted face to face, as the number of participants in a Q study is usually too small to permit identification of statistically valid relationships between viewing positions and other variables (Danielson, 2009). Furthermore, the results of a Q study cannot be generalized to the general population, as the P set is not representative of that population. Any findings are therefore specific to the sample of individuals studied. We will return to this issue later, as the question of external validity and generalizability of Q findings warrants further discussion. We would also note that there are various innovations currently occurring in online Q sorting and analysis of larger P sets that are likely to expand avenues for research that might well be able to generate statistically significant relationships between viewpoints and aspects of social location. This issue is also one we will address in greater detail below.

For those readers who are interested in how a Q study is written up, examples can be found in studies by Lindlof, Coyle, and Grodin (1998) on science fiction fans, Carlson and Trichtinger's (2001) study of responses to a racial incident in the television crime drama Law and Order, our own recent study (Davis & Vladica, 2010) which examines responses to an animated psychorealist documentary, and our Avatar reception project (Michelle et al., 2012). There have also been recent efforts to standardize and develop agreed procedures for conducting and reporting Q research, in an effort to enhance the validity of what is still a relatively new research method (see Dziopa & Ahern, 2011). In many studies, but not all, the written narratives that are constructed to represent each of the viewpoints identified through factor analysis incorporate some of the qualitative responses of those who loaded highly on each factor, as a way of illustrating the kinds of comments that tended to be made by group members in response to highly ranked Q items. As yet, there is no established process for selecting these comments. In our case, we sought to include remarks that best encapsulated or articulated the underlying sentiment that appeared to characterize that particular factor, or that illustrated most effectively a particular point that this group of respondents seemed to emphasize. For example, here is a brief excerpt from the analysis of Viewpoint 1 from our Avatar research, as reported in Michelle et al. (2012):

Here, we can see that the incorporation of qualitative data usefully illuminates and extends what would otherwise be rather dry reporting of aggregated findings, conveying little sense of the unique voices of individuals who nonetheless shared the same general subjective response to this film. It is this ability to convey something of each participant's individual perspective, as related in their own words, that some qualitative researchers perhaps fear losing through the reduction of rich expressions of human experience to quantitative factors. However, as we hope this example demonstrates, Q research can preserve at least some of that richness, while also offering a more systematic basis for the selection of qualitative data for inclusion – although such data obviously cannot take center stage. Further, while most Q studies present aggregated results, it is possible to focus intensively on individual responses to multiple Q sorts (Brown, 1993), or to examine prevalent viewpoints from the perspective of any particular individual participant, including those whose viewpoint may be regarded as of theoretical significance (for example, the author of a text), or even of individuals who do not load on any one factor, and thus have a very different perspective to any other participant.

What is Q Methodology's Potential Contribution to Answering Key Questions in Media Studies?

Q methodology is ideally suited to identifying commonalities as well as differences among the subjective perspectives, beliefs, experiences, and practices of different groups of media producers, audiences, consumers, and users. This is, after all, precisely what the method was intended to do. But we believe Q methodology may also prove very useful in efforts to address, in particular, the link between media reception, consumption, and use, and the location of individuals within the wider social structure; something that it is not designed to do, but could help to do, when used in combination with other methods.

Most media researchers now take for granted that media reception, consumption, and use are social activities shaped by specific and diverse contexts. Because it is inherently social and contextual, it seems reasonable to assume that our engagement with media should also be patterned at the levels of social group membership(s), discursive affiliations, and personal and collective experiences, in ways that produce distinct groups of individuals who share certain similarities in terms of how they receive, engage with, and use media. In the field of audience reception studies, for instance, there is now a considerable body of research exploring this social patterning of interpretation and response (seminal work includes Liebes & Katz, 1990; Livingstone & Lunt, 1994; Morley, 1980; Press, 1989, 1991). However, because most audience research, at least in the cultural studies tradition, has relied on qualitative methodologies such as ethnographic observation, interviews, and focus groups and thus (with a few exceptions) been relatively small-scale, it has been very difficult to generate clear and generalizable evidence of the kinds of relationships that are theoretically assumed to exist.

In this respect, one of the most important potential contributions of Q methodology for media studies lies in the way it provides a robust foundation for linking shared subjectivity to aspects of social location in a methodologically rigorous and structured way. Q methodology has particular potential to help the researcher identify whether certain “types” of media audiences, consumers, or users share similar orientations or attitudes toward media products based on actual similarities in their responses, rather than any preassigned categorization of individuals determined by what the researcher assumes will be the most salient aspects of shared identity or group membership – as typically occurs in focus group research, for example. Using the responses to a separate questionnaire seeking information about aspects of respondents' social location, sharing a generalized viewpoint can potentially be linked to other commonalities in terms of the socio-demographic characteristics, cultural competencies, experiences, and discursive affiliations of those who loaded on that factor. Furthermore, since identification of any common characteristics takes place “after the fact,” working backwards from the factor loadings to identify shared attributes between group members, it may be possible to isolate the particular aspects of identity, or the combination of social group memberships, that are linked to someone sharing a particular viewpoint or set of experiences with others. This process would provide a very robust and potentially fruitful means of establishing any correlations between social location and media consumption, reception, and use. For critical media scholars interested in exploring issues relating to media power and social inequality, a method of reliably identifying these kinds of correlations has thus far proven elusive. Q methodology, we believe, may be an important part of the solution to this problem.

There are various other advantages to using Q methodology, given the particular nature and strengths of this approach. Certainly, Q methodology's ability to map and systematically compare human subjectivities while preserving and apprehending the complexity of each individual's own multifaceted response is an advantage unmatched by existing quantitative approaches that seek to measure strength of association among variables or attributes. As Schrøder and Kobbernagel (2010, p. 123) suggest, the problem with quantitative questionnaires lies in “the atomized manner in which respondents answer the list of unrelated questions, thereby violating the coherences, interconnections, ambivalences and contradictions that are an inherent part of everyday life.” While the semantic differential method does more directly address human subjectivity, it similarly does so “in an analytic, not a synthetic context” (Stephenson, 1985, p. 238).

At the same time, the fact that Q utilizes factor analysis allows it to circumvent one of the major shortcomings of much qualitative research, in terms of “the opacity of its interpretive procedures of analytical generalization” (Schrøder & Kobbernagel, 2010, p. 123; see also Schrøder, 2011). While qualitative research can offer richly detailed insight into the subjective quality and content of specific encounters between media texts and audiences, consumers, and/or users, it can prove difficult to “reliably and validly generalize analytically” about similarities and differences within the very large and complex body of data that is often generated (Schrøder & Kobbernagel, 2010, p. 123). Q methodology, on the other hand, makes it “possible to incorporate and preserve qualitative data through a process of quantification” (Schrøder, 1987, cited in Schrøder & Kobbernagel, 2010, p. 123).

A related issue pertains to the way in which the qualitative researcher, faced with often very large amounts of richly detailed data, must attempt to impose some kind of order and structure on the material, and normally develops or applies a schema according to which data is selected and categorized, depending on their intended focus. Some data, perhaps even a significant amount, is often left out or discarded in this process – as we ourselves have both experienced and witnessed in conducting, reviewing, and supervising numerous qualitative research projects. It is virtually impossible to include everything that may or may not be relevant, and as in all research, whether qualitative or quantitative, the researcher's subjective perspective on the topic shapes various aspects of the research process to some degree. In qualitative studies, one potential consequence of this inevitable subjective influence is a degree of uncertainty about whether the categories identified reflect “natural” and purely inductively derived, or rather partially subjectively constructed, distinctions among participants (Brown, 1996; Cross, 2005). The factors that arise from a Q study, on the other hand,

Another key strength of Q methodology is that it is explicitly designed to preserve the integrity and richness of individual participants' subjective viewpoints, and because of this is perhaps less likely to produce findings that discount or minimize difference and diversity. Indeed, there are grounds to suggest Q methodology renders such diversity more visible and comprehensible in its full complexity (Stainton Rogers, 1995). Traditional quantitative surveys, for instance, are unable to capture subtle distinctions between the views of different participants; it is often very difficult to discern whether those who registered similar levels of agreement or disagreement with a particular item did so for the same reasons (Cross, 2005). With Q, it is possible to assess specific responses within the context of participants' broader subjective perspective, and to consider their response to one item in light of their responses to all others, as well as their qualitative explanatory responses, resulting in a more holistic interpretation. And while traditional qualitative methods such as focus groups, which are commonly used in media research, potentially allow for fulsome expressions of agreement and disagreement, practical difficulties arise when attempting to extensively probe all individuals to solicit more detailed and nuanced explanations of their own beliefs and opinions and then to construct rigorous comparisons between viewpoints. Nor it is easy to gauge and record individual responses to a constellation of related issues or questions simultaneously. On both counts, Q offers significant advantages over other methodological approaches.

One final key benefit relates to the potential applications of Q methodology for cross-national media research. An ongoing problem in qualitative research is the need to find a way to make more comparable observations that support stronger, testable generalizations that could potentially inform the development of theoretical understandings (Barker, 2006) of the nature of media production, reception, consumption, and use per se. It is particularly difficult to make linkages and generalizations across different qualitative studies conducted in relation to different texts and in different contexts (Schrøder, 2011). However, with Q methodology, the standardization of methodological and analytical approaches potentially offers a unique means of addressing some of the challenges of cross-genre as well as cross-national comparative research, as identified by Livingstone (2003). This is because, as Schrøder (2011, p. 22) argues,

Potentially, parallel multilingual Q samples could be constructed and administered relatively easily in FlashQ, and we are aware of one successful trilingual Q study. Adapting and expanding the capacity of Q methodology in this way, it may become possible to “share and compare qualitative data across national contexts” (Schrøder, 2011, p. 22). Given limited resources for the Avatar project, we were not able to undertake a large-scale cross-national study using multiple languages, but it became clear to us that such a study would be entirely feasible.

Hence, we are currently engaged in what we believe may be the largest Q study to date, which will draw together an international team of researchers to explore audience receptions of Peter Jackson's feature film The Hobbit: An Unexpected Journey (2012), using parallel Q samples in multiple languages. In this project, we will delve much deeper into the potential of Q methodology to productively fuse the more nuanced interpretative detail generated through qualitative methods with robust quantitative techniques that make it possible to identify broader patterns within qualitative data, and to reveal meaningful correlations between individuals, shared viewpoints, sociodemographic variables, and aspects of social location. By using very large P sets (at least in Q terms – in the range of several hundred respondents), we believe it may be possible to construct smaller representative samples from a range of theoretically relevant categories or combinations of categories, and to produce typal Q sorts that represent any significant shared viewpoints we find. In this way, it may become possible to describe and compare the viewpoints of younger versus older viewers, of fans of the Lord of the Rings trilogy versus aficionados of J. R. R. Tolkien, and to explore patterns of response between and within different national or ethnic communities, religions, genders, and socioeconomic classes. While this is a very new area of exploration, we believe that innovative adaptations of Q methodology potentially offer critical media scholars analytical capacities beyond those normally derived through the use of qualitative or quantitative methods in isolation from each other.

For these and other reasons, we believe that Q methodology is particularly well suited to the study of media production, reception, consumption, and use. It also has considerable potential to expand the contribution, relevance, and generalizability of in-depth qualitative investigations, while informing larger-scale surveys and complementing close textual analyses with data relating to production, reception, or both. Proponents also emphasize the ability of Q methodology to test theoretical assumptions and make new and unanticipated discoveries through the use of factor analysis (Brown, 2006; Stephenson, 1995–1996). There are, however, some important limitations to this methodological approach.

Q methodology's primary focus is on identifying and interpreting the subjective viewpoints and experiences of nonrandomly sampled participants, and it is commonly regarded as an exploratory, interpretation-intensive methodology that is primarily suitable for use with quite small numbers of respondents (Brown, 1996; Cross, 2005). On its own, Q methodology cannot be used to identify statistically significant causal relations between viewpoints and other variables; nor can the findings of a Q study be generalized to estimate the distribution of the identified perspectives within the wider population, as it does not use statistically representative P samples (Dziopa & Ahern, 2011; McKeown & Thomas, 1988; Stainton Rogers, 1995). In order to examine the distribution of subjective viewpoints within the general population and their possible link to sociodemographic or other attributes, it would be necessary to use the results of a clearly defined and robust Q study as the basis for a larger-scale representative survey in which random sampling is used.

This can be achieved in a few different ways, for instance through the creation of Likert items based on the distinguishing elements associated with each factor, which could then be used to measure the prevalence of perspectives within the wider population (Danielson, 2009). Alternatively, the researcher could use the method of narrative evaluation, where the factors identified through a Q study are transformed into short narratives that a representative sample are asked to rank and respond to in terms of their level of agreement or disagreement (Danielson, 2009). A representative survey based on a preliminary Q study would make it possible to both ascertain the external validity of the identified viewpoints, and independently link viewpoints to variables such as culture, ethnicity, gender, age, socioeconomic class, education, religion, and political belief, thereby addressing a key area of ongoing research and debate within media studies. Q methodology could thus form a key component of mixed methods research, perhaps serving as an essential bridging or translation device between in-depth qualitative interviews and survey research, in a way that makes it possible to combine qualitative and quantitative methods more productively.

One additional problem relates to what Schrøder and Kobbernagel (2010) refer to as the “validity cost” that arises from the participants having to reduce their complex and multifaceted discursive response to forced choices in the ranking of Q items, the selection of which, some contend, is open to subjective biases and also potentially limits or predetermines the range of possible responses that can be expressed (see also Cross, 2005; Dziopa & Ahern, 2011; Stainton Rogers, 1995). While this is true to a certain extent, the process of questioning participants about their selections can potentially offer insight into themes or topics that are not explicitly addressed in the available Q items. Another issue arises in that since the interpretation of the factors relies on the researcher's own analytical skills, existing understandings, and intuition, there is a potential risk of subjective biases at this stage, particularly when the researcher seeks to draw conclusions about the data (Cross, 2005; Stainton Rogers, 1995). This particular concern is not, however, specific to Q methodology, and is a problem that may arise in the interpretation of even purely quantitative data.

Conclusion

In sum, Q methodology offers considerable but still largely unrealized potential for media studies, and is particularly suited to research aimed at uncovering broad patterns, similarities, and differences in the subjective perspectives and experiences of media producers, audiences, consumers, and users. Q methodology may also have potential utility for testing general propositions about the nature and form of media engagement, consumption, and produsage. Certainly, our own research suggests Q offers considerable insight into specific areas of commonality and divergence within the receptions of audience members located in diverse national and cultural contexts; it also provides more robust insight into the likely basis for such variations in audience engagement and response, and clarifies pathways for future research. Q methodology is unparalleled in terms of its capacity to systematically identify and describe similarities and variations in human subjective viewpoints. Recent technological developments in terms of online Q sorting and computational software also make this method accessible to increasingly fragmented and dispersed media audiences and users, as well as to researchers who would otherwise not consider using a qualitative research approach “fortified” by factor analysis. Indeed, it seems that a method such as Q, which is adaptable to an online environment when warranted and which draws on the strengths of both qualitative and quantitative traditions, offers significant promise in terms of enabling emerging scholars to grapple more effectively with the complexities of media production, reception, consumption, and produsage in an increasingly convergent and pervasive digital media age.

NOTE

1 As shown in Figure 4.1, “+4” refers to the aggregate ranking given to this statement by this group of respondents, who “most strongly” agreed with the sentiment it expressed.

REFERENCES

Barker, M. (2006). I have seen the future and it is not here yet . . .; or, on being ambitious for audience research. Communication Review, 9, 123–141.

Bondebjerg, I. (1988). Critical theory aesthetics and reception research. Nordicom Review, 1, 21–32.

Brown, S. R. (1977). Political literature and the response of the reader: Experimental studies of interpretation, imagery, and criticism. American Political Science Review, 71(2), 567–584.

Brown, S. R. (1986). Q technique and method: Principles and procedures. In W. D. Berry & M. S. Lewis-Beck (Eds.), New tools for social scientists: Advances and applications in research methods (pp. 57–76). Beverly Hills, CA: Sage.

Brown, S. R. (1993). A primer on Q methodology Operant Subjectivity, 16, 91–138.

Brown, S. R. (1994). Q methodology as the foundation for a science of subjectivity Operant Subjectivity, 18, 1–16.

Brown, S. R. (1996). Q methodology and qualitative research. Qualitative Health Research, 6(4), 561–567.

Brown, S. R. (2006). Q methodology and naturalistic subjectivity. In B. D. Midgley & E. K. Morris (Eds.), Modern perspectives on J. R. Kantor and interbehaviorism (pp. 251–268). Reno, NV: Context.

Brown, S. R. (2008). Q Methodology. In L. M. Given (Ed.), The Sage encyclopedia of qualitative research methods (pp. 699–702). Thousand Oaks, CA: Sage.

Brown, S. R., & Ungs, T. D. (1970). Representativeness and the study of political behavior: An application of the Q technique to reactions to the Kent State incident. Social Science Quarterly, 51, 514–526.

Bruns, A. (2008). Blogs, Wikipedia, Second Life, and beyond: From production to produsage. New York, NY: Peter Lang.

Carlson, J. M., & Trichtinger, R. (2001). Perspectives on entertainment television's portrayal of a racial incident: An intensive analysis. Communication Review, 4(2), 253–278.

Cross, R. M. (2005). Exploring attitudes: The case for Q methodology Health Education Research, 20(2), 206–211.

Danielson, S. (2009). Q method and surveys: Three ways to combine Q and R. Field Methods, 21(3), 219–237.

Davis, C. H., & Michelle, C. (2011). Q methodology in audience research: Bridging the qualitative/quantitative “divide”? Participations: Journal of Audience & Reception Studies, 8(2), 527–561.

Davis, C. H., & Vladica, F. (2010). Consumer value and modes of media reception: Audience responses to the computer-animated psychorealist documentary Ryan and its own documentation in Alter Egos. Palabra Clave, 13(1), 13–30.

Durning, D. W., & Brown, S. R. (2007). Q methodology and decision making. In G. Morcol (Ed.), Handbook of decision making (pp. 537–563). New York, NY: CRC.

Dziopa, F., & Ahern, K. (2011). A systematic literature review of the applications of Q-technique and its methodology. Methodology, 7(2), 39–55.

Fiske, J. (1987). British cultural studies and television. In R. Allen (Ed.), Channels of discourse: Television and contemporary criticism (pp. 254–290). London: Methuen.

Grosswiler, P. (1992). Some methodological considerations on the use of multimedia Q-sample items. Operant Subjectivity, 15, 65–80.

Grosswiler, P. (1997). A Q methodology study of media and ideology orientations: Exploring medium theory, critical theory and cultural studies. Canadian Journal of Communication, 22(2), 1–17.

Hall, S. (1980). Encoding/decoding. In S. Hall, D. Hobson, A. Lowe, & P. Willis (Eds.), Culture, media, language (pp. 128–138). London, UK: Hutchinson.

Hannerz, U. (1990). Cosmopolitans and locals in world culture. Theory, Culture and Society, 7(2–3), 251–273.

Holbrook, M. (1999). Consumer value: A framework for analysis and research. London, UK: Routledge.

Internet World Stats. (2009). Internet usage statistics: The Internet big picture: World Internet users and population statistics. Retrieved July 15, 2013, from http://www.internetworldstats.com/stats.htm

Kuipers, G., & de Kloet, J. (2009). Banal cosmopolitanism and The Lord of the Rings: The limited role of national differences in global media consumption. Poetics, 37, 99–118.

Liebes, T., & Katz, E. (1990). The export of meaning: Cross-cultural readings of Dallas. New York, NY: Oxford University Press.

Lindlof, T., Coyle, K., & Grodin, D. (1998). Is there a text in this audience? Science fiction and interpretive schism. In C. Harris & A. Alexander (Eds.), Theorizing fandom: Fans, subculture and identity (pp. 219–247). Cresskill, NJ: Hampton.

Livingstone, S. (2003). On the challenges of cross-national comparative media research. European Journal of Communication, 18, 476–500.

Livingstone, S., & Lunt, P. (1994). Talk on television. London, UK: Routledge.

McKeown, B. F., & Thomas, D. B. (1988). Q methodology. Quantitative applications in the social sciences series, vol. 66. Newbury Park, CA: Sage.

Michelle, C. (2007). Modes of reception: A consolidated analytical framework. Communication Review, 10(3), 181–222.

Michelle, C. (2009). On (re)contextualising audience receptions of reality TV. Participations: Journal of Audience & Reception Studies, 6(1), 137–170.

Michelle, C., Davis, C. H., & Vladica, F. (2012). Understanding variation in audience engagement and response: An application of the composite model to receptions of Avatar (2009). Communication Review, 15(2), 106–143.

Morley, D. (1980). The “Nationwide” audience. London, UK: British Film Institute.

Press, A. (1989). Class, gender and the female viewer: Women's responses to Dynasty. In M. E. Brown (Ed.), Television and women's culture: The politics of the popular (pp. 158–181). London, UK: Sage.

Press, A. (1991). Women watching television: Gender, class and generation in the American television experience. Philadelphia, PA: University of Pennsylvania Press.

Rhoads, J. C. (2009). Immediate reactions to a film: The Believer. Journal of Human Subjectivity, 7(1), 61–79.

Robinson, D. C. (1975). Television/film attitudes of upper-middle class professionals. Journal of Broadcasting & Electronic Media, 19(2), 195–209.

Schrøder, K. C. (1987). Convergence of antagonistic traditions? The case of audience research. European Journal of Communication, 1(2), 7–31.

Schrøder, K. C. (2011). Audiences are inherently cross-media: Audience studies and the cross-media challenge. Communication Management Quarterly, 18(6), 5–28.

Schrøder, K. C., & Kobbernagel, C. (2010). Towards a typology of cross-media news consumption: A qualitative-quantitative synthesis. Northern Lights, 8, 115–138.

Stainton Rogers, W. (1995). Q-methodology. In J. Smith, R. Harre, & L. Van Langenhove (Eds.), Rethinking methods in psychology (pp. 178–207). New York, NY: Sage.

Stenner, P., Watts, S., & Worrell, M. (2008). Q methodology. In C. Willig & W. Stainton-Rogers (Eds.), The Sage handbook of qualitative research in psychology (pp. 215–239). Los Angeles, CA: Sage.

Stephenson, W. (1953). The study of behavior: Q-technique and its methodology. Chicago, IL: University of Chicago Press.

Stephenson, W. (1976). Q-methodology: Conceptualization and measurement of operant effects of television viewing. JCATS: Journal of the Centre for Advanced Television Studies, 4, 17–18.

Stephenson, W. (1978). Applications of communication theory IV: Immediate experience of movies. Operant Subjectivity, 1, 96–116.

Stephenson, W. (1985). Q methodology and English literature. In C. R. Cooper (Ed.), Researching response to literature and the teaching of literature: Points of departure (pp. 233–250). Norwood, NJ: Ablex.

Stephenson, W. (1995–1996). Quantum theory media research: Reception analysis. Operant Subjectivity, 19(1–2), 1–11.

Watts, S., & Stenner, P. (2012). Doing Q methodological research. Los Angeles, CA: Sage.