1Introduction: where media texts are found

Media texts have changed considerably in recent years. Passive information intake is no longer predominant; on the contrary, the media today are all about active participation, whether through interactive TV shows or social media platforms on the Internet. Similar to the behavior of the recipients moving towards participation or even production, media texts have developed further in how and where they appear. They may contain different components: (moving or static) images, verbal language (written or spoken), colors, symbols, emoticons. The list could be extended in many respects. The co-occurrence of images and language can be described as multicodal (cf. 2.4). This form of multicodality will be at the center of my further deliberations.

Let us consider, as examples, the online text types online forum and blog and audiovisual media texts in television and cinema. Just as in a blog, the contributions made to an online forum, serving the communication exchanges between users of a platform, consist as a rule of verbal signs (such as text) and nonverbal signs (images and emoticons). The latter can additionally be moving or static, for instance, hopping about or waving to the user. The inclusion of short video sequences is also possible in some forums. Not only nonverbal signs, but also verbal ones offer the user a whole range of design opportunities: the verbal text can be formatted in some forums and can thus be altered according to preferences or to emphasize the content (cf. Fröhlich 2015). In addition to italics, bold type and underlining, this may include the size, font and color of the verbal (graphic) signs. Therefore, blogs and forums are consequently both varied in their appearance and often actively variable in their design on the part of the user. Depending on the content of the forum or blog, the content may consist more of verbal text parts and additionally integrate nonverbal ones, although communication by image alone is also possible.

Images or pictures as a means of communication are not a novelty: they have long been used for communication, e. g., in cave painting. One characteristic of the modern “visual era” is the growing number of images and their frequent presence (cf. Lobinger 2012, 20), above all on the Internet. The information intake is carried out by way of the visual sensory channel. We cannot assume that a user has necessarily activated the loudspeakers of their Internet-enabled device (e. g., a laptop, a tablet; cf. Fröhlich 2015). Videos in forums are sometimes included with no sound and run without any user intervention on an endless loop. Thus, above all in online forums, images and videos often have an effect purely through their visual appearance.

When focusing on films or TV programs in this respect, we find a whole selection of appropriate material in these media texts. On the other hand, because the latter are usually invariable from the audience point of view, we cannot talk of TV users, but rather of passive viewers – insofar as it is not a question of an interactive TV show. What we see is, on the contrary, usually dominated by moving images, although verbal text can be present, fulfilling in most cases an additional function. The information in film and on television is passed on via the visual and auditive sensory channels.

To sum up, media texts in forums and blogs, compared to films and TV programs, are similar concerning their codes, but the dominance of the verbal and nonverbal parts varies, with the result that certain key aspects of research come to the forefront.

Images dominate in the cinema and on TV; any further investigation would therefore need an analysis of their visual content. In contrast, verbal language signs may dominate the images in online media texts and relegate them to the background – depending on the aims of the texts. The question is whether important information, although relevant for the aims, might be overlooked in a restrictive analysis of particular codes in media texts. Texts consisting of different signs transport their (whole) meaning through the totality of all the signs. This is the reason why this contribution illustrates the necessity of the holistic multicodal analysis of online media texts.

In the following, indications are given of how one may approach multicodal research into online media texts. Initially, I will go into the terms mode and code and explain the possible limitations of the concepts. Secondly, I will examine the image as a scientific concept. This will be followed by a concise presentation of the disciplines which deal with the image in partly differing ways. Subsequently, I will ask what we basically understand as image or picture. Further, I will describe my understanding of a multicodal text concept, in order to demonstrate, with the aid of an example of analysis sequence, why images as a component of a whole text should be considered as an important object of analysis.

2Mode and code

When working on code in linguistic research, it is impossible to avoid the widespread term mode or modality. Modus (plural: modi) can describe a procedural method. In linguistics, the term refers to the morphological, syntactic verb forms: indicative, subjunctive or imperative. From an information science viewpoint (e. g., Weidenmann 2002), mode and code refer to different fields of knowledge. Taking selected authors, I would like to take a closer look at different definitions of mode and code.

The two terms mode and code become blurred in use according to the branch of study and the author, as pointed out by, among others, Weidenmann (2002, 46), Schneider/Stöckl (2011, 25), Klemm/Stöckl (2011, 14s.). That makes a conceptual delineation of mode and code all the more important.

Fraas/Meier/Pentzold (2012, 56) thus consider images and language to be differently coded. The authors see the aim of the forms in the paraverbal code, such as bold type or capital letters, as “creating order” (Fraas/Meier/Pentzold 2012, 57). Following the subdivision in Weidenmann (2002), Fraas/Meier/Pentzold (2012, 64s.) initially also go along with the division into code, the type and arrangement of the signs, and mode, with a view to the sensory modalities which are employed for the perception of the signs. In social semiotics, especially through the research work of Kress/van Leeuwen (e. g., 2001), another understanding of mode and code is put forward, in which mainly the term mode is used.

In the following, the different characteristics are examined. Approaches which use these concept traditions in a slightly altered form are presented.

3The image in the world of science

Several research institutions work on images as the object of their investigations. Some pertinent disciplines are image science, visual communication research, visual culture and image linguistics.

Image science pursues the aim of “giving a general introduction to a basic understanding of pictorial phenomena”103 (Lobinger 2012, 33). It is not confined to particular pictorial media (ibid.). Researchers in this field in part come from the history of art or philosophy.

In contrast to image science, research in visual communication deals with specific pictorial media. The research focuses on, for instance, “spezifische(n) Differenzen zwischen einzelnen Bildmedien” [‘specific differences between individual pictorial media’] (Huber 2004, 16).

In the Anglo-American world, the reception of images is above all discussed within the research area of visual culture. The central issue is the relationship between the viewer of the image and the image itself.

Image linguistics arose in linguistics research that was carried out into the connections and references between language and image (cf. Klemm/Stöckl 2011, 9). For many years, linguistics did not concern itself with the image as an object of research (cf. Klemm/Stöckl 2011, 8). Multicodality can be adduced as an argument for the strengthening of this discipline. Images are used to communicate and/or often refer to language utterances, in the same way as language utterances can be supported or distorted by images (cf. Fröhlich 2015).

4The multicodal concept of text

With the help of culturally shared rules and processes, we can attribute a social and communicative sense to acts. “Human communication of all kinds is symbolic, interactive and social action”110 (Klemm 2011, 187). In terms of linguistic pragmatics, Klemm (2011) comments:

“Communicative action can be interpretatively attributed to action patterns or (in the terminology of speech act theory) “illocutions”, e. g., CLAIMING or WARNING. Actions are not, however, objective data” (Klemm 2011, 187; emphasis in the original).111

Based on Lenk (1978), he speaks of actions as “Interpretationskonstrukte” [‘interpretation constructs’] (Klemm 2011, 187), i.e., an action may lead to several, different interpretations. Additionally, speech acts can be carried out by means of different codes or sign systems (e. g., language and images). If several sign systems are used for the communication, then multicodal communication takes place (cf. Weidenmann 2002). For Klemm (2011, 187), this is the normal case. Similarly, online media texts can also contain multicodal communication. In the following, I shall be examining the question of what characterizes the textuality of these texts.

In order to classify online texts, it could be helpful to distinguish them from print texts. The textuality of online texts is, for example, for Pentzold, Fraas and Meier

“not only indicated and made visible by their linguisticality, but in relation to the updated action and sense horizon and to the implemented, resp. perceived, textual, paratextual and paraverbal aspects they are treated as texts” (Pentzold/Fraas/Meier 2013, 83).112

The possibility of updating the content is thus decisive for online media texts. It also becomes clear that the text-ness of online contributions or platforms is not exclusively confined to the verbal text (e. g., a posting), but that also the peritext (cf. Maaß 2014, referring to Genette 1987), like, for instance, information on the user and their status or paraverbal aspects (like the font, color and size of the print) plays an important role (cf. Fröhlich 2015). There are, too, depending on the individual contribution, the nonverbal elements such as (moving) images, smileys or emoticons. Compared to print texts that have appeared offline, online media texts can thus provide much information stemming from their “environment” which can be updated. On the basis of the textuality of online media texts that has been described above, the latter may be subsumed in a broad text concept which comprises different codes.

I consider not only images and emoticons, on the one hand, but also (verbal) language, on the other, as distinct codes, because they are built on different sign systems (cf. Weidenmann 2002). All information transmitted by different codes may in its entirety be considered as online media text. In consequence, we may speak of multicodal texts consisting of verbal, paraverbal and nonverbal signs and characterize these as a “Komplex aus Zeichen verschiedener Art” [‘aggregate of different kinds of signs’] (Fix 2011, 76; cf. also Weidenmann 2002). Through these different and partly intrareferential codes, it is likewise not possible to explain text comprehension alone with the verbal content. Multicodality has to be ascribed an important role here.

Multicodal text comprehension can be described as “the mutual activation and shaping of the meaning potentials of language and images with an aim of reconstructing a relevant, logical and minimal comprehensive message in context”113 (Stöckl 2011, 55). Comprehension is thus the result of cognitive and perceptual processes “which are based on language-image harmonization and a context-sensitive attribution of meaning”114 (Stöckl 2011, 55). We see that the interaction of the codes in use is decisive for multicodal text comprehension.

The following example illustrates multicodal text elements.

Elodie has been posting on her blog for many years. She lives in Paris and loves the world of fashion. She writes about fashion, beauty and lifestyle in her blog “Elodie in Paris”. She sees fashion as a continually unfolding new beginning:

“Analyser les tendances, les couleurs, faire attention aux moindres détails, anticiper, même si, en soit, la mode est un éternel recommencement…”115 (Elodie in Paris).

As may be recognized through the example of this blog, this online media text contains not only verbal signs (script) but also paraverbal (bold type, italics, underlining) and nonverbal signs (image116, a small heart and a star) and is consequently a multicodal (media) text. In the next subsection, I will be taking a closer look at the individual codes and making clear, through the interrelationship of the codes to one another, where the strengths of multicodal analysis lie.

5A multicodal analysis approach

For the scientific investigation of the (different) codes, it remains to be clarified not only which image definition is to be used (insofar as images are part of the texts to be analyzed). Also, the assignment to different codes is by no means trivial, but can, on the contrary, depending on the focus of the analysis, turn out quite differently. There may be agreement on assigning oral language to verbal communication, and, accordingly, written language will normally be assigned to a verbal code. This takes into account the fact that many media texts are oriented not only towards written, but also towards oral language use (cf. Koch/Oesterreicher 22011). We might think of an online forum, in which oral colloquial language is to be found in written form (cf. Fröhlich 2015). Vice versa, television programs may often contain rather formal written language in an oral form, e. g., when the content is read out or reported. Images can be assigned to a nonverbal code, because they do not contain any letters or anything similar in their pure form and, in fact, communicate in another way. There are different opinions on emoticons, for example, which are assigned to the nonverbal (cf., e. g., Fröhlich 2015), but also the paraverbal code (cf., e. g., Aldana Rueda 2011). The latter assignment is eminently plausible the moment we see them as oral forms of expression accompanying language. How far this is really the case, with regard to online media texts, is another question, because there they may also be seen as CMC-specific signs and not purely as compensation for the gestures and mimicry of oral face-to-face communication (cf. Döring 2003, 151s.; Fröhlich 2015). Different opinions on paraverbal phenomena are to be found, too. While discourse markers, like “Oh”, can be discussed under the heading of verbal code – emphasizing solely the verbal phenomenon – it is also possible to describe such markers as paraverbal, insofar as their context-dependent semantic meaning is at the center of attention and is intended to be highlighted in contrast to verbal signs with semantic content (cf. Fröhlich 2015). This follows the basics of oral communication. Therefore, in my opinion, the attribution to a certain code cannot be called right or wrong in one sweeping statement. The decisive factor is the reasoning behind the respective attribution with regard to the object under investigation.

Using the concrete example of the blog posting on “Elodie in Paris” (Fig. 1), it is possible to explain and illustrate the interrelationship of different codes.

The verbal code already directly refers in the headline “Paris, mon amour – Elodie in Paris” to the nonverbal code: in the background, the Eiffel Tower can be clearly seen. Elodie is posing in front of the Paris landmark, thereby expressing her close ties to, or maybe her “love” of, the city. The Eiffel Tower is depicted from top to bottom in the background, while in the foreground only half of Elodie’s face is visible. The famous landmark gains its importance from the central position in the picture.

In her verbal utterances, too, references to the picture are clear. While there is a hint in the French version with her comment “[. . .] malgré son côté tape-à-l’oeil” that she is playing on the striking color of her skirt, this supposition is confirmed in the English translation: “[…] even if it’s pink, metallic and not discreet at all [sic]”. The pink skirt is emphasized by the otherwise muted color mix in the nonverbal code and is the eye-catcher in the picture. Elodie uses bold type to highlight the names of brands and designers and even towns and thus uses the paraverbal code to make her references. On the one hand, the brand names refer to the clothes she is wearing, while on the other, she links them in part with their websites.

The English version of the preceding French verbal utterance is also printed in italics. The highlighting is not carried over. This may mean that the French version must be considered as the more important, whereas the English version is to be seen as a mere translation of the French or as an “extra”, and not as a verbal component of equal value. The peritext at the end of the posting sets itself apart from the main utterance in the text field by means of the typeface, using capitals, and assumes a framing position at the end of the posting.

The example shows that different codes can be harmonized to mutually complement one another.

6Conclusion

As explained in the previous sections, the separation of the definitions of mode and code has the advantage of emphasizing the different research aims and of being able to better profile the latter through careful use of the specialist terms. If the analysis refers to media texts, like films, for example, which are received through different sensory perception, then multimodality is at the center of interest, even if multicodality can play a certain role, for instance, in the analysis of images and spoken language. Therefore, the terms of denomination should be chosen consciously, according to the focus of the research.

Especially the investigation of multicodal interconnections, like those of language and images, is seen by Klemm/Stöckl (2011, 9) as an important task for linguistics, which can then contribute to the image sciences and, vice versa, image science findings can enrich linguistics. In the way that image science attaches little importance to language, on the one hand (cf. Klemm/Stöckl 2011, 11), the focus of linguistics, on the other, is too narrowly directed towards language and too little towards images. At the same time, it is images that simply cannot be ignored in present-day media texts.

Visual communication researchers like Lobinger (2012) plead for the recognition of the (private) image in public contexts, e. g., in social networks, as a relevant object of research within media and communication sciences. For their proponents, these images belong to the daily life of many people in the digital age (cf. Lobinger 2012, 16). In support of image linguistics (e. g., Klemm/Stöckl 2011), it is even possible to go a step further.

To go to the extreme, you could say that it is a distortion of reality to “filter apart” language and images, although they are commonly embedded in a whole text amidst other codes or modes. As is clearly shown in the multicodal research by Fröhlich (2015) with the example of forum communication, images represent an important component for the meaning of the whole text which should not be ignored. Images are used in part through their specific characteristics to communicate in other ways. Images can provide cause for (verbal) communication. They transport a share of the meaning in the whole text that is not to be spurned, and which ought to be investigated especially in the field of media linguistics. The analysis of the (whole) meaning of a multicodal (or multimodal) media text requires research into all the codes it contains and into their interreferentiality.

7References

Aldana Rueda, Virginia (2011), Verbale Konstruktion von Expertenstatus in asynchroner computervermittelter Kommunikation. Eine Untersuchung am Beispiel von Newsboards und Social-Networking-Plattformen, <http://www.as.uni-heidelberg.de/ejournal/si2/as-ejournal-si-2011_1.pdf> (17.01.2017).

Allwood, Jens/Ahlsén, Elisabeth (2012), Multimodal Communication, in: Alexander Mehler/Laurent Romary/Dafydd Gibbon (edd.), Handbook of technical communication, Berlin et al., de Gruyter, 435–460.

Boehm, Gottfried (2006), Die Wiederkehr der Bilder, in: Gottfried Boehm (ed.), Was ist ein Bild?, 4th edition, München, Fink, 11–38.

Brandt, Reinhard (2004), Bilderfahrungen – von der Wahrnehmung zum Bild, in: Christa Maar/Hubert Burda (edd.), Iconic turn. Die neue Macht der Bilder, Köln, DuMont, 44–54.

Colin, I. (1992), Optische Kodierung, Universität Frankfurt am Main (unpublished habilitation treatise).

Döring, Nicola (2003), Sozialpsychologie des Internet: die Bedeutung des Internet für Kommunikationsprozesse, Identitäten, soziale Beziehungen und Gruppen, Göttingen, Hogrefe.

Fix, Ulla (2011), Fraktale Narration. Eine semiotisch-textstilistische Analyse, in: Jan Georg Schneider/Hartmut Stöckl (edd.) (2011), Medientheorien und Multimodalität. Ein TV-Werbespot – Sieben methodische Beschreibungsansätze, Köln, von Halem, 70–87.

Fraas, Claudia/Meier, Stefan/Pentzold, Christian (2012), Online-Kommunikation. Grundlagen, Praxisfelder und Methoden, München, Oldenbourg.

Fricke, Ellen (2012), Grammatik multimodal. Wie Wörter und Gesten zusammenwirken, Berlin/Boston, de Gruyter.

Fröhlich, Uta (2015), Facework in multicodaler spanischer Foren-Kommunikation, Berlin/Boston, de Gruyter.

Genette, Gérard (1987), Paratextes, Paris, Seuil.

Gibbon, Dafydd/Mertens, Inge/Moore, Roger K. (edd.) (2000), Handbook of multimodal and spoken dialogue systems: Resources, terminology and product evaluation, Dordrecht, Kluwer Academic Publishers.

Halawa, Mark Ashraf (2008), Wie sind Bilder möglich? Argumente für eine semiotische Fundierung des Bildbegriffs, Köln, von Halem.

Huber, Hans Dieter (2004), Bild Beobachter Milieu. Entwurf einer allgemeinen Bildwissenschaft, Ostfildern, Hatje Cantz.

Klemm, Michael (2011), Bilder der Macht. Wie sich Spitzenpolitiker visuell inszenieren (lassen) – eine bildpragmatische Analyse, in: Hajo Diekmannshenke/Michael Klemm/Hartmut Stöckl (edd.), Bildlinguistik. Theorien – Methoden – Fallbeispiele, Berlin, Schmidt, 187–209.

Klemm, Michael/Stöckl, Hartmut (2011), Bildlinguistik – Standortbestimmung, Überblick, Forschungsdesiderate, in: Hajo Diekmannshenke/Michael Klemm/Hartmut Stöckl (edd.), Bildlinguistik. Theorien – Methoden – Fallbeispiele, Berlin, Schmidt, 7–18.

Koch, Peter/Oesterreicher, Wulf (22011), Gesprochene Sprache in der Romania. Französisch, Italienisch, Spanisch, Berlin/New York, de Gruyter.

Kress, Gunther R. (2009), What is mode?, in: Carey Jewitt (ed.), The Routledge handbook of multimodal analysis, London/New York, Routledge, 54–67.

Kress, Gunther R./van Leeuwen, Theo (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, London, Arnold.

Lenk, Hans (1978), Handlung als Interpretationskonstrukt. Entwurf einer konstituenten- und beschreibungstheoretischen Handlungsphilosophie, in: Hans Lenk (ed.), Handlungstheorien interdisziplinär, vol. 2.1, München, Fink, 279–350.

Lobinger, Katharina (2012), Visuelle Kommunikationsforschung. Medienbilder als Herausforderung für die Kommunikations- und Medienwissenschaft, Wiesbaden, VS Verlag für Sozialwissenschaften.

Maaß, Christiane (2014), On the markedness of communication on online message boards as part of a perception-oriented politeness approach, in: Kristina Bedijs/Gudrun Held/Christiane Maaß (edd.), Face Work and Social Media, Zürich/Münster, LIT, 237–275.

Martin, Jean-Claude/Schultz, Tanja (2012), Multimodal and Speech Technology, in: Alexander Mehler/Laurent Romary/Dafydd Gibbon (edd.), Handbook of technical communication, Berlin et al., de Gruyter, 189–254.

Mitchell, W. J. Thomas (1990), Was ist ein Bild?, in: Volker Bohn (ed.), Bildlichkeit, Frankfurt am Main, Suhrkamp, 17–68.

Müller, Marion G. (2003), Grundlagen der visuellen Kommunikation. Theorieansätze und Analysemethoden, Konstanz, UVK.

Nöth, Winfried (2004), Zur Komplementarität von Sprache und Bild aus semiotischer Sicht, Mitteilungen des deutschen Germanistenverbandes 51:1, 8–22.

Norris, Sigrid (2011), Identity in (Inter)action. Introducing multimodal (inter)action analysis, Berlin/New York, de Gruyter.

Pentzold, Christian/Fraas, Claudia/Meier, Stefan (2013), Online-mediale Texte: Kommunikationsformen, Affordanzen, Interfaces, Zeitschrift für Germanistische Linguistik 41:1, 81–101.

Sachs-Hombach, Klaus (2003), Das Bild als kommunikatives Medium. Elemente einer allgemeinen Bildwissenschaft, Köln, von Halem.

Schneider, Jan Georg/Stöckl, Hartmut (edd.) (2011), Medientheorien und Multimodalität. Ein TV-Werbespot – Sieben methodische Beschreibungsansätze, Köln, von Halem.

Stöckl, Hartmut (2004), In between modes. Language and image in printed media, in: Eija Ventola/Cassily Charles/Martin Kaltenbacher (edd.), Perspectives on multimodality, Amsterdam/Philadelphia, Benjamins, 9–30.

Stöckl, Hartmut (2006), Zeichen, Text und Sinn – Theorie und Praxis der multimodalen Textanalyse, in: Eva Martha Eckkrammer/Gudrun Held (edd.), Textsemiotik. Studien zu multimodalen Texten, Frankfurt am Main/New York, Lang, 11–36.

Stöckl, Hartmut (2011), Sprache-Bild-Texte lesen. Bausteine zur Methodik einer Grundkompetenz, in: Hajo Diekmannshenke/Michael Klemm/Hartmut Stöckl (edd.), Bildlinguistik. Theorien – Methoden – Fallbeispiele, Berlin, Schmidt, 45–70.

van Leeuwen, Theo (2005), Introducing social semiotics, London/New York, Routledge.

Weidenmann, Bernd (2002), Multicodierung und Multimodalität im Lernprozess, in: Ludwig J. Issing/Paul Klimsa (edd.), Information und Lernen mit Multimedia und Internet. Lehrbuch für Studium und Praxis, 3rd edition, Weinheim, Beltz, 45–62.