As humans, we are innately curious creatures. We seek meaning, connection, certainty, and clarification in the world around us—whether it's looking at signs as we drive or finding a vegetarian meal to order by scanning a menu marked with icons or trying to understand the pointed gestures of our friend showing directions. Beyond pattern finding, we seek to find meaning in the world around us. To find meaning, we must establish a connection and come to reasonable terms, just as the Deaf children in Nicaragua did, as described in the Part B opening.

While meaning and understanding the various relationships and patterns in the world is second nature to us, describing it in a way that can be formalized is rather difficult and abstract. This is where semantics comes into play. Semantics is the study of how we draw meaning in communication (Cann et al., 2019). With semantics, we can explore how words or signs combine to convey a concept, why icons denote specific ideas, or which gestures effectively support a message. We can take these interactions—between ourselves or machines—and abstract them to find common patterns across different languages, cultures, and experiences. We can use a variety of cues to affirm our message, or we can send signals that conflict, making it hard for others to understand the message.

In data visualization, incorporating semantic elements can be as simple as including an icon showing people where to click or how to interact. A known graphic is a specific “click” arrow that we have seen thousands of times, placed in proximity to a chart. Close your eyes and you'll probably even have a specific arrow in mind. This task is done without long explanations or training. This is semantics at work. We've taken a visual abstraction, agreed on its meaning, and put it to use as a shared symbol.

Sports uniforms, company logos, and traffic signs are symbols. In some cultures, a gold ring is a symbol of marriage. Placing your hands into the shape of a heart is also a symbol, one we can playfully toss to a friend across a variety of contexts. Other symbols are highly functional: stop signs, for instance, provide useful instruction for traffic flow. They are a ubiquitous form of what functional aesthetics provides, serving as pictographic representations of intent or purpose.

Figure 4.1 shows a variety of symbols. Which ones are meaningful to you?

These icons come in a variety of styles. When you see these out in the world, they are easy to spot, making them visually distinguishable. As you scan them, you can see the differences between all of these graphics. A symbol is perceptually distinguishable and semantically meaningful if you are part of the group that uses it. Even though we can most likely describe an abstract figure without understanding its goal, for these symbols to work, they must have semantic resonance in our lives.

Statistical Graphics Are Inherently Abstract

Charts abstract information. They make it easier to see patterns at a distance, compare, and extrapolate. Icon encodings are graphical elements that are often used to visually represent the semantic meaning of marks for categorical data. Assigning meaningful icons to display elements helps the user perceive and interpret the visualization easier. These encodings can be effective in enabling visual analysis because they are often rapidly and efficiently processed by the preattentive visual system (Setlur & Mackinlay, 2014). The human visual system spatially categorizes these icons in order to create a meaningful understanding of the visualization.

In his book The Dragons of Eden (1977), Carl Sagan showed a chart with the brain-to-body-mass ratios plotted for various animals, as seen in Figure 4.2. In order to make sense of this visualization, your eyes need to follow the dots and the labels to figure out a pattern.

Both label and icon indicate the semantics of the marks. However, the labels make it harder to see the centers of the dots, which makes it harder to see the correlation between body and brain masses. The icons, on the other hand, show the semantics of the marks while also helping the person to see the correlation. By associating each animal with a semantically meaningful icon (Figure 4.3), suddenly, the chart becomes more useful to understand and follow.

These mechanisms are far from perfect. While we can detect patterns, we don't always see or interpret the same things. Some of this goes back to our minds which will fill in patterns to better align with what we expect. Charts that ask you to draw your assumptions are powerful tools to combat some of these cognitive jumps. They force us to see what we expect compared to what the data shows.

As we abstract forms, we rely on shared semantic understandings of what is encoded in the chart. The COVID-19 pandemic led to a proliferation of visualizations: analysts showed daily counts, rolling averages, and accumulated totals over time and a myriad of ratios in a variety of ways. Yet, we didn't take away the same understanding of the data, and a small network leveraged this ambiguity to create more confusion.

Flattening the Curve

In 2007, researchers examined the impacts of non-pharmaceutical interventions within a pandemic (Hatchet et al., 2007). A single chart highlighted the differences between two cities: Philadelphia and St. Louis. These two lines are marked with a legend and two different styles, one solid and the other dotted (Figure 4.4, left). The paper highlighted the need for early intervention—mitigating the spread of endemic outbreaks by curbing interactions and using masks. In 2020, this paper fed one of the most iconic visuals of the COVID-19 pandemic: the graph showcasing our ability to “flatten the curve” with specific social interventions.

“Flatten the curve” is an abstraction. It takes curves seen from the cities and smooths them to two clearly distinct curves. One is a fast and high peak, while the other stretches across a longer span (Figure 4.4, right). It generalizes a fact and feeds our need for story: we can choose our path. Additional layers of annotation can highlight what feeds the spike: hospital capacity, our willingness to partake in social interventions, and the introduction of vaccines. Variations of “Flatten the curve” morphed to part of the visual vocabulary of 2020.

This symbol extracted a proven pattern beyond a literal example to that of a rally cry. We could use it to highlight healthcare system limits and the goal of delaying cases—we knew they would come—to perhaps when we had more capacity and more pharmaceutical options. Its ambiguity allowed its broad use but relied on clarification of what it showed. Explainers from Vox (Barclay et al., 2020), the Spinoff with Dr. Siouxsie Wiles (Morris & Wiles, 2020), and others used interactivity, showing tangible examples of each in isolation before showing both curves together. They highlighted the behaviors, leveraging comic-style graphics and pithy quotes to give an idea of attitudes that fed each curve.

Beyond “flatten the curve” graphics, reporters visualized case data in a number of ways. John Burn-Murdoch of the Financial Times shared his insights both through the news site and Twitter (https://twitter.com/jburnmurdoch), his charts highly annotated to help readers navigate the graphics. He featured a number of line charts generated across a variety of calculations to highlight his findings. Twitter provided a testing ground, a way to converse and derive a better understanding of how to annotate displays and showcase patterns.

As news organizations visualized COVID-19 data, they had to overcome hurdles associated with displaying ambiguous data. Case report dates varied, with weekend data delayed until the following Monday for reporting. This affected the visual pattern. Defining a case varied across geopolitical lines, deaths were not always directly listed as COVID related, and the delayed relationships between onset and death needed clarification for public understanding. Aggregated data hid location clusters and stratification-based discrepancies such as race, gender, and sexuality in outcomes. Spikes in data could be reporting-related, delayed by holidays, and are often smoothed out by rolling aggregations. These calculations needed to be exposed fully and explained to the broader public.

Along with what news organizations provided, analysts and other data workers took to chart making to find and highlight patterns. Sites crowdsourced and shared a variety of data points. As you saw in Chapter 3, the COVID-19 pandemic coincided with an infodemic. It wasn't just the onslaught of information, but the types of information. Skeptics leveraged the ambiguous nature of charts, pushing untrained individuals to do their own analysis on tabular data (Lee et al., 2021). This ambiguity feeds into data literacy discussions, which we'll explore in Chapter 6, and the blurry lines around how charts are interpreted.

Toward Meaningful Depictions

Symbolic representations manifest in various ways for effectively communicating a specific concept, as we have seen in the various examples so far. However, we know that communication is richer and more nuanced than that. Creativity is often seen as the ability to create novel ideas by making connections between existing concepts. It is the basis of how language has evolved and how we use graphics to share meaning. It plays an important role in graphic design not only in conceiving new concepts but also in visually representing them. As far as the visual representation of concepts is concerned, humans have been doing it since more than 77,000 years ago, starting with cave paintings (Relethford, 2008). These representations vary from being completely pictorial to more abstract.

The link between the visual representation and the conceptual connections behind it can be observed. Examples of this can be seen by looking at Chinese characters, more specifically at the ones categorized as ideogrammic compounds (Tung & Hopkins, 2012). These characters can be decomposed into others whose concepts are semantically related, belonging to the same (or at least similar) conceptual space. Figure 4.5 shows Chinese characters for root, tree, woods, and forest (left to right). Root can be obtained by adding a line to the tree character; the woods character can be obtained by using two tree characters; the forest can be obtained by using three tree characters.

These visual alterations are also how we mature semantics broadly in language. Deaf children in Nicaragua used similar paradigms to expand meaning in their newly birthed language, expanding, changing placements, and multiplying to enrich meaning (Senghas & Coppola, 2001). Spoken Korean is rich in vocabulary words that tie back to characters like these being combined to make new concepts. We will revisit some of these ideas in Chapter 6 with literacy. Beyond language, this creative response also flows into icons.

Some authors were inspired by this relationship between concepts to their visual representations. One of them was Charles Bliss, who developed a communication system composed of several hundred ideographs that can be combined to make new ones, called Blissymbols (1965), with various examples shown in Figure 4.6.

Several interesting things can be observed by looking at Blissymbols, such as variation in the degree of abstraction in the symbols. By combining symbols, new meanings are obtained (examples in Figure 4.6: pen + man = writer, mouth + ear = language); by using the same symbols in a different position, new meanings are obtained, as seen in symbols for water, rain, steam, and stream.

These systems work because of how we draw meaning with semantics: we create visual allegories to our intent by building up to the concept. These examples can be found in languages and other pictorial systems quite frequently. Researcher Neil Cohn (2014) has found a semantic lens in drawing. For him, drawing parallels between language fluency and “I can't draw” highlights a lack of semantic fluency.

We can help others draw meaning in charts as well. We can layer them, break lines and boundaries, and expand concepts to create new clusters of meaning. Figure 4.7 shows an example that intentionally blurs lines between the proportional brushing of the bar chart into the axes of the area charts. Proportional brushing is a technique where a proportion of the selected data is shown in relation to all the values rather than just filtering to the selection. Shared color helps obscure the boundaries.

This example highlights another tactic for creating meaning. By combining disparate charts into one that visually becomes more like one unit, we leverage semantics to expand our repertoire. In this case, we use color to provide a bit more clarification about the part-to-whole relationship.

Situating with Semiotics

When dealing with meaningful visual representation, aspects of a representation's meaning can be altered by modifying its visual characteristics; these characteristics are extensively explored in semiotics, the study of signs and symbols and their use or interpretation. Changing an aspect ratio changes meaning (Chandler, 2002). In Figure 4.8, Tableau Hall of Fame Visionary Kelly Martin analyzes animal strikes. Within the visualization, she embeds aspects of semiotics masterfully and deeply.

Deconstructing the dashboard highlights the three semiotic properties commonly applied to visually express meaning:

	One semiotic aspect, position, can be seen in the various positions and directions of the plane. When the image is flipped, our eyes and brain envision the movement that these visual representations convey, giving us “departure” and “arrival.” The descending plane is higher, while the ascending plane is lower, also supporting and enriching meaning.
	Shape is another aspect. The jarring shape affirms the topic of collisions, one leveraged in comics to highlight violent contact. Beyond comic design, an experiment to map shape and sound (Ramachandran & Hubbard, 2001) showed that we tend to associate sharp shapes with sharp, high-pitched sounds and smoother, organic shapes with softer sounds.
	The last semiotic aspect is color. The shapes are colored by time of day, leveraging a bright yellow for dawn, darkening to orange for day, and shifting to blues for the evening. At a glance, they align with the shifting natural light we experience.

The result is a delightful form of creative expression that cleverly blends the semantics of data with the effective use of color, shape, and space. The graphics in her view, are used to encode data values and show how those data values are connected to the world. Her shapes are particularly clever because the size of shape is used to encode the number of strikes, while the form of the shape connects to the violence of the event. It's visually striking, thoughtfully crafted, and visceral. We'll explore other elements within this visualization to further expose its craft. Semiotics is a useful tool for effectively depicting data in the form of semantic concepts in visualizations. The process of creating functionally aesthetic visualizations deals specifically with understanding the forms and functions of individual charts and skillfully compositing them into an effective representation.

Summary

Our knowledge of the world helps us understand the semantics of representation. It is a fundamental component of how we reason and make sense of everything around us. Our understanding of meaning is based upon a blend of perceptual, cognitive, and affective states arising from our direct sensory experience, familiarity, and understanding as seen with the various examples described in this chapter. While the world may be best measured and represented as concrete quantifiable entities such as meters and Fahrenheit, the way we interpret information is more loosely defined. We often use words like tall and warm as a way of describing concepts. The next chapter will explore language and its impreciseness and how to represent data as concepts that may be fuzzy around the edges.