The human visual system is quite intricate and remarkable. Cameras often serve as a common reference point when first starting to understand the mechanics of the eye itself. Like the lens on a camera, the cornea serves as the outermost lens and window to the world, protecting the viscous eye but letting light enter. The colorful iris acts as a shutter mechanism to control light. Similar to the camera metaphor, the eye can expand and contract to focus properly. While there are optical similarities between a camera and the eye in the way they capture an image, a camera does not have perceptual or cognitive abilities and the metaphor starts to break down when we get to how the brain interprets this information.

Seeing and Understanding Imagery

Beyond a picture that we perceive, our brains take this visual information and perform calculations on it. Think about it as you drive. You are analyzing in real time the placement of your car against the other cars as well as how fast the other cars are moving. When you buy a new car, your perception has to adjust from that vantage point: your new car might be smaller or larger, affecting all of your calculations and processes. Vision is an information processing phenomenon where neurons in our brain perform calculations on the signals so that we reason about what is in the imagery we see and how we act on it. In other words, it's not a simple picture or movie that we perceive.

Long before Isaac Newton discovered that colors were physical and quantifiable phenomena of light's spectrum, skilled artists figured out ways to trick our brain's circuitry. They often used color in clever ways to simulate illusions of the mind, whether it is creating the perception of depth on a flat surface or the sense of movement in a static object.

Claude Monet, an impressionist painter from the late 1800s, used unrealistic luminances in his masterpiece, Impression, Sunrise, which is shown in Figure 1.1. The scene deceptively looks rather simple; a few small rowboats at the port of Le Havre at sunrise in the foreground with the orange-red sun being the focal element. The layered hazy brushstrokes create the illusion of depth despite their imprecise details. Yet this piece is far more complex than it looks.

While it initially seems that the sun is the brightest spot in the painting, it turns out that it has the same luminance (i.e., the perceived brightness of a color) as that of the sky. Dr. Margaret Livingstone (2008), a neurophysiologist, showed that if you make a black-and-white copy of the painting as shown in Figure 1.2, the sun almost entirely disappears!

We can now barely see the shape of the sun. Rather than being drawn to the sun, the boats now stand out the most. Dr. Livingstone describes how the area of the visual cortex contains bands of cells that route signals coming in from the eye to specialized branches of neurons.

She presents a model of the neural processing of vision that argues that humans possess two distinct visual systems. The ventral stream (also known as the “what” pathway) leads to the temporal lobe and deals with visual identification and recognition through color and shape. The dorsal stream (also known as the “where” pathway) leads to the parietal lobe and concerns orientation and the location and movement of things in space. “We have all these modules in our brains for seeing things,” Livingstone says. “They make us an expert at seeing those things.”

Since the “where” pathway can only discern orientation and movement, it is color-blind. Like our black-and-white painting, it processes the sun and the sky and does not register the presence of the sun. The “what” system discerns color and can find clear boundaries between the orange sun and the blue sky. This mishmash of neuron pathways results in a perceived sense of motion—the sun cuts through the blue-gray hues of the sky and shimmers.

An effective piece of art can convey an artist's message by bridging the biology of seeing with an emotional response in the viewer. A combination of carefully laid-out brush strokes can provide a harmonious combination of beauty and purpose for the painting. Early impressionists were considered radicals in their time because they did not follow the rules of academic painting from the Roman and Greek periods. Rather, these impressionists used the canvas to go beyond photo-realism to something that was imaginative and magical as they teased our human visual system.

Too often, charts can be intimidating. They can seem rather anonymous, flat, and so obscure. As a chart maker, how can we think about the intentionality of color, shape, and placement of marks in a chart? A simple chart is no more than a set of statistics made visible, showing what happened in the past and perhaps what might happen in the future. But it can do more than just being functional. It can engage the viewer by capturing their imagination. Nigel Holmes has pushed the boundaries of infographics by advancing charts into evocative illustrations with embellishments that serve as a visual metaphor to generate a strong visceral reaction, often making the takeaway from the chart memorable (Holmes, 1984). We will discuss more about how charts can (and should) intentionally engage with the reader through careful design, semantics, and even creative embellishments. And perhaps, your dashboard may evoke an emotional response similar to that of an art lover staring at a Matisse or a Renoir in a museum gallery.

Color Cognition

Color is a key visual aspect of how we observe and relate to objects around us. A perceptual color space (what we see as color) is defined from descriptions of attributes of perceptions of about 6 million different colors in the real world. A cognitive color space (i.e., what we understand as color) would refer to the internal representations of the colors and would also include semantic color representations, that is, names representing the colors mentally. In a box of Crayola colors, for example, the color names and color values need to match. To do otherwise would be very confusing, potentially creating cognitive interference.

Language has a long history of resources for describing color. Firstly, a word can have a strong association with color, especially when color is a salient feature of the concept it refers to—for example, sky (blue), lemon (yellow). Secondly, color names pertaining to pigments and dyes are often derived from the source, such as indigo, from the Indigofera tinctoria plant. Thirdly, many languages have morphological and syntactic processes that create complex color terms out of simple color terms (for example, blue-green, yellowish, and pale pinkish-purple). Finally, many linguistically simple terms that denote subtypes or “shades” of colors are denoted by other terms. For example, scarlet, crimson, vermillion, puce, magenta, burgundy, and maroon are among the more commonly named shades of red. This suggests that linguistic data sources that consider the semantics of color names might provide for better reference, selection, and retrieval of colors for various tasks, including for categorical palettes in data visualizations.

In dealing with color-naming data, internal consistency is usually good but there are high variations between people. In fact, Randall Munroe of xkcd (https://xkcd.com) comic fame asked 22,500 willing participants to take part in a color-naming survey. Munroe found some interesting disparities between how people name colors, with women providing more descriptive modifiers than men.

Now, why is all this important or even relevant to data visualization? From the literature on categorical color perception, we know that only a limited number of colors can be internally represented and absolutely identified across different cultures and during different tasks. Berlin and Kay (1969) conducted a study with native speakers coming from a diverse set of 98 languages to decrease the possibility of borrowed color terminology. They established that for English (and many similar European languages), there are 11 basic categorical color names: red, green, blue, yellow, orange, purple, pink, brown, black, white, and gray. Languages with fewer than the 11 basic color categories all followed a set of “restrictions” regardless of the language family. For example, they found that all languages have words for black and white.

For languages with just three terms, the third term is always red, and those with four terms have words for either yellow or green. In other words, the lack of randomness of color terms over the studied languages suggested a series of evolutionary stages in the development of color words. While Berlin and Kay's work has demonstrated how the evolution of language has influenced color names in its vocabulary, one must point out that speakers can also signify differences in color, not by separate terms, but by variations in syntax, morphology, tone, and inflection of the language in question.

In the 1950s, the US National Bureau of Standards created a color naming dictionary [ISCC] to both define a standard set of color names by partitioning the Munsell colors (1919) and create a dictionary of commercial color names, defined in terms of their Munsell specification and their standard name. While this standardization effort was not widely adopted, the use of color names to represent colors in commerce continues today. Go to any paint store, and the color samples are labeled with both a technical code for constructing the paint and a semantic name, designed for some combination of descriptiveness and memorability. The Sherwin-Williams red collection (2021) provides examples such as Positive Red, Eros Pink, Radish, Brick, and CoralBells. HTML 5 supplies 140 color names, from the basic blue and red to the exotic chartreuse and BurlyWood (W3Schools, 2021). The power of these names is not their accuracy but their memorability and ease of use.

What is fascinating is that the cognitive and representational aspects of color in categorical color perception have been found to be closely related to visual synthesis and spatial organization. This means that these color names are used to describe objects around us—the red and black lacquered bento box, the green pines in Yosemite, or that yellow New York cab.

The Stroop effect is perhaps one of the most striking phenomena illustrating the cognitive aspects of color. Let's do a little exercise. Look at each word in Figure 1.3 and say the color out loud, not the word itself. It's rather tricky, isn't it?

This effect can be said to demonstrate a conflict between perceptual and semantic processing. When people are asked to name colors, they experience a conflict between the cognitive meaning of the word (yellow) and the perceived color (green). The Stroop effect also highlights a fundamental link between language and color cognition. This link demonstrates the importance of semantic coloring in the domain of the category.

Objects with strongly associated colors, like fruits, vegetables, political parties, and brands, also benefit from semantic coloring. Effective imagery must first determine the colorability of the objects in the category, then determine the appropriate coloring. Remember, we want to avoid that Stroop effect, which can trip up people. Semantic coloring is also defined by the context. For example, apple as a fruit is generally red, yellow, or green, but apple as a company brand is white or silver-gray. We will discuss how semantics plays a role in creating functionally aesthetic charts in Part B. Until then, Figure 1.4 is a teaser. After all, who doesn't love ice cream?!

Figure 1.4 is an example of a color palette automatically generated based on the ice cream flavor names in the data. The lovely cream brings out the vanilla flavor from the graph, making one's mouth almost water. There is a balance of intentionally delicate hues with a certain playfulness in that pink strawberry, all while making it easier to identify the different flavors in the graph. As we better understand how the human visual system works, our charts can in turn be made more meaningful. Often, we are made to believe that it's form or function, especially when visualizing data; but one can have the cake (or rather ice cream) and eat it too!

Saccade and Directed Attention

In addition to color, contrast plays an important role in visual perception and allows us to effectively differentiate contours, depth, and shape. The human eye makes quick scans when viewing a scene or performing any activity. Referred to as saccades, it is estimated that the eye makes about three scanning movements per second and that these tend to occur unhindered during both focal and distributed attention. Saccades generally involve focal attention and are facilitated by a short-term memory system; but what catches the attention of a saccade? Several studies found that contrast, including strong light-dark contrast, color contrast, and movement, attracts saccades and draws focal attention.

The human eye tends to notice and focus on objects that are bright or feature movement. Called the fixational reflex, this occurs when a brightly contrasting or moving object is deemed significant. In visual design, strong contrasts in hue, saturation, and/or tonal value can create fixational reflex and are often used to draw attention to significant areas of text or imagery or create some significant level of differentiation. To illustrate with the example in Figure 1.5, poor contrast in the left image tends to impact negatively on legibility and the text is more difficult to read. Strong contrast in the right image allows the text to be read quickly and effortlessly.

Just as color plays an important role in human visual perception, visual contrast also has the capacity to discern structure, logic, and patterns; it helps us make sense of the world. As a result, color and contrast are frequently harnessed in the design of visual communication to draw attention to key elements such as headlines, text, or imagery.

The Notion of Space and Spatial Cognition

People conceive of spaces differently depending on the functions they serve. A dining table could very well function as a school project space with glue sticks, pencils, and cardboard while in the evening, it can transform into a cozy family gathering for dinner and conversation. The space of external representation of pictures, diagrams, maps, and charts serves as cognitive aids to memory and information processing. To serve those ends, graphics may generalize information suitable to the context and message they need to convey. In many useful diagrams, similar elements appear with similar abstract meanings; their interpretations are context-dependent, as for many word meanings, such as line or relation or area or field. In diagrams, notably maps and graphs, lines are one-dimensional paths that connect other entities suggesting that they are related. Crosses are intersections of paths. Blobs or circles are two-dimensional areas that depict how much space is occupied by an element. These elements schematize certain physical or semantic properties while omitting others.

In the 1920s a group of German psychologists—Max Wertheimer, Kurt Koffka, and Wolfgang Kohler–developed theories around how people perceive the world around them called Gestalt Principles, where Gestalt is German for a unified whole. These principles help reason why the human mind has the natural compulsion to find order in disorder. We group similar elements, recognize patterns, and simplify complex images when we perceive objects. In other words, the mind “informs” what the eye sees by perceiving a series of individual elements as a whole. Gestalt Principles are an essential part of visual design. There are more than 10 overlapping principles; 5 of the more prevalent ones are listed here:

• Closure: We prefer complete shapes, so we automatically fill in gaps between elements to perceive a complete image. In Figure 1.6, we can interpret a cluster of black shapes set against a white background to reveal the familiar form of a soccer ball.

  

• Common region: We group elements that are in the same closed region. Related objects are placed in the same closed area or boundary to show they stand apart from other groups. They are assumed to share some common characteristic or functionality. In the graphic in Figure 1.7, the black circles appear related and as part of the same group even though there is no explicit boundary drawn indicating so.

  

• Figure and ground: We dislike uncertainty, so we look for solid, stable items, unless of course the image is ambiguous like the Rubin's vase in Figure 1.8. The visual effect generally presents the viewer with two shape interpretations, each of which is consistent with the retinal image, but only one of which can be maintained at a given moment. The brain begins "shaping" what it sees; the process involves higher-level cognitive pattern matching, in which the overall picture determines its mental interpretation rather than the individual effect of its constituents. We try to put together two distinct regions of the picture, that is, faces and a vase. Each makes sense in isolation, but when the brain tries to make sense of it, contradictions ensue and patterns must be discarded.

  

• Proximity (Emergence): We group closer-together elements, separating them from those farther apart. So when we cluster individual elements like the pairs of vertical lines in the graphic in Figure 1.9, we recognize each line pair as one entity standing distinct from the other pairs.

  

• Similarity: The principle of similarity states that when things appear to be similar to each other, we naturally group them together. We tend to attribute them to belong to the same functional category. For instance, in Figure 1.10, there appear to be two separate and distinct groups based on shape: the circles and the squares. The shapes are all equally spaced and are of the same size. Further, we also group them by color and categorize them into four groups, even though there's no rhyme or reason to their placement.

  

The Gestalt Principles are vital in design. Users must be able to understand what they see—and find what they want—at a glance. The principles of proximity and common region are good examples. Colors and graphics divide the content into separate regions. Without this, users struggle to make associations between unrelated clustered-together items. Graphic designers commonly use these principles in their websites as a way to organize content with well-placed elements that catch the eye as larger, whole images. The result is a presentation that is aesthetically pleasing, yet easy to understand.

Diagramming the World

How do we want to share a message with the reader and what do we want to say? So far, we have revealed the design principles, guidelines behind how visual cognition works—the dots, lines, boxes of the world, each of which has meaning, in context. Just as dough in “I will make a batch of cookie dough” does not have the same meaning as dough in “After I get paid, I'll have enough dough to go buy a pair of shoes,” a line on a map does not have the same meaning as a line on a bar chart. These forms need to be considered in the function with their meanings and inferences that they aim to promote. Appropriate colors, symbols, placement, and size are inherent to guidance, explanation, and storytelling.

Directing Attention Through Arrows and Signage

The visual environment around us generally contains more information than can be processed within a glance. We constantly look for signs around us to selectively focus on as we make sense of the world. As such, action and agency depend on mechanisms of selective visual attention for prioritizing an endless stream of sensory input. We rely on visual cues to help orient and give more weight to those objects, locations, or events that require immediate focus.

Our eyes and hands are our own arrows that we use for directing gaze and communication. The glances and eye contact you make with friends at a coffee shop or gesturing while providing directions to a stranger who needs help finding the nearest pharmacy are all ways of directing thought and attention. Arrows are specialized lines that depict paths, direction, and relation. Dr. Barbara Tversky (2019) describes the evidence of the arrow as an artifact in early cave paintings, with their depictions becoming a ubiquitous way of diagramming direction in the world. We see arrows as we get on an escalator or while driving a car on a one-way street, a means of establishing order and rules for our own safety.

Much of this book was conceived and written during the COVID-19 pandemic. Social distancing (or rather physical distancing) has become the new norm in these unprecedented times as we anxiously situate ourselves at a safe distance away from others, six feet away. We have grown accustomed to arrows to indicate the directionality of movement in an aisle of a grocery store as we plan not only our grocery shopping but our spatial plan of checking off items on our list through intentional movement. Businesses have placed markers on the floor to guide customers where to stand in line, often with additional creativity and branding to make the experience more personal. ZombieRunner Coffee is a local business that displays a marker with a cartoonized version of the owner holding his coffee cup and running. As shown in the example in Figure 1.11, the markers help guide customers and perhaps instill a notion of safety and trust as they order their favorite cup of joe.

Arrows are an example of a broad class of techniques for directing attention. For example, user experience designers have the phrase Call to Action (CTA) for design techniques that direct attention. These techniques include layout, size, and color. The presence of arrows affects how we interpret diagrams. Arrows provide visual perception of causality and action. In the example in Figure 1.12, the arrow indicates the direction in which force is applied to the lever, and we can almost envision the rock being lifted up and the level of effort that would take. Actions such as push, lift, move come to mind, adding movement to a static diagram. We can begin to understand the effect of such force on the rock and the fulcrum that supports the lever in play. A physical phenomenon that would take a paragraph of prose to explain is depicted in a diagram with the simple addition of the humble yet mighty arrow.

Illustrations

While arrows provide a sense of movement in space, illustrations are static yet powerful images drawn from a single viewpoint presented on a non-stereo medium such as pen on paper. They can depict objects in the world and their meaning (i.e., semantics) that the human visual system will quickly understand. The illustration in Figure 1.12 defines fulcrum for a person who does not know that word. People will also go rapidly from the depicted instance to functional categories. A fulcrum can be more than a triangle—anything that allows the board to move freely in the direction of leverage. Think of illustration as yet another means of diagramming the physicalities of the world into a single image. An effective illustration determines which lines should be drawn to maximize the amount of information conveyed while minimizing the total number of lines drawn. Edge lines consisting of surface boundaries, silhouettes, discontinuities, and creases to separate individual parts are used to suggest important features in the shape of each object. There are many line weight conventions from which the illustrator chooses based on the intent of the image. There are three common conventions for the use of lines in illustration:

• A single line weight used throughout the image
• Two line weights with the heavier one describing the outer edges
• Varying the line weight along a single line emphasizes the perspective of the drawing. Most illustrators use bold external lines, with thinner interior lines, which aid in the perception of spaces.

Continuing with the theme of coffee for a bit, Figure 1.13 is an illustration that explains the first design for the French press dated back to 1852 (Hoffman, 2014). Mayer and Delforge, two Frenchmen, describe their design in a patent showing the cutaway of the various labeled parts: the beaker, plunger, lid, base, and handle. It was a simple design that did not create a seal inside the carafe like the one you find today. Yet the illustration is functional and visually explains the components of the coffee brewer to the reader. The position of the labels shows the relative spatial relations of the parts across the two figures for cross-referencing. The outer edge lines are drawn thicker to accentuate the silhouette of the container. The thinner lines provide shading and contour information on the two-dimensional paper surface to indicate the curvature of the handle, plunger, and the beaker.

Maps

Similar to illustration, the map is another unique form of symbolic representation but is specialized for space. It depicts spatial relationships with objects in the geographic and physical landscape. By scaling down to a natural size conducive for reading, a map allows the viewer to see far more information in one glance than could ever be possible from ground level without the need for any pictorial realism. Creating a map involves integrating the 3D projection of landmarks and paths onto a flattened plane. While we may think that this translation is no ordinary feat, humankind has been creating maps as far as history can go, starting with the intentional arrangements of stones to understand the movement of the sun to the earliest cave paintings perhaps depicting a hunting terrain for the cave people. In fact, some of the early maps have rather clever representations of space and navigation.

Figure 1.14 presents one such example. Medieval mapmakers were aware of the Earth's sphericity, yet maps were schematic, as exemplified by the T and O map (or T-O map). It is a type of early world map that represents the physical world as first described by the seventh-century scholar Isidore of Seville in his De natura rerum. The map is rendered with a stylized T-form of the major water bodies separating the continents and the O as the ocean surrounding the world. The orientation with east at the top of the map was often used.

Maps have inspired the early roots of information visualization by depicting information as a communication process (MacEachren, 2004). While details of these depictions vary, all maps share a basic structure with an information source identified by the cartographer who determines what and how to represent that information to the reader. Cartographers have come up with filters that information must pass through on its route from the real world to the map medium, and ultimately to the user. These filters consist of goals, knowledge, experience, and context that influence the abstraction process by which information is put into map form. Cartographers have found ways to make maps more legible and useful. They often simplify or eliminate less semantically important features, exaggerate more important ones, and resolve visual clutter to improve information quality. This is a process known as generalization.

To provide spatial lucidity and overview, there exist various generalization operators, including simplification, aggregation, and exaggeration. These operators are applied based on the intended purpose and context of the imagery and are particularly useful when maps need to be drawn at different scales (think of a mobile screen vs. large display). Lines, space, and color are used for integrating and segregating properties, providing structure and organization to the geospatial information presented.

Exaggeration: As shown in Figure 1.15, spatial features are relatively enlarged so that the map's featured characteristics remain prominent at different scales or to draw the viewer's attention. This enlargement leads to a distortion in the representation of the object when compared to its true shape or size. An example of this is to make a small, slender spit of land larger in order to see its characteristics more clearly.

Simplification: As shown in Figure 1.16, shapes of retained features are altered to enhance visibility and reduce complexity or visual clutter. Often in a map, roads and region outlines are simplified, as seen in the image on the right. Small curves and points are removed to simplify lines and polygonal outlines.

Aggregation: As shown in Figure 1.17, aggregation combines features of similar characteristics into a single feature of increased dimensionality that covers the spatial extent of the original features. Data aggregation is used for summarizing, partitioning, and simplifying data.

Typification: The typification operation refines the density of marks in a spatial area. A set of marks (i.e., points, lines, or polygons) is replaced with a smaller set of the identical marks to depict the same number of relative distributions between clusters of marks but reduce visual clutter, especially when the map is drawn at a smaller scale, as can be seen in Figure 1.18.

Merge: Merging takes multiple objects and combines them into a single feature, as can be seen in Figure 1.19. Features that are too small are grouped together and represented as a unified feature. Other examples include combining a bunch of proximal buildings in a university together and representing them as a single building in a campus map.

Label prominence: The label prominence operator emphasizes place labels based on their relative order of prominence. For example, as can be seen in Figure 1.20, large cities are made more prominent (e.g., larger labels in bold) than smaller cities; interstate highways are made more prominent (e.g., lines are thicker) than local streets.

Route maps have succinctly applied various forms of generalization to effectively provide navigational direction. Generalization helps emphasize what is important and deemphasize the unimportant, showing points of interest, turns, and interactions such as pan and zoom. We will discuss more about maps and how they influenced the discipline of information visualization when we head to the next chapter.

Summary

The perceptual emphasis in this chapter sets the stage for discussion of how semantics and intent are linked to perceptual input in the interpretation of the use of charts. In the next chapter, we continue our journey of applying visual perception and cognitive processes to chart design and construction. These topics are critical to the design of functionally aesthetic charts to facilitate visual thinking, symbolization, and generalization.