“OMG!” I can hear Pecorino, my three-year-old Chihuahua mix, exclaim. “There’s someone on the other side of the door! Come here! Quick!” My dog, of course, neither speaks English words nor gives me specific directives. But he does communicate these messages very clearly through the pitch and volume of his bark, his dancing gestures, and his overall posture. I can read the pivot of his ears precisely and know where activity is happening—outside the front door or in the backyard. I can tell the difference between someone who’s just passing by or someone who is lingering by observing his jumps—he’ll pace around by the doorway a few times for a temporary passerby but escalate to a jumping frenzy if the person stays and knocks.
Pecorino and I communicate in a sort of shorthand: I can tell what he’s talking about, where it is, how he feels about it, and what he wants to do next, all at once. Dogs have been bred and domesticated over millennia to work next to people. They have evolved to be particularly good at reading people’s communicative cues and responding in kind.1 It seems like mind reading, but really, it’s body language.
Much like my dog, a product can express itself in many nonverbal ways, and as designers we can use this communication to quickly and intuitively offer feedback and other information. In this chapter we focus on ways that dynamic product characteristics can serve as a richly expressive yet highly efficient method of communication.
Just as we read subtle cues from our pets, we can read messages and emotion from our products, perceiving nuances of dialogue and a sense that the object is “alive” when it is actively interacting. Our washing machines can wiggle their doors to remind us to put wet clothes in the dryer. Our vacuum cleaner robot can perform a dance to show that it’s proud to have finished a room. These animated behaviors blend together in a magical way, and it’s human nature to decode them as if they are emanating from a living entity.2
As products continue trafficking in more and more complex information, the temptation to use speech or screens to communicate everything is powerful. While it’s popular to think of elaborate text or voice exchanges, often called natural language interfaces, as the best ways for people and products to communicate, this is often not the case. These explicit modes of communication require our undivided focus and are far less useful in environments where we might have demands on our attention, such as in a car or in the kitchen. While today’s Siri uses lengthy interstitial language, such as “Today’s weather will be …” and “It appears that you are …,” the most powerful aspect of our relationships with our products will be the split-second, near-telepathic exchanges that can happen with just a flicker of light, a sequence of tones, or a gestural movement—the kinds of messages that can benefit from our full attention yet can also take place in our peripheral vision.
Expression forms the second ring in the social life of products framework. In the last chapter, we talked about presence, the overall impression of a product based on its embodiment, considering aspects such as its color, the shapes of its parts, and the story expressed through its overall architecture. Progressing to the next ring in the social design framework, we take an intimate look at a product’s ability to use physical presence to communicate messages and respond to the people and environments around it, making apparent important details about its internal states. We’ll consider ways that a product can enhance its physical form and make the most of conditional abilities. And we will expand upon ideas around the social affordances of dynamic behaviors crafted by expressing information through light, sound, and motion behaviors, exploring the wide palette for expression that’s available to product designers.
In Turn Signals Are the Facial Expressions of Automobiles, Don Norman explains:
Facial expressions, gesture, and body position act as cues to a person’s internal states. We often call these things “body language,” the name indicating the communicative role. Body language makes visible another’s internal state. The blush of the cheeks, the grimace, the frown and the smile all act as readily perceivable external signals of a person’s internal state, making visible to observers what would otherwise be difficult or impossible to determine.… The lights and sounds of an automobile play a role analogous to the facial expressions of animals, communicating the internal state of an auto to others in its social group.3
Starting at the very core of a product’s essence are several messages I have observed in my work that products tend to communicate to the people using them on a regular basis. Some of these things take place so often that they demand as much shorthand as possible to avoid a constant and annoying barrage of messaging. Table 4-1 shows some examples.
“I’m alive.” |
Power, via cord or battery, is connected. |
|
---|---|---|
“I’m awake/asleep.” |
Standby mode status. |
|
“I’m waiting for some more info.” |
Pinging a server or other data source before an operation can be complete. |
|
“I need you to give me some more information.” |
Awaiting a person’s input through interface elements. |
|
“I heard you.” |
Confirming that a person has input the necessary information. |
|
“I’m in the middle of doing something.” |
A process is taking place that will require some time to complete. |
|
“I’ve just finished doing something.” |
A set of tasks is complete; product is ready for a new task. |
|
“Something is a little bit wrong.” |
An error has taken place, or there is another issue affecting normal performance such as the inability to read a sensor or charge a battery in time. |
|
“Something is seriously wrong.” |
An error has taken place that will interfere with performance. |
With a floor-cleaning robot, these messages might be seen as a core set; however, there might be many more that take place throughout the course of interaction involving both functional needs, such as scheduling cleaning times and identifying floor areas, and emotional needs, such as celebrating the moment a house has been fully cleaned or expressing regret for needing to be rescued from underneath a couch. The tone and content of the messages delivered will contribute to a product’s perceived character and therefore become important aspects of designing a product that feels holistic.
As products become more interactive and content-driven, such as Amazon’s Alexa or Apple’s Siri, they add speech to the other modalities when necessary, but even spoken words need to be integrated into the overall expression, with the understanding that they are part of an entity that is also expressing messages through light, nonverbal sounds, and movement.
When my little boy Massimo was one, I greeted him in his crib every morning to see him gesticulating with both arms toward the wall and explaining in baby talk that it was 7:00 a.m. and time to get up. While I’m tempted to tell you that my infant recognized numbers and learned to read a clock, what was really going on is that he and I shared the same vocabulary of lights: the dim blue hexagon meant it’s time to go to sleep, and the bright orange-and-yellow rainbow meant it’s time to get up. Behind the scenes, I had programmed the light system, composed of Nanoleaf Aurora modular flat tiles, to match our schedule so that the colors automatically mapped to certain times of the day. The end result was a language we both understood, consisting of abstracted messages composed entirely of light. Nanoleaf CEO Gimmy Chu echoed the satisfaction I experienced, telling me that customers used the panels as “extensions” of environments to create a certain feeling, from a calm forest, to a vibrant sunrise, to a creepy dark room.
People take for granted the number of messages that we intuitively read throughout the day through changes in light. The glowing button on the coffee maker tells us when the water is heated and ready for brewing. The light on the range top warns us that it’s too hot to touch. Flashing overhead lights in a theater tell us that intermission is over. Virgin Atlantic’s Boeing 787 Dreamliner touts subtle shifts in ambient lighting, from rosy amber to bright blue, as a programmed way to communicate shifts in time zones both consciously and unconsciously.4
When driving, we use lights as an extension of ourselves to indicate when we are turning left or right. Other drivers as well as pedestrians understand this language and use our signals as the basis for decisions regarding how and where they will move across a street. Light can be used like semaphores, to convey explicit coded messages, but maybe the more powerful way that light can be used is more subtle, functioning like the blush of a person’s cheek or the furrow of a brow to convey internal state or to react to external context. When mapped well it can direct attention, set context, and convey messages. It can be seen at a distance, so it is a good choice for products that may be positioned overhead or in a corner of a room to be viewed at a glance, such as situational monitors (like security cameras, thermostats, or Wi-Fi routers). It is also particularly valuable for robotic products that move around a space and may be far from the person because it can serve as a distant beacon that can communicate clearly.
The introduction of microprocessors along with the availability of LEDs has opened up the palette available to designers considerably by introducing the ability to control the intensity of the light as well as its color, enabling more complex and sophisticated messages to be communicated with a single element. Color and intensity of light add detail to underlying messages and can be mapped to specific values along a range, with red, for example, being 0 percent and green, 100 percent. The location of the light will draw attention to a part of an object, a gestural interaction, a feature, or a form detail that is critical to convey the object’s use and meaning.
The combination of lighted elements also allows for sequences of intensity and color to be programmed so that they emit an expressive animation. The front indicator light on the early MacBook Pro is a compelling example of an object that used a simple animation in light intensity in order to achieve a powerful effect: the illusion of being alive. A soft glow shone through the body of the computer, pulsing gradually from light to dark at a regular rate that mimicked human breathing. The effect was mesmerizing and intuitive—so much so that Apple even patented the breathing status LED indicator—letting people know the computer was still “alive”—that is, the battery had power, and the system was asleep.5
Multiple lights can be strung together to form low-resolution screens that can be positioned anywhere on a product, following the curve of a surface or even glowing from within. When series of lights are adjacent to one another, the changes in their color and intensity will be read as an animation that can be used to craft a variety of messages. Robotic toy products Dash and Dot from Wonder Workshop are remarkable in their use of light. Twelve LED lights positioned in a circle on the robot’s face create the illusion of one eye. It can show attention by pointing the light in the direction that it’s headed but can also use a sequence of light changes, from bright to dim, to indicate changes in expression. A happy response can be communicated by having all the lights flashing around the perimeter of the circle, whereas sadness or disappointment is shown through lights that are lit in sequence toward the bottom of the circle.
Designing products involves thinking about the overall form and how light will integrate with it. Will it emphasize a certain part of the product, making that dominant? Will it create a sharp spotlight that immediately grabs attention or a gentle wash of light that’s reflected onto a wall, table, or other surface? The size of the light can also emphasize certain features over others. Electroluminescent panels can glow, but they don’t emit or “throw” light, so they are useful in situations where only a highlight is needed.
Light is fluid and can fill a three-dimensional space and be integrated into a surface conforming to the physical space that it fills yet still maintaining the integrity of the larger object. When we design with it, we can think about filling a form rather than “painting” light onto a “page” that can only be flat. A voluminous light can imply a space that people can enter or avoid. A spotlight may create a column of brightness and/or color to emphasize a particular area or object. The ElliQ personal assistant has a range of light patterns that glow from the robot’s head, providing a spot of light or radiating rings. In a podcast interview, Dor Skuler, Intuition Robotics CEO, explained, “The light patterns are very simple cues to give the user an idea of when she’s talking, when she’s expecting you to talk, when she’s listening, when she’s thinking, when she’s stuck, etc.”6 The light in this case thus serves as a dynamic material, changing the object’s overall look and bringing attention to it on demand.
OBJECT LESSON
Glow Caps—“It’s Time to Take Your Meds”
The Glow Caps medication bottles tackle the large problem of prescription medication compliance using an embedded light that glows to indicate when it’s time to take a dose, along with an audible signal. The system also includes a wall-mounted plug-in light so that reminders appear in multiple places. This is a good example of a design that includes light features that consider multiple contexts and possible situations. When setting up the bottles, schedules can be set to map the light indicators to times of the day to correspond to when the medication needs to be taken. Since people tend to keep their medication bottles out on a counter or tabletop, having a light indicator allows it to be conspicuous. In the case when bottles may not be visible, there is also a plug-in wall light that allows the notification to take place. The lights on the wall mount and bottles draw attention to these items over others in the room. The condition of the light, whether on or off, maps to the state (medication needs to be taken/doesn’t need to be taken). The lights are also designed to serve one type of user yet play a role in the larger system, which takes into account situations in which the light indication is not enough, and it will notify caregivers of a missed dose. A history of compliance can be provided to a caretaker or doctor through the back-end system.
Some things to keep in mind when using light include:
On stage at the 2015 TEDxPeachtree conference, Dr. Andrea Thomaz gave the introduction to her talk with the robot Curi, poised nearby.7 I had designed the shells of Curi, Simon’s “cousin,” to have similar architectural features, with expressive round, glowing ears and large, sympathetic eyes. As an important task for a robot is to recognize objects in order to navigate the environment and manipulate things appropriately, the demo setup for the talk showed the advantage of having an articulated robot body when interacting with a person. To show the power of combining words with robotic gestures, Andrea prompted the robot so it used its robotic voice to ask, “Is the green object seventeen inches from the table edge a flowerpot?” Next, she prompted the robot to make the same observation but use its arms and fingers to point and ask the question more succinctly: “Is this the flowerpot?” The demonstration exposed how much more natural the communication was when movement was harnessed. While the elaborate mechanism of the robot may seem like a great indulgence, if it were a device in the highly stressful environment of a hospital setting, the efficiency of a gesture such as the one described above could save valuable time and cognitive energy on the part of the person using it.
We are also incredibly attuned to more implicit movement-related cues. Just the potential for action communicated through a person’s stance makes for great differences in our feelings of intimacy, comfort, or threat. Formal design details such as orientation are important—we’re much less wary of a stranger who is turned away from us than someone who is looking right at us from the same distance. A short bow toward a passerby, a nod to an empty seat, or a lift of the hands toward a door handle can indicate an unspoken offer to sit or to open a door. These are not canonical gestures in any sense, but we understand them and employ them without being taught. Movement is part of an intuitive language that even animals understand.
Because movement generally requires a lot more energy than sound or light to produce and because moving parts have a greater tendency toward failure, we don’t often see it used for expression in consumer electronic devices. As these gadgets take on more tasks that already require motion, though, its use will become more ubiquitous. The vacuum cleaner robot, for example, already has motors that allow it to be driven across the floor; it only needs to be reprogrammed to use those same motors to signal issues or request confirmations from the people using it. “What is that, Vroomie? There’s a block you can’t pick up under the sofa?”
The key to taking complex communications and translating them into the elegant nonverbal shorthand of movement is to become an expert in abstraction, similar to the way that animators do. They focus on identifying key emotional moments in a film and then draw a character that can exaggerate those emotional expressions. Designers can define key moments that are likely to take place during typical product interaction and then specify the way the product should behave in each case.
Students of animation begin with a standard exercise in which they are asked to create a flour sack that’s as minimal as possible in terms of form and decoration and then animate it to express a wide range of emotions, such as excitement, shame, shyness, elation, smugness, and so on. The assignment is restricted to a flour sack in order to focus the animator’s efforts on the movement above all else. It’s a formidable challenge, but talented animators manage to make that sack really come to life despite its lack of any real physical characteristics like a face, limbs, or head, evidence to the fact that nuanced movement can be used as a language to communicate emotional messages.
As product designers we challenge ourselves to do a similar thing using moving parts of products. If we design a simple overall form such as a cube or a cylinder, we’ll need to rely on dynamic behaviors to allow it to communicate like some of the social robots being developed in labs. Given its form, we will ultimately be relying heavily on abstracted or implied movement to convey emotional messages and may ask ourselves questions such as: How can we make a cylinder bow to express regret? Could a cube inflate itself triumphantly to show pride? Or cower to be fearful?
Movement is often the most exciting of the three modalities we explore here in that it can affect the entire architecture of a product in dramatic ways, adding richer meaning to the stories that a product tells. A solar-powered lamppost with a head that rotates can give the impression of yearning as the lamp reaches up to face the sun. A security robot that patrols a street gives the impression of greater awareness of its surroundings than one that stays stationary on a corner. Indeed, any object that can autonomously navigate a space gives the impression of agency based on the fact that it can move its body on its own based on cues from its surroundings.
As compelling as movement is, it’s challenging to develop because of the complexity of the electromechanical engineering required. Here are a few things to keep in mind when designing for movement:
OBJECT LESSON
Clocky the Runaway Alarm Clock—“Catch Me if You Can!”
Clocky, made by Nanda Home, is a simple alarm clock on motorized wheels. When the alarm is triggered, it rolls away, off the nightstand and onto the floor, shrieking its highly annoying alarm until somebody chases it down across the room to hit its snooze or off button. For those with a tendency to abuse the snooze button or wake up so foggy they can’t remember just why they wanted to wake up in the first place, Clocky can be the long-awaited solution to getting out of bed on time.
Clocky is not what we would call a “user-friendly” product, but it is effective. Its form boasts a rounded body, pronounced wheels, and an almost smiling face, which at first read seems friendly. After a few morning chases, however, the same form reads as far more menacing. Its rolling is an exhortation, “Catch me if you can!” Clocky’s ability to move is critical to its agency; because it does not sit and obey our sleepy orders, it can save us from ourselves in our weakest moments.
It is also not a particularly sophisticated product. Compared to app-based alarm clocks, it offers relatively few features. There’s no option for multiple days or changes in display aesthetics. Nonetheless, it still feels like a roving robot. Because its motorized wheels move around in the space, it gives the powerful illusion that it’s a creature moving of its own accord, thus triggering one’s defenses to wonder what’s invaded the bedroom, even if we can only see it peripherally.
In the ongoing search for efficient and engaging interactions, sound is a promising form of both feedback and input. Much the way my Italian cousin, Silvia, and I have a shorthand language that includes words, phrases, and fragments of sounds (“Uffa!” “Aiyee!” “Eh!”) that represent a variety of ideas, so, too, can a person and a product share a similar lexicon.
In some sense, sound is the most natural way for people and products to conduct social interactions because it’s at the foundation of how we communicate with one another as human beings. Whereas the semantics of light and motion require a translation from the original form into a message that we will decode (e.g., green lights indicate “all systems go”), sound can be delivered most directly to people in the form of verbal messages that we already know and understand. Whether they are language-based or warning sounds about what’s happening in our environment (the footsteps of an intruder, the imminent fall of a branch, etc.), we have a keen awareness of the sounds around us, and our ears perk up to look for meaning in them.
Sound is invisible, yet it can be ever present throughout a space, traveling across a room and filling a volume. Unlike light or movement, it can be detected by a person who is not looking at the product and thus can offer a freedom to multitask and layer messages on top of visual or tactile signals. It can be used by an appliance, such as a washing machine or dishwasher, to indicate when a cycle is complete but can also be used to deliver highly specialized and sophisticated messages such as news stories or detailed stock reports.
Sound is also an extremely efficient means of communication if we consider how quickly a very compact message can be conveyed and comprehended. In The Sonic Boom: How Sound Transforms the Way We Think, Feel, and Buy, composer Joel Beckerman describes the phenomenon of rapid recognition that we’ve all experienced at some point in life when we hear a fraction of a musical note, yet the timbre of the music and the pitch of the sound combine to evoke a memory of an entire song: “When the songs are more familiar, then a lot of these motor or so-called premotor brain areas became engaged. Within milliseconds, you’re not only recognizing the tune and feeling emotional about it but also rehearsing what action to take to respond to it.”9
In considering this phenomenon, a product strategy can include both foreground sounds that need to catch one’s attention and background or ambient sounds that can provide information if a person is actively listening but don’t demand immediate focus. A device monitoring a hospital patient’s heart rate, for example, can allow a nurse or caretaker to know the rate is within an acceptable range with an ambient sound that blends into the background but also alerts them when it goes beyond range with a louder, more strident sound that demands attention. What’s particularly tricky for nurses in this situation is that they can become accustomed to the alert sounds to the point where those, too, blend into the background while creating noise pollution for patients and family members, so it can be important to plan the sound design carefully to avoid the “crying wolf” effect of alarm fatigue.11
We can think of sounds in terms of two classes of messages: literal and representational. Literal messages are actual words, phrases, sentences, paragraphs, and so on that emanate from a product. Representational aural messages take place in the forms of tones, melodies, blips, bloops, and more. They are the bell tones that you hear in a car interior when the driver or front seat passenger hasn’t fastened his or her seatbelt or the beeping that emanates from your microwave oven when the cooking time has elapsed.
Sound is not an embellishment but a reflection of the heart and soul of the product and therefore must be given a good deal of attention during the design effort. Professional designers are increasingly turning to sound design specialists in order to invest an appropriate amount of effort into this essential aspect of interaction.
Spoken words command our attention. They are also very reliable because they do not need to be decoded or translated. They feel natural because they are in the same language we may use to command the product to do what we need. When we say, “Alexa, what’s the weather today?,” it seems logical for the device to respond in kind, using the same language. Furthermore, we don’t need to learn anything about an interface to interpret a spoken word message; simply knowing how to communicate as a human being is all the training we need to know how to use the product’s interface.
On the other hand, do we really need all those words? We are living in an era of heavy language usage in new products that employ conversational agents and speak to us in full sentences and even add some unnecessary sentences of their own in the effort to render themselves to us. Every time I set a timer with my iPhone, for example, I hear the standard tones that let me know that my request was heard, followed by a voice confirmation that says, “Your timer is set for XX minutes, and the suspense is killing me.” It’s surely cute, but is it necessary? Similarly, when I ask Siri what the weather is, the device says, “It appears to be raining right now in New York, with a temperature of fifty-five degrees.” While it can be comforting to hear sentences composed the same way a person might say them, the filler words like “It appears to be” are in many ways superfluous.
Dor Skuler of Intuition Robotics described a strategy around what I might call digital integrity. He explained a focus on transparency, including attributes such as a filter on the computer-generated voice to give it more of a robotic feeling, rather than trying to have the device mimic human speech. Led by Yves Bèhar’s firm Fuseproject, the design team leaned heavily on multimodal “body” language through lights, sound, and movement instead of simulated facial expressions. He explained, “It’s always an honest relationship that makes very clear what it is, what it isn’t and what it can or can’t do. It doesn’t try to fool you.”12
How can you develop this distinctive mode of communication? It’s helpful to literally write out the conversation in “longhand” form, as if it were a screenplay. While the intention may not be to have the product literally say the phrases you’ve written, it’s a great starting point for what is essentially a translation process. Your team can then place relative value on different parts of the interaction to determine which should be front and center and which may be more appropriate for the background of the user experience. Animator Doug Dooley describes this process in terms of the intonation an actor makes with the phrasing that travels up or down in pitch, with “up” at the end of the phrase being associated with more positive sentiments and “down” at the end of the phrase as being more negative.13
In addition, there are many practical and technical elements to keep in mind when crafting the sound output:
OBJECT LESSON
Jawbone Jambox Bluetooth Speaker—“I’m Connected!”
Minimal physical outputs enable most of the Jambox’s expression to take place through a series of sounds as well as spoken words. While there are buttons in the form of a chunky circle along with plus and minus signs, the interface doesn’t depend on a screen output for feedback. Instead, small blips confirm button presses, and spoken phrases like, “Jambox is connected!” alert people to events such as low power and Bluetooth pairing. It has a signature sound and voice but also allows customization with voice personality and language preferences. For a brand that has built its reputation on high-quality audio products such as fashionable Bluetooth headsets, it makes sense to demonstrate the core value of the product—its sound output—through every moment of interaction.
When planning a product, it’s essential to detail the ways that each of the modalities—light, movement, and sound—will operate in isolation in order to develop assets and assess overall messages. However, the experience a person has with a product will ultimately be created by a combination of all the dynamic characteristics at once. Below, I lay out the benefits of each as a guideline to deciding when to use each one.
In the next chapter, we will move to the next ring on our framework, interaction, which takes into account what we know about product architecture and dynamic expression and then adds to the product the ability to truly interact by sensing the people and environments around it. The product then can not only offer messages about its internal state but communicate real-time responses as well.
IN THE LAB
Using Sound
When working for the firm Smart Design, designing a floor-cleaning robot for a company called Neato Robotics, I asked the team to break down the product’s behavior into critical “moments” for personality definition. With people, we know that a person’s true character is revealed during moments of extremes—both negative (stress, anger, fear) and positive (pride, jubilation, satisfaction)—and so we used these moments as key elements to define the robot’s character. To create an abstract language of sounds, we detailed every message that the robot might have to convey in “human” language of words and phrases such as, “I’m done cleaning now!,” “My battery is running low,” or “Help! I’m stuck under the couch.” Types of sounds were categorized so that expressive moments such as wake up or cleaning complete could take place through melodies, but other sounds could be expressed as alerts or feedback blips. Just like we might work with a color or material specialist in a traditional product, here we worked with a music composer named Skooby Laposky to create a palette of sounds. The human language was translated into a language of musical phrases and tones that conveyed not only the content of the message but also the emotional aspect of it, from the distressful alerts to the jubilant celebrations.
When the unit is powered up, it has a “wake up” sound that is also the signature sound for the brand. It’s a short melody that’s memorable and catchy but takes place in under two seconds. When it’s beginning its cleaning cycle, it sounds another short melody, communicating the message, “I’m off to work!” A final melody takes place when the unit is back in its base and about to turn off, or go to “sleep.” During the cleaning cycle, there are several potential moments of interaction that relate to both core cleaning tasks and social exchanges with the person. The moments were then earmarked for three different types of sound that would vary in tone, duration, and volume, depending on the context:
35.171.182.239