That Feeling When (TFW)…

TFW you’ve replayed that cute otter video more than 10 times.

TFW you see yourself reflected back to you in your Spotify Discover Weekly songs.

TFW you made your Kickstarter goal much faster than expected.

TFW you discover your child having a heartfelt conversation with Alexa.

TFW your tweet goes viral and you don’t have a Soundcloud to promote.

TFW your WiFi goes down and takes your smart locks with it.

TFW Twitch subscribes you to yourself.

TFW you lose your teen’s Snapchat streaks while they are away at phone-free summer camp.

TFW your love language is texting links to those near and dear, and then you realize that you’ve become the next generation of your aunt sharing links on email or your grandmother sending newspaper clippings.

Our so-called life with technology is complicated. It’s becoming profoundly stranger at an exponential rate. And the most important parts of it are invisible.

At times, we’re keenly aware of the highs and lows of digital life. We feel an overwhelming sense of camaraderie after a serendipitous conversation. Or we feel a rage hangover after a particularly difficult news cycle. Sometimes, the emotional undercurrent shifts just below the surface.

Technology is deeply embedded in every aspect of our daily lives. It collects our secrets, it bears witness to our everyday actions, and it circumscribes our relationships. Yet somehow, it’s still radically insufficient for our messy, beautiful, emotional reality.

With all the focus on ease, efficiency, and convenience, we might miss what’s truly essential. We worry about our attention, while technology toys with our affections. We idealize calm, when tech takes us on an emotional rollercoaster.

Now more than ever, we need technology that respects what it means to be human. That’s what this book is about.

The True Story Behind This Book

It was the summer of 2014. Our team collected about 50 diverse characters—film crew, makeup artists, agency team, agency clients, agency clients’ clients, other consultants—in a space not uncommon in New York City: an airy loft with a bright-blue sofa, a whimsical display of globes, a carefully arranged bookshelf of faux tomes, a rough-hewn table in front of a chalkboard decked in flowcharts. It was the sort of space that you might rent out for wedding photo shoots or cooking shows or bespoke events.

But this was a research project: the kind of research that happens every day in little rooms and not-so-little rooms, with little devices and not-so-little devices, all over the world, but usually without the makeup touch-ups and outfit changes and pauses for microphone adjustments and careful rephrasing of questions for the final cut. We were trying to figure out that special future something, couched in a high-quality persuasive film for a corporate audience.

Here I was in a room full of cameras, bright lights, and the bustle of strangers, and into our study walks Matthew. On paper, the usual suspect. Millennial, tech-savvy, does all the right things online—shops, rideshares, streams. Ultimately, that tells me almost nothing about Matthew.

After some small talk, we huddle around his phone, and I ask him to give me a tour—kind of an ice-breaker. When he got to photos, he hesitated. “You are going to think I’m weird.” I assured him that “Weird is secret code for interesting.” He went on to show me photo after photo after photo of butterflies.

And I thought, well, if that’s weird then I’m truly odd. But I put on my best research face and asked him to tell me more. He told me about the last time he saw his father a butterfly had landed on his father’s shoulder. Matthew felt like it was sort of this magical moment when time stood still:

And I guess I was trying to bring that magic back. I don’t have many photos of my dad. So, I started taking pictures of butterflies, and then learning about butterflies, and then other people—people I’d barely spoken to in years—started sending me pictures of butterflies. I guess I’ve become a kind of a modern-day butterfly collector.

The room was suddenly silent except for the low buzz of film gadgetry. Observers leaned in. We were all connected. Everything changed. That interview convinced me that we all need to collect those butterfly moments—the rare and beautiful and meaningful.

As a designer and researcher, I didn’t start out studying emotion. But every time I looked at how people engaged with technology, there it was. Every experience was infused with sentiment, from intense emotions to fleeting feelings to lingering moods.

The past few years of my life have been devoted to the study of our emotional life with technology. Not always directly. After all, talking about feelings with strangers can invoke face-melting terror. So, I tried to get at it in all kinds of ways.

An “emotion exegesis” study had people annotate each tap and swipe with emotion. I encouraged people to sketch their experiences from memory. I asked people for their mantras, distilling their coping strategies into pithy advice. Just like the phone has “driving mode” to stop notifications or your laptop has “night mode” to soften the glow, I asked people to describe their personal modes for using technology. In another, I gifted people a magic wand that would instantly change the technology in their lives. I followed people charting new kinds of relationships with artificial intelligence (AI) companions. I hosted research playdates with kids to try out Cozmo and Aibo and Jibo. I orchestrated emotion tech hackathons.

I’d consider myself to be one of those highly sensitive people; you know the type—the one who winces at an unkind word, who carefully avoids watching anything violent (so we won’t be discussing the latest bingeworthy Netflix show), who absorbs the emotions in the room and then becomes easily overwhelmed. As a child, I’d been told I was too sensitive. As an adult, I cry too easily. Even with all these feelings, I was no emotion expert. Gradually, I learned.

The more I learned, the more I realized there’s still so much we don’t understand about emotion. You won’t find all the answers here either. My hope is that this book will draw more people into this conversation, especially people working with technology.

How You Fit into This Story

As someone who is somehow connected with technology (for the sake of brevity, I’ll call you a designer, but you might consider yourself an information architect, developer, product manager, strategist, entrepreneur, chatbot script writer, AI trainer, UXer, or even UI/UX unicorn), this is your story, too. You are probably an optimist when it comes to technology, at least a little bit. You feel like technology can make lives easier, make tiresome chores go faster, and make the world a better place. I feel this, too.

At the same time, maybe you are concerned with the way things are going. You feel the negative impact of technology in your everyday life. Or maybe you’ve been tasked with making something that started, or somehow ended up, at odds with living well.

So how do we sort out this paradoxical way of making technology? One way is to consider the invisible layer of experience. We’ve spent a lot of time and effort on the cognitive layer of technology, but less so on the emotional layer. The way we live in this new world, powered by the latest greatest technology is not rational, not by half. So why wouldn’t we reconsider the emotional?

Perhaps you wonder why you should read yet another book about design and emotion. Don Norman’s Emotional Design (Basic Books, 2005) is filled with wit and wonder, and the “robot chapter” is more relevant than ever. Aarron Walter’s Maslow-inspired hierarchy (Designing for Emotion [A Book Apart, 2011]) is near ubiquitous. It might seem like we have this whole emotional design thing locked. We don’t.

A lot has happened in the past decade or so when it comes to what we know, and we don’t know, about emotion. And technology aimed at deciphering emotion in new ways, some intriguing and some deeply misguided, is gaining momentum. Emotional design needs a reboot, too.

What if you’re a skeptic? I get that. You might think emotion is a people thing, not a tech thing. You might think tech is a neutral space, a means to an end. Or maybe you think emotion is just some marketing nonsense. This book is still for you.

Maybe it’s the Midwest in me talking, but I’m a pragmatist. Technology can’t help but shape who we are as individuals, as societies, as a species because it usually does. It’s not always obvious or evident, of course. And it’s much easier to see in retrospect.

AI isn’t the first technology to make a sweeping claim about consciousness, nor will it be the last. Just think for a moment about the compass and the crane, the loom and the lightbulb, the sewing machine and spacecraft. All technology that enables us to do new things is technology that enables us to feel new things.

Experiencing the world in a new way shapes our inner lives. So, even if you consider yourself a rationalist about technology, I’ll ask you to acknowledge your own emotional response to a book about emotion and read on.

We have an opportunity to rethink our approach to designing technology, by which I mean designing everything, really. In this new era of wearable, companion, ubiquitous, maybe-soon-ingestible technology, how will we design for emotional well-being? That’s the big question for this little book.

Here’s the Short Version

This book is a little strange for an O’Reilly book. It’s not a book all about cool new technology—that is, The Promise (or Peril) of Emotion AI. AI that detects some aspects of human emotion will be big business, no doubt. Emotion AI will practically force us to design for emotion in new ways. It will change how we conceptualize emotion. It will require us to think deeply about ethics. And it is coming on quickly.

Part of my motivation in writing this book is to get more people thinking about emotion AI while there’s still time to get ahead of it. But emotionally intelligent design isn’t just about new technology that will auto-magically reveal hidden truths. That’s not what the technology can do. And, more important, that’s not how emotion works.

It’s not a book filled with step-by-steps or code snippets or design best practices. Although there is a method to this madness, I approached the book as a way to draw together new research about emotion, new technology that engages with emotion, and new design practices. Think of it as a new way to consider something age-old.

Each chapter follows a certain path. I begin with a mashup of emotion wisdom and design method, giving a bit of background about each. Then, I consider how to bring them together, based on experimentation in my own research and experience with corporate and startup clients. So, the first part of most chapters is background, the second half how-to. Here’s how it breaks down chapter by chapter:

Chapter 1: Our Emotional Relationship with Technology

Our relationship with technology isn’t always, or even mostly, rational. While we zigzag from utopian dreams to dystopian nightmares, we rarely acknowledge the emotional impact. This chapter is a call to action for emotionally intelligent design.

Chapter 2: The History and Future of Emotional Design

Without an understanding of history, it’s difficult to work toward the future. So, in this chapter we look at the history of emotion, how it’s been translated through design, and how new tech is going to transform it once again. This chapter moves from Aristotle to AI, from sentimental education to sensors, from whales to wearables. Well, you get the idea.

Chapter 3: Designing, with Feeling

How might we bend our current practices toward emotional intelligence? With a mix of design thinking, mixed methods research, and emotion tech. Rather than come up with a completely new model, the idea here is to build on what’s already working.

Chapter 4: Cultivate Human–Machine Harmony

Let’s talk relationships. Not “building a bot” or “creating a character bible,” but thinking about how we will be increasingly living our lives with artificial friends. Framing our experience as a relationship changes the design process. Here, you’ll find a way to gauge whether your tech should have a personality, how much is enough, and how to evolve the relationship.

Chapter 5: Crafting Emotional Interventions

Digital well-being is often framed as taking a time-out from tech. A good start, but we could take it further. By blending well-being interventions with the grand tradition of design patterns, this chapter identifies positive interventions and negative antipatterns.

Chapter 6: Forecasting the Future with Feelings

What do affective forecasting, prospective psychology, and speculative design have in common? Each gives us a way to feel the future. In this chapter, we look at how to bring more emotion-sense to envision technology a decade into the future or more.

Chapter 7: Toward an Emotionally Intelligent Future

This final chapter looks at the implications of emotionally intelligent technology. Emotion impacts public policy, law, organizational culture, and society. We pause here to consider what will come next.

You’ll notice throughout that I keep the concepts of emotion and feeling, and even sentiment, loose. Some experts make a distinction between emotion as physical, instinctual, and basic, and feeling as perceptual, constructed, and complex. Others don’t. Either way, it’s complicated by the social and cultural layers of emotion. It’s kind of like that nature/nurture argument. I try to tease it out where I can, without stressing about it too much.

Likewise, you might find my use of emotional intelligence a bit unusual. Emotional intelligence comes in a few different flavors, but basically it comes down to understanding emotion in multiple dimensions. For now, that’s a very human enterprise. Soon, machines will be in on it, too. For me, both humans and machines together will formulate a new kind of emotional intelligence.

Feeling Forever Grateful

Writing this book was cathartic, exciting, frustrating, and humbling. It involved video chatting through bad hair days, half-read books lying face-down all over the house, lying awake in the middle of the night, scribbled notes in my chicken-scratch handwriting, equal parts coffee and wine, oh, and lists—lots and lots of lists.

At the heart of it all, generous, kind, brilliant people offered their time and wisdom in wonderful ways.

Thanks to all the people who shared a little bit of their lives with me, as participants in interviews, studies, and social experiments. Your generosity will have a long reach.

Thanks to my reviewers, Lane Goldstone, Anne Marie Léger, Sylvia Leotin, Cynthia Savard-Saucier, Alastair Somerville, and Mark Wyner for thoughtful commentary and tactful correction.

Thanks to sages who shared their work with me as I put together the book, especially Ben Bland, Elizabeth Buie, Rana el Kalouiby, John C. Havens, Simon Jimenez, Anya Kamenetz, Georgia King, Lucie Lemire, Andrew McStay, Taniya Mishra-Linger, Gawain Morrison, Dorian Peters, Anna Pohlmeyer, David Ryan Polgar, Caroline Sinders, Rob Strati, Lillian Tong, Marco Van Hout, Ben Virdee-Chapman, and Trevor von Gorp.

Thanks to Angela Rufino, my exceedingly patient editor at O’Reilly, who stayed with me through several different versions of this book.

Thanks to my colleagues Tony Alongi and Diana Sapanaro, who picked up so, so much slack for me.

Thanks to my parents, Andrea and George, who have provided loads of support over the years.

Thanks to my brother, George, who never ceases to surprise me with a fresh point of view.

Thanks to my spouse and partner, Steve, who made this book possible in all kinds of ways.

Thanks to my luminous daughters, Grace, Lizzy, and Eleanor. They are the inspiration for nearly everything in this book and in my life.

And thanks to you for reading, and hopefully, sharing your stories in the future.