Part I. Design and Art Across Digital Realities

We live in curious times. Of the nearly eight billion humans who live on Earth, for the first time in history, the majority are literate—that is, able to communicate with other humans asynchronously, with reasonably accurate mutual understanding.

But human expression goes beyond language. Design and art reflect that which might not be so succinctly defined. The unspoken behavioral patterns of the world, writ large, are reflected in excellent design. The emotions and social patterns that direct our unconscious brains are laid bare in art: sculpture, dance, paintings, and music. But until the digital era, these areas of human expression have been, in the end, always tied to physical constraints: physics, real materials, and time.

Computers are, in essence, our attempt to express ourselves with pure energy—light and sound beaming into eyes and ears, haptics buzzing, inputs manipulated any way we please. But, to date, much like design and art, computers themselves have been restricted to very real-world limitations; they are physics-bound glass windows beyond which we can see digital worlds, but to which worlds we cannot go. Instead, we take computers with us, making them lighter, faster, brighter.

In 2019, we find ourselves in another curious position: because we have made computers more mobile, we are finally able to move our digital worlds into the real world. At first glance, this seems a relatively easy move. It’s pleasant to think that we can simply interact with our computers in a way that feels real and natural and mimics what we already know.

On second glance, we realize that much of how we interact with the real world is tedious and inconvenient. And on third glance, we realize that although humans have a shared understanding of the world, computers know nothing about it. Even though human literacy rates have increased, we find ourselves with a new set of objects to teach all over again.

In this part, we review several of the puzzle involved in moving computers out of two dimensions into real spatial computing. In Chapter 1, Timoni West covers the history of human–computer interaction and how we got to where we are today. She then talks about exactly where we are today, both for human input and computer understanding of the world.

In Chapter 2, Silka Miesnieks, Adobe’s Head of Emerging Design, talks about the contexts in which we view design for various realities: how to bridge the gap between how we think we should interact with computers and real shared sensory design. She delves into human variables that we need to take into account and how machine learning will play into improving spatial computing.

There is much we don’t cover in these chapters: specific best practices for standards like world-scale, or button mappings, or design systems. Frankly, it’s because we expect them to be outdated by the time this book is published. We don’t want to canonize that which might be tied to a set of buttons or inputs that might not even exist in five years. Although there might be historical merit to recording it, that is not the point of these chapters.

The writers here reflect on the larger design task of moving human expression from the purely physical realm to the digital. We acknowledge all the fallibilities, errors, and misunderstandings that might come along the way. We believe the effort is worth it and that, in the end, our goal is better human communication—a command of our own consciousnesses that becomes yet another, more visceral and potent form of literacy.