High Expectations

Given holograms are so prevalent throughout modern entertainment, we collectively hold a very high bar for what feels right when it comes to experiencing them in real life.

Movie magic has done an amazing job of portraying how these digital stars should react to people. Special effects have come so far there’s almost nothing we can’t imagine holograms doing for and with us on screen. Directors and writers can dream up the most amazing situations for their characters to utilize holograms without worrying about whether it can be created or not. There seems to be no limit to what clever technical wizards can pull off convincingly in service of compelling narratives.

The Iron Man movie series is a great place to start if you want to fast forward to see and hear what holographic experiences will eventually be like. Those films offer a glimpse of what fluid interaction and seamless integration of holograms looks like. They are simply just another tool used by Tony Stark (Iron Man) to accomplish his goals — not a special environment or mode he enters to get work done. As depicted in the films, holograms are a very welcome replacement for the ridiculous movie interfaces of the past. They work for us, not against us. And because of that injection of intelligence, we love what they represent — the bright and shining future.

Sci-fi movies use projection, voice control, and gestural interaction as the primary ways to interact with holograms. Guess what? We caught up.

The holograms of Iron Man appear in both ambient environments and integrated directly into equipment, such as a helmet’s heads-up-display. Voice is often the primary input mechanism for Tony Stark, supported by articulated gestures for manipulating the holograms. Meticulously designed sound is also an important supporting element for the interaction sequences. Taken all together, there’s not much missing for what you’d want out of a next-gen computing interface (unless dealing with mechanical robots or androids is more your thing).

Now comes the hard part – translating all this convincing fiction into working software code that behaves as expected. As we turn our attention to bringing that amazing vision of holographic experiences to life via Windows Holographic, Microsoft HoloLens, or other mixed reality technologies, you quickly realize we’re still very much at the beginning of a long journey to discover how to make all this be as captivating as in film.

It’s harder than it seems, I assure you.

Paradigm Shift

Do you remember the first time you ever saw an IMAX movie with 3D glasses on?

I do. The IMAX movie Hubble 3D was so intensely beautiful, and so riveting, it literally brought tears to my eyes. The whole experience was so completely different than the normal way of seeing a film that I almost couldn’t pay attention to the movie itself. The spatial surround sound shook me to the core. I was mesmerized by the view of the Earth from space and the astronauts popping off the screen and appearing right in front of me – so close I could touch them. It was absolutely overwhelming. That amazing journey into space immediately changed my perception of what was possible as a storyteller.

Astronauts float above the audience in Hubble 3D (source: IMAX)

Holographic computing is doing the same thing for us today as IMAX 3D did for moviegoers when it arrived in the 1980s. The IMAX 3D technology redefined what a movie experience could be like for the audience. The integration of so many new elements together created a new standard, and raised expectations for impact, visual clarity, sonic landscapes, and proximity to the actors.

It completely changed things for movie makers, too.

And that is our biggest challenge in a nutshell – designing and developing holographic experiences is such a different paradigm than all the other forms of computing that have come before it, we don’t know what we don’t know. It’s almost like we’ve been transported to an alternate design universe where everything went from being flat to now fully 3D. All the foundational design elements are familiar, but the interaction models are all new, inverted even. It requires a bunch of experimenting to learn what works. Very few people get this right the first time. Don’t get me wrong, it’s fun to explore this universe, but it’s difficult to find your footing.

The one thing we do know for certain is that paradigm shifts like this one provide great new opportunities along with challenges. Holographic computing is no different. It’s a clear blue sky full of new ideas and possibilities, and no place for old baggage and inappropriate models.

The key to designing for mixed reality is completely letting go of the past.

If you are thinking about porting an existing piece of software or content into the holographic world as a “flat app,” please don’t. Let’s not repeat ourselves inside mixed reality. Now is the time to leap ahead, not fall back on old approaches or drag the past into the future. The world of mixed reality is a new frontier that deserves your best thinking. Take the time to reimagine what it could be like. In fact, that’s exactly what this book is all about.

Our challenge is to push forward in appropriate new ways. It’s hard, but necessary.

New Skillset

Another challenge when moving into holographic design is the feeling of starting over and needing to learn a bunch of new skills and methods. As Designers, our first thought when working in a new design space is to leverage our past experiences to get a quick start. After all, our design thinking, process, and techniques have delivered successful outcomes more than once in the past. These are tried and true methods we’re talking about. Sound thinking, right? Most of the time it is, but it’s dead wrong for holographic.

Designing holographic experiences requires its own skillset and techniques based on spatial thinking, natural inputs, and a bit of theater. Our standard way of solving problems for the web, mobile, or the cloud are not much use as we begin working with a medium so different, that’s still evolving, and certainly not uniform or predictable (just yet).

As we make the leap into fully immersive 3D spatial interaction, we should reconsider just about every aspect of our design thinking and development approach to better align with what’s needed here. Starting with a more human understanding of the person we’re designing for, and continuing all the way through the final execution of the tuned graphics and code, we should at least try to adopt the beginner’s mindset. We’re essentially starting fresh. And from that perspective, you might be better off as a young film student doing their first assignment than a software industry veteran who’s a bit set in their ways. Regardless, just open your mind and get ready to explore and learn every day.

It turns out that holograms need you to grow as a Designer.

A traditional designer’s skillset focuses on composition, typography, color, layout, and voice, which are not discarded here, but aren’t nearly as important as understanding your new holographic canvas (the physical space that you’re in right now, rather than the flat screen you are reading these words on). Consider how someone can place what you designed to be used on a tabletop up on the ceiling instead. How does that change your thinking?

Exactly. All bets are off. Mixed reality will stretch us to constantly adapt our design to the current context and rapidly changing conditions.

In that regard, becoming fluent in holographic design is just like learning a new spoken language — it has its own unique flow, idiosyncrasies, and accents. You just cannot get around the need to practice every day in order to become fluent. The only problem with this analogy is that you can’t go out and buy a conversational book to learn how get in synch with your audience. That takes some time and real commitment to experimenting, but you’ll get there.

Spatial Thinking

In order to express yourself well with this new medium, you’ll need to get comfortable thinking and acting in a three-dimensional space. That doesn’t mean you need to immediately learn 3D modeling (that’s optional). It has more to do with embracing space as your canvas rather than flat rectangular screens. The world around you is now the stage for your ideas, not the confines of windows or cards. Consider all of the air and space that surrounds you “inbounds.” Content can be placed and worked with anywhere in the actual room you are in, or even behind walls and under floors. Anything is possible within the space we occupy, even sharing experiences with other people.

As you’d expect, thinking spatially rather than screen based is a major shift, and is certainly harder in practice than it sounds for many of us.

Almost everything before this point in the history of user experience design has been all about designing and controlling what’s behind the glass —windows, menus, icons, dialogs, and buttons. Now we find ourselves sharing physical space with these life-sized digital objects that can appear above, below, and behind you. That takes a colossal mental pivot to start designing completely outside the glass and into your physical world. It’s the realm of spatial thinking, proximity, placement, legibility, and physical fatigue rather than pixels and resolution breaks.

The physical space all around us is our canvas for holograms (source: Microsoft)

Engaging in spatial design thinking raises all kinds of questions immediately – How does the size of the room and what’s in it affect your experience? Does physical proximity and positioning mess with your intended interaction model? Can people resize your objects to be bigger than the space you’re in? Why do people get to put things where they want?

There are tons of questions and few ready answers.

Natural Input

Because this is a relatively new design field, very few of us have mastered using the room itself as the interface and our whole bodies as the input mechanism. Yet, this type of natural input model utilizing our voice and whole body for interaction is foundational for holographic computing.

The closest we may have come to using this kind of natural input model in the past was with the Kinect for Xbox . When it was first introduced by Microsoft in 2010, the ultra-high tech Kinect sensor array brought full body interaction and voice commands to the Xbox console gaming experience without the need for extra controllers. The marketing at the time cleverly proclaimed “You are the controller,” and it was never more true. Kinect-based games asked players to jump, wave, turn, and use their hands instead of using the Xbox controller. The Kinect sensor would detect your body movements in real-time, and as a bonus you could finally speak naturally to your console and command it to do things. It was awesome. And it felt completely new.

Kinect for Xbox introduced voice and body gestures as primary inputs

Yet, despite huge initial sales of the unit, brilliant technology, and engaging gameplay, there’s little new activity around Kinect games today. It serves as a good example of how challenging it is to land a new interaction model successfully (as is the case with holographic computing). The difficulty of designing for that kind of full body input and interaction model, plus the physical space required to play the games themselves certainly contributed to a falloff in interest. With that said, Kinect was a huge technological milestone on the journey to holograms. It clearly paved the way for the mind-bendingly complex sensor, microphone, and camera technology packages needed to power holographic computing. And it introduced the general public to the idea of natural input and voice for controlling computing or mobile experiences before Siri, Alexa, or Cortana made their appearances as our voice-powered digital assistants.

Natural input still remains a challenge to do well.

Device Required

There’s a great deal of design and conceptualization work you can accomplish with just your imagination, but at some point there’s no getting around the fact that you’ll need to have an actual holographic device to create breakthrough experiences.

Holograms are too unique to be simulated accurately in software only. There’s a feel you only get when seeing them with your own eyes and it’s nothing like being inside the actual headset or a projection environment to sense what is happening firsthand. Today’s software emulators are great at getting us close, but they’re not accurate enough when it comes to nailing the scale, positioning, lighting, and presence relative to ourselves in the real world.

Any code that’s written for an emulator first is almost always tweaked for real world use in the device. The holograms appear too close or far away, aren’t quite lit properly, or aren’t blending in as expected.

This device requirement should not be viewed as a deterrent to proceed with your ideas, just note it as a challenge to overcome at the proper time.

Limited Data

Our industry craves usage data to help make better decisions. The more data the better. But, let’s be honest here and point out that unlike mobile, there are not currently billions of holographic computing devices out there in the world generating petabytes of usage data each day. We know the number of existing units will continue to grow each year as many major tech companies have placed huge bets on this space developing as the future of computing, but there aren’t tons of data points out there to help us learn why some approaches work and others don’t.

Data won’t save us this time. We need to learn as we go.

At this phase of the journey, you need to be comfortable going with your gut instead of wasting time searching for data before making a decision. Form your hypothesis based on experience, then run some experiments to see how real people react to the approach. Adjust as you learn more. Don’t get me wrong here. We should always try to instrument our apps and services to learn what we can from the real-time telemetry and usage data. Just be aware that trying to base your decisions on a body of previous data is not going to work out in many cases.

Our best data will come from working closely with people to understand what works, or doesn’t from their perspective, not from usage logs.

Real People

Unlike many other design spaces where you start from previous examples, telemetry data, or marketing personas to understand who you are designing for, mixed reality needs you to develop a true empathy for the people at the heart of your new experience. Real people’s emotions are the design center of gravity here, not the faceless “customer” cardboard cutout that agile teams obsess over as their target. There’s no amount of usage data or neatly organized focus group responses that will substitute for seeing a real person’s spontaneous reaction for yourself.

No amount of usage data beats seeing real people’s reactions to your designs up close

Holographic is truly personal computing.

The intimate feel of holographic experiences is partially due to the device sitting on your face, but more so from the digital artifacts sitting dead center in your field of view. Your brain is working overtime figuring out if these holograms are real and a threat. It’s kind of spooky. And that’s a game changer from an experienced design perspective. Not many of us believe that Apple’s Siri is a real person (yet), but these holograms that you see and interact with evoke emotional responses in a way we have rarely seen outside of “scare” virtual reality or immersive hardcore gaming. Pure emotion is a powerful thing to tap into. Mixed reality always has emotion turned on, ready to tap into.

The challenge then, and one of the key things to understand about moving into this realm, is that you have to (as the great golf philosopher Chevy Chase once said) “be the ball.” Put yourself right into the shoes of the person at the center of all this as often as you possibly can, through whatever means you can manage. It is the person using your ideas that matters, not the ideas themselves.

Holograms are nothing without people bringing them to life.

Envisioning

By themselves, none of the design challenges mentioned in this chapter are stopping us from figuring out what that next breakthrough holographic experience is. Combined, they are formidable but merely temporary obstacles to overcome by using a design technique called “ envisioning ,” which we’ll cover next.