Mindset

The best place to start when considering holographic design is trying to shift your own mindset to allow the ideas to flow freely and unconstrained by the past.

There is no question you need to think differently than you’re used to with Desktop, Web, Mobile, or Cloud models, but not for the obvious reason (i.e., 3D interaction). It’s the people part that needs deeper exploration. Our role in this computing model has completely changed.

No longer do people click buttons, pull down menus, scroll through windows, or drag icons around a screen to get things to happen. You do exactly what you do in the real world – manipulate objects with your hands and voice. Direct manipulation is no longer a complex set of interactions using double indirection that requires layers of translation. Want to see something better? Walk right up to it. Not in the right place? Pick it up and move it with your hand.

This inversion of the classic software interaction model means we need to focus on how people respond to their new environment and input model rather than the mechanics of a standard GUI interface. That aspect of holographic design is very similar to tackling the new conversational interface paradigm. Being completely voice based is a drastic departure for a visually oriented interface Designer. You can no longer rely on clever visuals to assist. Audio cues and spoken responses are your only feedback mechanisms. This seems deceptively simple at first, but nothing could be further from the reality of designing for voice input. It’s a tough mental shift to make and lock into.

You’ll find a similarly challenging mental switch when taking on holographic design. It requires imagination and empathy to put yourself inside other people’s thoughts, motivations, and expectations when they’re immersed in your scenarios. They are deeper into experiences than ever before. That’s a big responsibility. It takes reconciling the observations you make when mentally switching between thoughts of the participant, your role as an observer, and the qualities needed by holograms themselves to get this right.

The mindset required for holographic design revolves around these three inputs

Our mindset for holographic experience design has to be grounded in how we act in our physical spaces. This is not an artificial layer to interact with. It becomes part of the world itself. Our goal is to inject new elements that cannot be found in reality to create true value for the people who trust us with their time and responsibilities. Someday, our mindset will dictate that holograms are integral to our experiences. In time, we’ll forget it didn’t exist in this way before.

Approach

If we look closely at how high-tech products and services are traditionally created, we can identify a repeatable process that has yielded decades of successful ventures. Start by identifying a painful problem, figure out how to help reduce the pain, cash the checks. Easy. The approach for this new paradigm isn’t quite so straightforward.

You are actually at the center of everything in mixed reality

Very few of us have mastered using the room itself as the interface and ourselves as the controller and user interface (with the exception of avid Xbox Kinect game players). Almost everything before this has been all about controlling what’s behind the glass — now we find ourselves sharing physical space with crazy digital objects. You need to mentally pivot to design completely outside the glass in favor of being completely integrated into your physical world.

Historically, the way to approach this type of design would be from the “user” perspective. That’s how we’ve always done things in experience design. Figure out how the person will want to do whatever it is we’re designing, then optimize all of the user interface layout and interactions around that experience. The interface and user are separate things. Simple enough.

Conversely, the best way we have found to approach this holographic design exercise is to put ourselves in the middle of the action – “be the ball.” We need to figure out what the hologram itself needs out of this exchange. Weird, right? Why would we want to empower the thing we’d normally think was completely under our control and beholden to our commands? Great question.

This new approach is based on a very personal interface with the things that share our space. We need to connect with those things that we previously took for granted as merely objects. They were owned or mostly unimportant. Conversely, holograms are just as real as we want them to be. Therefore, we cannot assume they don’t have needs, expectations, and even feelings. That’s the big difference in the approach from previous generations of computing platforms to holographic.

Our new approach starts with trying to consider holograms as equals, regardless of whether they represent inanimate objects or living creatures. This is where empathy comes in. Something that is a non-living thing in the real world can very much be a vibrant and significant entity in the holographic world. Putting ourselves in the place of a hologram is a very different approach to working on interfaces. You’ll be surprised at how many innovative ideas you come up with by empowering and enabling the holographic actors in your scene to be more than what they appear to be visually. For example, things we consider simple tools can be humanized to feel like helpful friends or professional assistants instead of just objects to be picked up and used normally. They should not be expected going by initial appearance, but a pleasant surprise within the experience.

Language

To express yourself clearly in this new field, you’ll want to get comfortable thinking and acting in a three-dimensional space. There’s really no getting around that. You know this is true from your own experiences abroad. Practicing to speak a foreign language in the classroom or reciting phrases in your head as you listen to a language tutor on your smartphone is not at all like conversing with a native speaker in another country. That part of becoming fluent in 3D is just like learning a new spoken language — you learn by trying it out in the real world and making a few embarrassing gaffs along the way. Everybody has the same experience when trying to converse in a new language .

A great way to get comfortable with the language of mixed reality is to dive into using a 3D creation tool like Unity. Very popular with game and virtual reality developers, you’d be hard pressed to find a better introductory toolset. It’s a bit technical looking at first to someone who hasn’t coded or done any modeling, yet you’ll find it is quite approachable and easy to get results. The time you can spend getting a basic understanding of how people fly these tools will pay huge dividends in being able to discuss your thoughts in terms of a composed 3D scene using objects, planes, lights, materials, motion, and behaviors.

The Unity toolset is a great way to get started with creating 3D experiences

The language of holograms is a unique dialect of 3D which has its own flow, idiosyncrasies, and accents. It has a specific vocabulary and way to describe interactions. We’ll talk about how holograms can be interacted with using gaze, gestures, and voice a bit later. For now, it’s good to know that a similar language comes from theater and film direction. Mastering how to describe the thoughts and actions of holograms can be thought of as a director filling in an actor on their motivation for a scene.

All this said, let’s be honest here and point out this language is still evolving. The obvious problem is that you can’t go out and purchase Conversational Mixed Reality from Amazon to learn how get in synch with your holograms or audience. Fluency in this new language comes over time as you jump in and practice.

Skillset

At this point, you’re thinking I’m going to say that learning how to model and animate 3D characters is the most valuable skill you can learn to bring your holographic ideas to life, right?

Well, that’s a great guess. I can’t argue that having that production skill wouldn’t help things move along more quickly for any project. Modeling and animating sequences would be the most direct way to express your ideas about how holograms will interact with people. And it gets you closer to a final implementation.

But, modeling is really not the most important new skill to master for holographic design. Storytelling is.

If you haven’t needed to learn how to make an emotional connection through your verbal storytelling or visual storyboards I’d start there. Just as in filmmaking, being able to explain how the story captures the attention of the viewer is a foundational skill for anyone on the creative team. Making that emotional connection with the audience and explaining how it is achieved through story and action is a skill we can all use regardless of our role on the team. A great way to master this is by hand drawing storyboards that illustrate the scenes. It doesn’t have to be professional quality, the sketchier the better actually. The point is to convey the connection. That’s what it all revolves around with holographic design – how you engage the audience and capture their imagination.

The most important skills to develop for doing holographic design

Secondarily, learning how to quickly sketch in 3D is pretty useful. There are a number of ways to do this – paper drawings can be used, but VR sketching apps and coding directly in Unity are the most popular ways to dimensionally capture your best thoughts on how things play out in mixed reality. The goal is to show other people what you are thinking about the setup and subsequent interactions, so don’t worry about it being perfect. Just get the point across and move on. These sketches will capture the essence of what you are after, not the fine details. So, let it flow. Be quick. Be brilliant. Be done. Move on.

And finally, an incredibly helpful skill to master as we enter this new world of mixed reality is being a great participant. Jump into the headset as often as possible so you know what’s possible and what’s desirable. There’s absolutely no substitute for being there. Get right inside an augmented world someone has created for you to interact with. You will always find new things to consider about the decisions made and the promise unmet. No amount of thought, discussion, or debate will inform your ability to design breakthrough holographic experiences than being able to relate what you are going for to known examples.

There’s also no amount of projecting based on past experience that will work for you here. Imagine you are asked to describe how it felt for astronauts to see the Earth below them during a spacewalk. You can probably approximate the feelings, but I’m guessing anyone who actually had that experience was forever changed by the majesty and overwhelming sense of awe of being there. No one can conjure that amount of authenticity and be believable. Get in there and relate your own feelings of being in mixed reality with holograms.

Just Do It

Spending too much time figuring out which of the four areas mentioned above to start with when designing holographic experiences is kind of pointless. Just jump in anywhere. It really doesn’t matter in the end. What does matter is that you are actually doing it, not just thinking about it. If you gravitate toward the storytelling aspect that’s an awesome and valuable place to start, but diving headlong into creating 3D scenes as working prototypes for your ideas is equally great.

Next up, we cover how you can use the envisioning methodology to make some quick progress.