You can’t see the future in your rear-view mirror.
The whole point of this book is to help you approach your work in a way that will result in real breakthrough experiences. In practice, leaping forward is a bit harder than you’d think. First, considering yourself visionary and predicting exactly how things will unfold is fun, but far from reliable. Next, when looking around to see what you can copy from current competitors, remember that it isn’t going to get you anywhere in the marketplace. And finally, contrary to the myth, deeply analyzing the past is not the best way to predict leaps into the future.
Envisioning gives us a better way to tap into the future.
Combining the past, present, and future into an insightful perspective
For example, if we were looking at developing innovative sports coverage using mixed reality, a natural place to start would be with futuristic sci-fi movies that overlay information onto quick moving objects. That would translate well to augment the players. We can also see similar digital callout technology already being used in today’s broadcast sports. Looking back into the archives, we’ll notice a huge shift in how the action was televised with the advent of remotely driven camera rigs that could follow the action without being on the field and in the way of the players or officials.
Taken together as one combined viewpoint, you would suggest that our mixed reality leap forward for sports coverage would be to put us right into the middle of the action through elaborate camera tricks. This would be superior to completely immersive virtual reality broadcasts due in large part to being able to see and anchor yourself to familiar surroundings while watching. The real world acting as an anchor would potentially keep incidence of motion sickness down while enjoying fast-paced action. Not a perfect vision, but a good start.
Let’s go through some other examples where we’ve truly leapt ahead by combining different viewpoints and examine why they work.
Visualization
One the areas that will benefit the most from the rise of mixed reality is the visualization of data and complex concepts. We have made significant progress over the last decade in depicting data sets, but we have made little progress in being able to interpret them quickly. We keep drawing the same difficult to understand charts over and over again.
There are countless examples of data being poorly rendered in 3D or shown as awkward digital appendages of real-world objects, so 3D and AR are not the answer here. It’s also not the tech that’s holding us back, it’s more that we haven’t been able to leverage people’s abilities to quickly comprehend and compare based on our natural ability to deal with our three-dimensional world. That situation is about to change due to the much richer palette available to all of us using holographic visualization.
Room-scale Data
Imagine being inside a room-scale datacenter visualization showing storage problems
Working with data that is building-sized is another extraordinary aspect of mixed reality
BREAKTHROUGHS
The real leap room-scale data has more to do with the experience than the visualization itself. There will always be skeptics who point to the inefficiencies and distortion of data when it’s presented in three dimensions. What you don’t realize until you are standing knee deep in data (literally) is the visceral power of that experience. Our ability to make sense of the physical world and its contents kick in immediately. Coupled with even a passing familiarity of the data itself, the act of viewing room-scale visualizations is a bolt of lightning breakthrough for those of us who look for insights.
Understanding . One of the hardest parts of designing data visualizations is being clear enough upon first inspection to create understanding in the mind of the viewer. Many data charts are not designed to be easily read and analyzed. Holographic visualization has the built-in advantage of producing a feeling of “getting it” due to our ability to compare and contrast three-dimensional objects quickly. Used deftly, this aspect can reduce the time-to-understanding measures in this field.
Initial Impression . Holographic visualization also has another built-in advantage of producing a first impression of shock (in a good way). It’s hard to describe how it makes you feel when data envelopes you, or is spread out across a table. This qualifies as a real breakthrough moment. It suffices to say it’s nothing you’ll soon forget.
Visual Fidelity . We know that in most cases there’s little reason to use 3D charts over 2D to convey meaning. A well-constructed flat chart is plenty clear most of the time. But, there are those times when using 3D adds some other dimension of information that results in better clarity or insight. Holographic visualization has the property of allowing you to view it from new and more natural angles which creates a situation where insight surfaces more readily.
Direct Manipulation . You will immediately reach out to touch and try to directly interact with the data objects when looking at holographic visualization for the first time. It’s an instinctual reaction. They are right there. Why wouldn’t we want to see what happens? Well, now we can. It’s incredibly satisfying to use physical gestures to work with digital data objects.
Broadcast Media
Many of the early examples of pushing the envelope of mixed reality experiences focused on entertainment and education as many of their common scenarios lend themselves well to holographic content. It’s not hard to imagine being inside a game or an atom for that matter using this new technology.
As you’d expect, the seemingly more mundane area of broadcast news has been left mostly untreated. There were some interesting experiments during the 2016 United States presidential election coverage where pundits and reporters were beamed into the main studio set from remote locations, appearing as full-sized holograms. Interesting use of telepresence concepts and mixed reality tech, but not completely surprising or implausible.
This Vizrt demo beams interactive virtual graphics from Astucemedia live into the broadcast studio (source: Vizrt)
Breaking the Fifth Wall
In his insightful post Breaking the Fifth Wall , James Corbett notes we have recently been able to move past the charming cinematic technique known as “breaking the fourth wall” (where a film character surprises the audience by talking directly to them mid-scene) to an entirely new level of sophistication.
Using mixed reality techniques, digital actors can now not only break out of character to address viewers, but also inhabit the same physical space as the audience. People and things can now join you wherever you happen to be viewing from.
This is somewhat profound. We can now shift the audience from mere observer to active participant in close proximity to live action. This new capability called “breaking the fifth wall” by Corbett extends the original film metaphor to have characters literally leap out of the screen and into your physical space. There are nearly endless possibilities for this new mixed reality technique in broadcast news and sports.
BREAKTHROUGHS
The emphasis here is on cutting-edge video processing technologies for mixed reality. These new technologies enable the following techniques:
Real-Time Live Action Compositing. It’s one thing to be able to experience live action news reports or sporting events and feel like you’re really there on the field. Virtual reality has been able to create that effect for a while through 360-degree video. But, we haven’t until now been able to pull live action content into our own physical space – the complete opposite arrangement of what’s come before. Careful to not interfere with that fly ball or get swept up by that storm.
Come To Me. We are comfortable asking for the ability to travel great distances without leaving home. Again, that’s old news in the VR realm. What is not even thought about usually is asking for people and things to come into my own living room or office as digital objects that are perfectly placed, sized, and aligned with my world.
Dynamic AR callouts. Notice in the previous images the 2D textual information and 3D graphics that are deeply integrated with the live action content on screen. This level of alignment and tracking is possible today within mixed reality, but the difference here is that the capability now scales up to allow for many of these augmented reality style callouts to accurately track critical aspects of the broadcast.
Art
There is perhaps no area of creativity more interested in leveraging new technologies to educate the curious than the fine art world. Museums had traditionally been able to offer guided tours of their collections to patrons by using staffers and volunteers. They learn all about pieces of high interest so that they can answer questions on the artist, period, and contemporary works. As low-cost technologies came to market that would enable unattended tours, museums started investing in them. The state-of-the-art technology until recently was a multimedia affair using tablets to see video clips and hear narrator voiceovers. Now with mixed reality, it’s possible to explore completely different dimensions of these artworks .
Artist-in-Presence
One of the most important aspects of artwork is of course the artist. There are sometimes plaques and wall displays containing some background information on them if you look. Yet, far too many artists go unrecognized and unheralded because they are not featured as people along with their works. Somehow, we detached the person from the art, never to be reunited again except in art history books. Within galleries, the artist is predominately invisible. Until now.
Finally, artists can explain their motivation and methods to us in a personal way
BREAKTHROUGHS
Meet the Artist. Just as we never expected to be able to have a video phone in our pockets, the idea of having a conversation with an artist about their work didn’t ever seem to come up. Goes without saying that possibility approached zero after their death. Now, using easy 3D capture techniques, we can either beam in live artists to moderate discussions in real-time with students, or have artists posthumously explain their thoughts and process to gallery visitors on demand. It’s also easy to imagine vast amounts of digital content on the artist themselves being available right there along with the pieces.
Get Closer. By digitizing the artwork itself, it can now be viewed from any angle, at any size, from any location on the planet or off. The primary benefit of hi-fi reproductions is not the ability to zoom in to great detail, but rather to have the piece hold its quality constant as it is viewed and placed at any size that seems to fit the location it’s being viewed in.
View Related Work. A common question when an art patron comes across a piece they admire is “Are there any other pieces like this one in the collection?” By using voice input to ask these types of things casually, the overall experience and discovery of new artwork is greatly magnified. Serendipitous exposure to new ideas and styles is increased.
Entertainment
Music and dancing never go out of style, but it seems to be getting harder and harder to see the popular acts we like. If we do manage to get a few astronomically expensive tickets, it’s often so crowded or restricted at the venue that cutting loose and dancing up a storm is either impossible or frowned upon by security. Thanks to magic of technology, we could all don VR headsets and watch acts perform from the comfort of our living rooms, but what’s the fun in not being able to dance without smashing into your furniture? Immersive experiences are great, but much better to be able to see and interact with friends and loved ones while you enjoy the moments together in real life.
DJ Holo
With the ability to see everything happening around you in mixed reality situations, this scenario of dancing wildly around while listening to your favorite performers live with friends is entirely possible today. Using the beam in approach mentioned earlier in the Breaking the Fifth Wall example, we can literally bring the acts we’re dying to see to wherever we want to gather and rave.
There’s two parts to this of course – the ability to see and hear the holographic versions of the artists, and the ability of others to see the same thing. Done and done. Thanks to the shared experience code freely available for holographic platforms, we can synch the show for more than one person at a time. We can even add some special guests to the ambient environment in the form of holographic sidekicks or stage props.
Inviting the hottest DJ around into your lobby for live beats is just around the corner
BREAKTHROUGHS
Living Room Shows. Who hasn’t dreamed of having their favorite artists play a show in the privacy of their own home or backyard? Using the evolving volumetric video broadcasting technologies, performers can push out shows over broadband that can be subscribed to wherever you want on multiple types of devices. Whether on the go, or at home, you’ll be able to catch the show in life-size or scaled down to fit on a tabletop.
Virtual Tickets. Being able to broadcast and receive holographic performers on multiple types of devices opens up an entirely new market for live and pre-recorded content on-demand and via subscriptions.
Holographic Friends. You don’t really have to invite your friends or coworkers over to throw a party while catching the show. The same presence techniques that we use to broadcast and monitor the show can be used to pull other people into the space (or mute them if they get a little out of hand).
Emerging Technologies
One of the most fun things about envisioning is projecting your thoughts forward enough to catch a glimpse of things that won’t exist for a while. It’s usually not the case those things are impossible to create, it’s more likely the technology they are based on doesn’t exist in a form that can be put to that use immediately. Even though we try to never lead with technology as a driver rather than human needs or curiosity, it’s exhilarating to see what those combinations of emerging tech and novel approaches can create. It’s tempting to try and predict which will become true breakthroughs (even if they’re still on our mental drawing board).
Let’s go through that exercise of designing breakthrough holographic experiences based on the latest crop of emerging technologies listed below. The real question of course is how do holograms help turn these technology-influenced ideas into true innovations for people?
Sensors and the Internet of Things – What a fantastic opportunity exists to reinvent how we deal with dashboards of information reporting the exabytes of information pouring in each second from our billions of sensors around the world. Holographic displays bring the promise of using effective dimensional data visualization to help people better understand the state and trends within these incredibly complex systems.
What’s Your Breakthrough?
With all these features of other experiences bouncing around in your head, it’s time to dive into creating our own breakthroughs. Let’s get started by running through an easy to learn approach to constructing some working prototypes of our holographic innovations.