1

A New Reality and How We Got Here

For well over 100 years, audiences have looked into rectangular screens, ignoring everything peripheral to the edges of the frame. But in recent times, the edges of the screen have been removed. Narratives now have the potential to play out anywhere we can crane our necks to glance or stare. Like in life, any place we can walk to or journey toward becomes the screen for a story. This breakthrough in storytelling is changing the way audiences engage with the moving image as well as the ways we create content—and this is only just the beginning. Virtual Reality (VR) is one of the latest developments in the remediation process that has come to define digital media. According to theorists Bolter and Grusin, this process of remediation has become integral to the ongoing progress of media, which is now constantly commenting on, reproducing, and eventually replacing itself.¹

Immersive content is still in its infancy, but after Facebook’s $2 billion acquisition of Oculus, VR in the mainstream began marching forward at an entirely new pace. AOL’s purchase of RYOT to bring immersive video to The Huffington Post is equally significant in the move to bring VR into people’s homes. The affordability of Google Cardboard and demand for upcoming releases of Sony PlayStation VR, the HTC Vive, and Oculus Rift have quickly made 360-degree video something most tech manufactures are incorporating into future releases. Vimeo, YouTube, and Facebook are all already capable of 360-degree video upload and display. Samsung, GoPro, and Nokia all have significant VR products on the market as well. The technology is ready and available, and people are enamored with it. But there’s one problem. No one has really figured out how to tell a story with it.

Pixar Animation cofounder Ed Catmull famously stated to a reporter with The Guardian that Virtual Reality technology may not be the revolution in storytelling that some of its evangelists have claimed. “It’s not storytelling. People have been trying to do [Virtual Reality] storytelling for 40 years. They haven’t succeeded. Why is that? Because we know that if they succeed then people would jump on it. Linear narrative is an artfully directed telling of a story, where the lighting and the sound is all for a very clear purpose. You’re not just wandering around in the world,” Catmull said.

Critics have pointed out that Catmull’s approach—that storytelling in Virtual Reality would look and be structured like it is in cinematic narratives—is dated and lacks an understanding of the potential of the medium. Catmull has stood his ground, saying, “It’s good, but it’s not storytelling. The fact that you’ve changed the technology, and people are excited about it, doesn’t change the underlying difficulty of the compelling narrative story. Just like books aren’t the same things as movies. They don’t have to be,” he said.²

Media futurists such as Palmer Luckey, founder of Oculus VR, are out to prove Catmull wrong, just as Pixar proved the industry wrong with its 1995 animated feature and cultural juggernaut Toy Story. Luckey hailed the potential for VR films at the 2016 Web Conference, stating, “Telling stories in Virtual Reality is very different than telling stories through traditional films or even video games—it’s going to be a long time until Virtual Reality storytelling is nearly as refined as film. Decades.”³ John Gaeta, creative director at LucasFilm’s ILMxLAB, has agreed with Luckey’s sentiments, saying, “Cinema is a master storyteller’s art form. Until recently, a “fourth wall” has contained this form. Soon, however, we will break through this fourth wall, and cinema will become a portal leading to new and immersive platforms for expression. ILMxLAB is a platform for this expansion. We want you to step inside our stories.”

While Catmull, Luckey, and Gaeta lean toward the philosophical, others are making large financial investments that someone will figure out compelling ways to tell stories with VR technology. Investment researchers Piper Jaffray have forecasted that VR will be a major mega tech theme through 2030. “We liken the state of virtual and augmented reality today as similar to the state of mobile phones 15 years ago. It likely will take a decade before mainstream adoption as necessary improvements in displays and applications as well as lower pricing are needed to drive demand,” the company stated in 2015.⁴

Piper Jaffray is basing its optimism around consumers’ insatiable appetite for new tech experiences. They have suggested that on the virtual side, new immersive worlds will open up, including gaming, live sports, concerts, immersive cinema, and social experiences. They also forecast that eventually users will be able to virtually attend an NFL or NBA live game with a 50-yard-line seat, listen to a live concert of your favorite band with a front-row seat, watch movies optimized for VR, or visit friends in far-away locations. Further, they are predicting that classrooms will be able to virtually (and relatively inexpensively) tour the Great Wall of China, Egyptian pyramids, Stonehenge, the Coliseum, or the inside of a factory or laboratory. They have concluded in their report that Virtual Reality will encompass a $62 billion industry by 2025.⁵

There’s little debate that VR technology is here to stay despite the disagreements about if and how stories can be told with the medium, and it will only gain more traction as time goes on. As with any new medium, experimentation and creative thinking are what will push the tools to deeper depths of storytelling. New possibilities and perspectives will continue to arise as filmmakers forge the path ahead and find innovative ways to connect with their audiences.

Early Conceptual Ideas about VR

Virtual Reality can be and has been defined in a wide variety of ways. Some have suggested that even panoramic paintings were an early means of creating the illusion that a viewer is somewhere they are not by filling their entire field of vision.⁶ Many others agree that stereoscopic viewers, popularized in the early 1800s, were a step forward in creating a three-dimensional experience and somewhat resembled the head-mounted displays (HMDs) we encounter today. A century later, these devices were mass marketed by the View-Master company, and children have enjoyed the technology ever since.

In the 1930s, science fiction author Stanley Weinbaum wrote a story called “Pygmalion’s Spectacles” and seemed to foretell the VR we experience today. French poet and playwright Anton Artaud discussed how the props and characters in a play could be seen as la réalité virtuelle (Virtual Reality in English) in Le Théâtre et son double, a collection of essays he wrote about drama and the theater in 1938.

During the 1950s, Morton Heilig developed Sensorama, attempting to stimulate all the senses while audiences watched a movie. In the 1960s, engineers created the first HMDs and quickly added rudimentary motion tracking to the unit. A decade later in the 1970s, Myron Krueger coined the term artificial reality to describe his interactive immersive environments. By the end of the decade, MIT had developed the Aspen Movie Map, a hypermedia experience that allowed users to “wander” the streets of the Colorado town.

The Roots of the Movement

VR continued to pop up in science fiction, notably in 1982 when Damien Broderick’s The Judas Mandala was released. Atari founded a development lab for VR research that same year but closed it within two years. Finally, in 1987, Jaron Lanier, founder of the visual programming lab (VPL), popularized the term “Virtual Reality” to a larger audience than had been familiar with the concept before. VPL developed goggles, gloves, and HMDs that began to fully incorporate haptics.

In the early 1990s, VR machines began to turn up in some pop culture conversations. Brainstorm and The Lawnmower Man introduced a generation of moviegoers to the concept of VR. Video game arcades offered VR experiences, though many were short lived because of low latency, which often caused nausea. Later in the 1990s, both Sega and Nintendo announced VR technologies for their gaming systems. Both were racked with problems and quickly discontinued. Apple introduced QuickTime VR in 1994, a product that was widely available but never quite caught on. However, the end of the decade brought philosophical ideas about VR to the forefront of culture in the immensely popular film series The Matrix.

The Modern Era

The 21st century saw massive development in computer technology, graphics, and hand-held devices, all paving the way for where VR has landed presently. In 2007, Alex McDowell coined the phrase “immersive design” in a discussion around the growing field that was emerging around story-based media within the context of digital and virtual technologies.

Companies such as Kaiser and Canon developed progressive HMD and VR technology in the early 2000s, which would usher in the next generation of pioneers. At this point, the focus was still very much on the technology, and few were having conversations about how to tell a story with it.

In 2012, the Oculus Rift prototype was developed, and story conversations began. Smart glasses from Moverio and Google advanced conversations about Augmented Reality (AR) possibilities and affected the conversation surrounding VR as well. In 2014, Google Cardboard entered the public conversation around VR and offered an extremely inexpensive way for the average person to have a VR experience. Samsung, HTC, and Sony all released products that capitalized on VR’s progress and made it more desirable and affordable. However, that year is most significantly remembered as the year that Facebook purchased the VR startup, Oculus, for $2 billion. This acquisition is considered to have ushered VR into the mainstream market.

Navigating Mediated Experiences

Theorists have considered the psychology behind the rise of VR since its earliest iterations. Bolter and Grusin discussed the rise in terms of remediation, highlighting the ways that Virtual Reality has incorporated elements and aspects of other previous media forms. They suggest that culture moves toward media where the evidence of a mediated experience is less noticeable. They point to websites as being hypermediated experiences, offering photographs, video clips, hyperlinks, and other elements to guide the user’s experience, making users further aware of the force between themselves and the meaning behind the media experience they are engaged in. Bolter and Grusin suggest that users do not want an intervening agency but prefer to go beyond mediation to immediacy.

Technological advances that have led to greater and greater levels of immersion in VR tend to eliminate the evidence of a mediated experience, though one could argue there is an even greater level of mediation actually at work behind the scenes. Viewers in virtual experiences often respond as though they are having the immediacy experienced outside of the virtual world, suggesting that these virtual experiences have lowered the evidence of the mediated experience to levels almost indistinguishable from those they engage in without an HMD.

Further, Bolter and Grusin state that within a mediated experience, the meaning itself is not immediate. In order for it to be so, the viewer must be able to ignore the medium’s representational function. This most often occurs when the viewer encounters something from their world outside the virtual space, as opposed to something abstract. Since representational media such as film tend to rely on concepts, participants, and objects from outside virtual space rather than abstractions, such as those that might be found in fine art, it should be no surprise that cinematic VR—the combining of film techniques and principles with immersive media—creates an even greater sense of an unmediated experience and more immediate meanings for the viewer.

Changes in Perspective

For decades now, filmmakers have thrown themselves into film schools and other training venues to learn how to tell visual stories. With emerging technologies like VR, however, many of the concrete rules everyone has learned and spent time perfecting go out the window. For example, since the inception of film, we have forced a viewer’s perspective on our story by placing and moving the camera where we wanted it. This allowed us to use the edges of the frame to craft a window into the world we were creating for the audience. With 360-degree cameras, the field of vision is opened up to a dimension we’ve never had to deal with before. In essence, we’ve lost the edges of the frame and thus the traditional way we’ve manipulated the viewer’s perspective. We’ve also used lighting and lenses to further direct an audience’s attention toward who our protagonist is and what he or she is trying to accomplish.

The Two Philosophies

Shari Frilot, senior programmer and new frontier curator at the Sundance Film Festival, believes the only way to figure out how to tell stories in this new era is to cultivate artistic and social environments that disarm audiences when they enter the space of VR. “There’s not going to be just one way of telling a story,” she says emphatically. “There’s going to be different artists working in different media figuring out different ways to architect in this space. It’s still in a very nascent stage, storytelling in this new medium,” Frilot has said.⁷

Currently, there are two dominant philosophical storytelling approaches and perspectives in VR. The first allows the viewer to watch a scene that is played out in the space around them. They are immersed in the scene but not necessarily an active participant. This is engaging for a few moments but can become frustrating to the viewer when they want to more intimately interact with the world around them. Storytelling in Virtual Reality is less about telling the viewer a story and more about letting the viewer discover the story.

The second philosophical approach allows the viewer to actually become the camera, in a sense. Stories told in this format begin to blur the line with video games. However, VR storytellers are finding that not all gaming theory applies in this format either, as we might assume. Allumette from Penrose Studios premiered at the Tribeca Film Festival in 2016 and fully embraced the “viewer as camera” approach. The results were stunning. Most who have experienced the film have raved about it being a game changer for storytelling.

Former Pixar exec Tom Sanocki is creating his Limitless Entertainment brand with the same principles that made his former bosses storytelling giants. Limitless’s first film, Gary the Gull, is smart and funny and features a great lead character. Sanocki has said that storytelling is about figuring out how people solve problems. Now we have to figure out how they solve them in virtual or augmented realities. Sanocki has compared the process to the early days of filmmaking when experimentation was the key to moving the medium forward.⁸

Cinema’s journey began with much less promise than VR holds. Voices from Vaudeville and the more established theater community combined with objections of prominent technologists such as Edison opposed early cinematic storytelling, preferring to keep it confined to the realm of science. Experimentation with the film camera and the projectors that allowed its product to be shown for large groups eventually pushed through the opposition and created a new visual language for communicating, entertaining, and crafting narratives. It would take decades of innovations in both technology and creativity before cinema would arrive fully as the storytelling medium it is today. Sound, color, realistic special effects, and even the ability to edit would not become available until later.

While the development of immersive storytelling benefits from previous art forms, such as the cinema, it still may be some time before the full potential of this new medium is seen. The eyes of new generations and the fresh perspectives they will bring will continue to recraft the path. However, the initial foundation must be laid before bricks can be added to the wall.

Concepts to Consider from Jessica Brillhart

• Engaging with a viewer in VR is more of a dialogue or a dance than an experience that can be forced upon them.
• Characters are vessels for story.
• Creating presence is nuanced and can be difficult to achieve.
• You must determine what you want the viewer’s role to be in the virtual world and then communicate that to them somehow.
• Traditional three-act structure is useful in VR storytelling but must be unpacked through a kinetic lens.
• Creators should leave space for a variety of potential experiences the user may have—the gradient of a story.
• Energy, or the emotional journey of the audience through the experience, and perception, how the viewer is experiencing the world, should drive the technical decision making in VR storytelling.

Putting Ideas Together

Jessica Brillhart introduces concepts and ideas that will be key to understanding and developing new approaches to storytelling in virtual space. The concepts of engagement, presence, and energy will continue to arise and be explored as we delve deeper into the world of immersive narratives, as will story-related ideas such as character and three-act structure. As Brillhart highlights, the central concern must always be the experience of the audience. If they are not engaged, the novelty of our approach will not be sustainable. Every creative decision involving either narrative or technology must be to that end.

Forcing Perspective in New Ways

The word “perspective” is Latin in origin and comes from the words per, which means “through” and specere, which means “to look.” The rules of perspective in filmmaking are taken from the centuries of development in the arts of drawing and painting. The human form appeared more realistic when certain proportions were considered in relationship to others. Proportions were also considered in relationship to the subject’s distance from the viewer. The point of using perspective is to create a sense of depth. In cinematic storytelling, this is true both in a physical sense and in the metaphoric. We are using viewed images, juxtaposed together, to create a greater sense of meaning—a story. We are creating an extension of reality through art.

Forms in filmmaking, be they a person, an object, or a location, consist of three basic characteristics. They have size. They take up space or cover distance. And they extend in different directions. Keeping these ideas from classical art in mind as one begins to consider 360-degree virtual space and how characters and stories will work in it will be helpful. We sometimes refer to the relationship between the viewer and these forms as depth perspective.

Understanding how perspective worked in the early artistic disciplines gave us a head start when learning how perspectives would function through the lenses of cameras. While a camera is capable of reproducing an image faithfully, we are also able to manipulate the image and thus the perspective based on the movement of those lenses through spatial distance and time.

Manipulations of lenses affecting depth of field, focus, brightness, and contrast have long served as methods for forcing a viewer’s perspective on significant characters, themes, or moments in cinematic narratives. However, those tools are not as accessible in the world of VR filmmaking. Forcing perspective has allowed us to make certain characters seem larger in the frame and thus more dominant. Characters can be positioned below the viewer’s eye line or “down their nose” and thus made to feel looked down upon.

Regardless of our methods, the forcing of perspective is all about attempting to communicate a photographable subject to an audience, a baseline necessity in telling a story through filmmaking. In establishing their perspective, we are creating the audience’s eyes in the world of the story. Stories in the VR world are often quite simple, as the public’s VR literacy is just beginning. As their experience grows, the complexity of the narratives is likely to grow as well.

Edges of the Frame and the Gaming Expansion

Maintaining the rule of thirds, creating headspace, and establishing a 180-degree line have been standards of cinematic storytelling for decades. The goal in using these techniques has been to keep the audience immersed in the story. When the filmmaking process becomes visible to the audience, they are quickly taken out of the narrative and reminded that they are just observing technology. Their investment in the characters and the theme lessens.

In many ways, the edges of the frame have been convenient tools for filmmakers to rely on, knowing that they can hide what’s “behind the scenes” in the story they are telling. As the camera moves in a fashion motivated by the story, the viewer’s eyes and attention move along with it. The peripheral world to the edges of the frame is not visible and thus not important in the viewer’s mind.

Early efforts in video games resemble the structural and static nature of paintings more than that of great cinematic art. However, as gaming continued to develop, the technical rules experientially crafted by filmmakers began to appear more often. Eventually, games gave the player realistic first-person challenges. The user’s sense of presence and immersion increased as interactivity increased. The ability to control space inside a virtual world had not previously been experienced in any technology. This technique has proved to be important in the development of VR, as many early immersive experiences have duplicated the style and language of video games. VR experiences in gaming continue to be one of the most popular and effective uses of the technology, providing a canvas for storytelling that is still advanced beyond what can be achieved with cameras at this point in our technological development.

As visuals expanded in the world of gaming, the quality of and techniques for employing the accompanying audio expanded as well. Serious gamers invested in surround-sound audio systems or quality headphones that further immersed them in the virtual world. Advances in fields such as binaural audio provided further opportunities to bring users into the experience they found themselves in. All these developments would be useful in establishing the specs and guidelines for the best available VR experiences.

The Ultimate Empathy Machine

In a TED talk he delivered in 2015, filmmaker Chris Milk heralded VR as “the ultimate empathy machine.” Scientists are beginning to study VR and its relationship with emotions and even depression, believing that Milk may be correct.⁹ “Every time we build something new, more of my questions get answered, even if a few more questions come up. It feels like we’re getting better at it each time, and this is an evolving medium and an evolving process. In the long term, we do need to create a library of things that is not just about having 500 Citizen Kanes, because not everybody wants to watch Citizen Kane, and the people that do, don’t want to watch it all the time,” Milk says in his talk. “They want to watch Scream or whatever. Sometimes people want to watch things other than high art,” he concludes.¹⁰

Milk is pointing toward a potentially altered state of human consciousness being on the horizon. “Right now we’re still in the darkness of night, poking around with flashlights and trying to find our way there,” he has said.¹¹ As he suggests, VR storytelling and filmmaking will only progress as creators continue to experiment and share what they have learned to work. As with any new medium, learning what does not work will be just as significant as learning what does.

Elia Petridis, a VR filmmaking pioneer who founded the experimental filmmaking company Filmatics, believes he is seeing shards of light in the technological darkness surrounding VR storytelling. “There’s this element with VR of, ‘I don’t want to stand on a spaceship, I want to stand on the Millennium Falcon because that has meaning to me,’ ” he says. “It is the emotional connection, that something that was hanging on your walls is now right in front of you. As a storyteller, I find that aspect really crucial,” Petridis has said.¹² Other creators are also finding that establishing meaning leads to an emotional connection for the viewer in VR worlds. If the technology is indeed to fill this role as “ultimate empathy machine,” building this sort of connection will be crucial.

Concepts to Consider from Paul Meyhoefer

• Some VR experiences are captured through one or more spherical lenses that produce a 360-degree spherical image.
• Software is required to interpret and manipulate these spherical images.
• 360-degree video is not the same as Virtual Reality, though it can be used to create VR experiences.
• Technical challenges exist with using 360-degree and VR cameras, such as keeping lighting and sound equipment out of the shot.

Putting Ideas Together

Paul Meyhoefer mentions creators wishing to create more immersive virtual experiences. This idea will be unpacked in a variety of ways throughout the text. While technology provides the immersion for the viewer in VR, one of the chief concerns of and challenges to the technology is not breaking the immersion that it is providing. It is important to recall from the earlier interview with Jessica Brillhart that the emotional experience of the audience must remain at the pinnacle of our goals as creators. Breaking the immersion breaks the emotional journey and thus defeats the purpose of the experience we are creating. Technology must remain the tools we are using to tell our stories rather than the focus of the stories themselves.

Moving Backward toward the Future

Virtual Reality and its cousins, Augmented Reality and Mixed Reality, have not and will not only be used for the purpose of storytelling. However, the potential for storytelling in these mediums will be a significant factor in its long-term success. Film cameras and the accompanying technology would have continued to have uses, even if Edison and the theater community had gotten their way. However, film and later video would have likely not become the communications media they are today had we not figured out ways to tell stories with them. Even when these visual tools are used for purposes outside of the realm of narrative, the techniques and methods that storytellers created are still often used to employ them. Whatever technologies lie beyond us in visual communications, it is likely that immersive technologies and storytelling methods will be key to reaching them.

As our technologies have advanced, voices from Huxley to Postman have suggested that our mythologies have diminished. It is the opinion of this author that VR holds the potential for providing a platform for recreating them. Early mythologies were surrounded by rituals. These rituals had lasting impact because they provided those involved with experiences that not only resembled real-life mysteries and events but seemed to offer some connection to them that transcended the verbal languages people had come to rely on. The effect these rituals, and thus mythologies, had was visceral. It is entertaining to watch YouTube videos of those experiencing VR for the first time. The reactions could easily be described with the same term—visceral. The donning of an HMD headset and waving one’s hands about, reaching for objects that simply aren’t there, would certainly have appeared to be ritualistic, even cultic activity to the ancients.

The classic Greek myth of Demeter and Persephone is no longer familiar to the average citizen, as it was in ancient Greece. In the tale, Persephone is out gathering flowers one day when she is nabbed by Hades, god of the underworld. Her mother, Demeter, is crushed by the kidnapping. Persephone refuses to eat during her time in the underworld, having heard if one eats there, she can never leave. Eventually she succumbs when offered a few pomegranate seeds. Demeter convinced Zeus to send Hermes to negotiate a deal with Hades for the return of her daughter. The compromise required that Persephone would marry Hades and return to the underworld for a season every year. In the remaining months, she was free to live above ground, returning to her previous life. Demeter mourned every year when her daughter returned to the underworld. Being the goddess of the harvest, she used her power to keep the crops from growing, and all plantation died. She caused flowers to sprout again and plant life to return when her daughter arrived home each year. Several versions of the myth record that Persephone grew to love her time in the underworld, eventually preferring her time in the darkness to her time in the light. The experiences she had there offered powers she was unable to recreate anywhere else.

Entering virtual space through a VR headset is not unlike a trip to the underground. The virtual world holds possibilities beyond our natural landscape. It very well may make achievements possible that would never have been conceived in previous eras. Its dangers are equally real. Like Persephone, there may be those who begin to prefer the new world to the old, causing problems we can only imagine at this point. Still, virtual experiences may hold answers beyond those we’ve been able to grasp thus far in human development. VR has allowed scientists to view data in ways that have unleashed fresh insights and interpretations. The technology has allowed disabled veterans to visit war memorials, allowing emotional experiences they would have never had otherwise. The wonder seen embodied in those experiencing VR for the first time is reminiscent of the ecstasy those early rituals and mythological experiences provided according to ancient writings.

The myth of Demeter and Persephone allowed the Greeks to explain why seasons on the earth mysteriously came and went. It allowed them to make better sense of the world around them. It allowed them to have deeper insight into what it meant to be a human being on this earth. These are the things mythologies are capable of providing. Could there be any greater hope for what VR might offer us? Immersive experiences will not lead us to mythological insights about who we are unless they are accompanied by stories. Stories allow us to work in the language of metaphor, not necessarily allowing us to articulate the nuances of life as much as allowing us to experience insights into them in ways that seem to transcend language. But before we can transcend the language of VR storytelling, we must understand what others have discovered about it and how to speak it ourselves.