3

Science and Technology behind Storytelling

Though stories have been around since the beginning of human communications, scientific research into the particulars of the field is still relatively new. In his book looking at the science behind the power of story, Kendall Haven says that our minds were evolutionarily hardwired long before birth to think in specific story terms—that these internal neural story maps are not stored in our conscious mind but in the subconscious.¹ Beyond these concepts, children are read stories before they are old enough to grasp the words being said to well beyond the formative years. We never reach a point of maturity at which we stop telling each other stories. All this storytelling has mass effect on our brains and our lives.

Story and the Essence of the Brain

Brian Boyd contends that humans are “hyper-intelligent and hyper-social animals.”² By combining factors of intelligence, pattern seeking, alliance making, cooperation, and the understanding that others have beliefs and knowledge of their own, stories make us a stronger and more effective species. He suggests a story is “a thing that does” instead of “a thing that is.” Perhaps the most significant factor that Boyd mentions is pattern seeking. Stories that adhere to even loose structural frameworks are, in a sense, built around patterns. This is certainly a factor in the reason why story and its various narrative elements are attractive to the brain. A recent report in Frontiers in Neuroscience, available from the US National Library of Medicine and National Institutes of Health, states that pattern processing is the very essence of the evolved human brain.³ In light of the earlier discussions around story structure in this book, while nonlinear stories and those that stray further away from traditional patterns humans recognize in their narratives can still have resonance with an audience, those that rely more heavily on pattern-based structure have a neurological advantage when offered to an audience. This concept holds true with different forms of visual narratives, leading us to believe it will hold true for VR as well.⁴

Author Jonathan Gottschall suggests that another field of science in which a great deal of research has been performed might lead us to insights about our relationship with narratives. He states, “Dream researchers define a dream as—this is a quote—‘a vivid sensory motor hallucination within a narrative context.’ It’s a night story. It focuses on the protagonist, usually the dreamer, who has to overcome obstacles to achieve his or her desires.”⁵ In his book The Storytelling Animal, he suggests that people are interested in simulated worlds because they desire to live better in this one. While looking at storytelling in video games, Gottschall interviews one game designer who says that his games give players a daily vacation from the pointlessness of their actual lives, opting instead for the meaning-rich environment the game provides. Most research around narrative games and stories in general suggests that this quest for meaning lies at the base of the human compulsion to engage in these activities.

Science, Story, and the Body

The Virtual Human Interaction Lab at Stanford University conducts a wide array of studies around Virtual and Augmented Realities. Many of their studies are focused on issues of embodiment. One recent study asks the question, “Can an avatar’s body movements change a person’s perception of good and bad?” Specifically, the study looks at whether embodying a character in VR space has the power to change our minds about certain matters. While remaining clear that more research is needed, the study does conclude that VR experiences “may implicitly shape how users form mental representations about their digital experience and those outside the media interactions.”⁶ Other studies conducted by Stanford state that embodying animals and even nature in VR space can change the way we interact with those elements outside of the digital space.^7,8

Another area where science seems to be intersecting with story and the body is in how we actually view ongoing activity in the world. A study on the subject revealed that a steady diet of the type of editing used in film production has caused humans to begin to think in terms of segments or scenes.⁹ In other words, we expect a certain continuity in our lives and our activities, even as they unfold in real time. One report describes it as such: “Animal and human movement is integrated in rhythmic and graceful sequences of discreet units of activity, each with their own particular goal-orientation, which are coordinated by the purpose of a higher-order goal or project.”¹⁰ Many studies have begun to look at the effects of when human life does not so easily conform to these parameters and expectations, especially in digital environments. Users have documented dissociative feelings they experience since the early days of VR—termed “post-VR sadness.”¹¹

Concepts to Consider from Dr. Carolina Cruz-Neira

• Every VR platform has its own advantages, disadvantages, and applications.
• There are connections in the study of VR to the study of memory.
• Narrative elements have uses in VR beyond entertainment.
• Technical challenges are still steep and must be overcome for VR to advance.

Putting Ideas Together

As Dr. Cruz-Neira mentions, even those whose work transcends the primary goal of storytelling rely on narrative elements. Understanding these elements and how they can be used gives creators an advantage in the types of work they may choose to involve themselves in. Narrative elements are helpful when constructing methods and applications for solving a problem, be it psychological, physical, or digital in nature. Understanding the way that narrative elements function apart from, alongside, and together with other narrative elements gives a creator insight into world- and structure-building methods that may be most useful in developing a solution for a given issue. The answer may not always rely on developing a new technology to fill a gap but to look at using the existing technology to provide a different narrative.

Technological Challenges with Traditional Cinematic Storytelling Techniques

We are at a unique point in history where technology has been advancing more rapidly than our methods of production. This is true across the global culture but specifically true in areas of produced entertainment. Cameras were capable of resolutions higher than those that could be edited or displayed for a period. As fast as computer processing power was advancing, the average user is still unable to perform advanced operations with large data files in real time—rendering must still occur, slowing the rate at which content can be produced. VR has not been without its technological challenges as well. In fact, many say the largest tests that face VR are technological, as the human imagination has already dreamed a future far beyond what VR is capable of—for now.

Perhaps the most discussed challenge has been the logistical issue of just where to put all the supporting lights, audio, and grip equipment needed to produce the aesthetically appealing productions that audiences have grown used to. Certainly, YouTube and other online video platforms have diversified audience expectations when it comes to the production quality of video content they enjoy. However, when audiences are asked to pay for content, there remains a certain level of expectation that the production quality of what they buy will be at a level they consider professional.

Clever VR creators have already invented a variety of means to hide gear and crew members in plain sight within the VR space. Homemade helmets have been used to mount the camera above the director of photography’s head. Monitors have been hidden behind newspapers and disguised as televisions on the wall. Crew members have been disguised as extras in scenes—some with hidden microphones mounted strategically on their bodies. Stories that were originally meant to be told indoors have been moved outside to take advantage of natural light. All represent methods that creators are discovering in order to tell their stories with the technological challenges that presently exist in the medium.

Using Production Practices to Tell Better Stories in VR

Even with all the challenges, creators must use the production gear that presently exists to produce content in the medium. While eventually VR will likely develop its own tool sets, those used in film, television, theater, and the web must suffice for now. While the camera is likely the only piece of equipment currently on VR sets that is solely and specifically for use in the medium, this will quickly change. While the majority of equipment presently needed to tell VR stories will come from the production worlds of film and television, the discipline of immersive theater has developed tools and techniques that will be helpful as well. Creators in this field have a head start on hiding production elements that tend to break the immersion experienced by the audience.

Before discussing specific production practices, tools, and techniques, it is important to state a few ideas about the philosophical nature of the equipment used to tell the story. Far too often, the means of production becomes of higher priority than the content being produced. In other words, the production is aesthetically pleasing but the story the crew was telling doesn’t hold up to the aesthetics. While there is a certain audience expectation for the quality of content they are paying for, viewers tend to forgive simple and even problematic production elements but are less likely to forgive dissonance, confusion, or plot holes in the narrative. It is of utmost importance that every crew member realize that they are a storyteller. This is not solely the domain of the writer or director. Each production element tells a story within the larger narrative being communicated. Directors must combine these smaller stories to craft a larger one, much in the way a conductor combines all the sections of an orchestra to create a symphony. If one section of the musical collective is out of tune, the whole is in jeopardy. Dissonance created by one department can make the entire narrative confusing or difficult. One of the jobs of the director is to make sure that everyone is making the same story. If the lighting department believes the story they are telling is a horror narrative but the sound department believes it is a comedy, the final product will be unlikely to strike a chord in audiences—at least not one they enjoy.

How the Camera, the Frame, and Its Edges Affect Your Story in VR

As has been discussed, the camera is likely the sole element on the set that has been specifically designed for VR production. This production tool will make or break how the audience experiences the story. Its importance cannot be overstated. Depending on the scale of the production, VR cameras vary greatly in size, shape, and weight. Some cameras feature a single wide-angle lens that shoots up to 180 degrees. Creators then often take two or more of these cameras and tether them together to create immersion. Other cameras have multiple lenses that perform the same task in order to capture 360-degree space. It is worth offering a gentle reminder of a concept presented earlier in the text.

360-Degree Video Is Not the Same as VR

While 360-degree video may be used in VR to help create immersion, they are not, by definition, the same. You may recall this concept being introduced in the Spotlight on Cameras section in Chapter 1. Removing the edges of the square frame does several things in the mind of the audience. The square frame has long acted as a window through which we see the narrative world. When the audience has been repulsed or frightened by what they saw within the frame, they could look away. With the audience placed literally inside the frame, looking away is not an option. A viewer could close their eyes. However, this requires an even greater sacrifice of control on the part of the viewer. Most people would not close their eyes in a situation that frightened them unless they were a child. When scenes within the frame upset a viewer, they had the option of looking over at the person next to them, perhaps even taking their hand. With preset VR experiences, that agency has been taken away, which speaks to the importance of the impending social VR technology. The vulnerability established by the camera, the frame, and its edges affects the viewer’s experience of the story a great deal. This must be recognized when crafting narratives. It can be tempting for creators to press the boundaries of how deeply a viewer feels immersed in a world. However, there must also be an innate respect for the audience on the part of the creator—a consideration for the wide variety of audience members that may enter the world. Striking the balance between the deep immersion the audience feels in the experience and the level of comfort and enjoyment a diverse audience is capable of will continue to be of great importance when crafting stories in VR. It is also worth noting that even within the realm of animated VR narratives, the function of the camera is still in play, though there may not be a physical camera used in the creation. Animated worlds in VR remove many of the technological challenges that presently exist in physical production. However, the story challenges remain the same. The cautions mentioned earlier concerning audience immersion are equally important in animated VR stories.

Connecting with Concepts and Methods in Experimental VR Storytelling

Science and technology have become useful in the innovation of many art forms. However, the only way any art form moves forward is through experimentation and pressing the established boundaries. There’s a common saying in the film industry: “You need to know the rules before you can break them.” However, once one does know the rules, breaking them immediately becomes of interest. Experimental approaches in art rarely get much press. They are often only known to those working within the boundaries of the established industry. Those within the industry occasionally get bored with traditional approaches and look to those experimenting with the form for inspiration, taking approaches and techniques they find interesting and incorporating them into largely “safe” works. The original experimental approach often never gets acknowledged and certainly does not become financially successful. This pattern has been repeated over and over again in music, theater, literature, and film. While studied and lauded by academics, the writing and filmmaking of Guy Deboard in the early part of the last century remains unknown to most artists creating interactive and immersive work. In his book The Society of the Spectacle, Deboard unpacks spectacle as false representations seen in our real lives. He goes on to state that spectacle is a materialized world view, subjecting humanity to itself. He seemed to suggest that the more art resembled actual reality, the more significant its impact. In attempts to prove his theories, Deboard created films that forced his audiences to interact with the medium rather than remaining passive receivers of information. His methods were often extreme. His first film, Hurlements en faveur de Sade, consisted only of black and white screens. When the screen was black, there was no music or dialogue. When it was white, a speaker made a brief comment and quickly turned to black again. The periods between white screens sometimes lasted up to 20 minutes. Needless to say, the film was not a hit with audiences. The theory that Deboard was trying to enforce by breaking the rules, that audiences should interact with the images, remains quite relevant today. In this section, we will examine some of the potential methods of “breaking the rules,” as well as methods that reinforce them. These concepts are provided as templates that creators may play within or attempt to transcend while creating their stories.

Abstract versus Realistic Storytelling

There are a few basic terms and concepts that might be helpful to unpack before looking at the larger field of potential. The first is the tension between abstract and realistic storytelling. Obviously, visual mediums can be much more than just a “record of drama.” Abstract experiences push boundaries, stir emotion, and express the intangible feelings of the artist. They can also showcase specific effects or graphic techniques, explore sounds, emotions, and other more feeling-based ideas. Abstract experiences can push the boundaries of theme and structure in ways that more realistic approaches are unable to. Thematic abstract experiences explore narrative and cinematic concepts, whereas realistic narrative experiences usually tackle the specific issues of story and character. The line between abstract and realism is gray. Abstract films can have characters and realistic films can have abstract elements. In realistic experiences, audiences expect to see the rules of a certain genre adhered to. If that experience is documentary in nature, we wouldn’t be comfortable seeing the documentarian step from behind the camera and perform a show tune. If the genre is horror, we have certain forms that we anticipate the creators will maintain. With abstract experiences, the genre rules become less important, and simple feelings become the method of expression. It is important to note that abstract experiences may be created with photo-realistic images and realistic experiences may be created with artistic animation. The terms speak more to the conceptual approach to the work and its relationship with reality than to the technical medium employed to create the piece.

Breaking the Fourth Wall

Traditionally in theatrical experiences, an invisible wall has existed between the viewer and the stage players. The players are unable to see through this wall, and this gives the impression that they are unaware of our presence as an audience. This concept was brought over from traditional theater and employed in film as that medium developed. When a player, or actor, acknowledges the viewer, this is referred to as “breaking the fourth wall.” This concept has also relied on the assumption that the audience is only able to peer through one wall—the fourth wall. Even when we change angles or perspectives in film, we are still only looking through a single physical dimension. In immersive experiences, the viewer is placed inside the plane of action rather than watching it through an invisible wall. However, the player may still look directly into the camera and, in effect, break the fourth wall. While in a third-person experience, this would likely break the immersion, it actually heightens the immersion in a first-person experience.

Retrospective Storytelling versus Real-Time Storytelling

Beyond the abstract and realistic, there is another philosophical understanding of story that will be helpful to us in moving forward. Retrospective storytelling refers to narratives told from a perspective that looks to the past. In other words, the story being told is not happening at the time the storyteller is describing. There is an underlying assumption with retrospective storytelling that the events will add up to something or lead to a conclusion. Since the story is being told looking backward, the viewer assumes there has been a lesson learned or perspective gained on the narrative events that unfold. Retrospective narratives provide the “why” when we are curious about the present.

Real-time storytelling refers to narratives that are unfolding in the present. The viewer is experiencing them as they actually occur. This can be exhilarating for the viewer. However, it can also be disappointing, as there may or may not be meaning at the end of the experience. There is certainly not meaning while the experience is playing out. With real-time storytelling, sometimes the story simply ends without a dramatic conclusion or lessons learned. Usually, the viewer has the first-person perspective with real-time storytelling.

While meaning may not be readily available in real-time storytelling, the human mind is capable of creating meaning quite quickly, constantly looking back at the events that are playing out in order to retool the meaning that seems to be manifesting. The longer a narrative plays out in real time, the more information the user has to create meaning with. With retrospective storytelling, there is a set meaning to narrative events. With real-time storytelling, meaning is constantly unfolding, changing, and taking on new forms.

The Role of Memory in Retrospective Narratives

Researcher Kate McLean has done significant work in the relationship between the role of memory and retrospective narratives in human psychology.¹² Though her work was with adolescents, she states that there is every reason to believe her findings apply to expanding social networks. McLean suggests that rather than storytelling, we actually participate in memory telling after having experiences. This memory telling causes us to create a narrative identity about ourselves. To simplify, when we have experiences, we try to take our memories of what has occurred and string them together to try to create a coherent story about what we experienced. These memories may or may not have direct connection to what actually occurred. Sometimes the memories actually more closely conform to narrative concepts that help us make sense of what happened. Our brains look for stories. We need them to make sense of the world. Actual memories are secondary to story in our minds. For those wishing to delve deeper into these matters, the work of Gary Fireman and Ted McVay is an appropriate starting place.¹³ The work of Fivush, Habermas, Waters, and Zaman may also be helpful.¹⁴

The Role of the Environment in Real-Time Storytelling

When telling real-time stories, the audience immediately begins searching for information that might be helpful to them in orienting themselves in the new world they have entered. Environmental elements will be some of the first images the audience encounters and begins processing. The term environmental storytelling was popularized in the gaming world and refers to the act of “staging player-space with environmental properties that can be interpreted as a meaningful whole, furthering the narrative of the game.”¹⁵ Game designers have long stressed the importance of environmental storytelling in creating an immersive experience. According to Henry Jenkins of MIT, “Environmental storytelling creates the preconditions for an immersive narrative experience in at least one of four ways: spatial stories can evoke preexisting narrative associations; they can provide a staging ground where narrative events are enacted; they may embed narrative information within their mise-en-scene; or they provide resources for emergent narratives.”¹⁶ Mike Shephard expands on Jenkins’s thoughts, saying, “Many games rely heavily on the embedding of narrative information within mise-en-scene, a term used to describe design aspects in production. It examines set design, lighting, costuming, and so on in how they contribute to the narrative.”¹⁷

Creators of VR, AR, and MR experiences must consider the impact of environmental elements in their narratives in order to provide the most realistic and immersive stories. Much has been said about the role of and need for photo-realism in environments, characters, and objects in VR space. This conversation is nuanced by the inclusion of animated and stylized experiences, of course. The key question to ask when designing any environmental element is why? Why should the ground be sand as opposed to rock? Why should the story take place at night as opposed to day? Why are there fallen trees that must be climbed over? What caused them to fall? Why would this character not walk around the tree? These are the sorts of questions that open up space for storytelling and narrative design. There are entire volumes exploring the psychology of color, the impact of shapes on the brain, and the emotions associated with various aesthetics. For readers interested in exploring more depth about these issues, If It’s Purple, Someone’s Gonna Die: The Power of Color in Visual Storytelling by Patti Bellantoni is a sufficient starting place for novices and academics alike.

Passive versus Active Storytelling

Another lens we can view interactive and immersive storytelling through is created when we juxtapose passive storytelling and active storytelling. Further, there are a number of approaches when teasing out meaning from those terms. One approach is to leave gaps in the story that the audience fills in with their imagination, taking the narrative from passive to active. This approach is most effective in novels and short stories. When visuals begin to enter the narrative, we fill in many of those gaps for the audience. The more visuals that enter the narrative, the fewer gaps for the audience to fill in. With VR, the potential for the amount of imagery presented to the viewer is more vast than any sort of storytelling experience that has been created thus far. So this approach, if it is to be effective in VR, would need to leave thematic and narrative gaps as opposed to gaps in detail. It is worth noting, however, that strong narratives usually contain both passive and active storytelling elements. Passive elements give the viewer orienting elements. They set up conflicts and allow for later active storytelling elements in the narrative. Active storytelling elements are often what keep the audience engaged over time.

Passive and Active Storytelling in First-Person Experiences

These concepts look slightly different when we consider first-person experiences. When the viewer is the protagonist and at least partially responsible for crafting the narrative, the flow between passive and active storytelling may resemble the dance in which the creator/designer is leading the dance partner but there is still active participation required on the part of the other partner, as referred to earlier in the text. Creator and viewer are truly creating the story together. Passive storytelling elements in many ways become even more important in first-person experiences, giving the viewer rules and railings to keep them on track. They set up the active storytelling choices that the viewer will make. They will drive where they look, what they touch, and how they act in the space.

The balance of active and passive storytelling elements is significant in most narrative approaches. However, the balance takes on even greater significance in experimental story structures and perhaps the most significance in first-person experimental experiences, in which a linear story may be absent. Creators may choose to take away the familiarities of structural narrative from the viewer, but it is important to remember to give them something to orient. This could include any of the narrative shards, which are basically any of the elements commonly found in narratives that have been previously discussed, or an expression of agency that would allow the viewer to create the types of meaning he or she needs in order to navigate the experience enjoyably.

Embodiment in Passive and Active Stories

A distinction must be drawn between stories that allow for passivity or activity in the mind and those that allow for these concepts in the body as well as the mind. The incorporation of embodiment into VR, AR, and MR experiences will continue to grow in importance as the technology expands beyond simple room-scale experiences. In the earliest VR experience, viewers only saw through the eyes on one inside virtual space. They were unable to look down and see their own bodies, creating a lesser sense of immersion. Later, viewers were given “wands” or other tools that provided some agency. Eventually, viewers could see and respond with their hands. Technology that brought more aspects of the human body into virtual space was not far behind. The further we immerse users into their own bodies, thus embodying them in virtual space, the more impactful the stories in that space will become. Storytelling in future environments will bring new challenges beyond what we can even conceive of presently. Discussions of risk, ethics, and active storytelling requirements will only become more nuanced as the field expands. As with all other aspects of this emerging field, trial, error, and experimentation will be our only guides for navigation. As the potential for fully embodied experiences grows, what audiences anticipate in the balance between active and passive storytelling will likely change and develop as well.

Concepts to Consider from Jessica Shamash and Pete Billington

• We connect to stories through how we empathize with characters.
• VR storytelling connects to something primal within the viewer.
• The more layers included in an experience, the deeper the immersion in that experience.
• VR storytelling will continue to become more social in its opportunities.
• There are ethical considerations that should underpin the creation of Virtual Reality and its related technologies.
• Three-act structure should not be thrown out when creating VR stories.
• VR stories offer the opportunity to be immersed in the skin of someone of a different gender, ethnicity, culture, or class, thus creating a greater sense of empathy for humanity.

Putting Ideas Together

While Shamash and Billington introduce a variety of different concepts around VR storytelling, we can boil their comments down to issues of immersion. We experience empathy through characters. The more realistic and human-like the characters are, the greater the immersive experience for the viewer. Narrative and descriptive layers are what make those characters realistic and relatable. The more we can share those characters with friends and family and, perhaps more importantly, the experience we have with those characters, the greater the potential will be for even greater immersion. Our minds and bodies are what create these experiences of empathy and emotion. Understanding how they respond to and craft narrative is essential for creators who wish to bring their stories into this emerging arena.