8

Theory in Practice

Interviews and Case Studies

Since our earliest days, humans have sought new ways to express ourselves. Like language itself, our stories are one expression that have become more rich as time moved forward. Finding new ways to tell those stories has also been part of our journey of expression. Discovering new mediums, pushing them to their limits, and eventually transcending though not completely disposing of them has been the pattern that we’ve repeated over and over again with our storytelling efforts. Virtual Reality may hold the most potential of any medium that has come before it. However, it will still be subject to all the trial, error, experimentation, and eventual transcendence that its predecessors were. The speed at which this process occurs is largely dependent on us. The number of adherents and the volume of stories they choose to tell will determine the speed with which VR reaches its fullest potential as a storytelling medium, as well as the speed with which the next potentially related medium begins to show promise.

This examination of VR purposefully ends with a lengthy look at more storytellers. The most evident path for the emergence of a storytelling language in Virtual Reality will be found on the trail blazed by the early creators in the field—those doing the largest volume of and most imaginative work. Hearing what has worked and what has failed in their efforts saves us valuable time in our own experimentation. Listening to their processes, backgrounds in, and journeys toward effective and articulate communication with this medium will sharpen our own tools for creating within it. The insights of creators working within, and perhaps more importantly outside, the corners of the VR industry that we find ourselves in can expand our work in ways that don’t seem readily applicable at first glance.

Perhaps the single characteristic that unites every creator featured in these pages, besides the obvious connection to some area of VR, is the humility with which they approach the field. In nearly every interview conducted for this book, the creator was quick to state that no one has completely figured out this new language of storytelling, that all efforts thus far are simply the earliest efforts, and that the most important issue was to be aware of and learn from the efforts of everyone else creating work in this space. Holding VR storytelling somewhat loosely with an open hand was a practice this author frequently encountered. It is easy to become dogmatic about theories involving art forms. That dogma usually falls apart when the discipline of practice enters the picture. Recognizing that every story and experience in Virtual Reality should be to the benefit of the user is an ideal that should be fundamental but can easily become lost.

Style over Substance

One of the disadvantages of our rapidly advancing technological progress has been that the ability to look professional and engaging without having any real content to back up such characteristics has been put in the hands of all users, regardless of their level of expertise. This can lead to implementations of style over substance. Trailers for upcoming films promise stories packed with action, plot twists, and surprises. However, once audiences have paid for the product, they quickly discover that all that the trailer promised was actually in the trailer. The final product offered nothing more. While the novelty of VR is still fresh to many, we won’t be able to rely on the “shiny” factor for much longer. Audiences will demand more from every VR experience they engage with. If they fail to find it over a lengthy enough period, they will eventually reject the medium entirely—even if it offers an experience the user has never had before. Readers old enough to remember superior technologies such as the Betamax and DAT machine can testify to the graveyard of discarded technological mediums that has grown larger with each generation. Good stories are the best method to avoid such a fate for VR. If storytellers and VR creators are willing to invest the time in learning and executing the timeless arts of narrative, the technology stands a chance. It will be tempting to believe we already know all we need to know about story to create successful experiences. However, even the savviest of storytellers are constantly studying and developing new ways to express their characters and their journeys. In the quick progression of technology, it can be easy for well-developed and layered stories to get lost in the mix. If we believe in the power of this medium, we can’t allow this to happen.

While there are ancient elements and principles of narrative that coincide with the human psyche and will likely not change, there are other elements and principles that are constantly evolving. The ways that audiences change as they engage new media are countless. The most intuitive creators will find the ability to hold the never-changing principles of story in one hand while holding the ever-changing principles in the other. This has long been a path forward for creators in emerging media fields. Perhaps surprisingly, there is little argument about the importance of the timeless elements. There are few, if any, creators suggesting we attempt to tell stories without characters, for example. This seems to be one element, that when removed, takes away the “story-ness” from stories. The length and even structure of stories, however, has been and will be up for debate. Certain practices work in certain mediums and other practices work in others. As has been repeated, the experience of the end user and what they walk away with should drive such decisions.

Entering a Forest of Digital Trees

This examination began with the metaphor of a stone bridge covered in microchips. There is a final image and metaphor that may be helpful to consider as you move forward into telling your own stories in immersive space. Imagine a forest of digital trees. While their leaves appear to be real, they are synthetic. The green pulsing of their glow provided by the smallest of LED lights. Their stems are connected to branches of silicon. The branches are connected to trunks of reinforced steel. Beneath the surface, root-like wires wildly spread in all directions, tangling with each other and the wires of other nearby trees. There is a low hum caused by the electric soil that powers the roots and thus the trees themselves. The forest looks alive. In some ways, it even feels alive. But the life is manufactured, in a sense. Inexplicably, one day, a piece of fruit appears on a branch amid the synthetic leaves. The fruit is not digital. The fruit is organic. It is real. It is sweet and it is delicious. Day after day, more fruit appears on the trees in the forest. Some fruit rots and dies, as is the nature of living things. Other fruit remains and is enjoyed by all who enter the forest.

This, in some ways, is a picture of the field of Virtual Reality. The experiences created by storytellers in this space evoke real emotions in users. The journeys they make while immersed create feelings that are just as real as any they experience outside of the space. Amid all the technology and digital complexity, something real, organic, and meaningful emerges. Something that sparks a user’s imagination and sense of wonder. That which was never before possible suddenly appears before them in a quite realistic fashion. Opportunities arise that never existed before. Out of lifeless wires, silicon, plastic, and steel, a new reality is breathed into existence.

Interviews and Case Studies

The Storytellers

VR in a Galaxy Far, Far Away

An Interview with
Rob Bredow, Chief Technology Officer, LucasFilm

As the chief technology officer of LucasFilm, Rob Bredow oversees all technology operations for LucasFilm and Industrial Light & Magic. Bredow joined Industrial Light & Magic, a division of LucasFilm, in 2014 as a visual effects supervisor. In December 2014, he was named vice president of new media and head of LucasFilm’s Advanced Development Group. Bredow was instrumental in launching ILMxLAB in 2015, a new division that combines the talents of LucasFilm, ILM, and Skywalker Sound to develop, create, and release story-based immersive entertainment. Previously, Bredow was the CTO and visual effects supervisor at Sony Pictures Imageworks. He has worked on films such as Independence Day, Godzilla, Stuart Little, Castaway, Surf’s Up, and Cloudy with a Chance of Meatballs, as well as the Star Wars VR experience, Trials on Tatooine.

John Bucher:	I’m one of a billion people on the earth who have been a lifelong Star Wars fan. What sort of philosophical mindset does it require to take these characters that are such a core part of the culture into a new digital space with the work ILMxLAB is doing?
Rob Bredow:	That is one of the most fun things in my mind about getting to work at LucasFilm. If you look at the way George Lucas created and developed the world, he was embracing technology and new forms of storytelling all the time. That spirit of innovation has really just stayed alive in the studio, which has been one of the most fun things to discover as I joined—how innovative the thinking is, how fearless the crew is about trying new things, and experimenting with new forms of storytelling and technology—just to tell stories. That’s really been a part of the DNA of LucasFilm since the beginning and really holds true today.
John Bucher:	Can you talk about the Star Wars VR experience, Trials of Tatooine? How did the game come to be? As massive as the Star Wars universe is, how was it decided to set the game on Tatooine?
Rob Bredow:	Our initial creation of Trials on Tatooine was an ILMxLAB experiment. It was based on trying to get our heads around the question, “What does it feel like to be immersed in a simple story in Virtual Reality?” In our case, of course, a Star Wars story. We then would ask, “Is it fun to interact with that story when you have a first-person role in the experience?” There’s been a lot of VR done where you get to witness the story happening around you, but there’s been less VR done where you are present as a first person—present in the narrative. So what we wanted to do was find a simple, and hopefully satisfying, story that would let us experiment with what it feels like to be in Virtual Reality, and experience the story where you’re actually playing a simple role in it.
	We had the asset, the ability to use the Millennium Falcon, because we just finished getting that working from Episode VII. We thought what would be more fun than to have the Millennium Falcon come and land right on you? Some of us have gotten to stand under the real Millennium Falcon that we built in the studio, but not many people have gotten to experience that, and no one has gotten to experience what it’s like to stand under the Millennium Falcon when it’s landing practically on your head. So that was the very beginning of the experiment. We met with Kathy Kennedy and showed her that. Skywalker Sound came in and rigged up this really amazing sound system that was probably the kind of system that you’d usually use to fill an auditorium that holds 2,000 people, and it was just all pointed at Kathy as she was experiencing the Falcon landing on Tatooine right above her. It was one of her early VR experiences, too. She took off the headset, and of course everyone is looking at her to see what she thinks, and she was like, “That’s what I’m talking about! That’s a new kind of entertainment right there!” So we had the start of something that was going to be interesting to experiment with.
John Bucher:	Historically, with LucasFilm people have enjoyed the stories of Star Wars in a collective fashion—with community. We go into a movie theater and sit with a group of strangers and experience it as a group. We’ve sat in front of our televisions and watched the cartoons or the films with other people, as well. With these new experiences, we are on our own. We’re watching them in a headset where we are there with the characters by ourselves. Can you talk about the different approach of bringing someone into this more immersive environment?
Rob Bredow:	The kind of experiences you will see in Virtual Reality aren’t going to be solo experiences for very long. In fact, you’re already seeing a lot of work being done with multiplayer games. VR experiences will have multiple users at the same time very soon—which is going to really change the game. For this particular experiment that we did, it is a single person at a time, so we wanted to make sure the whole story was told around you so you felt like you still had social connection. You were listening to Han Solo give you instructions from the cockpit or interact with you from the Millennium Falcon, and you have other characters in the environment that you have to interact with to keep it alive and to make it as immersive as possible.
John Bucher:	LucasFilm is partnering with directors like Alejandro Iñárritu on new projects. Can you talk about what you’re hoping these masters of the cinematic realm are going to bring into VR space now that this new medium has started to arrive as a canvas they can work on?
Rob Bredow:	We’re really fortunate to get to work with people like David Goyer on the VR experience that we’re creating in xLab that will have Darth Vader in it. As you mentioned, we’re working with Alejandro on an upcoming project, as well. Our goal there is to work with some of the best creative minds in the world who are interested in building experiences in these spaces. There’s certain spaces that we can explore that happen to be the kind of projects these folks are interested in making, and we think there’s a really nice match there. So that’s really what we’re about, finding folks who have very ambitious and creative projects and experiments that need to be made and are best made in this format. These are people who can make movies or TV shows if they’d like, but there’s certain stories that may be best told in Virtual Reality or immersive experiences.
John Bucher:	Even though George Lucas was not the sole creator of Star Wars experiences, it seemed as though this mythological sensibility remained with LucasFilm and the projects that have been created, after he left. LucasFilm has obviously been masters at being the bridge between these universal mythological feelings we experience as human beings and the latest cutting-edge technology. How do you stay focused on being the bridge between the ancient and the cutting edge?
Rob Bredow:	That’s a really interesting question. There’s a huge opportunity there with Virtual Reality or Augmented Reality. For example, we’re doing a lot of work with Magic Leap right now. We know that we as people need something familiar to tie things back to—to relate the experience to. So we have those mythologies that people understand that are universal storytelling tools, and we also have this world of Star Wars that a large percentage of the world can relate to. When we put a Millennium Falcon or a droid in with you, that can be an immediate resonance that gives us a leg up when introducing someone to a new experience. You get something familiar in a way you’ve never seen it before.
John Bucher:	What opportunities does Augmented Reality present? What people are at that table in those creative meetings in order to come up with these ideas and execute them?
Rob Bredow:	It’s really a fantastic table to get to sit around and to be a small part of, because we do get to work with the best of the best. At LucasFilm, there’s this team called The Story Group that is responsible for the continuity of the Star Wars universe. We are really fortunate at xLab to have a close connection to The Story Group. They’re in all of our meetings. They help drive us creatively, which is really fantastic. All of us started at LucasFilm because we love the projects. We love this universe, so a lot of it is brainstorming the kind of stuff that we would want. We’d want these droids to be in our living room with us. We’d like to see what that’s like. What does that feel like? What can they do? What would those interaction models look like? Really it comes from a whole team of people who are Star Wars fans that want to invent these things that we want to experience ourselves.
John Bucher:	Is there any intentionality to connect the worlds that you’re building in the VR space at ILMxLAB and the future cinematic projects that are coming out? Will we see connections between those worlds, or will they remain different canvases that we see different stories separated on?
Rob Bredow:	Some of that I can’t speak to yet. The thing that I can say that’s specific to that is that we’re really fortunate to get to work really closely with the LucasFilm story groups, and they are the team that is responsible for the overall storytelling universe that we get to play in. I can’t say what specific experiences might relate to specific things in the canon, but it’s really great to have that close interaction with the team.
John Bucher:	Let’s talk about the idea of audiences experiencing empathy in these environments. Obviously, people have a great emotional connection to the Star Wars characters and the Star Wars universe. VR has been called the ultimate empathy machine. Do you have consultants in the science community that talk about empathy and how that’s achieved with the digital technologies, or is it more of an organic process of intuitively knowing how people will emotionally respond to this sort of content?
Rob Bredow:	I think our main focus is really about what a good story is in this world. The thing we’re looking for more than explicitly empathy per se is emotional resonance, which I think is slightly broader than just empathy. Empathy is great, but we’re looking for a story that has emotional resonance with the audience. And that’s often our threshold. Is this something that people would want to experience, and worth our time to build? We only get to build a tiny fraction of the total ideas we have, so we want to make sure when we’re building something, it’s going to be one of the best ideas we have. Really that vision comes from Kathy Kennedy. That’s been her mantra for the kind of projects that she expects to be made in LucasFilm.
John Bucher:	What makes a good Star Wars story?
Rob Bredow:	One of the biggest components is emotional resonance. Is there something that we universally can relate to as people?
John Bucher:	In Trials on Tatooine, players get an opportunity to hold a light saber and have that experience. As you mentioned, these are first-person experiences, as opposed to third-person experiences where we’re just a ghost observing these things in front of us. Can you talk about the differences in approach between something that we just observe and a story where we are the protagonist?
Rob Bredow:	There are quite a few differences, actually. I think the biggest one is the state that it puts the participant in. If you are just observing the things around you, you can tell very powerful stories that way, and there’s lots of stories that can be explored that way for sure. The moment you have agency in the story and you can interact with it, it really changes your overall perspective and what that story means to you, and the kind of emotions, and the impact you can have on people as they get to experience it that way. One of the experiments we did with Trials on Tatooine is to take it to a Star Wars celebration in London. That’s where a lot of fans go to get to meet the people involved in making the projects and hear about what’s coming up next. It’s a fantastic opportunity to interact with the fans firsthand. That audience was pretty excited to get to be in this world and experience what it’s like to stand under the Millennium Falcon, or to interact with R2, and of course hold a light saber. We had a lot of really positive responses and people who were very emotionally engaged in the experience.
John Bucher:	What would you say, on a big picture scale, would be the ideal opportunity that Virtual Reality will provide for LucasFilm? What is your ultimate hope in using these new technologies to continue to tell Star Wars stories?
Rob Bredow:	I think the big opportunity with Virtual Reality, or Augmented Reality, is being able to tell stories that are really well suited for this medium—that can only be told in this medium. We think there’s actually quite a few stories that are best, or perhaps only able to be told, with this sort of format.
John Bucher:	Will LucasFilm and ILM look through the VR lens beyond the Star Wars universe?
Rob Bredow:	I definitely wouldn’t rule anything out. ILMxLAB is working on projects that are outside of the Star Wars universe already because we think the Star Wars universe is a great place to tell these stories, but it’s certainly not the only one.
John Bucher:	You are someone who is a master storyteller that has worked for many years in these cutting-edge environments of technology. What are some of the key things that you have been able to bring from your background from working in film and special effects to the world of VR storytelling? What have been some of the key tools that you’ve continued to use with this new experience?
Rob Bredow:	Well, the same core emotional storytelling tools, I think, are completely applicable for cinema and Virtual Reality. I think the biggest differences are the things that have been the most interesting. I can give you a story from the making of Trials on Tatooine.
	We had it pretty far along, and the experience was quite a lot like what you see today, but it had some extra dialogue in it. We put Kiri Hart (SVP of development at LucasFilm) through the experience, and after she came out, she said, “Give me the script, I want to make a couple suggestions.” And she did an edit on the script where she tried taking out all the dialogue that wasn’t directed right at you the user—the player. The dialogue that was happening between Han and Chewy, in the cockpit, she tried striking it out. She tried striking out pretty much anything that wasn’t happening to you directly in first person in the story. She said, “Try it without these lines and see how it feels.” Ironically, some of those lines were some of my favorite lines that Pablo (Hidalgo) and I had written, so I was like, “Oh man! That’s a super-fun line. I don’t want to cut that.” But you have to try it and see what it’s going to be like.
	The next day we had that version up, without those lines in it, and it was the first time that our brains completely relaxed playing Trials in Tatooine. What we learned, at least in that moment and in this phase of Virtual Reality, is if you want somebody to believe that it’s a first-person story, you can do it. But you want to continually remind them that the story is directed at them. If you are in a virtual environment where you’re already imagining you’re standing and moving in the Millennium Falcon and then the storyteller asks you to imagine that in another room, let’s say the cockpit of the Millennium Falcon, there’s another conversation going on between Han and Chewy, that’s not directed at you but instead just overhearing—it’s pretty complicated to ask somebody to pick up on it and not get distracted by it. The moment we simplified it down to be all about you, the story got that much clearer and better as a first-person experience. That’s just an example of the kind of learning and experimentation that’s happening right now at LucasFilm and ILMxLAB with our storytelling process in Virtual Reality. We’re getting surprising results.

Acting and Directing in VR Stories

An Interview with
Tye Sheridan, Actor and Star of Ready Player One and Cocreator of Aether Inc.
Nikola Todorovic, Director and Cocreator of Aether Inc.

Tye Sheridan has been named one of Variety’s 10 Actors to Watch. Seen recently as Cyclops in X-Men: Apocalypse, Sheridan has played leading roles alongside Brad Pitt in Terrence Malick’s The Tree of Life, alongside Matthew McConaughey in Mud, and portrays Wade Owen Watts/Parzival in Steven Spielberg’s Ready Player One. Nikola Todorovic is an emerging director in the world of VR with a background in visual effects. Together, Sheridan and Todorovic created Aether Inc., a VR production and development company.

John Bucher:	Let’s start by talking about why VR has become so important. What is it about right now that, for whatever reason, makes this the right season for its large-scale adoption?
Tye Sheridan:	That’s a good question. I actually had a director show me a short that he did in 1998 or ’99, and it revolved around Virtual Reality where a guy had a virtual girlfriend.
Nikola Todorovic:	Millennials are so open to technology. When I was a kid, if you were doing something on the computer, your dad told you that you were wasting your time. Now everything is about technology. I think another reason VR is succeeding now is because the internet is faster. If you wanted to stream VR four years ago, you would have trouble. I think that’s a big issue. You have to have broadband that’s really fast to be able to do it, because everything is streaming now. People are also way more open to wearing an HMD now than they used to be. I think Google Glass was a good introduction to that, although it failed. People are still more open to wearable technology. The “Wow” factor in VR just completes that circle.
Tye Sheridan:	Especially for people who haven’t seen it before. I showed it to my 73-year-old grandpa and he takes the headset off, and looks at me and I said, “Well, Papa, what did you think?” He just can’t wrap his head around it and goes, “It’s different.” He just couldn’t comprehend what it actually was. It was funny to watch him inside the virtual world. I said, “Papa, you know you can look left and right and it will track wherever you look.” He starts to hesitantly turn his head left and then he realizes the frame is moving, and he can look anywhere, and it’s going to be filled by frame in a 360 environment. It was a great experience.
Nikola Todorovic:	I don’t like the idea that people have compared VR to 3D and its failure to be adopted en mass. 3D has existed for so long and does not really add that much to the average user’s experience. I think it’s quite different. It can’t be compared. This transforms you. It puts you in a world. It really does. We’re in such an early stage right now. This is early filmmaking.
John Bucher:	Chris Milk famously called VR the ultimate empathy machine. What do you think the relationship is between immersive environments and empathy? What’s the connection?
Tye Sheridan:	It really is a different plane of entertainment—a different experience. I remember being at Sundance 2014 and Oculus had a booth and I walked in and saw all these people trying Virtual Reality for the first time. They’d completely lost all awareness of where they were. I guess I’ve always been drawn to stories because it takes me out of my world, and this is just a totally different level of the empathy created by doing that. It’s that next level.
Nikola Todorovic:	I also think it’s such a psychological thing with our brains. Habituation, when our brains get used to activities, is a big problem. That’s why story is so precious. If you make a good story, you can move and inspire someone, or get someone to understand how kids in Africa live or how refugees in Syria feel. You have to make a really good story, because we see these worlds on the news all the time, and after a while, our brains become adjusted to it as normal. We can’t really feel that much empathy. Maybe for a second we’ll stop, but you see violence all the time on television. When you see someone getting killed on a TV show or a film, it really doesn’t affect you that much because you’ve seen it a thousand times, so you don’t understand how it looks and feels in real life. I think VR is so new that our brains are tricked into believing that it’s real.
John Bucher:	With other types of visual storytelling, be it theater, be it film, television, even video games, we experience those stories in a dark room with strangers or on the couch with family members or in an audience sitting next to people. With VR, presently, you have an HMD strapped to your head, and you’re very much alone in these environments. Can you talk a little bit about how you think the role of social VR will change storytelling?
Nikola Todorovic:	That’s Ready Player One, pretty much.
Tye Sheridan:	A much more glamorous, attractive world. You can pick and choose who you want to hang out with—the world you want to live in. Ready Player One is based on a novel by Ernest Cline, and in it, you see all of these kids who live on different planets based on their desires and likes, whether they like arcades or they like roller coasters. You can literally go to the amusement park planet or the sports planet. There are so many different options, and why would we live in a world where the limitations are restricting us from doing the things that we love to do all the time? If you can have that at your fingertips, then why not?
Nikola Todorovic:	Then comes the danger that I think that book speaks of so well—you can get lost.
Tye Sheridan:	You can totally get lost in that world. That’s one of the major themes in the story talks—staying true to reality and embracing it, because at the end of the day, it’s the only thing that’s real.
Nikola Todorovic:	I remember when everyone first began doing chat rooms. It changed the way we related to each other. Now, talking on the phone is getting weird, because we’re texting. You call someone that you’ve just met, they’re like, “Why is this guy calling me? Why doesn’t he text?” I think our kids are going to say, “Ha, you guys used to text. How ridiculous was that?” We lie to ourselves that it’s not moving as fast as it is. I’m not as scared about technology as many people are. I think we need to accept it. I think it will be good. There’s always the need for moderation.
Tye Sheridan:	I have a 16-year-old little sister, and in the past two years, I’ve seen a lot of that change represented in her life and the way she interacts with everyone and the way she’s evolved as a person. Back to Ready Player One, it touches on that a lot. It’s one of the major themes. If you get so comfortable being someone that you’re not, you start believing that you’re that person.
Nikola Todorovic:	I do think social VR and gaming’s going to be way bigger than entertainment VR. I think we’re going to be watching entertainment inside of these social experiences. For that reason, I don’t think people will be building avatars as often. They are just going to be themselves, I hope.
John Bucher:	Tye, how do you think VR is going to change the acting profession? Actors have been used to knowing, “This is my frame, and this is where I’m moving within a frame.” As that goes away, how is that going to be for actors?
Tye Sheridan:	That’s a great question. In recent years, I’ve become very technically aware. I believe that it allows me to do my job better when I have an awareness of what the frame is. I always ask to see the frame before I step into the shot or what lens you’re using, because when I have an awareness of the frame, it becomes clear in my mind. The easier that makes it for the cinematographer, the director, the gaffer. We can all start to work as one. We all understand each other’s job and we all understand the way the camera works and the way the set is supposed to function. With Virtual Reality, when you’re shooting in 360-degree video, it’s also super important that you stay technically aware because of issues like eye lines. It creates a new challenge for actors, but it’s also a new opportunity. In some ways, it’s more like stage acting.
Nikola Todorovic:	We shot a scene that was really hard recently. Someone who Tye’s character loves dearly is getting killed in front of his eyes and he cannot move. He’s tied up, so he’s screaming and crying. That’s really hard as an actor because he needs to keep that emotion going for the entire period. We can’t cut away. I think a lot of VR is going to be shot on blue or green screen for that reason. Then you will be able to do scenes like that with multiple takes.
Tye Sheridan:	There’s so many things to focus on. I know it’s also super difficult to direct.
John Bucher:	The two of you have formed this VR company. What are you hoping to bring to the VR space that will be unique?
Tye Sheridan:	I think there’s a huge lack of really strong narrative and story in Virtual Reality. People are getting distracted by the medium, and it is cool, but what’s the story?
Nikola Todorovic:	I always tell Tye, you ask yourself if you made this and put it on YouTube as a regular video, not in 360, would it be good? Story is story. It doesn’t change when you watch it on a different medium. Kids are watching multimillion-dollar movies on the iPhones. If you have a good story, it really doesn’t matter where you watch it. I think that’s one huge thing that we’re [Aether] focusing on. It’s story ahead of everything else.
Tye Sheridan:	Right, you can’t get distracted by the technology, just take us through the story.
Nikola Todorovic:	Once you call attention to the technology, I realize I’m experiencing something. I’m no longer in your story.
John Bucher:	It breaks the immersion.
Nikola Todorovic:	It totally breaks the immersion for me. I’ve seen experiences that were done in 180, and they’re great. I don’t always even need the entire 360 if the story is there.

Case Study: Baobab VR Studios

Eric Darnell, Chief Creative Officer
Maureen Fan, Chief Executive Officer

Established in 2015 by Maureen Fan and Eric Darnell, Baobab Studios is a VR animation studio, creating dynamic narratives and story lines. INVASION! was the studio’s first VR animated film, winning numerous awards including Tribeca Film Festival’s 2016 VR Selection of the Year, Cannes “Next Marche de Film,” and Toronto International Film Festival. Maureen Fan has held leadership roles in film, gaming, and the consumer web. She was vice president of games at Zynga, where she oversaw three game studios, including the FarmVille sequels. Previously, she worked on Pixar’s Toy Story and at eBay in product management and user interface (UI) design. Her most recent collaboration, The Dam Keeper, directed by Dice Tsutsumi and Robert Kondo, was nominated for the 2015 Oscar Best Animated Short.

Eric Darnell’s career spans 25 years as a computer animation director, screenwriter, story artist, film director, and executive producer. He was the director and screenwriter on all four films in the Madagascar franchise, which together have grossed more than $2.5 billion at the box office. He was also executive producer on The Penguins of Madagascar. Previously, Eric directed DreamWorks Animation’s very first animated feature film, Antz, which features the voices of Woody Allen, Gene Hackman, Christopher Walken, and Sharon Stone.

Focused on animated VR content, Fan explains why this medium has been their emphasis. “We believe that deep down inside, there’s still a dreamer inside everyone, and we know it’s true because it’s the reason that we go to the movies to this day. It’s to experience characters and stories beyond those that we meet in our common lives. And we believe that animation does this better than live action, because live action is still constrained by reality, versus animation, which is constrained only by the creativity within the director’s head. Animation, to us, it’s our emotion. It takes you to completely different worlds. It makes the world feel so real that you think you could reach out and touch it. The last two sentences that I just said, which is that it takes you to a different world and makes you believe it’s real, are the definition of Virtual Reality, which is why we think animation and VR were made for each other,” she said.

Darnell concurs. “We’re focused on interactive storytelling in Virtual Reality, and leveraging off of one of storytelling’s great strengths, which is the capacity to elicit profound, emotional experiences through the development of empathetic connections with the characters within the story. This is what storytelling has been doing for thousands of years. Storytelling has evolved with the human race. It’s in our DNA. Really, it’s what it means to be a human being, and it’s one of the reasons why these classic forms of storytelling, like literature and movies and TV and plays, can elicit these really powerful emotions in all of us. We’re talking about these powerful emotional journeys.”

Baobab has won over some of the most significant voices in visual storytelling. After seeing their content, Alvy Ray Smith, cofounder of Pixar, compared the protagonist bunny in their film to the power of VR itself. “It gives you the opportunity to believe that a character really exists, and really matters, and then be able to act on that behalf. You just can’t do that in any other storytelling medium. You know, if you look back at the movies, it’s remarkable how you can just sit in that room and it can elicit these really powerful emotional responses. It can make grown men cry, make an entire audience gasp in unison, lovers clutch others’ arms and children instinctively cry out for their mothers,” he said. Darnell agrees that story will be the key to VR’s success. “Through storytelling Virtual Reality, I believe we’ll be able to have the same kind of deep and profound emotional experiences that we have at the movies, and these experiences have the potential to be even more profound because of the fact that we’re actually living them.

“VR is not a movie. It’s not a game, at least the direction that we’re taking it. You know, there’s no camera, there’s no screen, there’s no rectangle. There’s no fourth wall to break, so when a character looks at you, they’re just looking at you. You are in their world,” he said.

Baobab plans to continue to focus its efforts around character-driven storytelling. “One thing that’s definitely true in film and certainly true in VR is that having great animated characters are always worth it. This is what storytelling is all about. It’s about connecting with these characters that you are in the world with, that you understand, not what they’re, not just what they’re doing, but you understand why they’re doing it,” Darnell said. “We need to know, we need to see what that character is thinking. Before they take an action, we all decide to take that action. We need to see that, and if you can deliver that to the audience and understand what’s driving these characters from the inside out, that’s what’s going to give you the ability to really connect with them, understand them, and empathize with them,” Darnell concluded.

Storytelling in VR through Journalism

An Interview with
Sarah Hill, CEO and Chief Storyteller, StoryUP

Sarah Hill is an Emmy-award winning, 20-year veteran of the interactive journalism industry. Before starting StoryUP, she built a successful TV feature franchise and the world’s first interactive news program based on Google Hangouts. She has produced content in Vietnam, Guatemala, Sri Lanka, Indonesia, and Zambia and collaborated with companies ranging from Google to the NBA to the U.S. Army.

John Bucher:	Tell me about how your professional interest in story began.
Sarah Hill:	Telling stories became important to me early on as a journalist in my fixed-frame flat world. I got into television in the early ’90s, and I did some stories about a few veterans that I fell in love with. We ended up opening up what’s called an Honor Flight Hub in Columbia, Missouri. These were flights that take aging veterans to see their memorial in Washington, DC—physical flights. I got into Virtual Reality because a lot of these veterans were calling us saying they were too ill to travel on physical flights to see the memorial in DC, so we needed an alternative. We began using Google Glass to try to take them there. We were doing live streaming tours and then we would take laptops to the veteran’s bedside. That had inherent problems with bandwidth and things like that. Someone suggested, “Have you ever heard of Virtual Reality?”
	I tried Google Cardboard. I watched Chris Milk’s TED talk and was totally blown away. I said, “This is the missing link, this is what we can do … Use it to capture these tours for these veterans.” After we did that first experience for the veterans, I noticed they were overcome with emotion, far more than a regular fixed-frame flat video. After watching them react to immersive video content, I knew, as a storyteller, this was a very important tool that we could be using. In addition to good writing and good video, good production values, good characters, all those kinds of things—the technology of it is really ripe for storytelling to elicit an emotional response. From there, I quit my excellent job with a well-paying salary, and we opened our own company. We are an immersive media company. We have a brand studio and we have a journalism studio.
	Primarily, our economic engine is working with foundations and charities to try to illustrate for people their problems so that they’re able to then use that content to raise money or raise support among their donors. That’s primarily the content that we do. We also do meditation and mindfulness experiences as well. Our app features a variety of stories, not only health-care stories, but experiences about what it feels like to experience stroke.
John Bucher:	Talk to me a little bit about why telling a story in VR space is different—what sort of opportunities that affords the storyteller that are different from our traditional media.
Sarah Hill:	It’s different, but it’s also the same. A lot of people think once they start telling stories in this sphere, “Oh this is great, there’s no frame,” and I was the same way. “Wow this is liberating. We can look anywhere,” and then you get into writing the stories and you realize, there is a frame. It’s just a frame inside the sphere and it’s moving left and right and it can be confining to the storyteller, because for decades we had the control. We were the ones who decided what they saw through that frame, and now we don’t have that control. The viewer has the control, and they have to decide where to look. As a storyteller, it’s a power struggle for us because it’s a different way of telling stories in that you don’t know what they’re seeing as opposed to what you’re seeing. You have to use positional audio. You have to use narration. You have to use graphics. You have to use tracking objects in order to gently direct the audience’s attention.
	It’s a different way of telling stories—a different way of experiencing them. We are not experiencing them any longer through the filter of the fixed rectangle. Storytellers had used flat words and pictures and video and still photos and all these media assets to try to place people inside a story, and now the technology actually exists that we can truly put you inside the story, which is an enormously powerful tool for a storyteller. We’ve actually done studies on the difference between fixed-frame video and spherical video. There’s higher viewer engagement. It’s shared more. It’s watched longer and it’s also watched again. The reason why is that they think that they might have missed something in looking around in this sphere because they can look everywhere.
	Which marketers love—the fact that people would actually want to watch their content again. There’s also some very interesting things that happen in the brain between fixed-frame video and immersive video. We have a psychologist who works with us. His name is Dr. Jeff Tarrant. He’s actually studying what immersive video does to the brain, so we give him our stories to work with. People aren’t just watching the video, they’re feeling the video, and so after you create an experience, you have to put it in the faces of a variety of individuals, even the most motion-sick-prone person in the room, and say, “How’d you feel watching that? Did this slight movement bother you? At any point did you feel queasy?”
John Bucher:	Let’s talk about the idea of narrative characters. We are familiar with the concept from traditional storytelling—plays and films and books. Can you talk a little bit about what you’ve brought from your traditional training and storytelling with characters and how you’ve brought that into the VR space?
Sarah Hill:	In journalism, we were never allowed to stage anything or use props or a set or anything like that. We just had to capture reality, and that’s what we’re really doing here with VR. You will see a lot of filmmakers who are staging or telling people what to say. Or bringing in props. And ours is more journalistic storytelling and less moviemaking. As journalists, we learned how to find compelling characters for our stories. That was what drove our stories in the fixed-frame world. It was very natural for us to be able to translate that concept into the immersive world, because when we tell stories, you can’t tell a story without a person. Sure, you can humanize around a tree or a dog or something like that, but you need characters, whether they be animate or inanimate objects. All of our stories are people driven, we call them CCCs, or central compelling characters in the story.
	With immersive video, you really have to decide who is the camera in this scene in the story. The camera doesn’t have to be one person throughout the whole piece. We learned that the hard way. We thought, “Oh, it’s all these pieces, they have to be point-of-view pieces.” Well, they’re not, because when you do that, you don’t get all sides of the story, and sometimes you totally miss the empathy. We thought, for instance, in our Zambia piece, the point of view we should adopt is the person on the ground and we should show that view, what that’s like at all times, but if you show that at all times, you miss the very important view of what it feels like when one of these individuals was crawling towards you. You also have to have that third-person perspective as well, or your audience sometimes feels a little bit confined. Sometimes you want them to feel confined in experiences, but sometimes you really want them to experience all of those different angles.
	Certainly, as a journalist, our default position is that we want the audience to experience a variety of different views. To answer your question from the fixed-frame world, as a journalist, what we carried over was the ability to find compelling stories. As journalists, this was how they trained us when there was no news going on. The assignment editor would say, “Go, get out of the newsroom and find a story.” So that was what we had to become accustomed to, and if you ask any feature journalist out there right now, chances are they have about three stories in their mind that they could tell on a day when there’s nothing really going on. Why? They talk to the people at the gas station. They talk to the people at the grocery store. They’re volunteering in their communities and they’re constantly asking people, “What’s news to you?” They’re constantly thinking about stories that they could cover.
John Bucher:	Let’s talk about the technical production of the pieces that you’ve done. In traditional, fixed-frame media, we’ve had the lower-third graphic that’s come up under someone. You’ve moved the lower third out into the space to put it next to the person or put it somewhere we still can identify who that person is. Can you talk a little bit about that process of finding the sweet spot of how to technically achieve good storytelling in VR?
Sarah Hill:	If you look at some of our early pieces, you will see that we did put it right where it’s always been in the frame, because that was where our comfort level was. We quickly realized that was not always a workable solution because viewers can look all the way around. Why have it covering the speaker’s body when it could be floating out to the side? When it could be embedded in a box that’s next to them? When it could be put in the side of a mountain?
John Bucher:	Can you tell us a little bit about where you’re taking this and where you want to see VR storytelling go?
Sarah Hill:	We have a project coming out with Empowered By Light and the Leonardo DiCaprio Foundation, and it’s all about solar energy—how a lack of solar energy threatens to flood sacred lands of a certain tribe in the Amazon. These sorts of projects that make the world a better place are where I’d like to continue seeing us go. We’re also shooting a project with Facebook and Oculus and a charity called Love Has No Labels. We’re playing with cropping. We’re playing with the sphere. We’re playing with mirrors. We are looking at implicit bias, using 360-degree video to illustrate for people our implicit bias, and how if we’re only seeing this much of a story or of a person, we’re not seeing their whole picture. We’re not seeing them fully, and we don’t have enough information to judge that situation fully.
John Bucher:	Finally, all technology takes us somewhere. It transports us somewhere. Where, in an ideal world, would you love to see this technology transport humanity?
Sarah Hill:	I would like it to transport us back to our home videos. I want the ability to put my fixed-frame video of my second birthday with my mom in a VR headset and watch it again like I was there. I know that some smart person will come up with that idea, but I think that would be really great to see.

Insights from the Storytellers

Storytellers provide looks at life that range from our own backyards to the farthest reaches of outer space. The tools at their disposal are advancing faster than ever. Some storytellers work with characters and archetypes very familiar to their audiences, like Rob Bredow. Others, such as Sarah Hill, are making the decision of what real-life personalities will be the characters in the story with every project. Both storytellers use characters, but in completely different ways. Understanding how characters work, as discussed in detail in Chapter 6, helps creators identify one of the most basic building blocks of narrative. We feel empathy for characters and the situations they find themselves in, not for environments or lifeless objects, such as costumes, important as those elements are. Bredow more specifically identifies emotional resonance as the sort of empathy he looks for in creating stories. Resonance suggests that two similar things share qualities. Emotional resonance would then speak to two individuals sharing qualities or emotions—the character and the audience member. Hill referred to these as the three Cs—central compelling characters. These are just the sorts of characters Tye Sheridan embodies when telling stories in VR. He models what is possible when an actor truly understands their role in the storytelling process. There’s no need for us to consider or move on to storytelling structure if we don’t have compelling characters in place.

While Baobab uses primarily animated animals in their stories, they demonstrate the same understanding of characters. They value the same empathy and emotional resonance that Bredow speaks of. There is a wide span between the sorts of stories told by Sarah Hill in Africa, Baobab in animation, and Rob Bredow in the Star Wars universe, and yet there is not. All the storytellers in this section understand that story begins with effective characters that people can identify with. All understand the power and even limits of the technology they are using but don’t let that become the focus of their work. They create characters that embody the best (and worst) of who we are and who we want to be. We learn lessons from the characters’ mistakes and take joy in their victories. Stories that are true, loosely based on real events, and completely fictional all have a place in the narrative universe, and certainly in the world of VR, as long as they hold the tenets of storytelling that have served humankind for thousands of years—characters, conflicts, and resolutions.

A number of storytellers in this section and in this book mention that VR will not be experienced by only a single person at a time for much longer. As social VR experience becomes available, the emotional resonance experienced through the technology will be held by an entire audience together, much as it is in a movie theater. Social VR experiences and stories will hold the capability to engage the smallest audiences of two or three, as well as the capability to tell stories to literally thousands of people at one time. These potentials will undoubtedly change the ways stories are crafted in VR space as they develop. These changes will make it increasingly important to rely on and implement the elements and methods that have seemed to work across centuries and mediums. As Eric Darnell stated, “Storytelling has evolved with the human race. It’s in our DNA.” There’s been no indication it will not serve us in all the same ways it has throughout our evolution.

The Technologists and Producers

Nonlinear Storytelling in VR

An Interview with
Jonathan Krusell, Google Daydream Producer

Jonathan Krusell is a creator with more than ten years of experience leading production, creative, and strategy for interactive entertainment. His career has included roles ranging from VP of production to studio director. Awards for some of the games he has worked on include IGN People’s Choice Award 2012 (Avengers Initiative), Best Family Game at E3 2009 (Disney’s Guilty Party), and GameSpot Best Use of Zombies at E3 2005 (Stubbs the Zombie). Other significant properties he has worked on include Disney’s The Haunted Mansion, Charlie and the Chocolate Factory, 50 Cent Bulletproof: G Unit Edition, and Spiderman 3. Currently, he serves as an executive producer at Mindshow and a producer for Google’s Daydream.

John Bucher:	Tell me about your background in gaming and how that led to VR.
Jonathan Krusell:	I’ve been doing games for 15 years—console, PC, and mobile. Games are what I set out to do, actually. I went to film school, was interested in movies, but to me, I felt like movies had peaked in the late ’70s. I don’t know if I was right about that, but I wanted to get into something that was still on the rise, so games are where I was focused for a long time. The thing that I find interesting about VR is that previously in games, the user is pointing things into a controller, the controller takes those signals, puts it into a system, the system interprets that and then triggers animations, right? If you think of the player as a performer and the avatar as the presentation of that performance, there’s a lot of layers between them. A lot of those layers collapse now with VR. On the high-end systems, and even the medium-level systems in VR, you basically have consumer-grade motion-capture systems. The HTC Vive has the lighthouses that capture your motion. Oculus has cameras. Daydream has accelerometers. They’re all capturing motion, but it’s actually your motion.
	Some games let the player directly drive the characters. That means that all those layers have collapsed now. What that means is that there’s this opportunity now for improvisation. You can do things that no engineer ever thought of. Previously, engineers, designers, and artists had to think about everything ahead of time and invest it all into the software so that you could trigger it. They don’t have to do that anymore. You can’t do everything yet, because it’s not full-body mocap, but it’s still pretty robust. In the near future, it will get more and more robust as newer technologies develop and converge. Things like that exist right now, but just on separate devices. As those things come together into a single device, it will be more and more robust.
John Bucher:	What were you able to bring over from the gaming world that works well in VR? What did you have to transcend?
Jonathan Krusell:	The biggest thing I think I bring to VR is just a stomach for software development. Software development is a very nerve-racking thing. Things are constantly evolving, and it’s hard to predict where it’s all going to end out. The first three times you work on developing a project, it’s terrifying. Eventually you get a stomach for it and you’re like, “Okay. Everything’s probably going to be okay.” I’m also comfortable with nonlinear storytelling. Actually, I’m super comfortable with it and prefer it to a three-act structure. A three-act-structure story means you’re taking agency away from the user. Video games tried to do that for so long with movie audiences. We were just trying to recreate a movie in a game. Nonlinear storytelling has served me very well in VR. I used to run a motion-capture studio. Similar to a theater in the round. We were constantly thinking about 3D volumes and 3D spaces and thinking about storytelling where you don’t know where the user is. Half-Life’s always done that really well. There’s a narrative happening with dialogue that’s preprogrammed, but you can move throughout the space and they react to you. On top of all that, I love the idea of immersion—trying to find ways to get the player to be engrossed in the experience. Not many other mediums have immersion at the level VR does. Books are the most immersive thing in the world, but they’re a particular medium that doesn’t include an audio/video experience or haptic feedback.
John Bucher:	In VR, we have a deeper level of embodiment. In some VR experiences, I can look and actually see my hands. Can you talk about embodiment?
Jonathan Krusell:	That’s something that is actually different from video games. The tropes that I’m used to from games—they’re evolving. They don’t apply the way they used to, whereas a third-person action game previously felt like you were driving an avatar, right. But now, a third-person experience in VR feels like me and you. There’s a character, and I’m a different character, and we’re working together to do something. That’s very different.
	First person is even more different. Previously, first person felt like I was in the action, but there was a barrier between me and the world—the screen. But now with VR, that barrier is dissolved and I am actually surrounded by it. It can be jarring and shocking for some users. As a designer, you have to ask, how uncomfortable are you willing to make the user, and does that become a barrier to other things you’re trying to accomplish?
	With some of the applications I’ve worked on, you could embody an alien, or you could embody a woman, or you could embody a man with completely different proportions, right. In all those cases, there’s going to be moments of, minutes of, or never-ending discomfort. Female characters in video games are super popular. But I wonder if in VR it will be a bigger barrier for some males—to embody a female character and get comfortable with that. You have to be thoughtful about when and how you’re going to do it and also to prepare the user for it. It can be jarring to suddenly be in a new space and have a little micro–identity crisis about what’s going on. That’s one area where it is very different than games, and there is still lots to figure out and learn about.
John Bucher:	You have developed content for Disney before. We often think about VR as being something that people 16 and up really engage with, maybe 16 to 40, actually, being the target demographic. When we think about putting a headset on children, do you think there are going to be different ways that we approach VR for different-age audiences? Obviously, content is going to be an issue, but do you think there will be differences in how we design or tell stories?
Jonathan Krusell:	Yes. I think there’s different ways that you would consider the ideal-use case—thinking through session length and intensity. I think Google Cardboard is something that is okay for a ten-year-old, because it’s not as intense. It’s something that’s easy to remove. It’s something that’s made with short sessions in mind. I think Expeditions is a great example of how you can make content that is only doable in VR—taking big field trips together. My kids are pretty young. I hesitate to put them in any VR, but I certainly wouldn’t put them in an HTC Vive, because it’s just too intense. I had a weird experience showing my mother VR. She’s over 70. I didn’t realize until it was on her how ridiculous it all looked because the Vive looks like you’re piloting a drone or something. It looks military grade.
John Bucher:	What do you think are going to be the ongoing challenges that storytellers in VR will continue to grapple with?
Jonathan Krusell:	When you’re in software development, the thing is that you never get comfortable. The technology’s always changing. You have to get used to that sort of cadence. I think that’s going to be challenging for people coming from other industries that are now learning the ins and outs of software development. At some point, there’s going to be some pretty big moral content decisions. Right now, it’s a pretty tight-knit community, and there’s some unspoken territory to stay away from. It’s not going to be like that forever.
John Bucher:	If I become comfortable having the experience of killing a very human AI, I’ve experienced the feeling of murder. What do you think are the ethical issues that we would do well to think about now? Most people agree, you can play video games and you can shoot people all day, and you’re not going to really want to go kill a human being. But at some point, with artificial intelligence in Virtual Reality, you’re going to feel like you’ve actually shot someone. How do we navigate it?
Jonathan Krusell:	Yeah, I agree with you. In the past, there was the barrier of the screen that arguably made everybody realize it was fantasy. That doesn’t exist anymore. How do we regulate that? I don’t know. How do current channels with lots of power self-regulate? Because, ideally, there’s some sort of system that self-regulates. I can’t immediately think of a system like that that works well. These are important questions to ask because they’re inevitable.
John Bucher:	I want to circle back to nonlinear storytelling. It can be argued that with all of human activity, we make sense of the world through narratives.
Jonathan Krusell:	We layer story on it, constantly.
John Bucher:	Absolutely. When we talk about nonlinear storytelling, are there narrative shards that we pull from a structured story where we say, “Well, it’s not a linear story, but we still have a good guy and a bad guy, or we still have this conflict”? How does nonlinear storytelling work? Can you tease out that a bit?
Jonathan Krusell:	There’s different kinds of nonlinear storytelling. If you think of the film, Clue, it has multiple endings. So does the board game. Nonlinear works well in the mystery genre. Where the board game Clue is simply context and progression, there’s the context of a game board, and there’s a context of the clues and the characters, and then there’s the progression as you move through and make your own deductions. The game doesn’t know what choices you’re going to make. The game doesn’t tell you how to perceive the clues that you find and make the deductions that you make. Now, the game does know who is actually guilty. It’s in the envelope. That doesn’t change the story that you’re telling.
	It’s all nonlinear until the end, because the ending is decided. But you’re able to create your own story in your head by the breadcrumbs that you pick up and how you interpret them. To me, that is nonlinear storytelling, which is like context plus progression, having choice in the matter, and being able to interpret and deduce what you believe. That’s what I think I’m really excited about in VR is the idea of mystery games. When you talk about Alternative Reality games, the murder mystery is the oldest Alternative Reality game there is.
John Bucher:	In video games, most stories have an end—you can conquer the game. There’s exceptions—world-building games that just go on and on. Do you see, with VR experiences, somebody entering a space where there’s not a clear ending or objective? Do you think that’s going to get frustrating for people, or do you think just being able to constantly explore will have the same effect as, say, a World of Warcraft that can just go on forever? What do you think we need to be designing experiences toward? Do we need to guide people towards an ending or just endless exploration?
Jonathan Krusell:	Well, I think it’s going to evolve as the medium evolves. I think, right now, an ending is fine. I think right now there’s going to be a lot of single-player experiences that are finite in length, maybe even episodic. These are all experiments right now, to learn about what is ideal. Then as it evolves, it can go in any number of directions. I definitely think that multiplayer or shared experiences are going to be huge. I think that taking on different characters, like in a murder mystery, where there is improvisation, will be effective. Let’s do it in a castle. No, let’s do it in a Victorian manor. That sort of thing, that allows people to customize the experience for themselves. Then, ultimately, I do think that these things would exist in some sort of shared universe where it’s looped within loops. Where you complete one loop and it just reveals a bigger loop. Which is just basically creating some sort of simulated reality. But a simulated reality would define rules. What matters the most, though, is what the promise is to the user when they’re going into it. Is there promise of an ending? Well, then you better give them an ending. Or is the promise a universe? In which case, you have to establish the rules and then let them make whatever choice they want within those rules. That’s where the real potential is—shared universes.
John Bucher:	Do you think rules are going to be necessary in order to ground Virtual Reality and cause people to actually have a truly immersive experience?
Jonathan Krusell:	I think rules are inherent. That’s why if you make something look photo-realistic, then you are setting a bar that everything else has to match, so the physics better be realistic, the lighting better be realistic, and the rule set better be realistic. Whereas if you make something that looks like VR Super Mario, then it’s a totally different rule set.
	I think it’s not a question of do we have to create a rule set; people expect rule sets and they take in what they see and they make rules already. They try to figure out what the rule set is in a scenario, and hopefully you meet that expectation. If you make a decision to dramatically surprise them with a rule they didn’t expect, that’s a risky move, because it could blow away immersion. If something did look photo-realistic, but then I couldn’t pick it up, I’m upset. In video games, you get away with that stuff because the expectations are different. You get away with a lot of things in video games that I don’t think you can get away with in VR. You have to be very deliberate. That’s why for right now, stylized content is a little safer, because then you can communicate the rules to the player as you go. I like starting small and then layering complexity on top of it. That’s just my style, because I think you can get to something more consistent that way.
John Bucher:	You brought up multiplayer experiences. In the history of VR, it’s been a very singular experience thus far. I put on the glasses and I’m alone. I’m figuring this all out myself. That will obviously change in the near future as we begin to have these multiplayer experiences with friends or family or strangers. How does storytelling change when we go from a medium for individuals to a group dynamic?
Jonathan Krusell:	I think it’s going to be like LARPing—where we all agree to live in this fantasy and we do our best to support each other in that fantasy, and if you break it, there’s some sort of social repercussion for that. I mean, we’re all doing that right now, every moment, in real life. It’s just not as fantastical. The trick with the VR, though, is going to be matchmaking—which is a really big deal. You have to have a critical mass of players at a certain time, at a variety of skill levels, that you can set up quality matches where everyone has fun, and then they show up again later. That’s how things are with console games.
	Now in VR, at least in my opinion of the immediate future, you’re looking at drastically shorter sessions—like an hour or less. How do you reach critical mass so that you can have a quality multiplayer experience? It’s tricky. It has to be very event driven. It has to be super event driven where there’s a reason. It’s like now with Netflix and all this streaming media where you can consume anything at any time. Now there might be this new VR event system where it becomes water-cooler talk. “Oh man, were you there last night?”
	There’s two things that are exciting, to me. There’s the shared fantasy, and then there’s the creation tools. I am intrigued by shared fantasy—role-playing. Then things like Google’s Tilt Brush that allow you to create in a way that you can only do in VR. If you can combine those two things, that’s the magic.
John Bucher:	Can you speak about what VR could look like as a storytelling medium once we are able to get away from the HMDs? Once it becomes truly augmented reality across the board, what is storytelling like then, and how thin does that line get between narratives and just actual reality?
Jonathan Krusell:	There’ll have to be some sort of visual language or some sort of adaptive density of fantasy where it can figure out the mental state you’re in and dial it in to what you need at that moment. When the technology is that seamless, it’s going to get real weird. Is there some sort of centralized service that is making sure that we’re all not in conflict, like our fantasies are harmonious, or at least reinforce each other? That we don’t walk off cliffs or something like that? I think that barring any extinction-level events, it will eventually get there, and that’s where we’ll need AI to help out a lot, if they don’t enslave us first.

Horror-Based Storytelling in VR

An Interview with
Robyn Tong Gray, COO and Lead Designer, Otherworld Interactive
Andrew Goldstein, CEO, Otherworld Interactive

Robyn Tong Gray is an interactive media designer who weaves new media together to explore narrative and empathy. Her projects have been featured at venues including the Independent Games Festival, IndieCade, and the Sundance Film Festival. Robyn is the director of Sisters, Otherworld Interactive’s horror franchise revolving around a pair of spooky twin dolls and their haunted home. She and Andrew Goldstein founded Otherworld Interactive, a Virtual Reality content studio located in Culver City, CA. Their company has been featured in publications across the industry, including a significant profile in PC Magazine.

John Bucher:	How did you become interested in VR? What brought you into immersive space?
Robyn Tong Gray:	Andy and I both finished our graduate degrees over at USC’s Interactive Media Program. They have a really high concentration of professors who came out of VR. When we were there, Mark Bolas was still on faculty. He’s now working on the Hollow Lens at Microsoft. We also met Scott Fisher. Both of these guys worked at NASA on some of the first early, practical VR in the 1980s.
	When I first started in the program at USC, I was a research assistant at the Mixed Reality Lab, which, at the time, Mark Bolas was the director of. I loved storytelling in all mediums. Interactive is particularly fascinating to me. Game markets were flooded, but there’s this whole new thing with all these new opportunities to actually make something that people will see and has a chance of standing out—and that was VR.
Andrew Goldstein:	Robyn had been making some Virtual Reality projects and had a lot of knowledge there and looped me in, saying, let’s see if we can push this forward and start a business. It was great because, at the time, we were one of the few people out there making VR, and we basically have rolled that into a company.
John Bucher:	Let’s talk about your most successful horror project, Sisters. How did that project come to be?
Robyn Tong Gray:	Sisters was originally a two-week project. I love horror movies, and obviously, VR is a lot about tapping into players’ gut reactions—something horror excels at. So we thought, why not make something spooky? I would say with Sisters we play a lot with tropes. We’re not really treading new ground when it comes to the scares there, but I think that’s actually important right now in VR—to make people understand that this is something they are familiar with on some level. Even if they don’t recognize the media they’ve suddenly been placed in.
John Bucher:	Did your classic story training and the storytelling tropes you mentioned translate well to VR?
Robyn Tong Gray:	I think it translated really well. With Sisters, it was taking a lot of the stuff you learned about when you make games. A lot about indirect control and then also a lot more pushing audio, for example, which is often ignored in games. I came up with the script really early on. I decided the interest curve we want, and that we don’t want to have too many plateaus in there where people lose interest. I think from there, it was mainly fine-tuning things, figuring out a good distance for the dolls to pop up from, the timing, and things like that. Also, playing around with jump scares. I love jump scares. They are really easy and cheap, but I love them. We also wanted to move beyond jump scares to incorporate details that might tell a little bit more about the story and that might give you a hint that there’s something beyond this two-minute, linear narrative.
John Bucher:	Let’s talk about the idea of agency in VR. It’s one of the things everyone is trying to figure out—what’s the perfect mix of giving the viewer a sense of local agency as opposed to giving a sense of global agency? Do people want to make the small decisions of figuring out what weapon they are going to be able to carry, or do they want to make the decisions that drive the entire narrative? Are you having those sorts of discussions in the things that you’re designing?
Robyn Tong Gray:	We’ve taken a lot of what we’ve worked on so far from the traditional gaming world. We are asking if we want to make branching narratives. Do we want to make this linear? As a player, I always want to feel like I have these awesome choices that I’m really going to affect the world that the story’s taking place in. On the flip side of that, I feel like nothing I’ve seen has done that in a way that makes me really satisfied about it in VR yet. I think it’s often to the detriment of the story at the moment.
	Even with The Walking Dead experience by Telltale. They really tapped into people’s empathy, really made you feel like you were struggling for these choices, but if you play it a second time around, you feel like nothing you decided on mattered because the other people got axed later. That’s horribly dissatisfying. Clearly, there was a story there that whoever made it really wanted to tell, and anything that diverted from that main vision didn’t feel great as a result. For us, we’ve been setting tone for this global story. We want the players to feel agency in that. They might not be able to alter the huge overarching fate of the world, but they can make decisions for themselves that feel good and feel like they did the best they could, just like in real life under those circumstances.
Andrew Goldstein:	Robyn brings up a really important point that Virtual Reality is about environmental storytelling. It’s not always about linear storytelling. Your job as a Virtual Reality creator is not always to create the best linear story possible. That’s sometimes the job of the film the experience is based on. Sometimes, our task is to create the best environment possible that adds to that story. A lot of times we reference things you embed in the environment that may be interactive, that you might catch, but add a slight level to you being in the space. They’re not necessarily linear pieces, but they add to the overall ambiance of being there.
	It’s creating a new language and new canon for the media. That’s where I think a lot of people don’t quite understand that Virtual Reality is not the next step in film or the next step in games. It’s its own separate thing, and we’re still tinkering with how to make it good. Specifically, with that, there are genres in Virtual Reality that play much better than other genres. Horror, sci-fi, stuff that’s really heavy on being there to understand it does so much better than comedy and drama. People who initially start out trying to do comedy have a lot tougher goal getting something out there because comedy is really specific with timing, or else you kind of miss it, whereas horror is completely ambiance. It’s about feeling scared.
John Bucher:	Let’s talk more about challenges. For several years, people have said the biggest challenge to VR storytelling were things like frame rates or latency—those types of technical challenges, which certainly still exist. But aside from the obvious technical challenges, what are you finding are the biggest hills to overcome?
Robyn Tong Gray:	I think it’s really about setting expectations. With a lot of traditional, say, inventor games, you have the expectation very quickly knowing that something’s purple, which means you can interact with it. You can grab it. Other things are not purple, so you can just ignore them. With VR, you’re putting people in this place, and if you’ve done it really well, then their expectations will be that everything is interactive in the world.
	It’s really about how you frame expectations to the user so that you have formed this version of reality in their mind with nothing that breaks that even on a subtle level. If they see a pencil setting on a table and want to pick it up, they need to be able to do that. And if they can’t, it’s really disappointing. On the other hand, things like physics are much more powerful, too, as a result. There’s not the expectation right now that you can bounce a ball, but if you can, that’s really delightful.
John Bucher:	Can we talk about the potential future of VR once HMDs are not the only way that everyone is experiencing it? Have you started to give thought to how your properties will work in AR environments?
Andrew Goldstein:	Every company in VR has to think about AR. You can’t be locked into this one specific genre. AR adds a whole new level of that storytelling. But if you’re doing it right in VR and the AR platforms come out, you should have no problem transitioning stuff over there. If you want to be an AR company as well as a VR company, you have to start figuring out how you tell stories that work in both worlds.
Robyn Tong Gray:	It’s very specific. It’s just like you wouldn’t make a computer game and expect it to run in VR, necessarily. Some of our stuff would translate really well. For others it would be very different. I personally think VR is the superior storytelling medium. I think there are very specific stories you’ll be able to tell in AR, but, personally, just the way I use technology in my everyday life, I want AR to give me all the practical things.
Andrew Goldstein:	We have a lot of internal conversations about how VR is a superior entertainment platform but AR is a superior tools platform. If you look at the disruptions in technologies over the years, the most disruptive things are tools platforms. That’s what becomes ubiquitous in daily life. We can use tools platforms as world builders. And, obviously, world building is wrapped into environmental storytelling.
John Bucher:	In one sense, my mind can’t tell the difference between the environmental storytelling that is happening to me digitally and somebody really scaring me in real life. Is that something that you think about? Does it go into the design process at all?
Robyn Tong Gray:	Definitely. That is why we like working in the horror genre. Because like I said, it’s really about empathy, and one of the easiest emotions to provoke out of people is being uncomfortable and being scared and not feeling safe. We’re playing on all those instincts all the time. We try to keep people constantly primed to be on their toes. You’ve removed the 2D screen from the equation, so now they don’t know what direction they’re safe in. They’re not safe on their couch anymore. Things could be behind them. That was definitely a very intentional choice. This genre automatically taps into that.
Andrew Goldstein:	Virtual reality tricks your senses because you’ve added that visual fidelity and added spatial audio. When you actually feel that you’re there, when you can actually walk in any given direction, you can feel and touch things—that’s when this completely new and different medium has come into its own. Right now, we’re just tricking.
Robyn Tong Gray:	And even though we are not quite there yet, we talk about the memories you make in VR being much different than the memories you make with a linear, 2D story. When you experience something in VR and you close your eyes later, you can really remember that the ceiling was a certain height. That distinguishes VR as a completely different medium alone.
Andrew Goldstein:	The example we always use in presentations is that if you watch a movie and you see a really key scene, you can’t say how large the room it took place in was. There’s so many visual tricks that are going on. You can guess, but you were never there. You never saw it for yourself. But if you were in Virtual Reality, you can literally say it was 8 feet. I was actually there. Your brain processes that in a completely different way, and stores it in a completely different way as a memory, too.
John Bucher:	One of the things that has continued to make the world of gaming really take off has been the ability to experience games socially—to play with other people in other parts of the world and build community within that gaming world. How important do you think the social aspect of VR is going to be to storytelling? Will it look like it does in the gaming world, or do you think we’re in for something really different?
Robyn Tong Gray:	I don’t know if it’s important to its overall success. I think it’s going to be a huge thing. I don’t think it’s necessarily going to be make or break, because a lot of the people who desire these VR experiences are just people who desire just to see something new, whether or not that’s with other people. I think the more it becomes integrated, the more people will want other people to accompany them.
Andrew Goldstein:	I would say it’s the direction of the experience. If content creators create great content that is meant to be social, then it’ll enhance the experience. If they create content that’s meant to be experienced as an individual, it’ll be really interesting to do that, too.
John Bucher:	We have first-person experiences in VR, and we have third-person experiences where we’re a ghost in the room that’s observing. Do you approach projects differently, or are you doing everything in first person or third person?
Robyn Tong Gray:	We stick with first person. Third person, where you are a ghost observing, is interesting to me, but I feel like it misses the point of VR. We really do love messing with first person and playing around with who you are in the experience. It’s definitely something we consider a lot. For example, with one of the upcoming projects we’ve been talking about, we were saying that we don’t really see mom characters in games enough. There’s been this huge surge of paternal characters, but you don’t really see moms. I like the idea of having people, especially within our demographic, that’s often men, having to step into the shoes of someone completely different. In this case, maybe a mom. You’ll get these people who could never actually be that in real life now forced to confront small parts about this character’s life and how they might think differently, or how they might therefore have to think to solve these puzzles or go through this story.
Andrew Goldstein:	Character identity is central to Virtual Reality because you now assign that. You’re not just a fly on the wall. You have to make sure that your character identity works with the narrative that you’re telling. Also, you have to be careful not to break the experience. The character’s role is essential in terms of helping to provoke a narrative. I’m really excited when there are games where you aren’t actually assigned a character, and you grow or become one out of that. It’s going to feel even more different because it’s actually you being in there, not just you having a relationship with an avatar. You’re not looking at an avatar, you are the avatar.

Case Study: Light Sail VR

Matthew Celia and Robert Watts
Managing Partners and Producers

Light Sail VR has been responsible for creating some of the most innovative content on the market. Their experiences hold one element above all others: story. Matthew Celia explains, “Storytelling, in my opinion, hasn’t changed since the dawn of mankind. Storytelling in its very nature is a linear process. It is someone who says, ‘Let me begin, let me tell you the middle, there’s conflict, there’s character, and then there’s a resolution.’ ” Light Sail has used this methodology to craft VR experiences including the Paranormal Activity VR Experience for Paramount Pictures, Reggie’s Garage for baseball Hall of Famer Reggie Jackson, and Fowl Seas—a VR pirate adventure for GoPro.

Robert Watts says, “For us, it’s all about motivating who you are in the scene and why you’re watching.” Each decision made in the stories that Light Sail tells is based on presence, which they define as how you exist in space.

“If I’m watching a VR experience and suddenly I think I’m a person, but then I’m lifted up high into the air, and I’m like, ‘Well, who am I? What just happened?’ Suddenly I’m no longer present in that scene. Suddenly I’m thinking I’m in a game or an app. Suddenly I’m no longer actually in that world. I’m in a game, and it reminds me that I’m in a game, and for me, that takes you out of presence,” Celia said.

In their Paranormal Activity experience, the team was challenged with hiding cuts in the postproduction process. Taking a cue from Alfred Hitchcock’s Rope, they hid the cut in a place that was organic to the story. “We put a flickering light in and hid the cut in the flickering light. It still feels seamless. It still feels like you’ve never moved, but we have been able to give you that seamless experience and make that part of the story effective. There’s a ghost, so of course the ghost is going to mess with the electricity. It’s all story driven,” Celia explains.

Watts agrees. “It fit in the world. In the Paranormal Activity universe, the camera flashes out, so we can use that to hide the guts behind the technology.”

Light Sail continues to find innovative ways to use the language of cinema in storytelling. Discussing the use of close-ups to establish emotional intimacy in traditional filmmaking, the team discovered that the same trope in VR actually made audiences uncomfortable and disoriented. However, having the actor look directly into the camera, breaking the fourth wall, from a slightly greater distance, had a similar effect as the close-up. “There’s a piece we did for Go Pro where we do a lot of eye contact, and that makes you feel very connected in the scene.” Celia said. Light Sail ran into similar problems using voice-over in some projects, pointing out that audiences sometimes spent the first few moments of the piece searching for the source of the disembodied voice and often missed a key story beat.

Watts and Celia also have continued to experiment with the effectiveness of various run times in their VR films. “I don’t think the length ultimately matters. It depends on how much you connect with the characters onscreen and what their emotional stakes are and what they’re going through. We are exploring short-form series, because right now, we’re dependent on the headset technology. Someone might not want to wear it for more than 15 minutes at a time, but it doesn’t mean you can’t do a whole TV series worth of content. You can still build your emotional arcs over that series length. You just have to pepper it in at the beginning to get people hooked, just like a normal TV pilot would,” Watts said.

Celia added, “I also think that audiences have matured in the way that they view and consume content. We’re much more a serialized society than we were 20 years ago. Twenty years ago, it was all about the feature film. I’m not saying feature film is dead, far from it, but I’m saying right now, the golden age of television in my mind is really due to the fact that audiences are craving deep emotional characters that they can identify with and that they feel connected to. I think you’re not going to have those watershed moments in VR until we start seeing more stories where people can relate to the characters onscreen.”

Presence and relatability continue to be the two big issues the team discusses with every project, considering how the audience will connect with the lives and nuances of the characters on screen. When scripts are crafted, the process and format look similar to that in traditional scriptwriting, with several notable exceptions. Actors will always be on screen on VR, so giving them business and motivation throughout every moment of the scene has been a useful technique, similar to the approach used in theater. “In a film, you cut away to the shot of the friends in the café and leave your other characters, and you forget about them for a second. In VR, you need to make sure you deal with them, because the characters are always on the screen, so you must give them entrances and exits that are story motivated. It’s a really important tiny nuance,” Celia said.

Expanding their stable of writers beyond just traditional scriptwriters, Watts also suggested that comic book writers have been successful at writing for VR because of their abilities to describe environments and build worlds.

Combining strong writing with the talents of a strong director aptly describes the creative approach that Light Sail is pursuing. Celia explains, “It needs the strong point of view of an artist to guide the audience on a story. It needs a storyteller, and that audience can’t be the person. Something I’m very passionate about exploring, and I believe strongly, is that I don’t believe that you as the viewer can be the protagonist in the story. I think it’s impossible. I think you can be the antagonist, I think you can be an observer, but I don’t think you can be the protagonist because there’s nothing for you to latch on and identify and relate to.”

Celia and Watts are quick to point out that all that they believe they have learned thus far about VR could change as the medium evolves. Watts said, “Just in general, I think people are still trying to figure it out. I would say beware of people who say that you can or can’t do this, because someone just hasn’t figured out how to do it yet, so in every piece we do, we try to explore a new way of telling the story and making it effective.”

Storytelling in VR through POV

An Interview with
Tai Crosby, Founder/CEO, SilVR Thread

Tai Crosby is the founder and CEO of SilVR Thread, specializing in first-person POV Virtual Reality technology. With its patent-pending camera systems, SilVR Thread created Lionsgate’s Nerve VR Experiences, which allow the viewer to embody the stars’ characters in action scenes that transport them onto a skateboard tied to the back of a moving police car and onto a ladder suspended between two New York City buildings.

John Bucher:	Why is storytelling important to understand in VR?
Tai Crosby:	At the fundamental core of storytelling is how we transfer knowledge. It’s survival. It’s how we as a society stay whole. If you look at where we are as a human race, we have more knowledge than ever to transfer, and we have accessibility to more knowledge than ever, but how do we keep the focus on what is important? How do we get back to experiential knowledge? Experiential knowledge is different than just, “I read it in a textbook.” How do you build wisdom? At SilVR Thread, we’re able to essentially capture a memory and experience, package it up, and allow somebody to experience that again as if it’s truly occurring for them. It’s the ultimate visualization tool. It’s a very specific place in learning, and it certainly doesn’t eliminate or change the teacher. The teacher’s more important than ever. It’s how we experience things we normally wouldn’t encounter.
John Bucher:	Talk about how story fits with the POV approach that you’re taking to create VR. How does it change the experience for the viewer to be the protagonist, as opposed to putting on a headset and just observing a narrative playing out?
Tai Crosby:	We believe it’s huge. It is the very core of our company. This is what we were created to be. Our core design philosophy began years ago with my experiences as a fashion and travel photographer. I was traveling through the Himalayas taking pictures, told a friend, “Oh, this is so cool, the photo’s so great, but if only you were there with me.” The photo speaks a thousand words, but it’s not enough. You have no idea what it was actually like to have dinner with this group of people that had never encountered anybody else and were completely off the grid in the middle of the jungle in India. You had to be there and to actually have somebody hand you that meal.
	In this little village I visited in India, the women aren’t allowed to touch the food, so they slide it to you with their foot on the ground. That’s considered polite. These are those moments you just have to experience. It’s not just seeing it, it’s going. To embody somebody in the experience changes the way I believe we relate to these things, and it takes us away from being able to step out of it, and forces us to truly feel what it’s like to be human in that scenario. From day one, we never set out to build a 360 camera. We’re not a camera company. It was always about how we capture the human experience and replay that for someone.
John Bucher:	Tell me about the Nerve VR experience, which was originally a feature film—a traditional narrative experience. How did your team walk through that process? What was the philosophy behind taking a traditional narrative and creating a VR experience?
Tai Crosby:	I think a lot of this is very market driven right now. I think this is something that we’re all pioneering. Everyone in this space has seen this evolve daily. A lot of it has to do with forming a critical mass of consumers. We’ve seen inventions be adopted swifter and swifter. If you go back for how long it took for the original radio to catch hold, it was almost a decade plus. Even cell phones had a seven- to ten-year adoption rate. We’re seeing it move quicker and quicker. The good news is there’s more and more VR headsets in people’s hands, but having said that, the paid gigs, the actual revenue where people are writing checks still involve a lot of marketing material.
	It can be concentrated delivery. It ties directly into another asset, and at this point, it’s still promoting other assets to make more money for the film. Now, from the storytelling place, we love this, because it enables us to go out and produce this great stuff—to produce these stories. We’re in a place (Los Angeles) where we can bring a lot of these traditional film elements in and apply them to this new medium and advance the medium much faster than if we were somewhere else in the world.
	We did two pieces for Nerve originally, both about what’s it like to be in the body of this teenager. We flew to New York City. We were on the set of the film and had our team of seven and we go in, a little surgical strike force, and rig up the stunt members 12 stories up. They put a ladder between two windows, between two apartments, in an apartment building. We’ve got the stunt woman rigged up, and we were quite literally recording her experience. She had to cross the ladder.
John Bucher:	Just like the scene in the film.
Tai Crosby:	Just like the scene in the film. We captured that scene, produced that, did everything behind the scenes, Lionsgate loved it so much, they said, “Hey, this is amazing. Can you do another one on top of this for the same property before it comes out?” We did another piece where we explored what it was like to ride a skateboard behind a cop car. It’s a great piece, everyone has to try it. I recommend experiencing it in VR over doing it in real life.
John Bucher:	Good suggestion.
Tai Crosby:	You look at great storytelling in great movies. The best movies are the ones where you walk away and say, “I didn’t even realize it was in the theater. I was so immersed in the story.” This is just a new tool that does that.
John Bucher:	I watched a piece your team created where my body changed over the course of the experience in the VR world. I was a DJ one minute, and then I was someone else a few minutes later. I was really surprised at how that didn’t cause me any problems. I very easily transitioned. Actually, I enjoyed transitioning from this character to that character. Can you talk about the thought process behind that? I have to believe that didn’t come by accident.
Tai Crosby:	Trial and error. Lots of error. Lots of failure along the path of success. Everything we’re doing right now is pioneering, and everything that we as a community are doing is pioneering. One of the core things with our technology is we’re constantly pushing to have the most realistic perception of your own body in the actual world. All of our footage is in true 3D. This isn’t algorithmic, but it’s true stereoscopic, which is just how we see, and we’ve got some secret sauce that, as you look around, makes you realize it’s not just 360, but you’re looking with a natural field of view of a human.
	What it does is, by being in a body, make your brain feel comfortable, and that actually alleviates a lot of motion sickness. We can do things with SilVR Thread technology in a movement sense that other 360 cameras just can’t do because the body doesn’t feel like it’s right. The moment you’re in a body, the brain feels very safe. The ability to jump between different characters is a thing that we’ve been looking at for a long time. It feels right.
John Bucher:	It does.
Tai Crosby:	The second you look down, it’s like, “Oh my God, this is incredible. What is it like to actually be in this beautiful girl’s body in a bikini partying on the other side of this incredible party scene?”
John Bucher:	I saw the scene differently at that moment.
Tai Crosby:	Exactly right. That’s where you go, “Hang on, this changes the game.” Telling the story from a POV is exactly what we’re doing. By definition, that is what we’re doing. It expands your experience as a whole because you’re seeing it from multiple angles as multiple people. How did that person experience it? Tons of fun. There’s so many other things we’re working towards, such as being able to navigate dynamically through things. It’s not just to choose your angle, it’s truly choosing whose perspective you want to be in as you’re experiencing this.
	One of my favorite classes back in school was the study of the gaze. Are you the watcher or the one being watched?
John Bucher:	Laura Mulvey’s theory?
Tai Crosby:	Exactly. From the storytelling perspective, how do you determine the behavior of the viewer? How do you influence the behavior of the viewer? Where are you empowering the actors and the other characters? This is now embodying that in a completely different way, because you’re embodied. You are there.
John Bucher:	Let’s move to the future. This technology is moving really quickly. VR experiences are headed towards a mainstream saturation like we’ve never experienced before. I’m not asking you to guess what the future holds, but what ideally would SilVR Thread like to move this VR world towards? Where are you guys driving the ship to go?
Tai Crosby:	It’s a wonderful question. Our core has always been POV VR and always will be POV VR. We’re constantly not only allowing that to be possible but trying to increase the fidelity of the experience and at the same time make it easier to produce so that more content can get out there, more experiences. Beyond the business, there are so many humanitarian tales that need to be told. We live in an age that’s more polarizing than anything, I think. We have a project we’re working on with a former world leader. He was president of a major country and he’s devoted his entire life now to kids. He set up a foundation, and he teaches a couple hundred thousand kids about leadership. They call VR the empathy machine. Well, if VR is the empathy machine, SilVR Thread is the way to inject it into that personal experience through POV VR. I think it’s remarkably powerful, and I can’t wait to see what lots of great artists and thinkers and activists do with this technology.

Storytelling in VR through Game Design

An Interview with
Adam Orth, Creative Director, Three One Zero

Adam Orth is a former creative director at Microsoft. He now heads up Three One Zero, an independent video game and digital entertainment studio. Their first title, ADR1FT, has been nominated for numerous awards and lauded as one of the top narrative experiences in VR.

John Bucher:	Tell me about your initial pull into this sort of storytelling.
Adam Orth:	Video games were actually my second career. My first career was a musician here in Los Angeles. I was a songwriter telling stories. I grew up playing video games and I was always a gamer, but when I began to focus on games, professionally, I got a job working on a James Bond game for the PlayStation One. The game industry was like the wild west at that time. I liked hanging around the studio and started being a game tester and realized very quickly that I could learn about and touch all the different disciplines of game development. I wanted to be a game designer because I love to tell stories, and this is a way for me to create interactive stories.
	I ended up getting a job at Sony and worked on God of War. Then I went to EA and worked on the Medal of Honor series. Then I got an opportunity to go and be the external creative director of Lucas Arts. I really wanted to go make good Star Wars games. I’d always been drawn to telling a really good, quality, emotional story built around good characters. I ended up working at all of these big places where there’s huge teams working on giant products, and it was really hard for me to fit into those environments, because I really wanted something small and personal, but wrapped in that kind of super glossy triple-A production packaging.
	After I left Microsoft, I had this space idea, and I realized, I can make this cool small space game and talk about some of the difficult things I’ve been through. I approached the storytelling like songwriting. I had something to say that was really personal and very meaningful to me, but I buried it behind some very thin metaphors, I think. When I was a songwriter, I always wanted songs that had meaning but were also accessible to a wide audience. That’s kind of how I approached the story of ADR1FT. I wanted to talk about normal life things that people experience. My goal was hoping that people would find some kind of common parallel there. Really, at the biggest level, my only goal was just to strike a chord in someone somewhere. ADR1FT is not a big multimeaning, circle of life kind of story. I just wanted you to put the controller down when you were done and go, yeah, I can relate to that. That’s how I approached it.
	I started thinking, there is a traditional way I can tell the story, and I embraced that, but there’s these side stories that are a bit nontraditional, and they’re all connected. The first thing I thought when I sat down was, most games tell a story—A to Z. You get to the points along the way, and there’s this arc of storytelling in video games that’s very stiff and requires a lot of hand holding. It’s very overt and very neat and tidy. I just didn’t want to do that. The idea of ADR1FT is really about minimalism. We’re a small team. We have limited time, limited budget, and we can only do so much.
John Bucher:	The chord that resonated in me while playing is that ADR1FT is inherently about loneliness.
Adam Orth:	Yeah, I wanted you to feel that, not only were you in a bad situation, but that virtually by hearing all these other stories that you’re a part of from people who are now deceased—it’s a very lonely feeling, because you find out, you matter to these people, or you didn’t matter to them. It’s almost like reading someone’s diary in a way. You feel weird doing it, and it makes you feel all these emotions.
	Anyways, I didn’t want to do the story traditionally. So I thought, if you were actually in that scenario, you wouldn’t possibly get a perfectly linear story. It would be messy, and you might only get 10% of it just based on the fact that you’re not going to stick around and smell the roses. Instead of telling this really fleshed out narrative, I approached the narrative bits as seeds. I just wanted you to get this little chunk and then to start growing whatever you think it is until you get to the next one or the next one. I think it was successful in that way. That’s one of the things in the game that I’m really proud of. It does have that feeling of this hollowness, because you’re learning that this is your fault in a way. Everyone knows what it feels like to mess up. Everyone knows what it feels like when you’re left out of the gang or just not included in something. I wanted you to have that feeling, and I feel like it’s there. Although I’m a little desensitized to it, just from playing it and making it, but many people have told me that that’s what they got out of it.
John Bucher:	Can you talk a little bit about your approach to characters?
Adam Orth:	Yes, so first of all, every bit of narrative in ADR1FT comes directly from my life. Every single thing. Some things might be twisted around to fit the fiction. But I talk about addiction. I talk about cancer, relationships, parents, all of those things. Everything, those stories all happened in some form to me. That’s why I think those characters feel authentic.
	When I decided to make the game, I also said, “I’m going to make it totally raw. I’m going to just put it all out there.” I feel like those characters are interesting. I hope that they are more interesting because they’re not totally fleshed out in the experience. They are fleshed out for me as the creator though. I wrote all of those characters’ life stories and then just cherry-picked some stuff to include in the game.
John Bucher:	I am really fascinated by the idea that you took people and situations and elements from your life and worked them into the story for this game. I think a lot of creators desire to do that, and I think the toughest thing to figure out is, when do I stay true to what happened in my real life, and then when do I need to not do that?
Adam Orth:	That’s a good question. It’s not something I struggled with, but I recognized it when it was happening. My thought process was, stay true to the character. Because you’re already putting them in this fantastic goal situation, and because the more true it is, the more real it is. That was my mantra when I was creating the game, just be true to the characters and be true to what you are trying to say. There is nothing wrong with mindless entertainment, but I just feel like if I have the opportunity and the audience, why not say something true?
John Bucher:	Can you speak about the relationship between external conflict and internal conflict? One of the things that I think’s really great about the game is there is this level of internal conflict going on that connects to the external conflict.
Adam Orth:	A lot of that has to do with the environmental storytelling. You have these mechanics that are working against you, and you have this beautiful world that you want to explore, but it’s dangerous to do that and it’s risky and you have to make bets. You know you’re in a bad situation and have to get out of it, but then there’s also the wonder of the entirety of humanity right there. The setting being in Earth’s orbit in that way, and being alone and having this totally beautiful but mangled environment that is also your enemy and your friend at the same time. There is some nice juxtaposition in those things.
John Bucher:	Why does story matter when creating games?
Adam Orth:	Well, because you’re asking people to buy into a fantasy. You can do that with just the environment, and you can do that with mechanics, but it’s not really a full experience until you have story. There are games and experiences where it’s just about walking around the environment and checking stuff out, and that’s awesome. I love those things. But do you want them to create their own story or do you want to tell them a story?
	I kind of like to try to be in the middle, because the agency that you have when you are doing something interactive can be important, and that does tell a story. I just think that, if I’m going to ask you to go fight a dragon, you want to know why, and I want to know why.
	When you play a game and there’s always twenty-five different side quests, those characters matter too. It’s not just that the goblin is terrorizing the village. Go a little deeper. That goblin used to be my brother and now he’s a goblin, or whatever. It’s important to me, and I think video games often get characters that are shallow. I don’t want to see that continue in VR. Now, I think there is a risk of giving so much agency in a VR experience [because] you are turning loose of the reins of really directing the experience. It can be the beauty of the experience too. Turning loose of those reins.

Insights from the Technologists and Producers

The most effective technologists and producers understand the elements and methods used in storytelling. They understand that the technology is a means to an end and that in its highest form, it is invisible to the end user. Jonathan Krusell mentions that there was a time in the past when audiences were more aware of the fantasy elements in well-told stories. We knew there was a screen that separated us from this other world. Technology has now allowed us to cause the audience to feel as though that screen has disappeared, fully immersing us in the worlds of our fantasies. Robyn Tong Gray agrees with pushing the audience into the new medium as far as possible and suggests that audiences be provided something they are familiar with on some level. That something might be familiar characters, genres that follow rules audiences have come to expect, or even environments that they have positive associations with. Successful VR experiences tend to embrace a mix of familiar elements and those completely new and foreign to the audience.

Robert Watts suggested that the key to this mix is communicating to the audience who they are in a scene and why they are watching. His partner, Matt Celia, believes that it is exactly the moment that the audience begins to question either of these concepts that the immersion is broken and the experience begins to fail. Krusell, Gray, and her partner Andrew Goldstein all agreed that the environment crafted by the creators of an experience plays a major role in answering these questions for the audience. How realistic or nonrealistic an environment appears to be provides different sets of rules for the world the user is immersed in. Tai Crosby and the creators at SilVR Thread approach VR exclusively from the first-person lens, allowing them, in a sense, to capture a memory, move through an environment, and then allow a user to experience it. Other creators throughout this book mention connections between memory and VR storytelling—a topic that deserves further exploration and consideration.

Krusell discussed the challenges with creating VR both now and in the future. These challenges range from the discomfort a viewer may feel when being forced to embody a gender, body type, or experience they are unfamiliar with to the ever-changing technological nuances of the coding world. In many ways, technical producers of VR stories, games, and experiences are put in the position of bridging the worlds of artistic storytelling and scientific practicalities, all while managing to create an emotional experience. Adam Orth stated that throughout the development of his project, his biggest goal was to strike a chord with someone somewhere and process his own loneliness. While not always possible, when creators do have the opportunity to create or work on projects that resonate with them personally, either emotionally or biographically, there does seem to be a level of energy in the final product that can’t be manufactured otherwise. All the technologists and producers in this section and throughout this book are united by the motivation and drive to share experiences with an audience. Tai Crosby spoke specifically about his experience of being in the Himalayas and wanting to bring his friends along so that they might all be able to see what he saw. When producers are able to work from a place of clear motivation, such as this, it gives the creator a metaphoric North Star for their future work.

The Artists

Storytelling in VR through Fashion

An Interview with
Angela Haddad, VR Artist and Producer

Angela Haddad is a VR artist and producer who got her start in the industry by producing story-like animated 360 videos out of hand-painted art under the handle One Third Blue. The studio creates original VR art productions for fashion brands and magazines, including Marie Claire. A VR Producer at SilVR Thread, Angela creates first-person POV stereoscopic VR experiences that inclusively capture the human body for moments that replicate the in-body human experience in its truest form. At SilVR Thread, Angela has coproduced VR experiences for Lionsgate’s movie Nerve, where three stunts from the film were recreated to be experienced by fans in POV VR, allowing for transformative, total human body presence. Angela has judged and spoken at 2016 LA Hacks, has been featured at SXSW, in WGSN, Mettle’s Blog, and is an active member of the SH//FT and Women in VR communities.

John Bucher:	Tell me about some of your early experiences with art and technology.
Angela Haddad:	I grew up in a very tech-oriented family. My dad was a tech entrepreneur his entire life. I reached college, and I felt like I needed a bit of a change and so I studied political economies and focused on the Middle East. I ended up with a concentration on the underdogs of Lebanon. I grew up drawing, and when I graduated college and started working in tech, I realized there was something missing. My dad always told me you need to get a hobby that keeps you creative, a hobby that makes you money, and maybe a hobby that keeps you fit. I really felt like I was missing that hobby that kept me creative.
	I picked up my brush, my pencils, and my calligraphy inks again. One day I was looking through Tumblr and found this fashion illustration that I wanted to print and frame on my wall, but I couldn’t because it was too low resolution, so I decided I was just going to draw it myself. From there I got hooked on fashion illustrations.
John Bucher:	Fashion is, in many ways, another form of storytelling. Can you talk a little bit about some of your underlying observations about the psychology of fashion and story?
Angela Haddad:	When we wear something specific, we feel differently about it, about ourselves, compared to when we’re wearing something else. Ultimately the fashion industry is not just selling you clothes, they’re selling you dreams and they’re selling you a story. The thing is, a piece of clothing is very rarely a piece of clothing by itself; it usually comes in a collection, and that creative collection is inspired by a creative story. When you look at, for example, Valentino Fall 2014, you really can see the collection is inspired by Greek mythology. Everything really does have to come with a story. Very rarely do you see a collection come down the runway without an immersive environment around it that can really put everything together. You’re really immersed in this greater environment. You’re immersed in a story, and the clothes are a part of it, a big part of it, but really, ultimately, just a part of it.
John Bucher:	You’ve got a very clear voice in your VR art. It’s very distinct and it’s a very feminine voice. It’s someone who’s speaking from a woman’s point of view. Can you talk a little bit about the voice of women in VR, art, and story?
Angela Haddad:	Looking back at the history of fashion art, it began as basically a way to illustrate Vogue covers. The earliest Vogue covers were not photographs. They were actually fashion illustrations. The earliest fashion illustrators were all men. Now, if you look at the top fashion illustrators, for example—Megan Hess, Megan Morrison, Katie Rogers—these are all women and they’re doing fantastic, fabulous work. Very few top fashion illustrators are men. I think that it’s always critical to keep women’s voices in mind, just as it is critical to keep it in mind with the tech industry. Right now, with the VR bubble, I think we’re all doing a fantastic job about keeping women involved and making sure that women have more and more of an equal opportunity. The Women in VR group is really strong. Women lend a different voice.
John Bucher:	Tell me about your initial experience with VR. How did that come into your world? What’s been your journey with it?
Angela Haddad:	My first experience with VR was with an Oculus DK2. My brother brought it home and said, “You have to try this.” I put it on, and my first demo was one of the default demos, and it was simply this 3D house that you could walk into. I remember that I had heels on and I had to take off my heels because I felt so off balance. The level of immersion was just insane. After that, the first thing I could think of was what if we could replay our memories in VR? What if we could, for example, record our wedding days in VR and our really special moments in VR and replay them for later on? Because of that, I set out to learn how to stitch 360 video. I took a class and I kept practicing, and then I released my first 360 video, which was monoscopic at the time. My first video was of me illustrating art around a 360-degree space. I put five easels around the 360-camera rig and I went around the rig and I drew a fashion illustration on each easel.
	Once I released that, I started thinking about what else I could do. I started thinking of ways of taking that to the next level. I played around for two months with various programs and software to figure out how I could incorporate my art into 360 videos. I didn’t want to just animate, I really wanted to animate in 360. The more and more I played, the more I figured out how to more easily build those environments, and eventually there came a point where I thought, okay, well, these animations have to tell a story now, and so now each of my videos is truly a cohesive story. They’re not just characters running around being animated in a software program.
John Bucher:	Can you talk a little about some of the things that you learned along the way?
Angela Haddad:	To me, the most critical thing in my 360 videos was the trigger points. What is going to trigger you to look around and move your head around 180 degrees to look at that other thing that I want to show you? In the beginning, that was really difficult for me to figure out. It didn’t come easily. For example, with my first 360 video, it was a live action, and I was going around the camera and drawing, and I wanted people to easily understand that I was going to stop at a certain easel, then I’m going to move to the left. I literally put arrows around to help guide the viewer. That to me was a good first try, but it wasn’t interesting enough. When I moved into creating art in 360, I really wanted to have something that wasn’t an arrow. Whenever I wanted you to look around, I wanted to make sure that it was part of the story. For example, with one of them, I have lipstick flying around. It’s flying around the walls, and the lipstick’s job is to paint each of the illustrations on the walls different colors. That was part of the story.
	With a recent piece for Marie Claire magazine, they wanted me to illustrate the three summer cover girls—Selena Gomez, Blake Lively, and Amy Schumer. What I did was draw each of these girls on their own set, within the 360 video, and the one cohesive point that made you look around the 360 was this table that moved around from set to set. It was the same table, but in each set, there was a different object laid on it. In the first set, I think it was flowers, in the second set, it was a different object and it was the same table, and that was the trigger point that made you look all around. This is applicable, I would say, to almost any 360 experience. With live-action experiences, for example, the most common trigger point would be like an audio cue. That is where 3D audio comes in.
John Bucher:	How do you come up with a trigger point like the table? Do you just brainstorm or do you talk to people or do you write things out? Talk a little bit about your process.
Angela Haddad:	Usually the greater story comes first. For example, with the Marie Claire piece, the general theme that they gave me was the three summer cover girls. I played around with different ideas, and once I set out to build three sets within those 360-degree spaces, that’s when the question starts coming in. It’s usually about, I would say, a quarter of the way through the storyboard. It goes hand in hand with what the animation process is like. What is the story on each set? Where does the animation start? Once I figured out that this is what Selena Gomez needs to do, this is what Blake needs to do, this is what Amy needs to do, then the table idea came through. I would say it’s usually a quarter of the way through the storyboard for almost every project. It is a really critical point, so it does have to come towards the beginning. It also goes hand in hand with how much a viewer is expected to look around. Do you want them to just look around 180 degrees? Or do you really want to take advantage of the full 360? Yes, it’s really all about the story.
John Bucher:	Let’s bring AR into the conversation. Some people are being introduced to VR initially through AR experiences.
Angela Haddad:	I think AR has a bit of a longer way to go than VR right now. I see a lot of uses for AR in fashion, especially in retail stores. For example, being able to see what the clothes look like live on you in AR before you purchase them without having to go into a fitting room.
John Bucher:	Let’s talk a little bit about the future. There’s plenty of talk in the VR space that eventually we’ll probably get away from HMDs and have a contact or a chip or whatever. What do you think are the advantages and disadvantages of creating a completely immersive experience for people? Is this going to cause problems? Are we going to have people that never want to leave their immersive environment? Can you talk about how you see the future of VR?
Angela Haddad:	Regardless of what I think, I think our society is ready for that kind of thing. I think when VR tried to break its way through a decade ago, our society was just not ready for that yet because we were not stuck to our smartphones twenty-four-seven. We wake up and for most of us, it’s the first thing that we do. As a society, we’re ready for technology where we’re completely immersed in it and completely lost away from our actual reality. I think that is inevitable.
John Bucher:	Why is story important when we’re working with technology?
Angela Haddad:	I think that’s the only way you can relate to people. I think it’s the only way you can resonate with somebody. If you don’t have a story, then what are you sharing? If you’re just sharing sound bites, for example, that’s not going to stick with them. That’s not going to be a memorable feeling. I think that if you want to create a true memory, you have to tell a story. Especially when we’re talking about something like Virtual Reality. If you throw somebody into a virtual experience and maybe there’s a lot of really cool effects and maybe there’s a lot of really visually intriguing things going on all around them, it’s very difficult for them to remember what they watched in a year or two years or five years or even ten years if there isn’t a cohesive story.

Storytelling through Immersion

An Interview with
Annie Lesser, Immersive Theater Director

• Getting to Know You
• (A)partment 8
• (B)arbershop

Annie Lesser is a writer, director, poet, and photographer. She has been recognized as one of the top creators of immersive theatrical experiences in Los Angeles. Her work has been recognized by LA Weekly, the Hollywood Fringe Festival, and the National Foundation for the Advancement of Arts. Several of her immersive theatrical projects are in development as VR experiences.

John Bucher:	Give us a little background on your creative development.
Annie Lesser:	From a young age, I was always making skits and putting on shows for my parents. I was very much the kid that played make-believe. I wrote a lot of poetry. I did anything I could do that was creative. When I was nine years old, I said “I’m going to be a big screenwriter one day.” You know how when you see an eyelash, you blow it and you make a wish? I always wished I was the greatest screenwriter in the world. It’s such a force of habit that I didn’t even think about it. When I went into high school, I started to get a lot more into writing theater. I wrote student plays that were in some theater festivals. I did a lot of slam poetry. I was a finalist in the Chicago Louder Than a Bomb poetry slam competition with my team from school and won some other poetry awards. I was a National Merit Poet from the National Foundation for the Advancement of Arts and an Illinois Merit Poet. I did one interactive class in college, where we, as a group at the end of the semester, put together a walk-through adaptation of Raymond Chandler’s The Big Sleep. I graduated NYU with an Excellence in Television Writing Award and had two of my plays in festivals. So, I was pretty busy. But when I came to LA, I was having a lot of trouble breaking into the industry, so I became a freelance photographer. It gave me a lot more freedom and flexibility to write and produce work on my own. I’d wanted to do some interactive theater for a while, so I began with a piece called Getting to Know You. It involved eight audience members and eight actors.
John Bucher:	Had you seen anyone else do interactive theater before?
Annie Lesser:	I always loved cheesy dinner theater growing up. My bat mitzvah was a murder mystery theater bat mitzvah. I have always been really supportive of Mick Napier. He always told me really nice things. Mick Napier is the founder of the Noise Beater. He’s an amazing improv, sketch comedy guy. He directed Highway 53, with Amy Sedaris, Stephen Colbert, and Paul Dinello. I saw Hotel Savoy at the Goethe Institute in New York. I loved just getting in there, and the freedom of relating to the actors, and trying to talk to them without there being the consequence of the rest of the real world. I wanted to create something like that, where someone feels something like I felt.
	I wanted to create something where someone asks “Was that person really hitting on me?” Or “Was that person really upset by what I said?” I wanted to create those moments. Obviously, I’ve seen Sleep No More. I’ve seen Then She Fell. I’d seen a couple things around LA. I’d been to two different versions of Delusion. I was at this point where I was like, “I want to create something, where someone feels that feeling that I had when I went to see the Hotel Savoy.” Because it was such an amazing feeling. I want to create that in people. I also really wanted to create things where people are introspective about themselves.
John Bucher:	With (A)partment 8, you moved out of the theater into an actual apartment. Can you talk about thinking through the idea of bringing people into an immersive space? How was that different than approaching something that was just in a standard theater?
Annie:	When I came up for the idea of (A)partment 8, I just wanted to create something in my apartment, since it was a location I already had access to. It gave me a lot more freedom in what I was able to do and when I could rehearse and when I could work with my actor. I have a vision in my head about what would evoke emotional response for me. I also go through the experience the audience will have first. I start acting like different worst-case-scenario audience members and then see how my actor reacts. I always worry that what’s going on in my head and what we’re actually doing aren’t the same thing, so I just have to hope and pray that people relate to it. Because people could say, “That didn’t make sense,” or “That was really pretentious.”
	I began with what would make me feel something. I started with a nugget. I started with the nugget of a dead actor in the bathtub, covered in blood. Then I write around that. I go into the space, because I write for the space, and add elements from that. Right now I’m dealing with the fact that I don’t trust a lot of people. Let’s make this a piece about trust, and how other people control our lives the way that God does. Let’s do that. I really have a lot of faith in Carl Jung and the collective unconscious and that we all have these similar emotional reactions to certain stimuli. I want to try and tap into that.
John Bucher:	One of the big buzzwords in immersive experiences with Virtual Reality is this idea of agency being given to the audience. So, with your experiences, the audience has different types of agency, depending on the experience. With (B)arbershop, it felt like the audience had the most agency. In (A)partment 8, the agency is very different.
Annie Lesser:	It’s internal for them. People react different ways. I basically try, with all my pieces, to create an internal agency more than an external agency. It’s placing them in a situation where you are creating this bond of trust. We are trusting you with our artwork, and you are trusting us that we’re going to keep you safe. I also feel if I’m good enough at my job, I create an emotional intimacy and trust with you and the actor. You should feel like this is something that is real, and you should have some sort of emotional relationship to the actors around you. I feel like 95% of the audience does what they’re supposed to do.
John Bucher:	Many audience members are used to watching a performance and not having agency. How do you guide people through the introduction of that power without forcing their hand?
Annie Lesser:	It’s always hard when someone doesn’t know what’s going on. Or even when someone does know what’s going on, but they want to take charge in a way that the piece wasn’t meant to support. I try to set a tone with actors asking the audience questions right away. They (the actor) might have a monologue, but within a minute or so, they’ll start asking you questions. They direct it at you, and they guide you to follow them, or things like that. For (A)partment 8, there’s a little bit of an intro where we say it’s okay to feel your own feelings. It’s okay to feel the feelings of the person whose skin you’re inside. It’s okay to draw on your own experiences. I feel like I try to do things that engage you right away.
John Bucher:	In Virtual Reality, one of the things that people are trying to wrap their head around concerns the camera. Previously, we’ve been able to use cinematic language like close-ups that force audience perspective. Being a photographer, as well as someone who works in immersive space, can you talk about the relationship between the two?
Annie Lesser:	Well, the forced perspective comes from the actors and comes from the language. I’m not forcing you to pay attention to a certain visual, but I’m forcing you to pay attention to a certain concept. It’s more like making sure that there are aspects of that littered throughout. In (B)arbershop, in the background, we had The Bachelor’s rose ceremony playing, and Bachelor Pad, Bachelor in Paradise, which you wouldn’t normally even pay attention to, but they are there. It plays to the theme of what’s going on. We choose where you’re sitting, and we place that. We create the whole tone of the space. You’re controlling the space. You’re trying to make everything fit with those themes you want to touch on. In the same way, when you’re reading a novel, you have all these little things that are symbolism and subtext, and you try and create an entire space that has that symbolism and has that subtext and has these ideas.
	You’re just trying to create an entire space where the actor is engaging enough that they’re holding your attention. There’s suddenly a knock at the door, and you could choose to pay attention to the new audience member entering the space or choose to pay attention to the actor, but you’re paying attention to that dynamic. With (A)partment 8, you could be looking anywhere. You could avert your gaze from Keight (the actress in the piece), not wanting to look at her, but you have her presence in the words that she’s saying to you. You have the image of when you first open your eyes, you’re like, “I get it, there’s a dead body here. This is a bathroom.” You have that. You have the intro of the audio beforehand, priming you for this experience. You hear something, and you go into it. You’re asked to meditate a little bit before you take off your blindfold and headphones in that piece. That prepares the audience. You’ve internalized yourself. You’re now in this space the way that you want to interact with it, but I have placed all these things that, hopefully combined, will elicit a certain emotional reaction. I think that’s what’s also interesting about the ABC project is that I’m looking for these spaces that lend themselves to that. I’m writing a piece of theater for this space that I think is the best way I can utilize that space to create and elicit an emotional reaction by adding things to it. But still, everything in this space, I try and use as much as possible.

Case Study: Jaunt Studios and the Invisible VR Series

Doug Liman, Director, The Bourne Identity, Edge of Tomorrow, Swingers
Melissa Wallack, Writer, Dallas Buyers Club, Mirror Mirror
Grant Anderson, Head of Studio
Tom Vance, Head of Content
George Kliavkoff, CEO and President

Jaunt VR was created in 2013 after one of the founders returned from a trip to Zion National Park and wished he could return for a brief jaunt at any time from any place. Focusing on realistic cinematic content, the company has produced the highest-scale, big-budget VR experiences available. Partnering on content from creators ranging from Paul McCartney to Disney, Jaunt focuses on developing stories that match the production quality of network television and film studios. Head of studio Grant Anderson explains the company’s model as having four main buckets: narrative, sports, music, and travel/adventure. “We’ve experimented with a broad range of content from the very beginning with different types of experiences, but narrative is what people want to see. Whether that’s in sports, or it’s in music, or it’s in travel. Since we are in the very beginning of VR, there’s a lot of experiential stuff, which was interesting, because it was the first time we’ve seen it in VR. But now, even in those genres, we need to have story. That’s what really keeps things interesting. We really wanted to expand on the type of narrative work that we’re doing and really bring even stronger narrative to VR,” he said.

Jaunt is betting that success on a large scale in the VR market will be in telling stories that combine elements of cinema, gaming, and interactive theater, as well as elements specific to VR that are still being discovered. “It’s about bringing all those parts together that’s really going to create this amazing experience. Video games have been trying to do this for 20 years. How do you tell a satisfying narrative where the viewer still has some sense of control? It’s a really interesting problem, and it’s what drew me to this, because I was a video gamer. I worked in 3D. I worked in visual effects. I worked in production, film. This is a combination of all of those, so it’s a really exciting challenge for how we tell these amazing stories that you can feel in your gut. This will be the way of the future in how we experience stories. It’s not quite what we’ve had in just linear narrative,” Anderson said.

Recognizing the value of what the tradition of cinema can bring to VR, Jaunt has been especially interested in tight narratives, the use of three-act structure, strong character development, and rich worlds—all of which have been staples of filmic storytelling. They have searched for writers who are comfortable with using these elements in nonlinear story environments while crafting their content.

One of the storytellers Jaunt has partnered with is Doug Liman. Liman has directed big box-office fair ranging from the indie cult classic Swingers to action tentpoles like The Bourne Identity and Edge of Tomorrow. His first project with Jaunt is a five-part VR series called Invisible. Liman brought in Melissa Wallack, nominated for an Oscar for Dallas Buyers Club, to write the project.

Tom Vance, head of content, explained the working relationship between Jaunt and Liman as a true creative partnership. “What makes Doug so remarkable and why he ended up being such a phenomenal partner is his willingness to take on a challenge, whether in The Borne Identity, completely upending and reinventing the editorial style of feature film, to Edge of Tomorrow, which is something studios just aren’t making—ambitious standalone original sci-fi movies. Doug brought all of those things to the table for us,” he said.

George Kliavkoff, CEO and president of Jaunt, agrees. “Doug’s unique process, in terms of shooting and editing in real time, was incredible. The whole process was a living document until we found the story. We knew what the story was going into production, we had a script, and that story is the story that ends up in the show, but in an effort to actually execute the story, we put the show together and broke it apart a number of times before we actually felt like we were communicating what was on the page when you’re in a headset. That process itself was amazing.”

Jaunt became interested in Liman when they heard he had been experimenting with VR on his own. The team knew they wanted to work in the genres that he excelled in. Vance is excited about working in genres with wide international appeal. “I think 45% of our total audience is global, so we’re not thinking about developing content for a territory-specific audience. Sure, there’re going to be pieces that are catered to some sort of geolocation, but with our high-concept and tentpole content, we want to make broadly appealing commercial concepts. We are asking, who are the internationally relevant actors? Who’s the biggest star in China that is forward thinking and wants to be part of a big kind of tentpole VR series? How can we bring those people together and create what is not just a show for North America and Europe but for the global audience? That’s something that we’re contemplating today, because the audience already is global,” he said.

Jaunt is looking ahead to the future, incorporating social VR narrative strategies on upcoming projects. “This bridges the cinematic VR space and the gaming VR space. Being in a film together with your friends will be a way that stories are told in the future. We plan to be on the cutting edge of that,” Anderson said.

Storytelling in VR through the Artist’s Lens

An Interview with
Mark Cordell Holmes, Art Director, Pixar

Mark Cordell Holmes has worked for 25 years as an art director and visual storyteller. He creates original narrative, interactive, and VR experiences. A significant amount of his work has been at Pixar, where his credits included A Bug’s Life, Monsters Inc., The Incredibles, Ratatouille, WALL-E, Toy Story 2, Toy Story 3, Cars 2, and Brave.

John Bucher:	Can you talk about your philosophy of VR storytelling?
Mark Cordell Holmes:	The interesting thing about VR is it’s a merger of different storytelling disciplines. It’s a new way of looking at this stuff, and so there’s going to be a limited value any one group can bring to it. There has to be almost this whole new art form or storytelling form that’s going to be a combination hybrid of these things in some sense and a still undefined combination.
John Bucher:	That’s exactly what I would love this book to be—the marrying of all these disciplines to figure out what storytelling in this discipline looks like.
Mark Cordell Holmes:	I can share with you how I learned to apply design to conventional visual storytelling through emotional storytelling both in a narrative context and interactive context, although I have less experience at that. VR definitely seems like a place where it’s definitely the merger of narratives and interactive experiences, immersive experiences. I definitely see people approaching it from either the narrative standpoint or the interactive standpoint and each falling short. I’m fascinated by that gap in between those. I think that’s where the sweet spot will be.
John Bucher:	Tell me about your background and what has brought you to the point you are at now with story.
Mark Cordell Holmes:	I’ve had some time to think about it because mine has largely been a career of circumstance in high school and college. I loved reading. I was a big reader. I loved science fiction and fantasy. I was a big movie fanatic. I was already loving stories. Probably around sixth grade, I stumbled into role-playing games with some friends. The thing that I fell in love with, though, was the dynamic storytelling and I just love the idea of creating worlds and characters and possibilities. It was like a vehicle for me to explore stories. I think that was more interesting to me than maybe the game mechanics themselves.
	I was also, was a voracious comic book reader. I loved early anime stuff. I just was always consuming story content in one form or another. I wanted to be a comic book artist in high school. I would even take my little portfolio with me to comic conventions, and I would always chicken out and not show anyone anything. It was a fantasy that I could be doing that, but I didn’t really have any idea how it could be possible to do that for a living. It was almost just always like a side project. I didn’t know what I wanted to be when I grew up. My mom was practical. She wanted me to maybe be in accounting. Once I got into college and I looked at course catalogues, it blew me away that you could actually study these things, let alone get a job in it.
	I was not a great student. I didn’t appreciate school, but my first week, I got a job as a paid graphic artist. I had to learn how to use a Mac and from there, it was pretty much my first job. I was already accruing skills that made me competitive. I ended up dropping out of school because I decided my job would be my education. My career would be my education. I found I was far more motivated if I was at the risk of being fired than I was of getting an F or a low grade. I got swept up into Silicon Valley. There were not many people that were trained on the computers or on the software, and I fortunately had been exposed to enough that not only did I have the drawing skills, but I had enough software knowledge to be competitive and get work. Then it just bounced around. I was always trying to find work that involved creating characters or worlds or a fantastic context or something was more interesting to me. I wasn’t really a big gamer per sé, but again, it was almost like those early role-playing game days. I was immersing myself and visualizing these fantasies.
	Again, I was just lucky enough that there was always opportunity. Right before Toy Story came out, I ended up getting a job at Pixar and working with their commercial group. I didn’t even know that they were making an animated feature. I just thought they were making commercials. I was working in the back office with a handful of people. I didn’t realize there was a warehouse filled with animators and technical directors. Then shortly after the movie came out, I was pulled into the feature group. The Pixar opportunity was far more fulfilling for me because I really love story. In college, I’d wanted to be either a filmmaker or a screenwriter.
	Pixar was definitely a step closer to what I wanted to be doing. I really wanted to be a story artist. I really wanted to be helping with the stories. I took some story classes while I was there, and I had a screenwriting mentor for years. I was a better artist and writer and was able to learn to contribute to the stories in a different way in a visual context, especially as I went from being a production artist, where my focus is on the nuts and bolts. How do you make something work? How do you design a character’s environment so that they can fit through the door and so that it could accommodate the action and the storyboards and all that stuff?
	To then get to art direct and production design where you’re responsible for not just the look of the film to support the story. That for me was really magical, and it began to give me a real fulfillment, because I felt like I was contributing to story in a very meaningful way. Then after a while, I’d been there long enough where I wasn’t growing in the ways I wanted to grow and then helped start up a game company, where I was able to take my learnings from Pixar and try to apply them to an interactive experience with a computer game. I’d been out of games for 16 years, and so it was like starting over but seeing it through a new light. I couldn’t map the learning between the techniques in a direct one-to-one way, but there were definitely places where it could apply. There were places where it didn’t apply where it challenged me to think. In a film, we would use lots of cameras on certain things. Here, we don’t have that control. It’s up to the player to explore. How do you create either the probability for things or communicate things where you are not reliant on composition?
	That was definitely a great learning curve, and I feel like I was just scratching the surface. Unfortunately, after we put out a game, our publisher had a strategic shift, and they ended up dropping the developers, including us. As a startup, we didn’t have any other financial backing, so we had to dissolve it. I was very excited at that point. I really wanted to get into VR just because it seemed like the perfect laboratory environment for how do we bring all of these assumptions we have and techniques and learnings we have of storytelling and these different disciplines, and how do we bring them together in a new combination? A new way to create a totally unique immersive experience that’s still fulfilling but fulfilling in a whole new way.
John Bucher:	Where does game theory apply and where does narrative theory overlap, and how do we tease those apart?
Mark Cordell Holmes:	In terms of immersive entertainment, there’s a couple of perspectives to which I’m seeing people approach it. Google is trying to repurpose the traditional narrative experience into their medium, and there is a whole host of challenges that come with it, of course. Each of the films they’ve done presents a different approach and different learnings and different failures, and they are all wonderful.
	Some others rely on novelty, but after a couple of viewings you go, okay, there’s nothing very substantial to hold you in or to make you want to go back to it. Other people are doing the games approach, so they are doing a first-person shooter. Where it seems like the obvious way you would use medium but to me it’s not pushing the medium, it’s not taking advantage of it in a way that’s different from how you experience forward. It’s almost just like watching a movie with stereoscopic glasses on. It gives you the illusion of immersion, but it’s not fundamentally shifting the experience in a way.
	I hate the idea of the helmet. I think you look stupid when you’re wearing it. It feels isolating. It’s like you have to have a huge space. There are a lot of barriers to entry to the experience that to me are a big turn off, and also I get motion sickness pretty easily. The discoveries will be made across different people trying different things, but eventually if the success is a little bit still down into more of a craft.
John Bucher:	Some have suggested there are two schools of philosophy for VR. One being the viewer as observer versus the first-person shooter where the viewer is actually participating or being active in that. Do you think those are geared towards different types of people or that different types of people will gravitate to one or the other, or do you think one will eventually win out the format? What are your thoughts on where that’s going to go?
Mark Cordell Holmes:	My instinct is that there is going to be many, many different kinds of experiences. These are our reference points right now. Third-person shooters, a third-person observational narrative experience. You can still have top-down VR experiences. You’re going to have the same reference points for how people experience things that will probably evolve in some capacity. I think some will be more successful than others because maybe they will lend themselves to the medium in a way better than others. There is going to be a process of natural selection that certain things will win out over others just because they are more engaging or more satisfying or what have you—more effective.
John Bucher:	The word that seems to be on a lot of people’s tongues when it comes to VR is “empathy.” You were associated with a company (Pixar) that has had the corner on the market on storytelling empathy—across the board. Obviously, you’re someone who knows something about communicating empathy from those experiences. Can you talk a little bit about empathy? Is how you’ve seen it working in the past the same as how that could work in VR?
Mark Cordell Holmes:	It’s interesting. My guess is that there is going to be different ways that you would use to create that but maybe not. From the Pixar standpoint, they are making stories. It’s narrative storytelling, and it’s actually quite a long process. It’s in the writing. It’s in the storyboarding. There’s a very iterative process there. At a certain level, it’s like there’s the writers, the storytellers. There’s a degree of themselves and their characters, or their own humanity is somehow going into it at some level. Whether it’s a direct personal experience or again it’s just, they are bringing their own empathy to the characters. There has to be a love for the characters.
	I think one of the differences between Pixar’s stories and other stories that maybe come off as being less empathetic or maybe less successful is that Pixar consciously enables the story iterative process to continue to the bitter end. If there’s time to make a change to make something better, they will do that. It’s expensive, and there is a lot of throwing out of work.
	As you go to the process of putting characters through their paces and bumping characters against each other and you have conflicting goals and obstacles, you start to learn things about them, and your ideas about them change, and you get into the process. You always get to the process where the story starts to write itself where you have to trust the process and let go of your preconceptions. What happens too is that until all these pieces come together towards the end where you have a context now. You’ve been with a character long enough that you’re starting to get to know them in a way where they are starting to reveal this dimensionality, and it’s even surprising to you—where you as the creators are starting to be surprised by these characters. It’s almost an unconscious thing.
	I think that Pixar allows themselves to get to that point where they have all this context, they’ve lived with the characters enough and there is that last little bit of evolution that starts to give them that real dimension. I think a lot of other studios, what they do is they want to lock the script early. The script is locked and let’s just make it. They didn’t allow themselves to have the time to be with the characters, to see them evolve, to give them space to evolve, and to surprise themselves. The result is that they have a cheaper movie with probably a higher profit margin, but it may not have the same humanity, the same depth of character that will stick with people for generations to come. The empathy I think comes from characters being believable and dimensional and of course being flawed and vulnerable. There is a lot of other craft sides of what you can do to make a character sympathetic. I think there’s a lot of important ingredients.
John Bucher:	I think it is interesting when you look back to the early days of filmmaking, we see the mistakes that they made, is there anything just intuitively that you’re feeling at this point, that people want to avoid as they are looking at interactive or immersive-type storytelling in this medium?
Mark Cordell Holmes:	If I’m understanding you correctly, you are talking about how there are mistakes early in the process, and the history of the storytelling craft. The first thing I would say is that there really are no mistakes. I’m going to go for the low-hanging fruit here. One of the iconic early pieces of filmmaking that literally helped to catapult the medium into the mainstream was The Great Train Robbery. Everybody ran out of their chairs because they thought the train was coming at them. We look at that now, we look back at that, and you think, God, you guys were idiots. It just seems so juvenile, but it’s when you look at the nativity, they were all experimenting with the medium. This was a brand new medium. There were no rules. They are taking baby steps and they would fall down and then get up. Someone else takes a different baby step and they fall. Eventually as there’s collective learning, each generation of storytellers or filmmakers takes the little bit of pieces that work and string them together, and then they make their own mistakes and they bring their new learnings to it.
	I feel VR right now is like we are making trains that are flying out in your face and straight at you. That’s all we are doing. It’s all novelty to an extent. You even look at the advent of 3D animation not that long ago. When you look back on them now, they are just these tacky, cheesy things to laugh at, but they were such milestones and it took those steps to lead to the next steps. Until you get this guy named John Lassiter whose job it is to make a text demo and instead of making a text demo, he makes a story about lamps or about a bee stinging a guy’s face.
	He was able to elevate beyond the novelty of the technology or the novelty of the medium to use it in a meaningful way as a form of expression, artistic expression. It then becomes truly empowered in a way that has relevance to people’s lives. You could look at a shiny ball and go, “That’s neat,” but then you’re going to forget about it. You go, “What am I going to do with that?” If you come up with this little lamp bouncing on the ball, that’s going to stick with people, and then they care about the medium. Everyone thinks that they know what they’re doing, and they do to an extent, but it’s such a brand-new field that people should be throwing all kinds of stuff at the viewer or trying all kinds of things. Hopefully, in five years, ten years, it could be three months for all I know. At some point, enough of what works and what people are responding to will connect.
	Honestly, we won’t know for a while because we don’t have any mass adoption of the technology yet, and there’s still a lot of barriers to entry. I think in time, though, we’re going to be discovering things that none of us could anticipate now that are experiences we just have never had before. That we have for the first time in this medium and it just blows us away and creates that into an art form or a new storytelling form or maybe as just some completely different application we can’t even imagine that becomes the new internet or the new way of doing things. Anyway, it’s a super-exciting idea. We’re still taking baby steps right now.

Insights from the Artists

Those bringing training and experience from other disciplines will continue to be important contributors to the field of VR storytelling. Artists with understanding of immersive principles in their respective fields can teach VR creators what has worked and what has failed in their experiences and potentially create significant VR work themselves. Angela Haddad’s background in fashion and design was useful in thinking through new ways VR could be expressed. Annie Lesser’s work with moving audiences through immersive drama has served as a basis for her beginning work in VR, AR, and MR. And of course, Mark Cordell Holmes’s talents with animation made for a natural bridge into creating experiences in which digitally produced characters engaged with the audiences in VR. All of these artists also share a common characteristic: a gift for knowing how an artistic element will impact the audience emotionally. Haddad called these elements trigger points. In her work, they are the elements that cause a viewer to turn their head and look in the direction she intends.

Artists in any medium, but especially in VR space, must become familiar with the tools at their disposal for moving an audience’s emotions and experience. Jaunt Studios chose to work with Doug Liman because of his ability to move an audience through a wide range of emotional space—from comedic moments to breathless suspense. Jaunt is transcending the traditional film studio model, creating both the technology that makes its product possible as well as the talent and understanding about how to tell stories in immersive space. Similarly, in her field, Lesser is meticulous about where she places actors interacting with the audience. She is aware that space itself is a tool the creators must use wisely. Liman demonstrates a parallel understanding of space in Invisible, placing his camera in positions and angles that defy traditional cinematography. His canvas had changed, and it became important to change the ways he used his tools. Lesser uses the location of her artwork as environmental storytelling, never missing a chance to let the objects in the room play their part in telling a story. Many of the immersive principles discussed in Chapter 4 unpack the ways that artists like these approach space.

Holmes compares the artistry in VR to the earliest works of filmmaking, relying often on novelty and not yet pushing the boundaries of the medium, which does take time. Most creators featured in this book stressed the importance of experimentation and risk taking in VR, recognizing that this path is the only way to keep moving the medium forward. It could be argued that the film industry has slowed its development as a medium due to the lack of commercial opportunities for films that push the artistic boundaries forward in ways that do not reinforce what audiences are already comfortable with. Lesser mentions the importance of symbolism, subtext, and thematic exploration as ways of pushing a medium forward along with the technological. Many times, these methods are more accepted by audiences simply because it gives them handles to grasp while trying to make sense of the new experience. Haddad relies on a similar approach with her work. Many experimental films use a few of the basic elements of traditional cinema in order to give their audience familiarity on some viewing level. Remembering that the experience of the audience is paramount will continue to be a guiding principle for creating VR stories.

The Visionaries

Storytelling in VR through World Building

An Interview with
Larry Rosenthal, VR Pioneer and Producer

Larry Rosenthal is a veteran of the VR industry, working in the field since the late 1980s. Currently he owns Cube3, a company focused on building virtual worlds and experiences.

John Bucher:	Let’s start by talking about what world building means to you.
Larry Rosenthal:	In today’s world, it’s an interesting concept because everyone has an idea now of what it means. It’s a buzzword. With certain properties like the Marvel universe, the money is there now to really build worlds. The lexicon has also moved on from Virtual Reality to virtual worlds back to Virtual Reality in a sense. I did a seminar in 1995 called “Places not Pages.” I was trying to get people to understand that the virtual world was going to be a bunch of social places more than home pages. At that point, everyone was trying to put a brochure on the internet. No one understood that it was a spatial relationship through Virtual Reality with a screen.
	World building is mediation. There’s nature. There’s a tree, the sky, the ground. There’s the man living in the cave who gets wet when he walks out. The minute you start to mediate the world by putting something over his head, going back into the cave with his wet hand, and putting it on the wall, it’s creating a mediated world. That’s what we did as humans. World building for us is mediation of the natural world. The nature of God as the world builder depending on your beliefs, full of everything we’ve been thrown into. World building to me always sets up an interface for the world. It can be used for all different sorts of things, often selling agendas. It can be used for entertainment. It can be used for teaching and learning. It’s a way of modeling.
John Bucher:	There’s a lot of people who say VR is the ultimate empathy machine. Why might that not be true?
Larry Rosenthal:	Because it’s the ultimate emotion mediator, but it’s not an empathy mediator.
	It’s an emotion machine, but empathy is up to the user to figure out. Someone was posting yesterday on Twitter the same thing. They said something like empathy is what you may feel compassion for but look at what you actually do. There’s no doubt that VR is the so-called last medium. One of the reasons I’ve stayed in it for so long is not just because I don’t want to be on the streets and broke. It’s because this is the atomic bomb. This is the third act.
John Bucher:	You talk about some of the uses of story and narrative in VR. Can you talk a little bit about what has been the history of story in VR, and more importantly, what you think is the future of how story will be used in VR?
Larry Rosenthal:	I like the term “narrative” better than “story.” The simplest form people understand is the “Choose Your Own Adventure” book. It began with the paper medium, and because we numbered it, we ordered it, we allowed people to basically skip ahead and branch off, and it only cost paper. They were books that were targeted for kids, but basically it’s where almost all the narratives in the digital industry come from. You had to let go of the attention. You had to let the viewer be a user, the star of the show, for lack of a better term. 3D games have already covered a lot of this. There’s no doubt in my mind that the people who are dealing with the worlds of RPG games and 3D games and theater outside of gaming are the most into what this will be. It’s their lessons that are going to work. The ones that are driven by a lot of the character-based ideas are the ones that are going to find what doesn’t work. They’ll find it again as everybody tries to become the Pixar of VR.
John Bucher:	Why has the character-centered narrative not worked as well in VR?
Larry Rosenthal:	Because they’re not the star. You are. In other words, the audience, the viewer has so much of the weight that you have to be the one who’s really driving the antagonism of the show. Otherwise, you’re watching a cartoon. You’re watching someone else do a performance, which is fine. The problem is right now and maybe for the foreseeable future that there are two things to consider: ROI and ROE. Everybody knows what ROI is: return on investment. But I’ve been using ROE for return on entertainment for just as long. That’s why a lot of the web 3D stuff and a lot of multimedia stuff fails, because you have to jump over too many hurdles. It ruins the entertainment.
John Bucher:	Just to build on that, why do you think our culture has become so hungry for experience as opposed to just entertainment?
Larry Rosenthal:	I think they’re both the same in some ways. We’re in an interactive age. We’ve gone from ear-eye to hand-eye as a culture. In the 1980s we got the clicker, so now we have control over the media. We’ve got 500 channels and nothing on. That began to change us from the ear-eye of having to sit through whatever was on TV to now where you can interact. You get anxious if you aren’t able to control your medium. The HMD, I think, is going to have the shortest lifespan ever as the mediation device, because I think within a decade, we’ll have gone from these into the chip. Now that we’re networking, acceleration factor, McLuhan, all this stuff kicks in. The kids will have the chips.

Storytelling in VR through Light

An Interview with
Paul Debevec, Senior Staff Engineer, Google VR

Paul Debevec has served as chief visual officer and led the Graphics Laboratory at the University of Southern California’s Institute for Creative Technologies. He is a research professor in the USC computer science department. He earned degrees in math and computer engineering at the University of Michigan in 1992 and a PhD in computer science from UC Berkeley in 1996. He directed a photo-realistic fly-around of the Berkeley campus for his 1997 film The Campanile Movie, whose techniques were later used to create the Academy Award®–winning virtual backgrounds in the “bullet time” shots in the 1999 film The Matrix.

At USC ICT, Debevec has led the development of several Light Stage systems that capture and simulate how people and objects appear under real-world illumination. Early Light Stage processes have been used by Sony Pictures Imageworks, WETA Digital, and Digital Domain to create photo-real digital actors in award-winning visual effects in Spider-Man 2 and King Kong, Superman Returns, Spider-Man 3, Hancock, and The Curious Case of Benjamin Button. The most recent Light Stage process based on polarized gradient illumination has been used in numerous films, including James Cameron’s Avatar, The Avengers, Oblivion, Ender’s Game, Gravity, and Maleficent. In February 2010, Debevec received a Scientific and Engineering Academy Award® for the design and engineering of the Light Stage capture devices and the image-based facial rendering system developed for character relighting in motion pictures. He is senior staff engineer at Google VR and recently worked with the Smithsonian Institution to scan a 3D model of President Barack Obama.

John Bucher:	Can you talk a little bit about your philosophy of bringing humanity to these digital creations? It seems to be a value you hold.
Paul Debevec:	I got into computer graphics pretty seriously in graduate school when I was at UC Berkeley. At the time, we didn’t really know how to do photo-realistic digital faces. It was clearly a holy grail of the field. We didn’t even know if it would ever be possible. Originally my work actually had more to do with architecture and lighting and digital photography. That’s where I found I could make contributions. As soon as it looked like we could bring human faces into the discourse of our research, with our first Light Stage, which was a way of reproducing the light of some other environment onto a human face, I knew that the work would get quite a bit more impactful.
	Another part of that is the inspiration from the filmmaking side. Back in the ’90s, I would try to do research in the fall to publish a paper for SIGGRAPH. Those were due in January. Their computer animation festival deadline was usually in April, so I had a few months to try to make a computer-animated short film that leveraged the technology and then have that be in the film showcase for the conference and get some good press marketing between the creative work and the technical.
	I made these films, The Campanile Movie in ’97 and Render of Natural Light in ’99, and I was really enjoying the filmmaking process, but I realized my movies don’t have any people in them. They were about architecture and lighting, and you can make an interesting two-and-a-half-minute abstract short that way but little more. The Campanile Movie technically did have a person in it, but that was live action. To go to the point where we could try to create a realistic, digital human face felt like the modern equivalent of the problem that painters had in the 1300s and 1400s and 1500s of getting to the point where a painting could show somebody that you felt was a realistic-looking person.
John Bucher:	In your TED talk, we saw digital Emily on screen. It’s difficult to tell if she’s real or a computer. Do you believe that we are approaching an age where we’ll be able to create digital actors and eliminate human presence in our films and storytelling?
Paul Debevec:	That’s a great question, and sometimes when we have an actor on one of the Light Stages, they’ll ask, “So this is going to make it so we don’t have to act anymore? Or they’re not going to need us anymore?” If it looks like a character is giving a good emotionally engaging human performance in a film or video game, the reason for that is that a real actor actually supplied that engaging human performance, and through performance-capture techniques, it was mapped onto the digital actor. There are a couple instances where key frame animation techniques were used to create the performance. For example, the Michael Jackson performance at the Billboard Music Video Awards, where he performed a song in one of these reflective Pepper’s Ghost images. We assisted with the project with the Light Stage. We didn’t do the animation. I met the animators, and they actually provided the performance with lots of reference from the real Michael Jackson. It’s incredibly well documented, so in a sense, it’s his performance up there, or it’s based on his performance. Lots of incredibly talented animators crafted it, so in that case, the animators are sort of the actors in that human element.
	If you go to any Pixar or Disney 3D computer animation, it’s the animators who are providing those very engaging performances of the animated characters. We’re not eliminating the human elements at all, in any of these cases. When you get to VR, the question does become a bit different. Once you have one of these headsets that you can look anywhere, and you really feel like you’re in a different world, a character that’s in that world with you, or that you see there—you really feel a lot more like that person is actually there with you. Since you can interactively look around and explore the scene, you feel this is a much more present experience, so your expectations that that character will react to you and that you can converse with them are much higher.
	Unless we’re doing a telepresence situation, where it really is another person over there, and I hope VR will produce great technologies for that, then we do have to figure out a way for this synthetic human to interactively react to you. To do so in real time, and believably—that’s something that is an area of research right now. It probably needs several years to get to the point where we’re blowing people away with what’s capable. I think VR, much more than the games, certainly way more than movies, is going to push the need for development of artificial intelligence technologies that can simulate engaging human behavior.
John Bucher:	I love that you brought up Pepper’s Ghost. If we look back at history and what people have been trying to accomplish with putting virtual presences in front of people, we must discuss Star Trek’s holodeck. Do you think VR is headed to a place where we take the glasses off and we experience this Virtual Reality in an even more realistic way?
Paul Debevec:	The holodeck, for a lot of us researchers, was the shining example of what we wanted to create when we created virtual environments and the kind of content creation techniques you need to generate what the Star Trek characters experienced, and that was a very reasonable vision, because they’re really these amazing 3D light field displays that can make you feel like you’re in a different place in that way. I think of it like the CAVE systems. They were early VR experiences with six screens around people that are projected and you have to wear 3D glasses. That was more kind of like a holodeck sort of experience.
	With this renaissance of Virtual Reality and the fact that now the screens are high enough resolution that you can make out a bit of detail in these environments and that the computation and the display is fast enough that it really feels like it’s reacting to how you’re moving your head in real time, I think within a couple generations of these headsets, we’ll have things that are incredibly satisfying VR experiences, and we’re going to say, “Holodeck? Why do you need a holodeck?” You can do this in VR. I don’t think that we actually need to invent the large-scale, 3D light field displays that the holodeck seems to have.
John Bucher:	You did a piece on the Parthenon. You’ve had an interest in Greek storytelling and what has come before. What do you think are some of the key things from our past we must remember and what we have learned over the last couple thousand years about story that’s important for us to bring into VR space?
Paul Debevec:	I love how, in our excitement about creating content for VR, we have thrown so much energy into the conversation of what is story, what is interactive storytelling, and how will we be telling stories in the future? I’m certainly of the camp that believes that with VR experiences, people feel like they’re in the present, generally. When you have the ability to affect what is happening in the environments and with the characters that are around you, it feels like this is happening now and it doesn’t feel like it’s once upon a time. “Once upon a time,” “in the past,” I think that’s storytelling. Stories are how we store information about history and the lessons that we can take from that history, allegorically. Maybe we should just call some of these things interactive experiences to differentiate it from storytelling—which is this whole other new completely amazing thing, where you feel like it’s the present—you feel that you have agency within this world. We’re only getting started with that. I’m so excited to be in this field for the next decade at least, because we’re going to see brilliant people coming up with brilliant stuff. It’s really even hard to imagine at this point as to what people will find fun and engaging to do in these virtual worlds.
John Bucher:	You’ve been called “Hollywood’s Master of Light.” Can you talk a little bit about the role of light in storytelling?
Paul Debevec:	I’m very happy that in my career, I have gotten to work with and talk to some of the great cinematographers in the film industry and learn a little bit of their craft. Lighting is incredibly important for setting the mood for a scene and the visual look of how wide the color space is, saturated or not saturated, the contrast you’re not seeing contrasting. Is it hard light or soft light? I’m conscious of light a lot more than the average person from studying it, but I can remember back before when I just wasn’t very conscious of light. I saw things in terms of objects and the colors of the objects rather than realizing that I don’t actually see objects, I only see light reflecting from them, and that light changes remarkably, depending on how things are lit.
	I find that lighting is a way to communicate directly to the subconscious experience. Take a look at Instagram, which is a multibillion-dollar company right now. It was founded originally on just taking a cell phone photo and altering the colors and the contrast in a way that it looked like a Polaroid from the 1960s, for example, and it completely changes your emotional relationship to that material, and that was enough to launch an enormously popular company, which does much more now but began with that simple idea. I’m looking forward to seeing what cinematography means in the Virtual Reality world, and I hope to get to work with some of these folks and experiment in the space myself.
John Bucher:	I would be remiss if I didn’t ask you to tell us a little bit about the experience of working with President Obama in the project that you did at the White House. Could you talk a little bit about how that worked within the larger scope that you’re doing?
Paul Debevec:	That was an extremely exciting project, and it was nerve-wracking. It was basically like playing a really important away game for us, because we usually have people come to our Light Stage in Los Angeles, to our own space, and if anything goes wrong, we have everyone who designed everything around. In the case of the scanning we did at the White House, which we did at the invitation of the amazing 3D digitization office at the Smithsonian Institution, who were our collaborators and decided that, from some talks I’d given, that using our techniques for the front of Obama’s face, ear to ear and down to the neck, was, at the time, the highest-quality 3D scan that could be done.
	We had to customize our system to take it to the White House to take the scan, and we had a very limited time to do that. We’ll have a major actor, like Tom Cruise or Angelina Jolie, for at least an hour, and in the case of President Obama, we had to make sure that we could do it in just a few minutes. One of the awesome things was, we were so fast, and the Smithsonian folks were so fast with their systems to get the back of the head and the shoulders for the 3D printed bust, that there was like 10 minutes left over, so he just hung out with us and we talked about the technology, and it was a pretty special time. I’m so glad technology happened to work flawlessly that day, and we were almost inappropriately exuberant afterwards in the White House once everyone had left and we realized that we got it done.
John Bucher:	In a perfect world, what would be your ultimate goal or ultimate dream for Virtual Reality?
Paul Debevec:	I think of VR and the content creation technology in general that I’ve been trying to help develop as enhancing the ability for people to communicate with each other and to tell stories to each other. I want it so that anybody, with anything that they want to show the world, to tell the world, can communicate that as realistically as possible, as fleshed out in terms of the environments and the characters as possible, and that we’ll be able to make experiences on the epic scale of Avatar with a relatively small group of collaborators who are able to do the key creative things to get that stuff done.

Case Study: Lucy VR Series

Written and Directed by Matt Thompson

Matt Thompson created the VR series Lucy as an exploration of life’s biggest questions. Why are we here? What happens after we die? “The idea of the afterlife and the idea of a story taking place starting at the point of death was really interesting to me. I’m particularly interested in spirituality. To me, spiritual stories are the stories, that started stories,” Thompson said.

Following the loss of his father at a young age, Thompson had explored the theme of loss in his work for some time. With Lucy, he wanted to explore what his father’s journey might have been like entering death. VR presented a new medium to explore this theme in.

“Virtual Reality is not the new film, it’s just a new medium. For the afterlife, there’s a lot of stories, a lot of narrative written, but there’s never been an opportunity to step into the afterlife and be meditative about it. That’s what I was looking to do,” Thompson said.

The script for Lucy became an evolving document, changing as different scenes were crafted in virtual space. The dialogue for each scene remained roughly the same, but the capture process constantly presented new challenges as well as new opportunities. Combining VR computer graphics with two live-action sequences, Thompson storyboarded using five circles on a page to represent the different angles available to the viewer in each scene in order to craft the narrative.

The production process leaned heavily on practical lighting and hidden microphones to keep gear out of the shots. “We coined the term ‘DPP,’ the director of pan-optic photography, for Matt, our cinematographer. We were intentional to adopt a philosophy that recognized it’s not virtual cinema, it’s Virtual Reality,” Thompson said. “It’s a lot more about context. When I walk into this room, I feel a certain way about you, and not because of how you’re lit, although it helps, but it’s because of the context that I know you in. That’s what I started with to create the environment and world,” he continued.

Because Lucy is a first-person experience about a protagonist that loses their body, Thompson decided to envision the camera as a vessel. “If people are going to put something on where they can now look all around, it is now inhabiting a body. You can make these experiences where you can still be omniscient and you’re watching people, but I don’t think that makes sense,” he explained.

Logistically, tempo and pacing are among some of the most difficult elements to control, according to Thompson. The loss of certain editing techniques in the post production process forced the crew to think differently about how to establish a rhythm in the story. Audio became a driving force in the process. The audio team developed a spatial mix for the project for viewers on the Oculus platform as well as a standard stereo mix for viewers on more basic systems.

Future episodes will explore death experienced by a different person from a different place in the world, exploring their cultural, religious, or spiritual beliefs. The ongoing theme will examine how planet Earth thinks about the afterlife as a whole. “When you can experience it (afterlife experiences) side by side, I think you can get a real sense not only of perspective but empathy,” Thompson said.

Thompson remains focused on simply telling good stories in VR space as opposed to crafting content around the medium. “I think that if I can just focus on making really compelling content, the brilliant engineers of the world will keep making the technology better and easier,” he said.

Storytelling in VR through Agency

An Interview with
Brian Rose, Google VR Team, Community and Outreach

Brian Rose has worked as community and outreach manager for Niantic, creators of AR mobile app games Ingress and Pokémon Go. He’s now in developer outreach and community for the Google VR team, working on AR projects such as Project Tango and VR projects such as Daydream.

John Bucher:	Let’s talk about some of the more philosophical ideas when it comes to Virtual Reality. One is the idea of agency. I’ve noticed that Google has been leading the path on providing agency to people, especially with programs like Tilt Brush. Can you talk a little bit about the way that you and your team approach agency when you’re trying to design something for the end user?
Brian Rose:	On the Google VR team, we’re working on a lot of different projects. On the YouTube side, we have things like Spotlight Stories. One of the things that came out of the YouTube Spotlight Stories project was this idea of branch story lines. In normal Google Cardboard 360 content, you’re essentially just stuck in one place and you can look completely around you, but a lot of our creators had asked for some kind of interactivity. Some kind of either look-based or gaze-based interactivity. We created animation with Aardman Studios that had something like forty-five different branch story lines or possible story lines. When I thought about a lot of narrative games which I really love—games like even Life Is Strange for example, or any Choose Your Own Adventure book, a lot of times I’d have a lot of fun in the process of reading the stories or playing through the games, but at the very end, it always felt like it always came down to this binary (a) or (b) path. Whenever I hit that end point, I don’t know if I ever really felt satisfied with being able to choose, “Do I save the town or do I save my friend?”
	So, because of that, what we decided to do on the Spotlight Story side was always have the same beginning and ending to the video, but how you get to that ending or the subplots or substories that you see along the way—those are the things that are different and can change anytime. It might be unique in the way I watch it or you watch it, but we’ll always have the same beginning and ending. We have that shared experience, but the way that we get there could be different meandering paths.
	I do think agency is very important. We created a narrative called Pearl. We had talked about a 2D version where it’s just something that you could watch on a TV or on a computer. Or perhaps a 360 version that you could watch on mobile VR with the YouTube player. But on the HTC Vive, you get a greater sense of agency. You can actually walk around the car or stand up when the little girl pops out of the sun roof to catch the fireflies. That agency provides some powerful, emotional moments that come from that.
	There’s a difference between what I refer to as ghost stories, where the characters don’t know I’m actually there, and stories where you have presence with the characters. In the first type, I’m essentially a ghost in the story. I’m just passively watching things happen as they happen versus actually being able to interact with characters in the story or affect the environment, affect the other characters in the experience. But those are both paths that we’ve been exploring.
	We’re doing so many things in VR. We try to do a little bit of everything to see what works and what doesn’t. Also to see what works for specific platforms. There are some limitations that we have with mobile VR that we don’t have with desktop-tethered VR. Being on the Vive gives us things like hand controllers that Cardboard doesn’t, for example. Being able to use those hand controllers, being able to take advantage of room scale and figuring out how to incorporate those into the story in a natural way, they’re all parts of things that we’re working on with the Google VR team. I wouldn’t really say at this point there’s one right way or a wrong way to do it. There are certain things that work better on specific platforms, and because VR is so many things, we end up doing a little bit of everything.
John Bucher:	As you and your team are looking at and designing experiences, when do narrative elements rise about “this character” or “this conflict”? Can you talk about what some of the conversations are around designing an experience when it comes to narrative elements?
Brian Rose:	A lot of what we try to do is create tools to enable creators to tell those stories and then make sure they can reach as wide an audience as they want to. On our team, part of our thinking is that if we are going to create these platforms, it would probably also be helpful for us to provide some high-quality content so that we can show people, “These are the things that we’re looking for.” Or, “These are things that we think are possible on the platform.” To that end, we do have a team in New York that’s working on content. Jess Brillhart is our principal filmmaker, and she can talk much more in depth about her point of view or her direction in terms of storytelling and narrative in VR.
	On the Google VR team, a lot of what we’re focused on is the technical aspects rather than on trying to tell creators, “Here’s how to or how not to create content.” I will say though that with the hardware itself, I don’t think it is what’s most important. At the end of the day, although we are working hard to bring mobile VR to the masses of Cardboard users, we’re working hard on bringing up Daydream with higher-quality mobile VR. Story is such a key part of that. As much as we can, we work with the community to figure out if there are tools that content creators need that don’t exist today. If there are, please let our team know and we’ll work to build those into the platform. Things like those branch story lines or things like interactivity through hand controls or gaze-based and look-based interaction.
	Storytelling in VR—it’s incredibly important. Those rich examples of the great stuff you can do with VR, really come down to creating compelling immersive stories. That’s why we’re so invested in trying to support this community.

The Future of VR Storytelling

An Interview with
Ted Schilowitz, Futurist, 20th Century Fox Studios

Ted Schilowitz was a founding member, first employee, and integral part of the product development team at RED Digital Cinema. He currently spends his time focused on the future of the cinema experience. He has a studio deal at 20th Century Fox as their futurist, working directly with senior leadership at the studio on the constantly evolving art and science of the digital age of motion picture creation. He advises and creates strategy on future technology and vision of cinema for the next generation of movie entertainment.

He has been featured in Wired, Variety, The New York Times, The Wall Street Journal, The Hollywood Reporter, NBC, American Cinematographer, MacWorld, Popular Science, LA Times, and in countless other trade publications and the mainstream press, discussing technology advancements in his areas of passion and exploration.

John Bucher:	I’d love to start by talking on a macro level about VR. Why now? Why is it that we suddenly hit a breaking point in the adoption of VR, and what does that have to do with storytelling?
Ted Schilowitz:	I think about this a lot. I think about what the many answers to that might be, much more than the raw technology itself. I think about humanity. We spend a lot more time touching technology in a more intimate way than in generations past. You look around this world and see how many people are holding a little piece of silicon and functioning to achieve things through that. Eight to 10 hours a day, right? Or maybe longer. With kids, probably longer. The question is, “Are we ready as a society for the next step in a visualization experience? For a more intense and modern experience than what a phone offers?” I believe that we likely are.
John Bucher:	Perhaps a more embodied experience than what we currently have with the internet?
Ted Schilowitz:	When we talk about the curving of technology and the use cases of technology, this becomes this very popular discussion, which I’m sure you’ve had many times, about what makes it “Virtual Reality.” What takes us to a point of presence, to a point of awareness that our optic system, our brain, our limbic system, and our physical body are actually connected into that world? That’s when you start to get into the world of the Ready Player One metaphors.
	That starts to get into this interesting discussion of the dystopian or utopian universe. What I like to say, and I think this is a fairly popular point of view, is that humanity tends to find a way to drift toward more of a utopian future than dystopian future. The more and more powerful the tools get, the more and more chance that the dystopian part can actually be more powerful and more meaningful. You end up in this interesting ethics discussion about connecting all the different systems of our physical body as a new form of entertainment or a new form of communication. Do I believe it’s going to happen? The answer is yes. I’m already seeing touchpoints every single day.
	It’s been a year, maybe more, the first time I did VR karaoke. I actually stood up in front of a virtual crowd and actually sang just like I was in a real karaoke bar, but it’s virtual. So, my brain is being tricked. You get the same nervousness. You get the same amount of tension. You get the same performance anxiety. I’ve done a VR talk show where I actually sat with a group of people on a virtual stage just like it would be at a real event that I do at real places all the time. I interacted with them. Essentially we were all avatars at that time, but that’s going to change as well, as far as body scanning and so forth. It’s spatial. That’s actually a big point of something we might want to talk about. The sense of what we call true spatial sense in Virtual Reality and augmented reality is fundamentally different than flat-screen media or all the media experiences that have come before it. I think it’s not discussed or talked about nearly enough.
John Bucher:	I’d love to talk about that.
Ted Schilowitz:	What we see now is that the head-mounted displays that exist today do a really good job at surrounding our visual experience and firing our visual cortex. You’re firing a lot more brain cells in VR than you are when you’re watching a screen in front of you. You start to get that sense of believability. You start to attach the first step in the equation, now very commonplace, which is your hands. But you’re still using controllers for your hands. It’s a metaphor from the past, which comes from video games. You’re still clicking buttons.
	I think when you put your futurist hat on, where it really starts to go is what we refer to as the ghost hands—when you’re not actually using a physical device. The sensors are just good enough that it can actually track what you’re doing. If I was in front of a virtual panel, getting ready to go into a sci-fi adventure, I would move my hands around Minority Report style. If I was going to be in an active shooter game, as opposed to using a game controller, I would just literally have a physical prop, and that prop would be tracked in space. It would feel like it’s a real gun and act like a real gun.
John Bucher:	If you own a 3D printer, then you can easily create those props.
Ted Schilowitz:	You see companies doing this now. None of this is exotic anymore, and that’s when I ask, “Why is this now?” The confluence of all these things. We talked about humanity a bit. We also should talk about the fact that you can actuate and create things that deliver on that dream without having to go to huge amounts of expense or exotic tool sets to do that.
	Now, the next step in the equation is when you actually start to make a choice to wear enough sensors on your body that you have what we would call full-body presence. There are a number of different companies that I’m very closely involved with that are touching this. Again, not all that exotic. You’re talking about motion-capture tracking systems and wearing little balls. With enough of those data balls, which is like performance-capture equipment, but simpler, you get a sense that “I can see my own body in VR.” If you think about it, what we’ve always been doing in entertainment are these magic tricks. We’re creating illusions, and the more you can blend technology into the illusion-creation system, the more of a chance you have to actually really fool people. It’s no different than going to a magic show. The better magicians, you fully believe, and you’re scratching your head going, “How the hell did they do that?” In the world of VR, there are moments that I have, even as a practitioner of this now, a conjurer in my own right, where the illusion is almost complete. You set the stage for “Wow, look what just happened to me. I started here, and I ended here.” Historically, if you want to reference the practitioners of that, for a number of generations, they have been people that were creating theme park entertainment.
	I’ve often made this funny little trite reference that now we have the ability to wear a theme park on our face. Take it with us in our carry-on luggage. Put it in our living room or our dining room or our game room. Whereas before I had to go to a real theme park to get a theme park experience. To get into a simulation ride or experience. Technology is now merging that with an at-home or mobile device, or anywhere I want to be in experiences. That’s the real power. That’s the lightning bolt. When you say, “I could only do this in a very exotic place.” Now I’m moving to what I would call a semi-exotic place.
John Bucher:	You’ve brought up theme park experiences. You’ve brought up viewing experiences. We certainly have gaming experiences. Let’s talk for a second about the narrative elements that we find in all of those experiences. We are bringing together the world of gaming, the world of theme parks, and the world of film and television in VR. It’s merging into this new medium. Can you talk about the narrative elements in this space?
Ted Schilowitz:	Sure. I’ll take this in a slightly different direction than a lot of people would take it. I have slightly different kinds of language points. Why do we like cinema? Why do we like gaming experiences? What are their similarities and what are their differences? This is interesting to me. You’re watching something play out on a screen. The fascination is there, even though you know it’s already been captured. It’s been created. As you watch it for the first time, you are just a fraction of a step behind what we would call reality. That’s what’s so intriguing for us as humans. I’m watching a story play out in front of me, and I don’t exactly know what’s going to happen in the next moment. Now, I make all kinds of predictions about what could happen in the next moment based on the last 5 seconds, 5 minutes, 30 minutes, 1 hour. When it takes a sharp right turn or sharp left turn, and I didn’t understand for a moment until my brain caught up why that happened, that’s to me, the magic of cinema.
	We can talk about why are twist endings so important. Why does the horror genre work so well? Why does action-adventure work so well? It’s because even though you think you know what you know, they put a big twist in. It’s that, “Oh! You got me” moment. That’s storytelling. It’s when you’re always right on the razor’s edge of thinking you know what’s going to happen but not exactly sure what’s going to happen. That’s real life. That’s the mirror of real life.
	The part of our brain that gets stimulated from this passive entertainment where we know the choices have been made for us. We’re actually not going to get to make any choices here. We’re going to watch, and our brain is going to wander down the path as the story’s catching up to us. There is a different part of our brain that actually functions in the decision-making process, which is a more active part of entertainment. Look at who likes games versus who likes passive entertainment and why there are not a lot of people that would say they’re super passionate about both. It is because people that like to make decisions when they entertain themselves drift toward interactive entertainment. They drift toward console gaming, PC gaming, mobile gaming. They want to be an active participant in their own destiny.
John Bucher:	They want agency.
Ted Schilowitz:	They want agency. Agency often gets misused a little bit in VR. It’s like, “Well you have agency.” Well. Not if it’s not crafted right. You have pseudo-agency.
John Bucher:	Then you have global agency and local agency and all these other ideas with that.
Ted Schilowitz:	I think that’s a super-interesting way to think about the world. It’s interesting when it comes to the age dynamics of why we see that younger kids, at the end of their day, want to entertain themselves with a gaming experience versus people that are a little bit older want to entertain themselves with a passive experience. I have a theory on this. It’s not proven. I just have a theory.
John Bucher:	I’d love to hear your theory.
Ted Schilowitz:	My theory is that as a young kid, you’re not often asked from the minute you wake up in the morning to the minute you get home at night to make a lot of decisions. You sit in school and they’re putting an educational curriculum out for you. You’re asked to do things and stay on the path. Pay attention. We’re going to deliver a lesson plan. You’re not given a lot of agency. In the more creative schools, you get more agency. I think that’s actually why the superstar kids come out of schools that have figured out that they need to be more interactive and more demanding on their students in terms of making their own decisions. Even with that being said, the end game is usually set. It’s like, “Yes, we want to give you the illusion that you’re doing this on your own and you’re going to come to your own conclusion. But ultimately as educators, we think we know the conclusion.”
	As a kid, you’re not often asked to make a lot of decisions in the first 8 hours of your day. When you get home at night, the entertainment you want to do is more about wanting to be the master of your universe. You want to control because you didn’t get to make 200 decisions from eight in the morning till five thirty at night. My mom puts dinner on. I don’t get to pick. She just made dinner. If I don’t eat it, I go up to my room. When I get to break out, I want to get into the Call of Duty world. I want to get into the Grand Theft Auto world. I want to get into the CS Go world. I want to because now I have the ability to make my own choices. To hide or shoot. To get better at something. To activate with my friends in a certain way. It fires a completely different part of our brain. Now, on the flip side as grown-ups, you’re often asked to be in an environment where you have to make decisions all day long that are affecting other people in your world, that are affecting your job, they’re affecting money. At the end of the day, as a grown-up, you want passive entertainment. You don’t want to make any more decisions. Which is why the appeal of a game, a console game, a PC game is much less so for someone that had to make decisions all day long. They now want to turn that part of their brain off. Kid’s want to turn that part of their brain on at the end of the day. Adults mostly want to turn that off. Now, I would say, we’re starting to see a conflict. We’re starting to see a new generation of kids that grew up in that world of, “I don’t get to make decisions here but I get to make decisions there” now becoming adults. They’re blending those two worlds in a different way. They want to entertain themselves in a different way. They don’t find the idea of sitting in an office all that compelling. They want to move around throughout the world and have agency in everything that they do. At the end of the day, they want theme park entertainment. They don’t want passive entertainment.
John Bucher:	I think that leads to a discussion on presence, because in a movie theater, that’s a passive experience. We turn the lights off so that you don’t even see yourself. There’s no sense of presence.
Ted Schilowitz:	The noise of your neighbor bothers you because now that decision-making process in your brain is turned on, unfortunately. When I see the light of a text message in that dark theater, I now have to make a decision whether to tell that person to shut that thing off or to try and ignore it. It throws you out of that passive relaxation of entertainment.
John Bucher:	Everyone is going for the deepest sense of presence and immersion that can be accomplished in VR space, but there are some who would say, “We’re actually trying to move towards AR world rather than taking you out of the present reality.” A sense of presence changes as we move to AR because we’re here. There is presence. How do you think we are going to deal with presence in an AR world?
Ted Schilowitz:	I’m doing a lot of VR and I’m doing a lot of AR. I actually think it’s two different forms of art. I think VR is literally creating a universe that is trying to include as much of the real world as possible and take you to another place. That magic trick is all about trying to convince me that I have left the real world. With AR, it’s a blending of worlds. In a world where we’re playing a mobile game on a screen, the real world is kind of there with me. I have to multitask a little bit. As I start to wear it on my face, that changes.
	If I was wearing some sort of AR glasses, in the future, if I don’t have time to go to another world but have five or seven minutes to kill some aliens in the parking lot I’m walking through, I can do that. That’s fun. I don’t have to leave my world to augment my world. HoloLens is doing it. Our friends at Magic Leap are doing this. There’s a lot of companies that are starting to figure that out as a form of entertainment. I think it’s extremely powerful. I also think it’s actually very different than VR. They’re almost two different artistic pursuits.
John Bucher:	How does gaming figure into VR and AR from a storytelling perspective?
Ted Schilowitz:	Games have figured out another way to tell stories. People sometimes don’t think gaming is a storytelling medium, and I violently disagree with that. Everybody’s played Pac Man. I don’t care how old or how young you are, at some point in time, you’ve played Pac Man. You have agency in Pac Man. You have a device that allows you to make choices in Pac Man, and you have to run away from ghosts. It’s a flat metaphor that is telling us a story. You got to get away from the ghosts, hide, eat some food, and then progress to the next level in life. It’s a story.
John Bucher:	It really is a story.
Ted Schilowitz:	It is, right? In the most basic understanding, I just told you a little story. Okay, so here’s a story. You’re going to be this creature and you have to find a way to survive and ghosts are going to chase you. Well of course it’s a story. Space Invaders is a story. Galaga is a story.
John Bucher:	Pac Man is like the oldest story of humankind. I’m going to hunt for food and try and get away from ghosts. That’s like a caveman’s story.
Ted Schilowitz:	That’s modern storytelling, too. What I find really interesting in the world of Virtual Reality is the intersection of the cinematic type of story and the interactive story of any of these game references. With Virtual Reality and Augmented Reality, you get to take those two areas of pursuit and combine them together into one. Anybody that plays games enough knows what a cut scene is or a cinematic scene. For a very long time, you would play the game at a various state of fidelity. Then when you got into the cut scene, the fidelity would go up. You wouldn’t get to do anything, you just now have to watch. The controller has no function right now. It’s an achievement, and it’s propelling the story forward. It always was a bit of a weird break point. If you’re activating and making decisions in a game, when suddenly you’re asked to make no decisions for two and a half minutes, the two sides of your brain conflict with each other. It’s hard to tell a story if you need to interrupt active storytelling with passive storytelling.
	Whereas, in the world of VR, once I’ve brought you into that world, I can blend those things in a much more nimble and effective way. I can point to story elements and capture you without having to remove you from the interactive moment. I can have a character start to address you and give you some narrative and propel you in a certain way. That’s hard to do in a console game. It’s innately perfect to do in a VR game because you’re asking to create reality. Theatrics. Magic. The chemistry’s better.
John Bucher:	What are the big challenges that we’ve got to overcome in VR to tell better stories?
Ted Schilowitz:	Getting the storytellers up to speed on understanding that this is a new medium, a new way to tell stories. I’ll correct myself. I wouldn’t actually say it’s all that new. I’ve made reflections into the theme park world—into what Walt Disney saw when he was making movies and realized he could do more than make movies. He realized he could actually bring people into a world. That’s actually the physical space of a theme park. Everything about that is a creationist’s world. The minute you walk through that gate. The minute you go down Main Street and the minute you go into Frontierland or Adventureland or Tomorrowland, he’s created a spatial sense of entertainment where you drop into things. He and the people in Imagineering had a realization that you could break those boundaries down. They recognized that the illusion is not just a screen. The illusion is every single thing about what happens when you get into an environment. Now we can do that artificially. We can create new environments in VR. That will enable us to begin to craft better stories.

Insights from the Visionaries

In any discipline or medium, we must have those who look past the current state of the craft to the future. Those individuals act as lighthouses in the storms of experimentation. We need their forward thinking for direction and guidance as we attempt to transcend what we know to be possible and look to what instead might be. Anyone in the world-building arena of VR likely has some amount of vision. However, leaders will always emerge that point the way toward what will likely lie ahead. Larry Rosenthal compares world building to our relationship with nature, saying it is simply mediation. Matt Thompson’s VR experience, Lucy, is certainly a mediation with what an afterlife experience might be like. Paul Debevec’s work equally strives to create a mediated experience with the viewer, aiming at making that mediation as transparent as possible. All of the visionaries in this section and throughout the book take the responsibility of mediated experiences between the creator and the viewer very seriously.

Branch storytelling was mentioned by Chris Milk in Chapter 5 as one concept that will likely be important in the future of VR storytelling. Brian Rose speaks of the idea in similar terms. The team at Google continues to look for ways to explore the potential of combining social VR with branch storytelling within the confines of current technology. While the limits confine the process now, creators are confident that these barriers won’t exist forever. They want to be ready with these more complex stories that allow the audience even greater agency, as soon as engineers and scientists are able to solve the tech issues. Rosenthal is confident that the progress the gaming community has made with branch storytelling will likely initially inform the early VR efforts. This reminds us of the importance of working with those both inside and outside of immersive communities in pushing forward VR techniques and methods.

Ted Schilowitz stated that humanity tends to find a way to drift toward more of a utopian than dystopian future. The more and more powerful the tools get, the more and more chance that the dystopian part can actually be more powerful. The section on ethics in Chapter 5 could not be more important in an age when our technology could quickly take us into uncharted waters we never dreamed possible. The stories we tell ourselves about who we are and who we might become take on even greater significance when our mediums of storytelling become more transparent and indistinguishable from reality. Our technology must continue to drive us toward a greater capacity for good, empathy, and humanity. Anything less leads to a reality that is not only emotionally empty but potentially bankrupt of that essential quality we desire as people journeying through life—meaning. Virtual Reality has offered us a new way to share our thoughts and experiences. It is a new canvas for artists to create with. Paul Debevec suggested that stories are how we store information about history and the lessons that we can take from that history, allegorically. The stories we craft within VR will remain a record of who we were on a great number of levels for generations to come.