Introduction and Definitions
For most people, virtual reality (VR) is first and foremost a gaming medium. Is it true that numerous studies show how massive and important this market is about to become with the release of the PlayStation VR headset as well as the HTC Vive with its dedicated VR-ready Steam interface. For others, VR will revolutionize healthcare, architecture, education, etc. However, what about storytelling?
This book is specifically about “VR filmmaking,” which means it is about how to create virtual reality experiences for entertainment purposes, whether it is a fiction or documentary, and whether it is shot with a VR camera (this is called “live-action VR”) or created in game-engine software (“game engine-based VR”).
The book is divided into two parts: Part I is devoted to theoretical fundamentals and techniques. You will find there a summary of VR history, an in-depth exploration of the current technologies for both live-action VR (cameras, workflow, sound, etc.) and game engine-based VR (software, photogrammetry, volumetric capture, etc.), and a chapter dedicated to the VR headsets and other human-VR interfaces. In Part II, we focus on the storytelling aspect of VR in order to provide a comprehensive and global description of it. We compare VR with other existing arts; explore what VR brings to the table in terms of storytelling, and how to use it. Finally, the last chapter describes the making of a VR experience from start to finish, from script to distribution.
You have in hand two books in one and I warmly encourage you to focus on the chapters that interest you the most, even if it means skipping some others if necessary.
Shari Frilot, Founder/Chief Curator of New Frontier at Sundance Film Festival
I never really thought about this but I had a flashback moment where I remember starting Sundance’s New Frontier VR selection and one of the reasons why I think we were able to succeed was because we were able to make the case with YouTube becoming a new venue for filmmakers and YouTube also becoming the second largest search engine after Google. You were starting to see films evolve in their role in society. They were not just films that we watched and were entertained. They weren’t even things that we learned from, but they were things that we were using to communicate to one another. We would send shorts back and forth to describe and communicate things, and that has very much come to bear in terms of how we communicate with one another using the moving image as the media landscape and the infrastructure. The media architecture, the story architecture has enabled the role of cinematic storytelling to evolve.
VR presents the same kind of thing. We’ll see how it’s adopted on the consumer market and how many people get into headsets, but certainly the technology of being able to go online and share VR is here and will get exponentially better.
VR as a gadget that delivers an experience is really compelling but it can also change our storytelling culture profoundly. Not only how we tell stories but why. The role of storytelling as a communications platform.
First and foremost, let’s define what virtual reality really means as well as other terms that are used in this book. “Virtual reality” is an ensemble of visuals, sounds, and other sensations that replicate a real environment or create an imaginary one. A person using virtual reality equipment like a VR headset is able to “look around” the virtual world and sometimes interact with it (see below).
Virtual Reality and 360° Video
There is an ongoing debate regarding the difference between the terms “virtual reality” and “360° video.” Some like to say that VR only applies to game engine-based experiences as it allows for freedom of movement. The most common opinion – and the one adopted in this book – is that we use the term “VR” when the content is watched in a VR headset and “360° video” when it is watched on a flat screen, using a VR player. This means that the same content can be both VR and 360° depending on which platform is used to display it.
VR platforms include (but are not limited to) Steam, Oculus Store, or Samsung Gear VR. 360° platforms include YouTube 360, Facebook, etc.
Figure 0.1 VR vs 360°
In VR, the audience has a more active role. Or should we say the players? Virtual reality as an art form seems to be the missing link between gaming and traditional filmmaking, and the audience is neither exactly a spectator, nor a player. This is the reason why we will use the term “participant” to describe the viewer/user/audience from now on.
Cinematic VR and Interactive VR
Maxwell Planck, Founder Oculus Story Studio, Producer of “Dear Angelica,” “Henry”
I believe that cinematic VR and interactive VR are two different media. The visitor feels dramatically different when they can only look around vs. when they can move around in a space. Both have amazing potential for art, but I personally find interactive VR to give the visitor a stronger feeling of presence which I find most compelling in VR. The challenge of interactive VR is that our visitors have such a strong feeling of presence that they want agency as well. It’s much easier to tell a story where the storyteller is in full control and the audience is a passive participant. But, even though it’s an incredibly hard problem, I believe that if you can create an experience where a visitor feels like they are part of a story, and not simply listening to a story, it can have a memorable and powerful impact.
We’ve started exploring how to tell a story and reward agency at Oculus Story Studio by making stories that allow our visitors to have different views of the story since they can move around, sit down, lean forward, etc. We’ve also played with having story moments that would wait for the visitor to look in a particular direction, or get close to a character or set piece. And on “Dear Angelica,” we actually change how the illustration behaves depending on how close our user gets to the line work.
I’m a big believer that we’ll be able to get to storytelling with agency in VR, but it will take time and lots of experimentation. And the best way to experiment is to create many experiences, innovating with each project.
Virtual reality is divided into two categories: cinematic VR and interactive VR. Cinematic VR is closer to filmmaking than gaming, and interactive VR is the other way around. In a cinematic VR experience, the participants are immersed in a 360° sphere and the only agency they have is to look around. In an interactive VR experience, the participants can sometimes interact with objects and characters, choose various outcomes of a situation, and move freely in the environment. Cinematic VR is akin to traditional storytelling, where the audience is invited to “sit and relax” and witnesses the story being told. Interactive VR is akin to gaming, where the audience must have a more active role. This passive/active separation does not necessarily mean cinematic VR is third-person only. Some cinematic VR experiences are indeed first-person: The participant is a character in the story; he/she is acknowledged but cannot interact with the story. Conversely, an interactive VR experience can very well be third-person.
AR, MR, and AV
Virtual reality is only the tip of the iceberg when it comes to new immersive technologies and the impact they will have on our daily life. Augmented reality (AR) is projected to become a much bigger market than VR, especially in industries such as defense, architecture, engineering, and health. Augmented reality overlays digital imagery onto the real world with the use of dedicated glasses like the Microsoft HoloLens, Google Glass, or Magic Leap. MR means “mixed reality.” As of today, the definition of what MR really is remains unclear. Some say it is the merging of real and virtual worlds where physical and digital objects co-exist and interact in real time. The term “mixed reality” is also used when a participant in VR is filmed and composited (comped) in the virtual world he/she is in. This technique is sometimes used to advertise VR headsets and apps.
Figure 0.2 A mixed reality system
Last but not least, AV means “augmented virtuality,” which is a form of mixed reality where physical objects are dynamically integrated into, and can interact with, the virtual world in real time. This book focuses solely on the art and craft of virtual reality, and not AR, MR, or AV.
Let’s dive in!