Introduction

Virtual Reality (VR) and Augmented Reality (AR), as well as their related subcategories, Mixed Reality (MR) and Augmented Virtuality (AV), are categorized by software, hardware and content. VR artist and author Celine Tricart (2018) describes “Virtual Reality” content as “an ensemble of visuals, sounds and other sensations that replicate a real environment or create an imaginary one,” while AR content “overlays digital imagery onto the real world.” When VR and AR are used together, we call it “mixed reality.” The final subcategory is AV, which includes interaction with physical objects via Haptics, which refers to a sense of “touch” usually via a controller or input device as “many haptic systems also serve as input devices” (Jerald, 2016). Finally, we “use the term ‘VR’ when the content is watched in a VR headset,” but call it “360° video when it is watched on a flat screen using a VR player” (Tricart, 2018). VR or AR will be used here, with the understanding that these include their subcategories of MR, AV and 360° video (also referred to as “Magic Window” in the industry).

Typically, VR is watched on a stereoscopic (a distinct image fed to each eye) head mounted display (HMD) that employs inertial measurement unit tracking (IMU) that “combine gyroscope, accelerometer and/or magnometer hardware, similar to that found in smartphones today, to precisely measure changes in rotation” (Parisi, 2016). HMDs fall into three broad categories: smartphone-based, tethered (requiring a computer and software development kit or SDK), and all-in-one (also referred to as stand-alone, similar in function to a tethered HMD, but not tethered to a computer). Tethered and all-in-one HMDs allow for Active Haptics, 6 Degrees of Freedom (DOF), which means you can move around, and Agency, which means you can guide the experience. Playstation VR or Oculus Rift are examples of tethered HMDs that come with controllers (Haptics).

The all-in-one HMD is relatively new, with many, including Oculus Go, Vive Focus, and others (Conditt, 2017) announced for release in 2018. Smartphone based HMDs use a smartphone compatible with the HMD, but have only 3 DOF and no Agency, which translates into limited movement and user control of the experience. Google Cardboard and Samsung Gear VR are both smartphone HMDs. VR can also be projected with a Cave Automatic Virtual Environment (CAVE) system, “a visualization tool that combines high-resolution, stereoscopic projection and 3D computer graphics that create the illusion of a complete sense of presence in a virtual environment” (Beusterien, 2006). What all VR systems have in common is that they block out the real world and are therefore considered immersive.

AR is viewed through glasses or smart devices such as a smartphone (not housed in an HMD), which do not occlude the real world as seen by the wearer and are therefore not immersive. As of this writing, the number of AR glasses in use is limited, with the notable exception of the 2015 release of the Microsoft HoloLens, which uses “a high-definition stereoscopic 3D optical see-through head-mounted display (and) uses holographic waveguide elements in front of each eye to present the user with high-definition holograms spatially correlated and stabilized within your physical surroundings” (Aukstakalnis, 2017). However, the popularity of Pokémon Go proves that AR can be engaging, even through a smartphone.

AR is projected to be a major market in the future, and there are a number of recent software development kits (SDKs) that should drive content creation. “A software development kit (or SDK) is a set of software development tools that allows the creation of applications for a certain software package, software framework, hardware platform, computer system, video game console, operating system, or similar development platform” (Tricart, 2018).

VR content can be “live” action, computer generated (CG) or both. Live action VR content is captured via synchronized cameras or a camera system using multiple lenses and sensors, with individual “images” being “stitched” together to form an equi-rectangular or “latlong” image (similar to how a spherical globe is represented on a flat map). VR experiences are usually posted in a non-linear edit system (NLE) or Game Engine and Digital Audio Workstation (DAW). VR content is also streamed live for sporting events (usually viewed as 360° video) though this application has required specialized high end professional VR cameras such as the Ozo 360° stereoscopic camera (Kharpal, 2015).

VR games are built in game engines, for example, Unity 3D or Unreal. Since game engines include “video” importing and exporting capabilities, VR content is also created in game engines. These game engines are free and have been very popular with both professional and amateur content creators, but 3D (stereoscopic) 360° VR cameras have been very expensive, and therefore only affordable for big budget productions

While this chapter primarily addresses entertainment applications, Table 15.1 lists many examples of VR/AR companies and products for non-entertainment industries. These examples were sourced from Practical augmented reality: A guide to the technologies, applications, and human factors for AR and VR (Aukstakalnis, 2017) where full descriptions can be found. In particular note that real estate, architecture and flight simulators are robust markets for VR.

Background

“Virtual reality has a long and complex historical trajectory, from the use of hallucinogenic plants to visual styles such as Trompe L’Oeil” (Chan, 2014). Even Leonardo da Vinci (1452–1519) “in his Trattato della Pittuara (Art of Painting) remarked that a point on a painting plane could never show relief in the same way as a solid object” (Zone, 2007), expressing his frustration with trying to immerse a viewer with only a flat frame.

Then in 1833 Sir Charles Wheaton created the stereoscope, a device that used mirrors to reflect two drawings to the respective left and right eye of a viewer, thus creating a dual-image drawing that appeared to have solidity and dimension (Figure 15.1) Wheaton’s device was then adapted into a handheld stereoscopic photo viewer by Oliver Wendall Holmes, and “it was this classic stereoscope that millions used during the golden age of stereography from 1870 to 1920” (Zone, 2007).

Source: Wikimedia Commons-public domain

“Google’s Cardboard as an inexpensive mass-produced VR viewer is almost identical in purpose and design to its analog ancestor, Holmes’ patent-free stereoscope, from over 150 years ago” (Tricart, 2018). Later stereoscope was popular in early films. However, by the 1930s, film moved from capturing reality to more complex narrative styles, while stereoscope was still primarily for “reality” gimmick films. It is the limit of stereoscope for story-telling, more so than the need for glasses, that noted stereographer and author Ray Zone (2007) argues led to its marginalization by the mid-1900s.

Between the mid-1900s and 1960, there were a few “mechanical” VR devices, most notably Edwin Links’ flight simulator created in 1928, with 10,000 units being sold principally to the Army Air Corps by 1935 (Jerald, 2016), but the next real big innovation involved the computer.

In 1968 Ivan Sutherland and Bob Sproull constructed what is considered the first true VR head-mounted display, nicknamed the Sword of Damocles, which was “somewhat see through, so the user could see some of the room, making this a very early augmented reality as well” (Tricart, 2018). By the 1980s NASA developed a stereoscopic HMD called Virtual Visual Environment Display (VIVED), and Visual Programming Language (VPL) laboratories developed more HMDs and even an early rudimentary VR glove (Virtual Reality Society, 2017). During this same time VR creators, most notably VPL laboratories, turned their attention to the potential VR entertainment market. By 1995 the New York Times reported that Virtuality managing director Jonathan Waldern predicted that the VR market would reach $4 billion by 1998. However, this prediction ended with “Most VR companies, including Virtuality, going out of business by 1998” (Jerald, 2016). At this time limited graphic ability and latency issues plagued VR, almost dooming the technology from the start. Still, “artists, who have always had to think about the interplay between intellectual and physical responses to their work, may play a more pivotal role in the development of VR than technologists” (Laurel, 1995). During this same time, artist Char Davies and her group at SoftImage (CG creator for Hollywood films) created Osmose, “an installation artwork that consists of a series of computer-generated spaces that are accessed by an individual user wearing a specialized head mounted display and cyber-vest” (Chan, 2014). Osmose is an early example of the “location based” VR experiences that are re-emerging today.

In 2012 Oculus released the Oculus Rift SDK I, a breakthrough tethered stereoscopic HMD (software and hardware) with a head-tracking sensor built into the lightweight headset (Parisi, 2016). Oculus Rift SDK1 and SDK 2 (both tethered HMDs) were released to the public, but between the cost of the hardware and the required computer, the final cost remained well over $1,000 and required that the user have some computer knowledge. AR was also introduced via Google Glass in 2012, but many issues from limited battery life to being “plagued by bugs” and even open hostility towards users (because of privacy concerns) led to them being quickly pulled from the market (Bilton, 2015). Then in 2014 Facebook bought Oculus for $2 billion (Digi-Capital, 2017a) while Google cardboard was introduced at the Google I/O 2014 developers’ conference. The Google cardboard SDK was available for both Android and iOS operating system’s and could be handmade from simple cardboard instructions available on the web (Newman, 2014). In 2015 Microsoft announced HoloLens, “a high-definition stereoscopic 3D optical see-through head-mounted display which uses holographic waveguide elements in front of each eye to present the user with high-definition holograms spatially correlated and stabilized within your physical surroundings” (Aukstakalnis, 2017). Meanwhile, Valve, which makes Steam VR, a virtual reality system, and HTC collaborated to create HTC Vive, a tethered HMD (PCGamer, 2015). During the same year, Oculus partnered with Samsung on an Android smartphone HMD, the Samsung Gear VR, releasing a smartphone HMD in November 2015 (Ralph & Hollister, 2015) with this lower cost smartphone HMD outselling both Vive and Occlulus Rift tethered HMDs (Figure 15.2).

By 2016, Sony released the Playstation VR, which, while “tethered,” could be used with the existing Playstation 4, selling the most “tethered” HMD units in 2016 (Liptak, 2017). During these years, most VR or AR was proprietary, with manufacturers only working with specific smartphones or only supporting their own platforms. The exception was Google Cardboard, which was biggest seller in the VR and AR market, outselling all other HMDs (tethered and smartphone) as well as AR glasses combined, with estimates by 2017 of over 10 Million sold (Vanian, 2017). While it is clear that cost, ease of use, and cross platform compatibility strongly affected consumer adoption, it is most important to note that the overall costs dropped and access to content increased, thereby increasing overall consumer adoption of VR and AR technologies.

Source: H.M.V. Marek, TFI

In North America’s Silicon Valley in 2015 alone, “AR/VR startups raised a total of $658 million in equity financing across 126 deals” (Akkas, 2017). In Vancouver, “Creative B.C. invested $641,000 through its interactive fund in 14 different VR companies. By July 2016, there were 168 mixed reality companies, and 5,500 full-time employees in B.C.” while “4 billion RMB ($593 million USD) has been invested in VR in China in 2015 and 2016; a large majority funded by the Chinese government” (Akkas, 2017).

In addition, although not directly related to sales at the time, in 2014 the Mozilla team (Mozilla Firefox web browser) and a member of the Google Team First announced “a common API between the two browsers…using this new API, the programmers could write VR code once and run it in both browsers. At that moment WebVR was born” (Parisi, 2016).

Recent Developments

The release of the smartphone HMD drove a rapid increase in sales, including even more newcomers such as the Daydream, a higher end smartphone HMD announced at the Google I/0 16 developers’ conference (Holister, 2016). However, when 2016 actual sales numbers were released in 2017, they fell far short of projections for 2016. “Where at the start of the year we thought 2016 could deliver $4.4 billion VR/AR revenue ($3.8 billion VR, $0.6 billion AR), the launch year’s issues resulted in only $2.7 billion VR revenue. This was counterbalanced by Pokemon Go’s outperformance helping AR to an unexpected $1.2 billion revenue, for a total $3.9 billion VR/AR market in 2016” according to the Digi-Capital Corporation, a Swiss company managing equity investments in the entertainment industry (2017a).

AR exceeded projections in 2016 but must overcome some major inhibitors to move from the early adopter stand to a majority stage in the consumer market. According to Digi-Capital (2017b): “5 major challenges must be conquered for them [AR glasses] to work in consumer markets: (1) a device (i.e., an Apple quality device, whether made by Apple or someone else), (2) all-day battery life, (3) mobile connectivity, (4) app ecosystem, and (5) telco cross-subsidization.” A major inhibitor to VR growth has been the need for a specific app, headset and browser. In a 2017 interview, Antti Jaderholm, Co-founder and Chief Product Officer at Vizor, said, “We see that currently about 15% of viewers view the experiences in VR. That number has remained stable for a while, haven’t seen a big change in the last year or so…85% consume WebVR experiences from our site without any headset” (Bozorgzadeh, 2017). 2017 saw several advances addressing the above inhibitors to consumer adoption.

In 2017 Google announced advances to WebVR and the release of ARcore, while Apple announced ARKit. On the VR hardware side, Oculus announced the Oculus Go, an all in one HMD priced under $200 and HTC announced the Vive Stand Alone (Greenwald, 2017). On the AR hardware side, Magic Leap announced all-in-one consumer AR glasses (Conditt, 2017). Combined, these addressed the major inhibitors to VR and AR adoption.

Source: SuperData Research

Source: Strategy Analytics

At the Google I/O 2017 conference, Megan Lindsay, Google’s WebVR Project Manager, in her presentation “Building Virtual Reality on the Web,” (2017) announced advances to WebVR that “opens VR up … to content from any device through any web browser,” then Lindsay went on to say that “content is absolutely critical to the success of any new ecosystem…WebVR opens VR up to the largest developer in the world…web developers.” While initially available on Daydream, Google’s own mobile VR platform, Google is working cross platform with other HMDs and other browsers such as Mozilla (who initially started WebVR and involved Google) towards all HMDs working on all VR-enabled browsers.

Who are web developers? Everyone, and that is the point. We are seeing an increase in WebVR development tools, for example Amazon’s Sumerian (in Beta at this writing) “lets you create and run virtual reality (VR), augmented reality (AR), and 3D applications quickly and easily without requiring any specialized programming or 3D graphics expertise” Amazon Web Services, 2017).

Apple and Google also introduced their AR frameworks ARKit and ARCore, respectively, to allow developers to create mobile AR experiences. “The wide support of mobile AR presents an exciting opportunity for marketers. AR is no longer limited to social media apps like Snapchat; and instead a set of natively integrated tools on both iOS and Android, enabling developers to build cool, engaging applications” (Yao and Elchison, 2017).

SDKs, whether you are developing content on the web, iOS or Android, have many uses including VR, AR or mixed MR (Alexandar, 2017).

It is also important to note that while free game engines such as Unity 3D or Unreal, which work with 3D 360° video, have been very popular with both professional and amateur content creators, stereoscopic VR cameras were priced only for big budget productions. Affordable live action 3D 360° video VR cameras, such as Human Eye’s Vuze camera, priced under $1,000 (see figure 15.5) have entered the market, with a new Vuze+ camera to be released in 2018 that even allows for live VR streaming for under $1,200 (Antunes, 2017).

Source: Human Eye’s Ltd.

VR has evolved from early suggestions that it might be the new “film” to its own category, Cinematic VR Experience, with most major film festivals (Sundance, Tribecca, Sundance, etc.) having a VR category. The Academy of Motion Picture and Sciences awarded its first VR award “in recognition of a visionary and powerful experience in storytelling” to director Alejandro Gonzalez Iñárritu for his virtual reality installation “Carne y Arena” (Virtually present, Physically invisible) (Robertson, 2017a). Top cinematic VR experience companies include Felix & Paul, who produced among many, Obama’s White House tour, “The People’s House,” which won an Emmy (Hayden, 2017), and companies such as Jaunt VR with multiple Cinematic VR originals (Jaunt Studios, 2017), plus major film and television studio and sports news partnerships.

Finally, distribution has opened up. For example, “Paramount has partnered with MoveoPlus to provide Moveo Virtual Reality Simulators in theaters and theme parks (BusinessWire, 2017). Disney has partnered with Jaunt, a VR company currently boasting roughly $101 million in investment capital, to create content (Ronald, 2017) and the Void, which builds location-based VR experiences” (Fickley-Baker, 2017). Much of this is an extension of the concept of transmedia, i.e., reselling the same story franchise across multiple experiences. As just one example, Star Wars VR experiences such as Star Wars Droid Repair Bay, powered by Steam VR (a virtual reality system) are available for home use, or “bigger” (e.g., on a better HMD, CAVE type environment or simply as an “outing”), at IMAX theatres for about $1 a minute (IMAX VR, 2017). Other Star Wars experiences, such as Star Wars: Secrets of the Empire are also at Void locations (The Void, 2017).

Current Status

Recent developments in hardware (all-in-one displays, including both HMDs and AR glasses) and software (SDKs) are fueling predictions and current sales. According to industry analyst Canalys (2017), 2017 Q3 saw shipments totaling over 1 million for the first time in any quarter, namely companies creating both infrastructure and content.

Factors to Watch

Now that someone has tasted VR, how much more will they want? For example, how many of the 85% of 360° Magic Window viewers will purchase an HMD to watch VR on the web. Of course, it is not as simple as just having the headwear. In looking back historian Ray Zone noted stereoscope wasn’t “killed” by the technology, but content, as stereoscopic film was used primarily as “gimmick” or “novelty” (Zone, 2007) Fortunately, VR/AR content creators today are actively exploring how “plot in the classic sense may not be the only type of high-level narrative structure” in VR and AR worlds (Aylett, 1999). Will affordable 3D 360° camera’s increase the number of indie-artists making VR experiences as DSLR cameras increased the number of film makers? Certainly, developments including WebVR, ARcore, and Arkit represent a collaborative push by the major companies to make content easier to create. Faster data rates could make a difference, too; for example, Google’s Seurat demonstrated at Google I/0 ‘17, compressed content that used to take an hour to transmit down to 13 milliseconds, noting how this enabled easier “content creation” by VR and AR artists. Now we must watch to see if more content creation will lead to more quality content, enticing both consumer adoption and longterm consumer interest.

We also need to watch emerging potential problems with widespread VR and AR usage. For example, there is some concern over long-term effects of VR and AR on an individual and society. In the medical field, it’s been demonstrated that VR is so powerful that it is actually more effective than opioids in pain management (Hoffman, 2004) but the medical field is also studying negative effects. “Neuroscientists at UCLA have been conducting research on how the use of virtual reality systems affect the brain… the hippocampus, a region of the brain involved in mapping and in individual’s location in space showed 60% of neurons shut down” (Aukstakalnis, 2017). This is only one study, so it will be important to see the results of more studies. Clearly, more Big Data will be collected if individuals are constantly online via smart glasses, so concerns about Big Data’s use—or mis-use—will intensify. There is also a phenomenon known as the “Uncanny Valley” effect, in which CG people appear too “real” resulting in an “evil” quality many viewers find unsettling. Finally, there are some practical questions of safety while using the various headgear. This is not a comprehensive list, and many of these types of concerns may dwindle or be overcome, but when a medium is predicted to be as pervasive in our lives as VR and AR, potential adverse outcomes should be watched.

Getting a Job

VR and AR will exist across many industries and so there are multiple points of entry: programmers, designers, creative coders, cinematographers, animators, visual special effects artists, audio engineers and mixers, editors, special effects editors and many of the same jobs you see in movie credits today. As of this writing, the mode of entry is very closely related to modes of entry into other media areas, such as film, television and gaming. A good place to start is an institution of higher learning where students study the fundamentals of media technology, craft, professional workflows, artistry, and theory. It is important to note that there is a big gap between student skill level and professional production, so internships are also very important as a gateway to employment.