Chapter 6. Fully Immersive Displays
Virtual reality is here. No longer limited to university and government labs, major advances in several core enabling technologies have given rise to the first generation of high-quality, mass-produced head-mounted displays that are now available to general consumers. In this chapter we look into the details of the latest generation of commercially available, fully immersive head-mounted displays across several classes ranging from PC and console-driven devices to lower end systems based on modern smartphones.
PC-Console Driven Displays
We are living in an extraordinary time in history. There have been few instances since the introduction of the basic desktop computer mouse that we have experienced a genuine paradigm shift in the manner in which average individuals interact with computers and other complex information systems. Although there has been excitement surrounding the very concept of being able to enter high-fidelity, computer-generated worlds for some time, such experiences have been limited to those facilities and organizations with deep pockets and specific missions to fulfill. But times are changing. In this section we will explore a number of PC and console-driven head-mounted displays currently dominating the general consumer market as well as overall public attention. A basic overview is provided for each highlighting unique characteristics and features, as well as a detailed table of device specifications.
Oculus Rift CV1
What started in 2012 as a crowd-funded product development effort for a small company known as Oculus has resulted in a massive resurgence in the public’s awareness of, and excitement for, fully immersive virtual reality systems. Following two prerelease developer models (DK1 and DK2) and the start-up’s acquisition by social media giant Facebook, Inc., the first consumer version of the display is now shipping to customers and it is, without a doubt, an impressive piece of engineering.
As shown in Figure 6.1, the Oculus Rift CV1 (Consumer Version-1) is a lightweight stereoscopic wide field of view (FOV) head-mounted display based on two low-latency AMOLED flat panels with a 1080 × 1200 resolution per eye. Of particular note within this display is the use of highly specialized, free-form hybrid Fresnel lenses, a side view of which is shown in Figure 6.2. In addition to the irregular lens surface, the Fresnel diffraction pattern is extremely tight. This combination of a nonrotationally symmetric surface and Fresnel pattern make these lenses extremely difficult to manufacture, although highly effective at providing the user an immersive 110° horizontal FOV with minimal distortion or artifacts.
Credit: Image by eVRydayVR via Wikimedia
Figure 6.1 The Oculus Rift CV1 (Consumer Version -1) is the first commercial headset produced by Facebook-owned Oculus VR.
Credit: Image courtesy of iFixit
Figure 6.2 This side view of an optical element from the Oculus Rift CV1 (Consumer Version -1) shows the nonrotationally symmetric inner lens surface.
The asymmetric optics used in this display also enable partial binocular overlap, which makes the horizontal FOV appear wider at the expense of binocular rivalry.
Position and orientation of the device is tracked using the combination of an internal IMU (accelerometer, gyroscope, and magnetometer) as well as a “constellation” of IR LEDs built into the display housing and support strap that are visible to a separate IR camera, even when the user is turned away from the sensor. The system is designed for use in either a standing or a seated position and has an effective operating area measuring 5 × 11 feet.
More detailed specifications for the Oculus Rift CV1 display hardware can be found in Table 6.1.
HTC Vive
Roughly concurrent to the early development efforts surrounding the Oculus Rift, entertainment software developer Valve Corporation of Bellevue, Washington, and HTC, the multinational manufacturer of smartphones and tablets headquartered in New Taipei City, Taiwan, also began a collaboration to develop their own high-end, PC-driven virtual reality display. Initially unveiled to the public in early 2015 and formally referred to as the HTC Vive, the system is now also shipping to customers and is itself an exceptional display.
As shown in Figure 6.3, the HTC Vive is a lightweight stereoscopic wide field of view head-mounted display based on two low-latency AMOLED flat panels with a 1080 × 1200 resolution per eye. Using its own set of proprietary optical elements with a Fresnel diffraction grating, the Vive provides the user with a solid 110o horizontal FOV with minimal distortion or artifacts.
Credit: Images courtesy of HTC
Figure 6.3 The HTC Vive is a first-of-its-kind virtual reality system designed from the ground up for room-scale immersive experiences.
At this point, the Vive and Oculus part ways as other key system functions are handled using very different methods. From the beginning of design efforts, the HTC Vive has been developed with “room scale” tracking technologies in mind in an effort to allow users to navigate naturally by walking around, as well as to use motion-tracked hand controllers to intuitively manipulate objects and interact with the simulation.
As opposed to using a camera to track an array of IR LEDs mounted on the display, the Vive system uses a reverse approach. First, two IR laser base stations are positioned at opposite ends of a room and emit precisely timed IR pulses and X/Y axis IR laser sweeps. An array of IR sensors embedded in the display housing detects these IR pulses and laser sweeps in various combinations. From this information, the system is able to calculate the position and orientation of the display with considerable speed and precision. This system is covered in greater detail in Chapter 11, “Sensors for Tracking Position, Orientation, and Motion.”
Another impressive capability of the Vive system that takes user safety into consideration is the “Chaperone” feature, which uses the front-facing camera in the headset to capture and display a representation of the user’s physical environment, including walls and obstacles, if the user gets too close.
More detailed specifications for the Vive display hardware can be found in Table 6.2.
Sony PlayStation VR
Years in the making and intended for console gaming, Sony Computer Entertainment Inc. has introduced PlayStation® VR (PS VR), a virtual reality system that uses the PlayStation® 4 (PS4) gaming console to deliver highly compelling immersive simulations. Originally referred to as Project Morpheus when initially teased in 2014, the new high-performance peripheral has generated significant excitement because it does not require a high-end gaming PC to operate, such as in the case of systems like the HTC Vive and Oculus Rift CV1 mentioned earlier in this chapter.
The PS VR package consists of two major components: a visor style, stereoscopic wide FOV head-mounted display and what is referred to as a breakout box. The display (shown in Figure 6.4) is based on a single shared 1920 × 1080 AMOLED panel providing 960 × 1080 per eye along with proprietary optics providing an approximate 100-degree FOV. The system uses built-in inertial sensors and the PlayStation Camera to accurately track the position and orientation of the display (that is, the user’s head) as well as a PlayStation Move (PS Move) Motion Controllers in three-dimensional space.
Credit: Image courtesy of Dronepicr via Wikimedia under a CC 2.0 license
Figure 6.4 Unlike PC-based virtual reality hardware, Sony’s PlayStation VR headset is immediately compatible with more than 30 million PS4 game consoles already in use.
The breakout box serves as an interface between the display and the PS4 console. Although the console itself easily handles all graphics-generation tasks, the breakout box handles object-based 3D audio processing, the reverse warping of imagery from one eye, and its output via HDMI to a television for what Sony refers to as social screen so others present can share in the gaming experience.
The PS VR system also provides what is referred to as cinematic mode, which allows users to enjoy a variety of nonstereoscopic content on a large virtual screen seen while wearing the headset. This content includes standard PS4 games, 360° photos, and videos captured by devices such as omnidirectional cameras via PS4 Media Player (Stein, 2016; Crossley, 2016). More detailed specifications for the PS VR display hardware can be found in Table 6.3.
OSVR–Open-Source VR Development Kit
One of the surefire ways for the newly emerging virtual and augmented reality industry to slow its own growth is by limiting interoperability between applications and peripheral devices. If gaming or professional software applications, or peripherals, are limited to only one platform or system, things become highly problematic and inconvenient for end users and adoption slows.
By inspiring developers to adopt basic standards early on, greater interoperability is achieved, resulting in faster growth of the industry. It is this particular strategy that has led to the development of OSVR (Open Source Virtual Reality), an open source hardware and software ecosystem for virtual reality development efforts.
The hardware system, known as the Hacker Development Kit and shown in Figure 6.5, consists of a high-quality stereoscopic, wide FOV head-mounted display and the associated cabling. The core of the display is based on a single low-persistence 1080p AMOLED panel (divided in half for separate left- and right-eye views) and crisp, custom design dual-element aspheric optics (lenses whose surface profiles are not portions of a sphere or cylinder). A powerful addition to this display is the diopter (focus) adjustment, a feature lacking in most other head-mounted displays covered in this chapter. Tracking the position and orientation of the display is enabled using an internal nine degree-of-freedom IMU along with a camera that tracks a faceplate embedded with IR LEDs.
Credit: Images by Maurizio Pesce via Flickr under a CC 2.0 license
Figure 6.5 Developed by Razer and Sensics, the OSVR development kit is an open licensed ecosystem intended for use in creating VR experiences across any operating system, including Windows, Android, and Linux.
Other innovative features for this display include the ability to convert a regular desktop video signal into side-by-side mode so that it can be viewed in the goggle, as well as the ability to accept both 1080 × 1920 and 1920 × 1080 video signals. This lets you use the display with a wireless video link.
The intent behind the release of this display hardware is to provide developers a wide open, nonproprietary platform for their own system development and testing. All aspects of the display have been designed to be hackable, including the actual designs themselves, which are freely available for download. More detailed specifications for the hardware components can be found in Table 6.4.
OSVR is also a software framework. The OSVR API is a platform-agnostic, standardized interface for virtual reality display devices and peripherals. Intended to serve the role of middleware, the OSVR application programming interface (API) is a set of high-performance rendering and device abstraction services enabling augmented reality/virtual reality (AR/VR) applications to provide near-universal support between head-mounted displays, peripherals such as input devices and trackers, and operating systems.
Smartphone-Based Displays
It is without question that one of the key fields enabling a resurgence of virtual reality is mobile telephony. From the ever-increasing performance of mobile processors, to display manufacturers continually packing more pixels into a given screen area, to the introduction of entirely new classes of sensors, these developments have broken through some of the long-standing barriers haunting this field, and in the process, unleashed a wellspring of creativity. One of the highest impact results is the development of simple tools that enable the use of smartphones themselves as the primary platform upon which to drive immersive simulations.
In this section we explore two of these smartphone-based display systems. Although there are dozens of manufacturers for these devices, each with its own take on the overall design, optics, materials, and added capabilities, most have been introduced as a result of the profound success of those appearing in this section.
Google Cardboard
Disruptive innovation is a term originally coined by Harvard Professor Clayton Christensen. It describes a process by which a product or service takes root initially in simple applications at the bottom of a market and then relentlessly moves up market, eventually displacing established competitors.
In the case of Google and its mid-2014 introduction of a smartphone-based virtual reality headset made of cardboard, cheap biconvex plastic lenses, a magnet, washer, and a few tabs of Velcro (see Figure 6.6), shockwaves were sent through high-tech centers around the globe, and in particular, through the meeting rooms of companies still trying to get their higher-end, PC-driven virtual reality displays into consumer hands. Now anyone with $15 and a smartphone could experience virtual reality (albeit in relatively crude form) and even get started developing their own applications with the basic software development kit (SDK) released at the same time.
Credit: Images by othree via Flickr under a CC 2.0 license
Figure 6.6 A Google Cardboard headset is a cheap and easily accessible standard for experimenting with virtual reality.
After the initial shock passed, most industry participants realized that Google’s action, while possibly a joking prod at Facebook for its $2 billion dollar purchase of Oculus not three months earlier, was also one of the most intelligent and high-impact moves this field could experience. Suddenly, there was a means available to allow the masses to experience at least a taste of what high-performance immersive virtual reality systems were capable of delivering once they entered the marketplace. The net effect was a massive jump in public awareness and enthusiasm, not to mention the rapid formation of several dozen new companies looking to get involved at some level, be it through producing their own kits with more durable display housings for sale or new content and app providers. As will be covered in Chapter 18, “Education,” Google itself has launched a program known as Google Expeditions within which educators in the K-12 sector now make use of the cardboard viewers to engage their classes in virtual field trips. As of November 2015, more than 100,000 students in schools around the world have used Expedition cardboard viewers in their courses.
Samsung GearVR
Whereas Google Cardboard and similar devices represent the low end of the smartphone-based display market, the Samsung GearVR device clearly dominates the higher end market. Developed in collaboration with Oculus VR, the headset was first unveiled in late 2014 and released for sale in 2015.
As shown in Figure 6.7, the GearVR headset serves as a viewing device and controller into which any of Samsung’s flagship mobile devices can be securely mounted. The system optics expand the display to encompass a 96o horizontal FOV with adjustable focus. Movement of the user’s head is tracked across three axes (roll, pitch, and yaw) using an internal IMU with a higher precision than that provided by the IMU within the mobile device. The power for this sensor as well as the side-mounted controls including a touchpad and back button is supplied by the mobile device via a microUSB connection.
Credit: Images by S. Aukstakalnis
Figure 6.7 Samsung Gear VR is a mobile virtual reality headset developed by Samsung Electronics in collaboration with Oculus VR. The device serves as a housing and controller for any of a variety of Samsung smartphones that acts as the headset’s display and processor.
More detailed specifications for the Samsung GearVR headset can be found in Table 6.5.
CAVES and Walls
A variety of large-scale, fixed location immersive displays exist for use within the scientific and professional communities. These systems come in a variety of geometries and sizes, including multisided (or multiwalled), rear-projection, or flat panel-based displays, single and multiprojector hemispherical surfaces, and more, each typically displaying field sequential stereo imagery at high resolutions. Most are designed to accommodate multiple users, each of whom wear LCD shutter glasses controlled by a timing signal that alternately blocks left- and right-eye views in synchronization with the display’s refresh rate. Most systems incorporate some method of tracking the position and orientation of a lead user’s head to account for movement and to adjust the viewpoints accordingly. In such multiuser scenarios, all other participants experience the simulations in 3D, but passively. An example of one such display is shown in Figure 6.8.
Credit: Image courtesy of Idaho National Laboratory
Figure 6.8 In this image a research scientist with Idaho National Laboratory views a subsurface geothermal energy model within a computer-assisted virtual environment (CAVE) display.
Located at the Center for Advanced Energy Studies in Idaho Falls, Idaho, the FLEX CAVE (computer-assisted virtual environment) is a four-walled, reconfigurable, projector-driven 3D display system that allows scientists to, quite literally, walk into and examine their graphical data. Within minutes, the system can be reconfigured into a flat wall display, an angled theater for team collaboration, an L-shape to serve as both an immersive 3D display alongside of a section for 2D data, or a closed room for exploring spatial structures and training applications.
A powerful variation on this reconfigurable room-sized display theme is shown in Figures 6.9 and 6.10. Known as the EmergiFLEX and manufactured by Mechdyne Corporation of Marshalltown, Iowa, the display is composed of a large articulating wall of 24 high-definition panels that can be moved into a variety of configurations, including that of a corner with one extended wall or an open-sided room-like display with an overhead, downward-facing projector.
Credit: Image courtesy of Mechdyne and UALR
Figure 6.9 The Mechdyne EmergiFLEX CAVE accommodates multiple users for collaborative design, engineering, and visualization applications.
Credit: Image courtesy of Mechdyne and UALR
Figure 6.10 The EmergiFLEX CAVE display is composed of an articulating wall of 24 high-definition Barco LED DLP projection cubes as well as overhead projectors to create floor imagery.
As shown in Figure 6.10, each of the three vertical wall sections consists of 8 Barco LED DLP projection cubes. Each cube operates at a 1920 × 1080 resolution. When the 24 cubes are configured as a single wall, the overall display surface measures approximately 10 feet high by 34 feet wide. Two high-resolution overhead projectors are used to create a blended floor image. The floor itself is made from a washable rigid material that has been optimized for front projection (in this configuration it is actually a downward projection) and for color matching with the side walls.
The system incorporates a 24-camera, six-degree-of-freedom optical motion tracking system (see Chapter 11) and is capable of both head and hand tracking regardless of the display system configuration. Here again, users wear polarized glasses to see the 3D graphics presented on the screens.
Initially developed by researchers at the University of Illinois at Chicago in the early 1990s under the name Cave Automatic Virtual Environment, a large number of variations for the overall room-sized immersive display model have been implemented by a handful of companies. Along with the varied number of implementations have come a seemingly equal number of names and acronyms, including computer-assisted virtual environment (shown above), computer-aided virtual environment, COVE, iCube, and still others.
A variety of these displays are detailed in different application case studies found later in this book. Examples of such uses covered include architectural walk-throughs, engineering design reviews, and additional examples of visualization of complex scientific datasets. A list of suppliers for these systems is provided in Appendix B, “Resources.”
Hemispheres and Domes
Large format displays are in no way limited to rectilinar geometries. Hemispheres and dome displays are a popular option in many industries, and in particular, those requiring considerable precision in visual representation, room for multiple users, as well as the need to be untethered. As an example, Figure 6.11 shows the advanced joint terminal attack controller training system (AJTS) used by the U.S. Army for skills development and training of joint terminal attack controllers (JTACs). The system simulates virtually any environment and most aircraft and weapons systems utilized by the JTACs.
Credit: Image courtesy of DoD
Figure 6.11 The advanced joint terminal attack controller training system (AJTS) simulator is a domed visual display system used by the U.S. Army to train joint terminal attack controllers (JTACs) and combat controllers.
Developed for the Army by QuantaDyn Corporation of Herndon, Virginia, the core AJTS configuration consists of a domed visual display system with high-resolution projectors, as well as a powerful and intuitive Computer Generated Force (CGF)/Semi-Autonomous Force (SAF) application. A high-fidelity image generator renders scenes in multiple spectrums. A dynamic aural cueing system adds highly realistic audio to supplement the visual display. The simulator is also capable of networking with other offsite simulators, enabling JTACs to communicate and train with real-life pilots in their own simulators at distant locations (QuantaDyn, 2013; Bruce, 2014).
It is important to note that within such large-format displays, it is not always necessary or even desirable to use stereoscopic imagery. As depicted in Figure 6.11, within many simulations there are often no near-field scene features necessitating an enhanced 3D view. Even with this feature’s absence, the wide FOV nature of the display still provides users with a highly compelling visual sensation of presence and immersion within the simulation.
Conclusion
Within this chapter we have explored a diverse number of fully immersive displays ranging from handheld and head-mounted stereoscopic viewers to large-scale, fixed placement multiuser systems. In each instance, the majority of the user’s FOV is filled with computer-generated or video imagery. Although most head-mounted devices provide a 360-degree viewing experience without interruption of the displayed scene, most of the large-format systems described in this chapter do not because CAVEs and domes have entrances or because the specific applications for which the system is intended do not require this feature.
As will be seen in the application case studies provided later this book, these displays are actively being used in a variety of entertainment, engineering, scientific, and training endeavors with outstanding results.
Most of the flat panel-based head-mounted displays currently available on the market will likely be considered “old technology” within just a few short years. Each of these devices has been engineered around displays originally intended for other purposes. As such, features such as the associated electronics, large, heavy optical elements, and oversized housings are required to bring about functional systems. Although impressive engineering feats and adequate for this stage of the developing industry, this category of displays will be rapidly displaced by new systems designed from the ground up with this application in mind. These trends are discussed in greater detail in Chapter 23, “The Future.”