Chapter 4. Component Technologies of Head-Mounted Displays

Head-mounted displays for virtual and augmented reality come in a variety of forms, from single eye information displays to fully occluding stereoscopic headsets such as the HTC Vive and Oculus Rift. At the heart of each are two primary component assemblies; image displays and optics. In this chapter we explore the diverse range of these core enabling technologies, identifying the strengths and limitations of each solution.

In this chapter, our primary emphasis will be to build foundational concepts concerning the technical performance of these basic subsystems. To this end, we begin with defining basic display categories, followed by reviewing the most common image displays and optical systems.

A summary of these display categories and the primary advantages and disadvantages of each is shown in Table 4.1.

As shown in Figure 4.2, an LCD is an electronically modulated transmissive display device composed of two sheets of polarizing material, the axes of which are aligned perpendicular to each other. Between the polarizing sheets is an array of cells arranged in rows and columns containing liquid crystal molecules. These cells are enclosed on each side by thin glass substrates, upon one of which are printed millions of precisely positioned tiny transistors. Electrical current passed through the liquid crystal in each cell causes the crystals to come into and out of alignment. Varying the current passing through the liquid crystal material allows each cell in the array to serve as a shutter modulating the passage of light.

For color LCDs, an additional substrate is added to the stack containing red, green, and blue filters, or masks, each of which is precisely positioned over an individual cell. Three RGB liquid crystal cells (known as subpixels) form one pixel. When an electrical current is applied, each cell modulates the passage of light based on the intensity of the signal. The combined output from the three subpixels produces the color of the light seen by the observer. If all three subpixels are fully open, the result is a white dot, the color of the backlighting, which is explained next.

Because liquid crystal materials do not emit their own light, the light must be provided by a separate source. Although small LCDs such as those in watches often use reflected ambient light, modern color LCD panels such as those found in some head-mounted displays typically rely upon illumination provided by blue LEDs at the edge of the display panel, which is filtered through a yellow phosphor, and then spread via a diffuser to create a pseudo white light. This light is, in turn, further filtered into different colors by the color mask layer of the liquid crystal elements.

There are two main types of OLED displays: Passive Matrix OLED (PMOLED) and Active Matrix OLED (AMOLED). The difference is in the electronics used to drive the display. A PMOLED display uses a complex electronic grid to sequentially control individual pixels in each row (or line) in the display and does not contain storage capacitors. Thus, update rates are slow and the power draw is high to maintain a pixel’s state. In general, PMOLED are used for simple character and iconic displays.

Most applicable to near-eye virtual and augmented reality displays is the AMOLED, a basic design for which is shown in Figure 4.4. Similar to the control of an Active Matrix LCD, an AMOLED is driven by a thin film transistor (TFT) layer that contains a storage capacitor to maintain each subpixel’s state. This provides a significant level of control over individual pixels, including the ability to completely switch them off allowing for deep blacks and high contrast.

In many ways, AMOLEDs are far superior to LCDs. These advantages include the fact that AMOLEDs can be very thin because there is no need for backlighting (they produce their own light), and they have lower power consumption, fast refresh rates, low persistence, excellent color reproduction, and high resolutions. As will be see in Chapter 6, “Fully Immersive Displays,” most commercially available head-mounted displays on the market at the time this book was prepared utilize AMOLED modules as their image sources.

As shown in Figure 4.6, the DLP DMD chip architecture and small size enables considerable flexibility in their integration within near-eye displays for both virtual and augmented reality. In one orientation, the DMD micromirror array is positioned directly in front of the user’s eyes. The array is then illuminated from either the side or the bottom by low power 3-in-1 RGB LEDs, with the light then transferred via optics into the user’s eye to form an image on the retina. A second orientation places the DMD chip off to the side, with the reflected light transferred via waveguide (see the next section) into the user’s eye. In either configuration, the eye itself serves as the last element in the optical chain, with the retina serving as the display surface (Bhakta et al., 2014).

DMDs represent one of the fastest display technologies in existence. The speed of color refresh and low latency, the high resolution for such a small form factor (a 0.3-inch array diagonal enables a 1280 × 720 image), flexible illumination direction, and low power requirements make DMD an ideal imaging solution for head-mounted displays

As shown in Figure 4.7, an LCoS device is a combination of both transmissive and reflective technologies. Constructed on a silicon wafer overlaid with a highly reflective coating, LCoS uses nematic or ferroelectric liquid crystal to modulate field sequential colored light reflected off this backplane. Because control circuits are beneath the reflective surface behind each pixel (as opposed to surrounding each pixel as in traditional LCD architectures), there are no obstructions in the light pathway, enabling the generation of substantially clearer images.

Similar to the DLP DMD solution covered earlier, the LCoS chip architecture and small size enable considerable flexibility in their integration within small form factor near-eye displays for both virtual and augmented reality. A powerful example of one such implementation is the Google Glass monocular head-mounted display.

As shown in Figure 4.8, a side-projecting red, green, and blue (RGB) micro-LED array is used to produce field-sequential color illumination. This light is steered through a wedge fly-eye microlens element (shown in Figure 4.9), where it illuminates the reflective inner surface of an LCoS chip. Upon exiting the chip, the resulting modulated light (for each color channel) is reflected off a polarizing beamsplitter and further down the optical path of the display, where it is ultimately steered into the user’s eye.

Lumen—A lumen (lm) is the unit of measure for quantifying luminous flux (defined next), the total quantity of visible light energy emitted by a source. Most light measurements are expressed in lumens. You will frequently see this unit of measure used in relation to displays based on projection technologies.

Luminous flux—Luminous flux is a quantitative expression of light energy per unit of time radiated from a source over wavelengths to which the human eye is sensitive (380 nm–740 nm). The unit of measure for luminous flux is the Lumen (lm). One watt of radiant power at 555 nm, the wavelength to which a healthy human eye is most sensitive (a yellowish green color), is equivalent to a luminous flux of 680 lumens (Meyer-Arendt, 1968).

Luminous intensity—Luminous intensity is the luminous flux per solid angle emitted or reflected from a point. The unit of measure for expressing luminous intensity is the lumen per steradian, or candela (cd).

Candela—A candela is the unit of measure for quantifying luminous power per unit solid angle emitted by a point light source in a particular direction.

Luminance—Luminance is the measure of luminous intensity per unit area projected in a given direction. Luminance describes the amount of light that passes through, is emitted or reflected from a particular area, and falls within a given solid angle. The unit of measure for expressing luminance is candela per square meter (cd/m²). Readers will also frequently see luminance expressed in “nits” or footlamberts (fL). 1 fL = 3.426 cd/m².

Illuminance—Illuminance is the luminous flux incident on a surface per unit area. The unit of measure is the lux (lx), or lm/m² (lumen per square meter). For a given luminous flux, the illuminance decreases as the illuminated area increases.

Brightness—Brightness is purely a subjective attribute or property used to express the luminance of a display. Brightness is perceived, not measured. From a technical standpoint, the words brightness and luminance should not be interchanged (although they frequently are). Brightness is not a measurable quantity.

Spatial resolution—The term spatial resolution refers to the number of individual pixel elements of a display and is presented as numerical values for both the vertical and the horizontal directions. For instance, the spatial resolution of the Oculus Rift CV1 is 1200 × 1080 resolution per eye.

Pixel pitch—Pixel pitch refers to the distance from the center of one pixel (RGB triad, LED cluster, and so on) to the center of the next pixel measured in millimeters. This concept is illustrated in Figure 4.10.

Fill factor—When referring to pixel-based visual displays, regardless of type, the phrase fill factor refers to the amount of black space between individual pixel elements. Because these types of displays are typically composed of a precision array, an excessive black area between pixels can give the visual effect of a fine grid overlay or “screen door” effect when the display surface is viewed close up.

When expressing the phenomenon, the phrase high fill factor refers to minimal black spacing between pixels. A low fill factor would be an expression of excessive black spacing between pixels. This phenomenon is illustrated in Figure 4.11.

Of the four primary imaging technologies covered in this chapter (LCD, AMOLED, DLP, and LCoS), DLP has the least perceptible spacing between pixels (or the highest fill factor) due to the exceedingly small size of the individual mirrors comprising the reflecting surface of the chip.

Persistence—The term persistence refers to the length of time a pixel remains illuminated. Full persistence refers to when pixels remain illuminated for the time span of an entire frame. Low persistence refers to when pixels remain illuminated for only a fraction of the time span of an entire frame. Increasing refresh rate decreases persistence time. Persistence is typically represented in percentages.

Latency—The term latency refers to the elapsed time (measured in milliseconds) between the movement of a head-mounted display and the actual display elements updating to reflect the new scene view. Latency is by no means just a display issue, but the totality of time required for all system components to complete their processing tasks and updated imagery to be displayed.

Response time—The term response time refers to the measure of time a pixel takes to change. Measured in milliseconds, the longer the response time, the more pronounced are the visible image artifacts. Low response times are crucial for head-mounted viewing devices.

Color gamut—The color gamut of a display defines a specific range of colors a device is capable of producing. Although there are a variety of methods used to express a display’s color gamut, the most common are the xy chromaticity diagram of the XYZ color system established by the International Commission on Illumination (CIE), as well as the percentage of the NTSC (RGB) color space.

Contrast—Contrast is a measure of the relative difference between light and dark areas of a display. LCDs tend to be low contrast display, whereas AMOLEDs are high contrast.

Collimation of light such that an image source appears at a distance greater than its physical distance from the wearer’s eyes;

Magnification of the image source such that an image appears larger than the actual source;

Relay of light patterns from an image source, sometimes located off axis, to the wearer’s eyes (Melzer et al., 2009).

As detailed in the previous section, there are a variety of imaging and display technologies employed within head-mounted viewing devices. Each separate implementation requires its own specific optical design, whether it is to enable users to focus on a flat panel array positioned within inches of their eyes or to relay imagery generated by a microdisplay in an off-axis position such as the side of a display device and into their field of view. In many ways optics are the greatest challenge, and often the limiting factor, in the design of head-mounted viewing devices. Each optical element added to a system means an increase in size and mass of the overall display.

Another significant challenge comes as a result of the fact that head-mounted displays are human-centered systems. The complexity and sensitivity of our sensory mechanisms demands precision and accommodation; otherwise, problems arise, including discomfort, nausea, and an overall poor end user experience.

As shown in Figure 4.12, for a healthy adult, the average horizontal binocular FOV, which is the foundation for stereopsis and is important for depth perception, is 120^°, with a total FOV measuring approximately 200^°. Our vertical FOV is approximately 130^°.

As discussed in greater detail in Chapter 21, “Human Factors Considerations,” proper IPD settings are extremely important because poor eye-lens alignment can result in image distortion, the effect of which can be eye strain and headaches and may contribute to the onset of nausea (Ames et al., 2005). Incorrect settings can also impact ocular convergence and incorrect perception of the displayed imagery.

Although single lens, nonpupil forming optical designs result in lighter, more compact displays, with a large eye box, the trade-off is that they impose significant distortion in the process of bending the light field. The effect is known as pincushion distortion and is illustrated in Figure 4.14. To counteract this effect, images displayed within these viewing devices must be rendered with an equal and opposite barrel distortion to cancel out the effect.

On the other hand, pupil-forming optical system architectures such as those frequently used in combination with DLP DMD and LCoS microdisplays offer significantly greater flexibility in the overall design of head-mounted viewing devices due to the ability to fold, bend, or otherwise manipulate optical tracks to accommodate different design objectives. Figure 14.15 shows a number of optical elements, beam splitter, and combiner configurations used within these architectures.

Although the basic concept of internal reflection is fairly straightforward, from this point the display technology grows in complexity, principally in the means through which light from an image source is introduced into (the in-coupling) and out of (the out-coupling) the waveguide. In general, there are four primary types of waveguides currently employed within existing commercially available augmenting displays:

Holographic Waveguide—Holographic waveguides use holographic optical elements as the in- and out-coupling mechanisms. This technique is currently used within the Sony Smart Eyeglass displays and is illustrated in Figure 4.16.

Diffractive Waveguide—A detractive waveguide contains surface relief gratings with features sometimes measuring in the nanometer range. Detractive waveguides, a basic illustration for which is shown in Figure 4.17, are currently used within Microsoft HoloLens, various Vuzix displays, and a variety of defense and avionics optical systems.

Polarized Waveguide—A polarization waveguide such as shown in Figure 4.18 is a multipart assembly embedding a parallel array of partially reflecting polarized surfaces and multilayer coatings to direct light toward the eye of the viewer. This method was developed by and used within the Lumus DK-50 AR glasses.

Reflective Waveguide—A reflective waveguide such as illustrated in Figure 4.19 uses a single planar light guide with at least one semireflective mirror. This technique is used by Epson in its Moverio product line as well as Google Glass.

But these are necessary steps. It is because of these limitations, compromises, and trade-offs that researchers have spent decades digging into the problems, learning how our visual system functions, and deciphering the best ways to format and present structured light patterns to our eyes. Given the host of new display technologies under development, from depth indexed projections using oscillating optical fibers to holographic imaging technologies, it is abundantly clear that the flat panel-based head-mounted displays capturing the attention of the public today will be considered relics in just a few short years.