9.2 3D Capture and Display Systems
Chapter 4 discussed 3D content construction techniques. These constructed 3D contents can be represented in different formats including the conventional stereo video (CSV) format, the mixed resolution stereo (MRS) format, and video plus depth (V+D) format. Figure 9.2 shows an example of using CSV, MRS, and V+D to represent a 3D image. The three formats also use two different ways to capture the real world: one is to capture the real world with two sensors where one represents the left eye and the other represents the right eye. The other is to capture the real world with one traditional sensor and one depth sensor. In the following, we give a short description of two-sensor capturing systems and color-depth-sensor capturing systems.
On 22 July 2009, Fujifilm announced its FinePix Real 3D System which is a complete 3D solution to capture, edit, and display stereoscopic images and videos. The first device in the system is a digital 3D camera called the FinePix Real 3D W1. There are two CCD sensors on the camera to capture the world in stereoscopic images or videos. An autostereoscopic 3D LCD is integrated into the camera to preview the possible shots and watch the captured images and videos. The previewing ability allows users to adjust the 3D effect of images and videos interactively. The second device is a digital 3D viewer, FinePix REAL 3D V1, which is also implemented with autostereoscopic display technologies to see and edit the captured stereoscopic images and videos. For general consumers, Fujifilm provides a total solution from 3D contents shooting to 3D contents viewing. Furthermore, Fujifilm continues to improve its 3D cameras' capture and display abilities and announced its FinePix Real 3D W3 in 2011. Panasonic, Sony, and JVC have also launched their own stereo camera projects and announced their own 3D cameras which are Panasonic TM-750, Sony HDR-TD20V, and JVC GS-TD1. In addition to buying a new 3D cameras, users can also choose to attach an auxiliary 3D lens to certain cameras for capturing 3D contents. For example, Panasonic has released an auxiliary 3D camera lens, Lumix Micro G Microsystem, for their G/GF series cameras.
Figure 9.2 The top two images record Chiang Kai-Shek Memorial Hall with the left view image and right view image. The middle two images record Chiang Kai-Shek Memorial Hall with the full-resolution left view image and half-resolution right view. The bottom two images record Chiang Kai-Shek Memorial Hall with the left view image and corresponding depth map of the left view.
In addition to 3D cameras, Maya which is a famous 3D animation software, also provides 3D camera functions (as shown in Figure 9.3) to create pairs of stereo cameras in the virtual 3D environment. Designers could first create a pair of stereo cameras for rendering the virtual scene in the stereoscopic images. Then, they can adjust properties of the pair of cameras. For example, the interaxial separation, which is the distance between two eyes, can be adjusted to give different depth perception. Figure 9.4 is an illustration of an exemplar stereo scene created by Maya.
Figure 9.3 (a) The user interface to initiate the stereoscopic camera setting for animation creation. (b) The exemplar camera setting condition.
Figure 9.4 (a) An anaglyph image rendered by the camera setting using the Autodesk Maya system. (b) The capturing condition and scene for the left 3D stereoscopic image.
According to the discussion in Chapter 5, video plus depth is a popular choice for transmission. Therefore, a depth map or “disparisty map” must be constructed or captured along with the color map. For those videos which were shot in 2D, the corresponding depth map for each frame must be created. This process is called 2D-3D conversion, and is discussed in Chapter 4. However, automatic 2D-3D conversion algorithms are still not mature enough to convert just any 2D images and videos. Therefore, manually creating the depth map for each frame in those 2D videos is still the only robust algorithm. Wang et al. [1] provide an interactive interface called Stereobrush to use strokes of different depth values to construct all possible layers with different depth values. The application allows users to construct the depth map for each frame efficiently and effectively.
Manual depth map construction is generally used in the movie industry to transform 2D movies to 3D, but this post-processing procedure is still tedious work for general consumers. Therefore, more and more depth capturing devices are being introduced and developed to generate a depth map of the world. When combining a color sensor and a depth sensor, the video plus depth 3D contents can be directly constructed without any post-processing procedure. The most famous and popular one is Kinect which is designed to be a new user interface to control game characters. Kinect provides not only intuitive control of elements in the 3D gaming environment but also abilities to capture 3D video plus depth contents. Therefore, Kinect has become a popular human–computer interaction device in the worldwide and therefore a large number of applications have been developed. For example, Kinect can be used to assist surgeons to check the conditions inside the patient and help them visualize the twists, turns, and branches of the aorta for navigating catheters inside the blood vessels during the surgical procedure. This surgical procedure has successfully been conducted in Guy's and St Thomas's hospitals in London. Benko et al. [2] propose Mirage Table which uses Kinect as a control in an AR application to directly interact with the virtual object in the 3D environment. Developing other types of human–computer interaction based on Kinect is still a hot research topic. Additionally, the precision of real-time video plus depth capturing devices is still limited. Mechanically and algorithmically improving the precision is also another popular research topic.