Organization of the book
This book is organized as three parts:
- principles of 3D visual systems: 3D graphics and rending, 3D display, and 3D content creation are all well covered
- visual communication: fundamental technologies used in 3D video coding and communication system, and the quality of experience. There are discussions on various 3D video coding formats and different communication systems, to evaluate the advantages of each system
- advances and applications of 3D visual communication
Chapter 1 overviews the whole end-to-end 3D video ecosystem, in which we cover key components in the pipeline: the 3D source coding, pre-processing, communication system, post-processing, and system-level design. We highlight the challenges and opportunities for 3D visual communication systems to give readers a big picture of the 3D visual content deployment technology, and point out which specific chapters relate to the listed advanced application scenarios.
3D scene representations are the bridging technology for the entire 3D visual pipeline from creation to visualization. Different 3D scene representations exhibit different characteristics and the selections should be chosen according to the requirement of the targeted applications. Various techniques can be categorized according to the amount of geometric information used in the 3D representation spectrum; at one extreme is the simplest form via rendering without referring to any geometry, and the other end uses geometrical description. Both extremes of the technology have their own advantages and disadvantages. Therefore, hybrid methods, rendering with implicit geometries, are proposed to combine the advantages and disadvantages of both ends of the technology spectrum to better support the needs of stereoscopic applications. In Chapter 2, a detailed discussion about three main categories for 3D scene representations is given.
In Chapter 3, we introduce the display technologies that allow the end users to perceive 3D objects. 3D displays are the direct interfaces between the virtual world and human eyes and these play an important role in reconstructing 3D scenes. We first describe the fundamentals of the human visual system (HVS) and discuss depth cues. Having this background, we introduce the simplest scenario to support stereoscopic technologies (two-view only) with aided glasses. Then, the common stereoscopic technologies without aided glasses are presented. Display technologies to support multiple views simultaneously are addressed to cover the head-tracking-enabled multi-view display, occlusion-based and reflection-based multi-view system. At the end of this chapter, we will briefly discuss the holographic system.
In Chapter 4, we look at 3D content creation methods, from 3D modeling and representation, capturing, 2D to 3D conversion and, to 3D multi-view generation. We showcase three practical examples that are adopted in industrial 3D creation process to provide a clear picture of how things work together in a real 3D creation system.
It has been observed that 3D content has significantly higher storage requirements compared to their 2D counterparts. Introducing compression technologies to reduce the required storage size and alleviate transmission bandwidth is very important for deploying 3D applications. In Chapter 5, we introduce 3D video coding and related standards. We will first cover the fundamental concepts and methods used in conventional 2D video codecs, especially the state-of-the-art H.264 compression method and the recent development of next generation video codec standards. With common coding knowledge, we first introduce two-view video coding methods which have been exploited in the past decade. Several methods, including individual two-view coding, simple inter-view prediction stereo video coding, and the latest efforts on frame-compatible stereo coding, are presented. Research on the depth information to reconstruct the 3D scene has brought some improvements and the 3D video coding can benefit from introducing depth information into the coded bit stream. We describe how to utilize and compress the depth information in the video-plus-depth coding system. Supporting multi-view video sequence compression is an important topic as multi-view systems provide a more immersive viewing experience. We will introduce the H.264 multiple view coding (MVC) for this particular application. More advanced technologies to further reduce the bit rate for multi-view systems, such as the multi-view video plus depth coding and layered depth video coding system, are introduced. At the end of this chapter, the efforts on the 3D representation in MPEG-4, such as binary format for scenes (BIFS) and animation framework extension (AFX), are presented. The ultimate goal for 3D video system, namely, the free viewpoint system, is also briefly discussed.
In Chapter 6, we present a review of the most important topics in communication networks that are relevant to the subject matter of this book. We start by describing the main architecture of packet networks with a focus on those based on the Internet protocol (IP) networks. Here we describe the layered organization of network protocols. After this, we turn our focus to wireless communications, describing the main components of digital wireless communications systems followed by a presentation of modulation techniques, the characteristics of the wireless channels, and adaptive modulation and coding. These topics are then applied in the description of wireless networks and we conclude with a study of fourth generation (4G) cellular wireless standards and systems.
To make 3D viewing systems more competitive relative to 2D systems, the quality of experience (QoE) shown from 3D systems should provide better performance than from 2D systems. Among different 3D systems, it is also important to have a systematic way to compare and summarize the advances and assess the disadvantages. In Chapter 7, we discuss the quality of experience in 3D systems. We first present the 3D artifacts which may be induced throughout the whole content life cycle: content capture, content creation, content compression, content delivery, and content display. In the second part, we address how to measure the quality of experience for 3D systems subjectively and objectively. With those requirements in mind, we discuss the important factors to design a comfortable and high-quality 3D system.
Chapter 8 addresses the main issue encountered when transmitting 3D video over a channel: that of dealing with errors introduced during the communication process. The chapter starts by presenting the effects of transmission-induced errors following by a discussion of techniques to counter these errors, such as the error resilience, error concealment, unequal error protection, and multiple description coding. The chapter concludes with a discussion of cross-layer approaches.
Developing 3D stereoscopic applications has become really popular in the software industry. 3D stereoscopic research and applications are advancing rapidly due to the commercial need and the popularity of 3D stereoscopic products. Therefore, Chapter 9 gives a short discussion of commercially available products and technologies for application development. The discussed topics include commercially available glass-less two-view systems, depth adaptation capturing and displaying systems, two-view gaming systems, mobile 3D systems and perception, and 3D augmented reality systems.
In the final chapter, we introduce the state-of-the-art technologies for delivering compressed 3D content over communication channels. Subject to limited bandwidth constraints in the existing communication infrastructure, the bit rate of the compressed video data needs to be controlled to fit in the allowed bandwidth. Consequently, the coding parameters in the video codec need to be adjusted to achieve the required bit rate. In this chapter, we first review different popular 2D video rate control methods, and then discuss how to extend the rate control methods to different 3D video streaming scenarios. For the multi-view system, changing the viewing angle from one point to another point to observe a 3D scene (view switching) is a key feature to enable the immersive viewing experience. We address the challenges and the corresponding solutions for 3D view switching. In the third part of this chapter, we discuss the peer-to-peer 3D video streaming services. As the required bandwidth for 3D visual communication service poses a heavy bandwidth requirement on centralized streaming systems, the peer-to-peer paradigm shows great potential for penetrating the 3D video streaming market. After this, we cover 3D video broadcasting and 3D video communication over 4G cellular networks.