Skype Multimedia Codecs

Skype supports video chat and voice call services. The name “Skype” was derived from “sky” and “peer-to-peer.” Skype communication uses a microphone, speaker, video webcam, and Internet services. Skype was founded by Niklas Zennström (Sweden) and Janus Friis (Denmark) based on the software created by Ahti Heinla, Priit Kasesalu, and Jaan Tallinn (Estonia) in 2003. On August 29, 2003, the first public beta version of Skype was released. In June 2005, Skype made an agreement with the Polish web portal Onet.pl which enabled integration into the Polish market. On September 12, 2005, eBay, Inc. agreed to acquire Skype Technologies SA for approximately $2.5 billion, and on September 1, 2009, based on Skype’s value of $2.75 billion, eBay announced it was selling 65% of Skype (to Silver Lake, Andreessen Horowitz, and the Canada Pension Plan Investment Board) for $1.9 billion. By 2010, Skype had 663 million registered users. On May 10, 2011, Microsoft acquired Skype for $8.5 billion and began to integrate Skype services into Microsoft products. On February 27, 2013, Microsoft introduced Office 2013 which included 60 Skype world minutes per month in the Office 365 consumer plans, which was for home and personal use as well as universities. On April 8–30, 2013, the Windows Live Messenger instant messaging service was phased out in favor of promoting Skype (but Messenger use continued in mainland China). On November 11, 2014, Microsoft announced that Lync would be replaced by Skype for Business in 2015, where later versions included combined features of Lync and the consumer Skype software. E–organizations were given the option to switch their users from the default Skype for Business interface to the Skype for Business (Lync) interface (https://en.wikipedia.org/wiki/Skype).

Skype is based on a hybrid peer-to-peer model and uses a client and server system architecture. Supporting applications include VoIP, video conferencing, instant messaging, and file transfer. Skype video conference calls can be made from various platforms using smartphones, tablets, and PCs. Desktop Client Operating Systems (OSs) including Windows and HoloLens as well as other desktop OSs which include OS X (10.6 or newer) and Linux (Ubuntu and others) support Skype.

Other mobile device OSs that support Skype include iOS (for Apple’s iPhone and iPad), Android (for various Android smart devices), BlackBerry 10, Fire OS, Nokia X, and many more (including support for selected Symbian and BlackBerry OS devices). Skype calls to other Skype users within the Skype services are free. Skype could also be used to make traditional phone calls to landline telephones and mobile phones using debit-based user account charges based on Skype Credit or by subscription. Similar technologies that support VoIP and video conferencing include Session Initiation Protocol (SIP) and H.323 based services which are used by companies like Linphone and Google Hangouts.

Skype video codecs before Skype 5.5 used VP7 as the video codec, but after 2005, Skype started to use the True Motion VP7 codec. In early 2011, Skype 5.5 moved to VP8. The video codecs True Motion VP7 and VP8 were developed by On2 Technologies for Google. Skype 5.7 uses VP8 for both group and one-on-one standard definition video chatting. But after Microsoft acquired Skype in 2011, the video codec was replaced to H.264.

The Skype video call quality is based on the mode being used. The standard mode has a video resolution of 320×240 pixels with 15 fps; high-quality mode has a resolution of 640×480 pixels with 30 fps. High definition (HD) mode has the highest resolution of 1280×720 pixels with 30 fps. Other H.264 codecs are also used in group and peer-to-peer (one-on-one) video chatting.

Skype initially used G.729 and SVOPC as its VoIP audio codec. Skype added SILK to Skype 4.0 for Windows and other Skype clients. SILK is a lightweight and embeddable audio codec created by Skype, where more details are provided in the following. Skype’s Opus is an open source codec that can integrate the SILK codec for voice transmission with Constrained Energy Lapped Transform (CELT) codecs for higher-quality audio transmissions (e.g., live music performances).

SILK is an audio compression format and audio codec that was developed by Skype Limited to replace the SVOPC codec. SILK was initially released in March 2009, and its latest software development kit (SDK) version 1.0.9 was released in 2012. SILK operation specs include the audio sampling frequencies of 8, 12, 16, and 24 kHz (i.e., 24,000 samples/s) resulting in a bitrate range of 6~40 kbps (bps = bits/s = bits per second). SILK has a low algorithmic delay of 25 ms, which is based on a 20 ms frame duration plus a 5 ms look-ahead duration. The reference programming of SILK is in C language, and the codec is based on linear predictive coding (LPC).

Skype also uses Opus as its audio codec. Opus was developed by Xiph and is standardized as RFC 6716 by the Internet Engineering Task Force (IETF). It was designed as a lossy audio coding format to efficiently encode speech and general audio into a single format. Opus was designed to combine SILK with CELT, in which switching between or combining SILK and CELT could be conducted as needed for maximum efficiency. Opus maintains low-latency sufficient for real-time interactive communications and was designed to have low complexity to run on low end ARM3 processors.

CELT is an open royalty-free lossy audio compression codec software algorithm format used by Skype. CELT has very low algorithmic delay, which is very beneficial in supporting low-latency audio communications. CELT was designed using lower latency Modified Discrete Cosine Transform (MDCT) technology.

YouTube Multimedia Codecs

YouTube is a global video-sharing website that uses H.264/MPEG-4 AVC, WebM, and Adobe Flash Video technology. Three former PayPal employees (Chad Hurley, Steve Chen, and Jawed Karim) created YouTube in February 2005. The cofounder Karim said that the inspiration for YouTube first came from the difficult experience in finding online videos of Justin Timberlake and Janet Jackson’s 2004 Super Bowl halftime performance exposing incident and later the 2004 Indian Ocean tsunami, where the need of a video-sharing site was realized. Cofounders Hurley and Chen said that the original idea for YouTube was an online dating service based on videos. On February 14, 2005, the domain name www.youtube.com was activated and on April 23, 2005, the first YouTube video titled “Me at the Zoo” was uploaded, which shows cofounder Karim at the San Diego Zoo (https://en.wikipedia.org/wiki/YouTube).

In September 2005, a Nike advertisement featuring Ronaldinho reached one million views for the first time. In November 2005 to April 2006, YouTube received a $11.5 million venture investment from Sequoia Capital, and on December 15, 2005, the www.youtube.com site was officially launched. In July 2006, YouTube announced that every day there were more than 65,000 new videos uploaded. In November 2006, YouTube was bought by Google, Inc. for $1.65 billion. In 2010, the company headquarters was moved to San Bruno, California, USA. In April 2011, YouTube software engineer James Zern revealed that 30% of videos accounted for 99% of YouTube’s views. In 2014, YouTube announced that every minute, 300 hours of new videos were uploaded. In 2015, YouTube was ranked as the third most visited website in the world (based on Alexa and SimilarWeb). In 2015, YouTube was attracting more than 15 billion visitors per month making it the world’s top TV and video website (based on SimilarWeb). In January 2016, the San Bruno headquarters (CA, USA) was expanded to an office park that could support up to 2,800 employees based on a facility investment of $215 million.

YouTube is a representative Internet video OTT service, which enabled user-generated content (UGC) video uploading. UGC is also called user created content (UCC), which is a video that is made and uploaded mostly by amateurs. This broke the traditional video service model of professionally generated content (e.g., TV programs or videos made in studios and broadcasting companies) and direct movie/video sales (e.g., video streaming companies like Netflix, Disney+, etc.). Now, video streaming companies also use OTT service models.

YouTube video technology used to require an Adobe Flash Player plug-in to be installed in the Internet browser of the user’s computer to view YouTube videos. In January 2010, YouTube’s experimental site used the built-in multimedia capabilities of HTML5 web browsers to enable YouTube videos to be viewed without requiring an Adobe Flash Player or any other plug-in to be installed. On January 27, 2015, HTML5 was announced as the default playback method. HTML5 video streaming uses MPEG Dynamic Adaptive Streaming over HTTP (DASH), which supports adaptive bit-rate HTTP based streaming by controlling the video and audio quality based on the status of the player device and network.

The preferred video format on YouTube is H.264 because older smart devices can easily decode H.264 videos more easily with less power consumption. Some older smart devices may not have a H.265 decoder installed, and even if it may be installed the power consumption and processing load may be too burdening for an older smart device. Therefore, if the wireless network (e.g., Wi-Fi, 5G, or LTE) is sufficiently good, H.264 could be the preferred video codec to use. This is why in some cases, when a user tries to upload a high-quality H.265 video to YouTube, the uploading YouTube interface may convert the H.265 video into a lower-quality H.264 format or multiple H.264 formats, which are used in MPEG-DASH. More details on MPEG-DASH are described in Chapter 6, and the technical details on YouTube recommended upload encoding settings can be found at https://support.google.com/youtube/answer/1722171?hl=en.

Netflix Multimedia Codecs

Netflix started multimedia streaming in 2007 using the Microsoft video codec VC-1 and audio codec Windows Media Audio (WMA). Currently, Netflix uses H.264 AVC and H.265 HEVC. H.264 AVC is used to support the CE3-DASH, iOS1, and iOS2 Netflix profiles. H.265 HEVC is used to support the CE4-DASH Netflix profile for UltraHD devices. CE3-DASH and CE4-DASH commonly use HE-AAC and Dolby Digital Plus for audio codecs, and iOS1 and iOS2 both use HE-AAC and Dolby Digital for audio codecs (https://en.wikipedia.org/wiki/Technical_details_of_Netflix).

Among existing Netflix encoding profiles, the following are deprecated (which means that although they are available, they are not recommended): CE1, CE2, and Silverlight use VC-1 for video and WMA for the audio codec; Link and Kirby-PIFF use H.263 for video and Ogg Vorbis for the audio codec; and Vega uses H.264 AVC for video and AC3 for the audio codec.

Netflix uses the multimedia container file formats of fragmented MP4 (FMP4) and the MPEG-2 transport stream (M2TS). Multimedia containers enable multiple multimedia data streams to be combined into a single file which commonly includes the metadata that helps to identify the combined multimedia streams. This is why multimedia containers are also called “metafiles” or informally “wrappers” (https://en.wikipedia.org/wiki/Container_format). More details on containers, Dockers, and Kubernetes will be provided in Chapter 8.

For digital rights management (DRM), CE3-DASH and CE4-DASH use PlayReady or Widevine, and iOS1 and iOS2 use PlayReady or NFKE. DRM enables control and policy enforcement of digital multimedia legal access and is based on licensing agreements and encryption technology. DRM technologies help monitor, report, and control use, modification, and distribution of copyrighted multimedia and software. There are various technological protection measures (TPM) that enable digital rights management for multimedia content, copyrighted works, and proprietary software and hardware (https://en.wikipedia.org/wiki/Digital_rights_management).

Disney+ Multimedia Technology

Disney Streaming started off as BAMTech in February 2015, which was founded through MLB Advanced Media. In August 2016, one-third of the company was acquired by the Walt Disney Company for $1 billion. In October 2018, BAMTech was internally renamed to Disney Streaming Services, and in August 2021, via Twitter, it was announced that the company was renamed to Disney Streaming (https://en.wikipedia.org/wiki/Disney_Streaming).

Luma and Chroma

Since we are studying video technologies, it is necessary to understand the terms “luma” and “chroma.” In simplest terms, “luma” is used to represent the image’s brightness level, and “chroma” is used to represent the image’s color, which is combined through red (R), green (G), and blue (B) light sources. This is why luma and chroma are commonly paired together.

The general term “brightness” is commonly used as a perceived concept and is not objectively measurable. This is why “luminance” and “luma” are used in science and engineering. Luminance is defined in a way that is both quantifiable and accurately measurable. In technical terms, “luma” is different from “luminance.” Based on the International Commission on Illumination (CIE)², application-wise, “luma” is used in video engineering (like the H.264, H.265, H.266 codecs and standards) where “relative luminance” is used in color science.

Luminance represents the level of brightness of a beam of light towards a direction. Luminance is represented by the luminance intensity of a beam of light based on the unit of candela per meter square (cd/m²), which is defined by the International System of Units. The unit candela (cd) is the Latin word for “candle,” which is the basis of brightness for this unit. One candela (i.e., 1 cd) is the luminous intensity of approximately one (ordinary) wax candle.

In video engineering, “relative luminance” is sometimes used to represent a monitor’s brightness. Relative luminance is derived from a weighted sum of linear RGB components. On the other hand, luma uses gamma-compressed RGB components.

The Gamma (γ) encoding applied in the luma computation helps to optimize the bits encoding efficiency of an image. Gamma encoding is based on the gamma correction nonlinear operation used to encode and decode luminance (Y) and luma (Y’) values. The basic equation has the form of Y’=AY^γ (other equations exist), where A=1 is commonly used. If γ<1, the encoding process is called gamma compression. If γ>1, the encoding process is called gamma expansion (e.g., γ=2.2). For the special case of γ=1 and A=1, there is no change, resulting in Y=Y’. The same rule can be applied to RGB as well using R’=AR^γ, G’=AG^γ, and B’=AB^γ (other equations exist).

Examples of how luma is computed using the weighted sum of gamma-compressed RGB video components are provided (please note that the actual process is more complicated and other equations exist). In the ITU-R Rec. BT.709, relative luminance (Y) is calculated based on a linear combination of R, G, B pure colorimetric values using Y=0.2126R+0.7152G+0.0722B. However, luma (Y’) is calculated in the Rec. 709 specs based on the gamma-compressed R, G, B components R’, G’, B’, respectively, using Y’=0.2126R’+0.7152G’+0.0722B’. For luma calculations, the commonly used digital standard CCIR 601 uses Y’=0.299R’+0.587G’+0.114B’, and the SMPTE 240M equation Y’=0.212R’+0.701G’+0.087B’ is used for transitional 1035i HDTV³.

Chroma is Greek for the word “color” and a shortened version of the word “chrominance.” The chroma components notation of Y′C_bC_r is commonly used to represent a family of color spaces, where the luma is Y′, the blue-difference is C_b, and red-difference is C_r (https://en.wikipedia.org/wiki/YCbCr).

For example, based on the ITU-R BT.601 conversion 8 bits per sample case, the digital Y′C_bC_r values are derived from analog R’G’B’ using the equations Y’=16+(65.481R’+128.553G’+24.966B’), C_b=128+(-37.797R’-74.203G’+112B’), and C_r=128+(112R’-93.786G’-18.214B’).

Human vision is more sensitive to notice changes in brightness than it acknowledges changes in color. This effect is used in “chroma subsampling” technology to reduce the encoded bitrate while maintaining the same perceived quality of an image to the human eyes (which means that there is no difference in the quality of the video to the viewer). Therefore, in chroma subsampling technology, more bits are assigned to the luma (Y’) representation, and less bits are assigned to the encoding of the color difference values C_b and C_r (https://en.wikipedia.org/wiki/Chroma_subsampling).

Chroma subsampling schemes are classified using a three number sequence S₁:S₂:S₃ or a four number sequence S₁:S₂:S₃:S₄. Chroma samples are commonly defined in a matrix format that has two rows of pixels and S₁ columns of pixels (which is commonly 4). S₂ represents the number of chroma samples (C_r, C_b) in the first row of the S₁ pixels. S₃ represents the number of changes of chroma samples (C_r, C_b) between the first and second row of the S₁ pixels. S₄ represents the horizontal factor. Figure 5-1 presents simplified examples of the popular chroma subsampling formats, which include 4:4:4, 4:2:2, 4:2:0, and 4:1:1.

A graphic of a 2 cross 4 heat map of Y prime is added with C subscript r and C subscript b to give a 2 cross 4 heat map of Y prime C subscript r C subscript b.

The H.264, H.265, and H.266 video codecs can use one or three color plane arrays of the pixel sample value bit depth. The luma color plane is the primary color plane, which is used to represent local brightness information. Chroma planes can be used to represent color hue and saturation. In the 4:4:4 format, all three color planes have the same resolution. In the 4:2:0 format, compared to the luma plane, the chroma planes have half the width and half the height. In the 4:2:2 format, compared to the luma plane, the chroma planes have the same height but half the width. The 4:4:4 format is used by computer desktop sharing and wireless display applications, and the 4:2:0 format is used by many consumer applications (https://ieeexplore.ieee.org/document/9503377).

H.264 Video Profiles

There are four H.264 video profile groups, which include the (1) non-scalable 2D video application profiles; (2) camcorders, editing, and professional application profiles; (3) scalable video coding (SVC) profiles; and (4) multiview video coding (MVC) profiles (www.itu.int/ITU-T/recommendations/rec.aspx?id=14659). As new updates to the standards are continuously published, the reader is recommended to recheck all standard updates before making a profile selection, as only the basic profiles are listed in the following.

H.264 AVC Encoder and Decoder

In this section, more details on the H.264 AVC encoder and decoder are provided (https://doi.org/10.1007/978-3-030-62124-7_12).

H.264 divides an image into a sequential group of macroblocks based on raster scan order, which each group is called a slice. Figure 5-2 presents an example of an image divided into slices.

A graphic of a square is divided into four uneven segments labeled slice 0, slice 1, slice 2, and slice 3.

A simplified H.264 AVC encoder is presented in Figure 5-3. The H.264 encoder includes integer transform, variable block-size motion compensation, quarter-pixel accuracy in motion vectors, multiple reference picture motion compensation, intra-frame directional spatial prediction, scaling, quantization, in-loop deblocking filter, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), entropy coding, etc.

A block diagram of H.264 A V C Encoder. It defines how the input image macroblock with an error is encoded via coder control and is processed under Intra or Interswitch.

A block diagram of H.264 A V C Decoder. It defines how the encoded bit stream is decoded via entropy decoding, reconstruction, Intra or Interswitch, and memory devices.

A simplified H.264 AVC decoder example is presented in Figure 5-4. The decoder operates in the reverse order of the encoder in reconstructing the original image from the compressed encoded bit stream. The decoder uses entropy decoding, inverse quantization, transform of residual pixels, motion compensation, inter-fame prediction, intra-fame prediction, reconstruction, in-loop deblocking filter, etc. in reconstructing the video images.

More details on the H.264 AVC encoder are provided in the following. File size compression is conducted using both intra-frame prediction and inter-frame prediction in H.264. Intra-frame prediction is used to compress the file size of an image using parts mostly within that individual frame. Inter-frame prediction is used to compress the file size of an image using parts mostly from other images (e.g., P and B frames), where more details on I, P, and B frames will be provided later in this chapter. In a video, motion changes take place over multiple images, so motion compensation must use inter-frame prediction algorithms. Since images of a video are interrelated, in-loop processing is used to provide the feedback loop operations needed to compress the video file (in the encoder) and decompress the video file (in the decoder) of the H.264 video codec system.

Integer Transform: After the macroblocks are obtained, integer transform is applied. H.264 uses a simple 4×4 discrete cosine transformation (DCT) with integer-precision. This is based on the fact that the residual pixels have a very low level of spatial correlation in the H.264 I and P frame prediction schemes. Earlier video codec standards used larger and more complex DCTs, which were a burden to the graphics processor and resulted in prediction shift errors due to the rounding errors that occurred in the floating-point calculations that were conducted in the DCT and inverse DCT (IDTC) transformation processes. An inverse integer transform is performed before the deblocking filter and will be used in the feedback loop of the H.264 encoding process.

DCT is a process that converts a time-domain signal into frequency-domain signals, which is a spectral signal decomposition technique. DCT changes a time-domain signal into a sum of multiple frequency domain signals (that are represented with cosine signals) that have different frequencies. Integer transformation is an integer DCT process, which is an integer approximation of the DCT. The integer approximation process enables selectable block sizes to be used, which results in exactly matching integer computations. The integer approximation significantly helps to reduce the computation complexity, which results in low complexity and no drifting. DCT alone (without integer approximation) is known to create undesired prediction shifts due to the rounding errors in the floating-point calculations conducted in the DCT and IDCT processes. Integer transform helps to improve the intra-coding spatial prediction accuracy, which is followed by transform coding.

Scaling and Quantization: After the integer transform, H.264 conducts a combined scaling and quantization process. The integer transform values go through a quantization process using a 4×4 quantization matrix that maps values to quantized levels. Then the scaling process is applied, which is a normalization process of values, where all values are mapped to values within 0 to 1. A scaling and inverse quantization process will be performed before the inverse integer transform is conducted in the feedback loop.

Entropy Coding: Entropy coding is a lossless data compression scheme that has the characteristics to approach the limit of data compression without losing any information or resolution of the image. H.264 uses three entropy coding schemes, which are the Exponential-Golomb (Exp-Golomb) code, context-adaptive variable length coding (CAVLC), and context-adaptive binary arithmetic coding (CABAC). H.264 entropy coding has two modes that are distinguished by the “entropy_coding_mode” parameter settings, which apply different encoding techniques. If the entropy_coding_mode = 0, the header data, motion vectors, and non-residual data are encoded with Exp-Golomb, and the quantized residual coefficients are encoded using CAVLC. Exp-Golomb is a simpler scheme than CAVLC. If the entropy_coding_mode = 1, then CABAC is applied, which is a combination of the binarization, context modeling, and binary arithmetic coding (BAC) processes. BAC is a scheme that results in a high data compression rate. Binarization coverts all nonbinary data into a string of binary bits (bins) for the BAC process. Context modeling has two modes, the regular coding mode and the bypass coding mode. Regular coding mode executes the context model selection and access process, where the statistics of the context are used to derive probability models, which are used to build context models in which the conditional probabilities (for the bins) are stored. Bypass coding mode does not apply context modeling and is used to expedite the coding speed.

CAVLC has a relatively complex process of quantizing the residual coefficients. CABAC is very effective in compressing images that include symbols with a probability greater than 0.5, in which algorithms like CAVLC are inefficient. For example, H.264 main and high profiles use CABAC for compression of selected data parts as well as the quantized residual coefficients. This is why CABAC is continuously applied in H.265 and H.266 entropy coding, but CAVLC is not.

Intra-frame Prediction: H.264 conducts a more sophisticated intra-frame directional spatial prediction process compared to earlier video codec standards (e.g., H.262 and H.263). Neighboring reconstructed pixels from intra-coded or inter-coded reconstructed pixels are used in the prediction process of intra-coded macroblocks. Each intra-coded macroblock can be formed using 16×16 or 4×4 intra prediction block size. Each intra-coded macroblock will be determined through a comparison of different prediction modes applied to the macroblock. The prediction mode that results in the smallest prediction error is applied to the macroblock. The prediction error is the difference (residual value of the subtraction process) between the predicted value and the actual value of the macroblock. The prediction errors go through a 4×4 integer transform coding process. In a macroblock, each 4×4 block can have a different prediction mode applied, which influences the level of the video encoding resolution in H.264.

Deblocking Filter: Due to the macroblock-based encoding mechanism applied, undesirable block structures can become visible. The deblocking filter has two general types (deblocking and smoothing) which are applied to the 4×4 block edges to eliminate the visible undesirable block structures. The deblocking filter’s length, strength, and type are adjusted considering the macroblock coding parameters and edge detection based spatial activity. The macroblock coding parameters include the reference frame difference, coded coefficients, as well as the intra-coding or inter-coding method applied.

Variable Block-Size Motion Compensation: In H.264, the default macroblock size is 16×16. Alternatively, dividing the 16×16 macroblock into four 8×8 partitions can be used. Each macroblock or partition can be divided into smaller sub-partitions for more accurate motion estimation. Based on the 16×16 structure, there are four M-type options (sub-partitions) that can be used, which include the 16×16, two 16×8 sub-partitions, two 8×16 sub-partitions, and four 8×8 sub-partitions. Based on the 8×8 partitions, there are also four 8×8-type options (sub-partitions) that can be used, which include the 8×8, two 8×4 sub-partitions, two 4×8 sub-partitions, and four 4×4 sub-partitions. These eight type options are presented in Figure 5-5, which are used as the basic blocks in the H.264 luma image inter-frame motion estimation process. In H.264, the motion compensation accuracy is at a quarter-pixel precision level for luma images. This process is conducted using a six-tap filtering method applied to first obtain the half-pixel position values, and these values were averaged to obtain the quarter-pixel position values.

Using these H.264 video compression techniques, I, P, and B frames are formed. I frames are intra (I) coded independent pictures that serve as keyframes for the following P and B inter-frame prediction processes. P frames are predictive (P) coded pictures obtained from inter-frame prediction of previous I and P frames. B frames are bipredictive (B) coded pictures that are obtained from inter-frame prediction of I, P, and other B frames.

An illustration of H.264 Macroblocks describes the 16 by 16, 16 by 8, 8 by16, and 8 by 8 M types, and 8 by 8, 8 by 4, 4 by 8, and 4 by 4 type segmentations.

Group of Pictures (GOP): Based on H.264 and related MPEG standards, the first and last frame in a GOP are I frames, and in between these I frames, there are P frames and B frames. The reference frames of a GOP can be an I frame or P frame, and a GOP can have multiple reference frames. Macroblocks in P frames are based on forward prediction of changes in the image. Macroblocks in B frames are predicted using both forward prediction and backward prediction combinations. In Figure 5-6, I₀ is the reference frame for P₁, I₀ and P₁ are the reference frames for P₂, and I₀ and P₂ are the reference frames for P₃. In addition, I₀, P₂, and P₃ are the reference frames for B₁.

H.264 can have a GOP structure that has no B frames, or use multiple reference frames, or apply hierarchical prediction structures.

A H.264 GOP with no B frames would reduce the compression rate, which means that the video file size would be larger. But not using B frames would enable the H.264 video playing device to use less memory (because B frame processing requires more memory to process) and require a lighter image processing load. A H.264 GOP with no B frames is presented in Figure 5-7, where there are four reference frames, which are I₀, P₁, P₂, and P₃.

A graphic of 5 frames of I subscript 0, P subscript 1, P subscript 2, B subscript 1, and P subscript 3 describes how the H.264 G O P is formed.

A graphic of 5 frames of I subscript 0, P subscript 1, P subscript 2, P subscript 3, and P subscript 4 describes how the H.264 G O P is formed.

A graphic of a hierarchy of frames from layer 1 to layer 3 are combined to form a set of frames in combined G O P.

Hierarchical Prediction Structure: A H.264 hierarchical prediction example is presented in Figure 5-8, where the GOP starts with I₀ and ends with I₁₂, where I₀ is the reference frame for P₁. Layer 1 is formed with three frames I₀, P₁, and I₁₂. Using layer 1 frames, the layer 2 frames B₂ and B₅ are formed. B₂ is predicted using I₀, P₁, and I₂₁. B₅ is predicted using P₁ and I₂₁. Using the layer 1 and layer 2 frames, the layer 3 frames B₁, B₃, B₄, and B₆ are formed. B₁ is predicted using I₀, P₁, and B₂. B₃ is predicted using I₀ and B₂. B₄ is predicted using I₀ and B₅. B₆ is predicted using I₁₂, B₂, and B₅. For technical accuracy, please note that the “+” sign used in Figure 5-8 indicates a combination of pixel information of image parts used in the inter-prediction process, and not that the actual image pixel values were just all added up. For example, “I₁₂+B₂+B₅” means that B₆ was predicted using the frames I₁₂, B₂, and B₅. In addition, in order to control the video compression efficiency, larger quantization parameters can be applied to layers that are formed later down the hierarchy sequence.

A graphic of 10 frames in 4 rows illustrates how the process of H. 264 M V C is mapped from view 0 to view 3 through views 1 and 2.

H.264 supports multiview video coding (MVC), which enables users to select their preferred views (if this option is available). MVC is especially important for new XR systems and advanced multimedia applications. For example, Free Viewpoint Video (FVV) services use MVC. Figure 5-9 shows a H.264 MVC process example based on four views. MVC takes advantage of the hierarchical prediction structure and uses the interview prediction process based on the key pictures. Among the views, View 0 uses I frames as its reference frames, whereas View 1, View 2, and View 3 use P frames which are based on View 0’s I frames (and a sequence of P frames), which creates a vertical IPPP frames relation among the different views. Within each view, the same hierarchical prediction structure is applied in the example of Figure 5-9.

Hologram Generation Technologies

Holography generation methods include laser transmission, electroholographic displays, parallax techniques, and microelectromechanical system (MEMS) methods, which are described in the following.

Laser Transmission Holograms

Lasers are commonly used as the coherent light source of the hologram. For hologram generation, two laser beams are commonly used. A holograph is formed of numerous points located in a 3D domain that have different illumination colors and brightness levels. Each of these 3D points has two laser beams intersecting to generate the modeled interference pattern, which becomes the holographic 3D image. As presented in Figure 5-15, the laser beam is divided into an object beam and a reference beam (using mirrors and lenses), where the object beam expands through lens and is used to illuminate the modeled 3D object. By illuminating the hologram plate with the same reference beam, the modeled object’s wavefront is generated. Diffraction of the laser beams at the modeled object’s 3D formation points contained in the interference pattern is how the holograph is reconstructed (www.researchgate.net/figure/How-reflection-hologram-works-B-Transmission-holograms-Transmission-holograms-Fig-2_fig1_336071507).

A process defines how the user sees the hologram image from the Laser via shutter, beam splitter, mirror, diverging lens, reference, and object beam of the real object.