10.3.1 Layered (Backward-Compatible) HDR Video Compression

Layered HDR compression methods are designed so that legacy decoders, which can cope only with standard dynamic range bit depths and LDR displays (displays that are unable to render HDR content), are still able to decode the base layer and display a tone-mapped version of the HDR image/video. HDR-capable decoders would be able to decode the full stream and deliver the higher-performance HDR content. Fig. 10.1 shows a general block diagram describing the structure of a typical backward-compatible HDR encoder. The base layer encodes a tone-mapped eight-bit LDR representation of the HDR input using a fully compatible legacy encoder/decoder. The enhancement layer contains the difference (residual) between the inverse tone-mapped base layer and the original HDR input. This residual is used for reconstruction of the HDR content to be used with HDR devices.

The TMO plays a significant role in the performance of layered HDR video coding methods. The main goal of tone mapping is to reduce the dynamic range of the input HDR image/video while preserving perceptual aspects in the resulting LDR image/video. Several TMOs have been developed for different purposes, such as photographic tone reproduction (Reinhard et al., 2002), photoreceptor physiology–based modeling (Reinhard and Devlin, 2005), simulation of artistic drawing (Tumblin and Turk, 1999), gradient domain dynamic range compression (Fattal et al., 2002), display adaptive tone mapping (Mantiuk et al., 2008), and user interactive local adjustment tone mapping (Lischinski et al., 2006).

Different visual and coding results are obtained when HDR content is mapped with different TMOs. A number of studies have evaluated, objectively and subjectively, the visual impact of TMOs (Yoshida et al., 2005; Eilertsen et al., 2013; Yeganeh and Wang, 2013; Narwaria et al., 2013a). Narwaria et al. (2012) investigated the relation between TMOs and visual attention. They performed psychovisual experiments that assessed the impact of eight TMOs on visual attention. The results of their experiments suggest that TMOs can modify fixation behavior, with contrast greatly influencing saliency.

In most video tone mapping cases a TMO is applied independently to each frame of the HDR video. This produces brightness level variation between consecutive tone-mapped frames, causing flickering artifacts. Hence, when designing a TMO for HDR video, one has to take the temporal dimension into consideration. Lee and Kim (2007) proposed a time-dependent adaptation method for the TMO of Fattal et al. (2002) which exploits the motion information and prevents flickering artifacts. A real-time tone-mapping system for HDR video was proposed in Kiser et al. (2012). This adapts to changes in the scene and reduces the amount of flickering artifacts in the resulting tone-mapped HDR video. The method was implemented in a field programmable gate array for use as a real-time tone-mapping system. Boitard et al. (2014) classified the temporal artifacts generated by TMOs when tone mapping video into different types and surveyed methods designed to remove such artifacts. Aydin et al. (2014) designed a local TMO for avoiding visible artifacts in the spatial and temporal domains. The proposed method decomposes the signal into a base layer and a detail layer. A temporal filter is applied on the detail layer and a spatiotemporal filter is applied on the base layer. An extensive review of TMOs can be found in Reinhard et al. (2010) and Banterle et al. (2011).

Mantiuk et al. (2004) were the first to propose layered coding for backward-compatible HDR video compression. Their method was designed as an extension to the MPEG-4 (Part II) compression standard. The method applies tone mapping (perception-based HDR luminance-to-integer encoding) to form the LDR base layer. The input to the MPEG-4 codec consists of eight-bit RGB data and HDR data in the CIE XYZ color space. The CIE XYZ color space enables encoding of the full visible luminance range and color gamut. The luminance values are quantized with a nonlinear function that distributes the error according to the luminance response curve of the HVS. A color space transformation is introduced that facilitates comparisons between LDR and HDR pixels of different color spaces so that the chrominance data of the residual (enhancement) layer can be determined. Mantiuk et al. (2006a,b) extended this work to include a contrast sensitivity function (CSF)-based prefiltering of the residual stream that removes imperceptible high spatial frequency information, thus reducing the bit rate of the coded sequence.

Mai et al. (2011) proposed a backward-compatible method that aims to find an optimal tone curve for mapping the input HDR video to a backward-compatible eight-bit LDR video format. The optimal tone curve minimizes the quality loss due to tone mapping, encoding/decoding, and inverse tone mapping of the original video. To compute the optimal tone-mapping curve, closed-form and statistical solutions were proposed. The LDR video can be compressed by a conventional video codec such as H.264. The reconstructed video can either be displayed on a conventional LDR display or can be inverse-tone-mapped and augmented by an optional enhancement layer containing an HDR residual signal (also compressed by the codec) for display on an HDR display.

Koz and Dufaux (2014) presented a backward-compatible HDR video compression scheme that uses the tone-mapping method of Mai et al. (2011) and the perceptually uniform mapping of Aydin et al. (2008). Perceptually uniform mapping converts real-world luminance values into perceptually uniform encoded values. The aim of the proposed coding method is to minimize the mean square error between the perceptually uniform-encoded values of the original HDR signal and those of the reconstructed signal. To constrain flickering distortion in the tone-mapped HDR video, their method restrains the average luminance of neighboring frames within the just noticeable differences (JND) interval. Using perceptually uniform mapping, Koz and Dufaux define the perceptually uniform peak-signal to noise ratio that accepts perceptually uniform-mapped values as its input for estimation of the quality of the HDR (or LDR) content when that is displayed on bright displays (Aydin et al., 2008). The experimental results indicate that this method provides a good trade-off between HDR video compression performance and the flickering distortion introduced by the tone mapping applied in most backward-compatible HDR video compression approaches. Compared with the method of Mai et al. (2011), this method offers better performance in terms of the perceptually uniform peak-signal to noise ratio but achieves a smaller objective score when quality is measured with the HDR-MSE and HDR-VDP metrics.

The method proposed in Le Dauphin et al. (2014) adjusts the TMO depending on the coding type of the video frame (intraframe or interframe) in order to maximize the compression efficiency. The experimental results presented suggest that this adaptation can lead to more than 20% bit rate reduction for the LDR layer and more than 2% reduction for HDR video.

The scalable extension of High Efficiency Video Coding (HEVC) — that is, SHVC, offer both bit-depth and color-gamut scalability. Bit-depth-scalable approaches have been proposed in Winken et al. (2007), Wu et al. (2008), and Liu et al. (2008). Typically, the proposed methods create a backward-compatible eight-bit base layer and an enhancement layer for higher-bit-depth reconstruction (ie, 10-, 12-, or 14-bit). In Bordes et al. (2013) and Duenas et al. (2014), color-gamut scalability was been investigated in SHVC. The proposed method addresses the case where the original enhancement layer uses a color gamut different from that used by the base layer. A prediction tool for color differences between the base layer and the enhancement layer is thus required. This can be useful, for instance, for deployment of HDR or wider color gamut services compatible with legacy LDR devices. Typically, the LDR or high-definition TV uses the ITU-R Rec.709 format, while UHD (maybe HDR) is likely to be based on ITU-R Rec.2020.

10.3.2 High-Bit-Depth (Native) HDR Video Compression

High-bit-depth HDR video compression methods directly code the input high-bit-depth HDR video, as opposed to first separating it into base and enhancement layers through tone mapping. They rely on existing codecs that can handle a high-bit-depth input (eg, JPEG 2000, H.264/AVC, HEVC) with added preprocessing steps and/or modifications/extensions that aim to exploit HDR-specific aspects of the HVS for compression gains.

Motra and Thoma (2010) proposed an adaptive-LogLuv transform which can be used with an existing video encoder such as H.264/AVC. Their approach can represent the luminance channel at any specified bit depth. Eight bits are used for the chrominance channels. This method requires that side information is stored/transmitted with each HDR frame. Round-off quantization noise produced by the logarithmic color space transformation used in this method can have repercussions on visual quality. An extended version of this work was presented in Garbas and Thoma (2011), wherein a temporally coherent adaptive dynamic range mapping was proposed. This generates temporal weights and shift parameters for each frame and makes use of the weighted prediction tool specified by the H.264/AVC codec (Boyce, 2004). An electro-optical transfer function is proposed in Miller et al. (2013) which acts as a perceptual quantizer that maximizes the perceptual quality when one is creating HDR content with bit depths of 10 or 12 bits.

The HVS exhibits nonlinear sensitivity to the distortions introduced by lossy image and video coding. This effect is due to the luminance masking, contrast masking, and spatial and temporal frequency masking characteristics of the HVS. Efficient perception-based compression of HDR imagery requires models that capture accurately the masking effects experienced by the HVS under HDR conditions, so that bits are not wasted coding redundant imperceptible information. Fig. 10.2 shows the general structure of perception-based HDR image and video coding.

Zhang et al. (2012a,b) proposed a perception-based HDR image and video compression method for both JPEG 2000 and H.264/AVC. The method uses a discrete wavelet packet transform (DWPT) and applies coefficient weighting factors that are derived from luminance and chrominance CSFs. The aim of the weighting factors is to reduce the amount of imperceptible information that is sent to the encoder. Fig. 10.3 shows the relationship between luminance and chrominance CSFs and a five-level two-dimensional discrete wavelet packet transform decomposition. The perceptual CSF model characterizes the relationship between contrast sensitivity and spatial frequency. In the luminance case (black curve), the HVS is more sensitive to middle spatial frequencies and is less sensitive to lower and higher spatial frequencies. In the chrominance case (blue and red curves), the HVS is more sensitive to lower spatial frequencies and less sensitive to high spatial frequencies. Psychophysical experiments reported in Barten (1999), Mullen (1985), and Mannos and Sakrison (1974) have quantified this phenomenon and describe the ability of the HVS to recognize differences in luminance and chrominance as a function of contrast and spatial frequency. The weighted wavelet coefficients in the case of HDR images are passed to a JPEG 2000 encoder, whereas in the case of HDR video, they are inverse-transformed and the resulting video frames are fed to an H.264/AVC encoder.

The experimental results presented in Zhang et al. (2012b) indicate that the method offers visually lossless quality at a significantly reduced bit rate compared with JPEG 2000 (Table 10.1, Fig. 10.4). The HDR images listed in Table 10.1 were adopted from (Debevec and Malik, 1997; Drago and Mantiuk, 2005; Reinhard et al., 2010). Quality was measured with the HDR-VDP (Mantiuk et al., 2005) and HDR-VDP-2 (Mantiuk et al., 2011) quality metrics. The method also outperforms that of Motra and Thoma (2010) for coding HDR video (Fig. 10.5).

The method of Zhang et al. (2012b) assumes that HDR content exhibits the same CSF as standard dynamic range content. However, HDR content and HDR displays can offer a much larger contrast than conventional imaging. Compared with conventional LDR imaging, HDR imaging (HDR images/video displayed on HDR displays) exhibits considerably larger brightness variations. From a psychophysics point of view, this significant contrast extension has the potential to change the noise visibility threshold in HDR content.

Zhang et al. (2013a) performed psychovisual experiments using a Dolby HDR display (prototype code name Seymour) and HDR stimuli (luminance range 0.103–3162 cd/m²) (Zhang et al., 2013a) to examine HDR luminance masking. The experimental results indicated that the noise visibility threshold increases in both low and high luminance background levels, and especially so with a low luminance background. These results suggest that quantization errors could be hidden by a background luminance masking effect in the darker and brighter areas of HDR content.

To exploit this varying distortion sensitivity, a luminance masking profile can be used to map the average luminance intensity to the quantization step of each coded block. Naccari et al. (2012) proposed the intensity-dependent quantization (IDQ) method for the HEVC standard. Their method applies coarser quantization to darker and brighter image areas without introducing coding artifacts that are noticeable to the HVS. The mapping from average pixel intensity to quantization step is performed through an IDQ profile that is fixed for all sequences and is a modified version of the profile proposed in Jia et al. (2006).

An IDQ profile scaling based on a global tone-mapping curve computed for each HDR frame was proposed in Zhang et al. (2013b). A global tone-mapping curve is a function that maps HDR luminance values either to the displayable luminance range (Mantiuk et al., 2008) or directly to LDR pixel values with eight bits per channel (Larson et al., 1997). In Zhang et al. (2013b) the latter case was considered with the histogram adjustment TMO of Larson et al. (1997). Scaling the IDQ profile using a global tone-mapping curve not only enables adaptation to any HDR content, but also offers a more accurate adaptation with respect to the HVS.

With this IDQ profile, the quantization step can be perceptually tuned on a transform unit basis. However, the intensity differential Quantization Parameter (idQP) value depends on the average pixel intensity (μ) of the original content. Given that the original content is not available to the decoder, for the coded bitstream to be properly decoded, μ would have to be communicated to the decoder. This creates significant overhead that increases the resulting bit rate.

To avoid this overhead the method of Zhang et al. (2013b) estimates μ from the predictor used for each coded block, thus avoiding the need for additional signaling. The predictor can be spatial in the case of intraframe coding or temporal for interframe coding. Estimation of μ from the block predictor avoids the introduction of overhead but creates dependencies in the decoding process that can significantly complicate/delay it. More specifically, the inverse quantization process at the decoder is now dependent on the availability of the predictor. For the case of a temporal predictor, the processing required by motion compensation may delay significantly the output of the pixel data since several memory access operations would have to first be performed.

To prevent such data dependency the method makes use of the intensity-dependent spatial quantization (IDSQ) proposed in Naccari et al. (2012). As can be seen in Fig. 10.6, IDSQ does not require availability of the predictor during the inverse quantization step. Instead, perceptual quantization is applied at the point of final reconstruction of the block, when prediction is added to the residual.

With IDSQ, the inverse quantization process is decoupled into two main steps (see Fig. 10.6):

1. Step 1: Over the quantized coefficients c, perform inverse quantization as specified in the HEVC standard with quantization parameter (QP). Then perform inverse transform to obtain r′.

2. Step 2: Before r′ is added to the predictor, perform inverse quantization with idQP(μ) to obtain the rescaled residuals $\widehat{r}$.

The inverse scaling performed during IDSQ (Fig. 10.6) is performed in the spatial domain, which is the reason for the acronym IDSQ.

IDSQ was integrated into the HM reference codec considered for the HEVC range extensions (HM-10.0-RExt-2.0), and its performance was assessed by measurement of the bit rate reduction relative to the codec without perceptual quantization (Zhang et al., 2013b). The test material used in the performance evaluation consisted of six sequences captured by Zhang et al. (2013b) using a RED EPIC camera using its HDRx function (Fig. 10.7A–F. All the test sequences were then downsampled to 1920 × 1080 and provided as OpenEXR, half float representation files. The input bit depth for the luminance channel was 14 bits, with two eight-bit chrominance channels in LogLuv color space (Zhang et al., 2011). The luma/chroma sampling pattern was 4:4:4. The coding configurations selected for the experiments were the ones described in Flynn and Rosewarne (2013): all intra, random access, and low-delay B. The QP values considered were 12, 17, 22, and 27.

HDR-VDP-2 was used to assess the quality of the reconstructed video. The results obtained (HDR-VDP-2 predicted mean opinion score (MOS)) suggest that the proposed HDR-IDSQ method provides quality perceptually equivalent to that of the anchor but at a significantly lower bit rate. An average bit rate reduction of up to 9.53% was observed when compared with the anchor, with the largest reduction (18.55%) being achieved for “Carnival2.”

As with most objective metrics, HDR-VDP-2 has certain limitations in terms of its correlation with subjective results. The experimental results of Narwaria et al. (2013b), for example, suggest that at higher bit rates HDR-VDP-2 scores do not increase in proportion to the subjective scores. To overcome this, recent work by Zhang et al. (2015) focused on conducting objective and subjective quality tests on an extended HDR dataset. The results of these tests support the conclusions in Zhang et al. (2013b), demonstrating a bit rate reduction of up to 42% for the HDR-IDSQ codec compared to an HEVC anchor across various coding conditions.