12.4.1 IDR Coding

As introduced above, part 6 defines two orthogonal extension mechanisms, one in the DCT domain denoted as refinement coding and one in the spatial domain, called residual coding. Both possibilities have already been discussed in the video coding community. Residual coding is similar in nature to the HDR extension for MPEG video proposed by Mantiuk et al. (2007, 2006), and refinement coding is not unlike the MPEG extension proposed by Zhang et al. (2011). Both mechanisms can be combined if needed, including the possibility to extend residual coding by refinement coding.

Refinement coding will be discussed first. For that, it is helpful to recapitulate how the progressive coding mode of legacy JPEG works (Pennebaker and Mitchell, 1992). Image quality is here progressively improved in two possible directions: the spectral selection mechanism allows the encoder to select parts of the frequencies in the JPEG zigzag scan pattern to be encoded first, with additional frequency components to be included in later scans if desirable. The successive approximation mechanism improves the bit precision of the DCT coefficients by first coding a subset of their most significant bits, then allowing the encoder to include additional lower-order bits in later scans.

It is important to note that the first scan of a successive approximation scan pattern uses a coding mechanism very similar to regular (sequential) coding. Progressive coding uses first a regular sequential scan — with only very minor extensions to skip over empty blocks quickly — to encode the most significant bits, and all subsequent least significant bits are encoded by an alternative subsequent approximation entropy coder. If the spectral selection includes all frequencies³ and block skipping is not used, the entropy coding mechanism of the first scan of a successive approximation scan pattern is identical to that of the sequential Huffman scan.

Refinement coding makes use of this identity by splitting the coding of a high-bit-depth DCT block into two parts: a legacy sequential or progressive coding part that includes all the necessary bits required to reconstruct an eight-bit approximate image, and refinement bits using the successive approximation scan of the progressive coding mode to extend the bit precision as required. The difference between refinement coding and the legacy progressive coding mode (Fig. 12.3) is that the latter signals the number of least significant bits in the syntax elements — the start of scan marker — by which the decoder has to upshift the reconstructed DCT samples as output from the first scan, whereas the latter hides this information in the syntax elements (ie, boxes) of JPEG XT that remain invisible to legacy implementations. In other words, legacy applications would see a syntactically correct eight-bit stream, whereas extended applications increase the bit precision by the known successive coding mechanism already specified in the legacy standard.

If one wants to limit the number of bits required for the DCT coefficients to 16 and the number of bits required to perform an integer-to-integer DCT approximation to 32, it is not hard to compute that at most four refinement scans can be included; this constraint is included for purely practical purposes in the current committee draft of the standard, although extensions to more bits would be straightforward.

To introduce residual coding, one should first model the merging step of base and extension layer as a similar extension of bit depth through least significant bits, though this time in the spatial domain. In a very simple application of residual coding, the eight most significant bits stem from the legacy JPEG codestream and provide an approximation of the IDR image, whereas as the least significant bits are represented in a side channel making up the residual image IDR images would then be reconstructed by upshifting the samples of the base layer and adding the least significant bits from the residual image. These two operations are done by the addition “$\oplus$” on the right-hand side in Fig. 12.2 and in the box denoted as “Base NLT Point Trafo.” The dotted boxes are not used, the box denoted as “Output Conversion” clamps the reconstructed sample values to their valid range, and the box denoted as “Color Transformation” is the identity.

This elementary approach does not, however, withstand a closer analysis of the requirements. IDR images are typically encoded without gamma correction (ie, sample values are proportional to physical intensity), whereas LDR images are gamma-corrected, tone-mapped versions of them. A simple downshift to generate an LDR image from the most significant bits of the IDR image would not work because of the lack of a gamma correction. However, if for image reconstruction purposes the upshift by the LDR image bits is replaced by a simple lookup table that provides a global, simple approximation of the tone mapping applied at the encoder for the generation of the LDR image, both a useful LDR image and an IDR image can be carried in the same file. In such a case, the lookup table (the box denoted as “Base NLT Point Trafo”) has 256 entries if the legacy image does not make use of refinement coding and hence has eight-bit precision; any type of error due to the approximation of the tone mapping is then captured by the extension layer. In the case of refinement coding, the lookup table is 2^h+8 entries long, where h is the number of refinement scans “hidden” from the legacy decoder. Here a finer table approximation allows a more precise approximation of the IDR image, and precise reconstruction with refinement coding alone is possible only if the original tone mapping for generation of the LDR image was a simple global operation. Otherwise, additional errors remain that require correction by a residual scan.

Leaving color issues aside, the reconstruction algorithm for IDR grayscale images defined in part 6 of the JPEG XT standard is hence

$\begin{array}{l}\hfill \text{IDR}(x,y)=\Phi (\text{LDR}(x,y))+\text{RES}(x,y)-2^{R_{\text{b}}-1},\end{array}$

where IDR is the reconstructed IDR image, LDR is the base image visible to legacy decoders, RES is the residual image hidden in applications markers, and $2^{R_b-1}$ is an offset creating a residual signal from RES(x,y) that is symmetric around zero. The constant R_b is the bit precision of the IDR image generated. In the presence of refinement coding, the LDR signal will have a bit depth of 8 + h instead of 8, where h is the number of refinement scans.

Let us briefly discuss the encoding of the extension layer. Since the extension layer is never seen by a legacy decoder, it is not necessary to constrain entropy coding to the legacy eight-bit Huffman mode of JPEG. If its resolution is not sufficient to reach the desired quality, one can either use the 12-bit mode of JPEG or encode the data with the eight-bit Huffman mode and extend the resolution of the extension layer by refinement scans, as one can for the base layer. In principle, even other entropy coding methods could have been considered, but the committee decided to constrain JPEG XT to algorithms that are as close as possible to existing JPEG technology to ease their implementation.

12.4.2 Enlarging the Color Gamut

Even though the original JPEG standard did not define a color space and only specifies a mechanism for how to encode sample values into a codestream, JFIF — later standardized as ISO/IEC 10918-5 — did (Brower et al., 2011). JFIF selected ITU Recommendation BT.601, originally a TV standard, both as a color space and as a decorrelation transformation from an RGB-type description into an opponent chrominance/luminance description. Unfortunately, even this selection is not fully consistent with typical uses of JPEG where images are represented in sRGB rather than Recommendation BT.601. Both color spaces are related but not exactly identical.

For reasons of backward compatibility, part 1 of JPEG XT also specifies the legacy ITU color space. Its color gamut is, however, quite limited, and it is often desirable to encode images expressed in larger color spaces. Simply mapping the primaries of larger color space to the Recommendation BT.601 primaries may be an option in some simple cases, but this will typically not map the white points onto each other; in other words, the colors of the legacy image encoded with wrong primaries when interpreted with Recommendation BT.601 colors might be off, creating an overall undesirable image; such color alterations can be especially irritating in the reproduction of skin colors and should be avoided. They can be corrected, however, by use of the extension layer. Any undesirable color shift can be compensated by the addition of a suitable error signal which will shift the reconstructed sample values to their desired color position. From the perspective of coding efficiency, this approach should, however, be avoided, as the error signal can become large enough to compromise the performance of the code.

To address this predicament, JPEG XT offers three options. First, the LDR primaries can be used consistently for the base image and the full IDR image, then requiring the use of negative color coordinates for out-of-gamut colors. Due to the offset shift in Eqn. 12.1, such sample values are representable by a minimum LDR value and a small, but positive residual error.

Second, the legacy transformation between YCbCr and RGB can be replaced by another transformation that maps the legacy samples into a larger gamut color space. This generalized transformation is denoted as “Base Transformation” in Fig. 12.2; it can be considered as the combination of two subsequent linear transformations. The first transforms the Recommendation BT.601 YCbCr opponent sample values into an RGB color space, with color primaries defined as in the mentioned ITU recommendation; the second maps the coordinates of the ITU Recommendation BT.601 color space into the coordinates relative to the target color space. From a color science perspective this transformation is not strictly correct. The transformation is applied in the nonlinear (gamma-corrected) Recommendation BT.601 color space, hence creating errors in the chroma reproduction; the larger the nonlinearity in the inverse tone-mapping process, the larger the errors. Again, such errors can be corrected by an appropriate extension layer, and in the worst case compromise the coding performance, but not the quality of the image.

The third option overcomes this problem by introducing an additional linear transformation after inverse tone mapping, denoted as “Color Transformation” in Fig. 12.2. Here, the decoder first performs the legacy YCbCr to RGB transformation in the nonlinear color space as required by JFIF, and then maps the samples into the (typically) linear target color space of the IDR image by the base nonlinear point transformation, before it finally applies a linear transformation to map the colors to the target space.

While the third approach seems to be ideal from a color science point of view, it comes with problems of its own. None of the transformations can be implemented with infinite precision, and the second color transformation creates additional numerical errors that also require correction. Astonishingly, it is sometimes beneficial to use the second method instead of the third to maximize the coding performance. JPEG XT thus allows the use of all three approaches, even combined, and leaves it to the encoder to signal the desired mechanism.

The color decorrelation in the extension layer is much less critical. At first sight, it seems plausible to transform the residual data also first from YCbCr to RGB with Recommendation BT.601 primaries, and then further transform the data into the target color space. While such transformation are allowed and even present in some profiles, see Section 12.4.3, they not only require one additional matrix transformation, but also add further loss due to numerical inaccuracies. IDR coding as defined in part 6 does not make use of this transformation. Instead, one can take advantage of the fact that the residual image is never displayed directly and does not need to be backward compatible with legacy JPEG applications. Residual coding can be considered as a compressor of three sample values per pixel, without any color space information. While error residuals could be compressed by the residual stream directly without any transformation, it turned out that a decorrelation transformation identical to the Recommendation BT.601 RGB to YCbCr transformation improves compression efficiency as well. This transformation is applied in the box denoted by “Residual Transformation” in Fig. 12.2. The residual nonlinear point transformation is typically only a scalar multiplication or an upshift, necessary to use the full range of eight bits to represent the error signal. It serves no additional purpose, and is even absent for lossless coding.

12.4.3 From IDR to HDR: HDR Coding

The coding mechanisms introduced above are, within the limits the current draft of the standard sets, able to encode images consisting of integer samples of up to 16-bit resolution. Part 7 of the JPEG XT standard is an extension of the coding tools of part 6 to allow reconstruction of images to floating point samples, as required for the representation of HDR images. Similarly to part 6, these extensions are based on two elementary types of operations: a pixel-based nonlinearity, implemented either by a lookup table or by a parameterized curve, and a linear transformation defined by a 3 × 3 matrix.

These tools, while being part of a common decoder framework, enable several encoder architectures to tackle the problem of HDR coding. As for many other standards, the set of decoding tools is structured into profiles, each of which defines a set of allowable decoder tools for codestreams conforming to the corresponding profile. Even though the corresponding encoder architecture is not specified by the standard, it is still helpful to start the discussion with typical encoder designs.

Profiles A and C make use of the approximately logarithmic dependency between the physical luminance (stimulus) and the corresponding response of the human visual system, which has been known for a long time as “Weber’s law” (Netravali and Haskell, 1995). Profile C represents images in the logarithmic domain directly, using a piecewise linear approximation of the logarithm that is exactly invertible and hence enables lossless coding, which is, however, standardized in a separate part. The coding algorithm in the logarithmic domain is then identical to part 6 (ie, profile C part 7 is a minimal extension to part 6).

Profile A applies the logarithmic map only to the luminance channel; that is, the HDR image is represented as the product of an LDR base image and a luminance scale factor μ that is encoded logarithmically (see Fig. 12.2). The relation to profile C becomes more apparent if one recalls that this multiplication in the image domain performed by profile A is equivalent to an addition in the logarithmic domain as performed by profile C decoders (ie, the equivalent to the multiplication step by μ in profile A is the addition step immediately before it in profile C; see Fig. 12.2). Profile A also makes use of this addition step, although here to correct small residual errors in the chrominance channels.

Unlike profiles A and C, profile B encoders split the HDR signal along the luminance axis. Samples below a luminance threshold are represented in the base layer, and samples above the threshold are represented in the extension layer. The composition of the base and extension layers in the decoder is performed by a division operation; this division is represented by an addition and logarithmic and exponential preprocessing and postprocessing. Decoders do not need to go through these processing steps, of course, and may apply the division immediately when they detect a profile B codestream. From the perspective of the architecture of the standard, the addition step is here the same addition as applied in part 6, and the step involving multiplication by μ is absent as in profile C.

A somewhat more formal and more precise discussion of the profiles is given now. Profile A, allowing both the prescaling and postscaling nonlinearity but no refinement or secondary nonlinear transformations (Fig. 12.2), reimplements a decoder similar to JPEG-HDR, a backward-compatible JPEG extension originally proposed by Ward and Simmons (2005) and Dolby. The reconstruction algorithm in this specific case is given by

$\begin{array}{l}\hfill \text{HDR}(x,y)=\mu ({\text{RES}}_0(x,y))(C\Phi (\text{LDR}(x,y))+\nu (SC\Phi \text{LDR}(x,y))\cdot R\text{RES}(x,y)).\end{array}$

In Eq. (12.2), μ is the postscaling factor depending on the luminance signal l of the extension layer. The function μ(l) relating the luminance of the extension layer to the scaling factor is typically an exponential ramp function of the form $\mu (l)=exp(al+b)$ with suitable constants a and b (ie, the extension layer luminance l depends logarithmically on the scale factor μ). This nonlinearity is, from the perspective of the standard, implemented by the postscaling nonlinear transformation.

The transformation S is a 3 × 3 matrix that extracts the luma signal from the RGB base image, and ν adds an offset shift to the base layer luma signal extracted through S. The matrix C finally maps the primaries of the Recommendation BT.601 color space of the base layer onto the primaries of the larger HDR gamut.

The offset shift through the function ν is represented in the standard decoder architecture by the prescaling nonlinear transformation. As said, one typically has ν(y) = y + N₀, where N₀ is the noise floor. It allows for chroma variations due to camera noise even in the absence of a luma signal.

R is an affine (and not linear) transformation that converts the chrominance signal of the extension layer to an additive RGB residual. It does not take the luminance channel of the extension layer into account, which is instead split off before to compute the scale signal μ.

Because of postscaling by μ, the nonlinearity Φ in the base layer, represented by “Base NLT Point Trafo” in Fig. 12.2, does not need to compensate for the effects of tone mapping and is here typically an inverse gamma transformation. It maps the gamma-compensated base image back into a linear radiance space.

Encoding in profile A with prescaling and postscaling transformation enabled is somewhat more complex, and we will next derive the encoder algorithm. The input of the encoder is an LDR-HDR image pair from which a suitable extension layer must be computed. For that, first note that RRES(x,y) is, by design choice, a pure chrominance residual and does not include any luminance information. Thus, if we denote by P_L the projection onto luminance space, we find that

$P_{\text{L}}(C\Phi (\text{LDR}(x,y))+\nu (S\text{LDR}(x,y))\cdot R\text{RES}(x,y))=P_{\text{L}}C\Phi (\text{LDR}(x,y)).$

This allows the encoder to determine first μ(RES₀(x,y)) and then, as μ is a known function, RES₀(x,y) by the quotient of the reconstructed base image luminance and the original image luminance:

$\mu ({\text{RES}}_0(x,y))=\frac{P_{\text{L}}C\Phi (\text{LDR}(x,y))}{P_{\text{L}}\text{HDR}(x,y)}.$

In the last step, the chroma residuals can be computed:

$R\text{RES}(x,y)=\frac{1}{\nu (S\text{LDR}(x,y))}\left(\frac{\text{HDR}(x,y)}{\mu ({\text{RES}}_0(x,y))}-C\Phi (\text{LDR}(x,y))\right).$

The reader may now verify that the right-hand side lies indeed in the chroma-subspace of the color space — that is, the luma signal is zero (or close to zero, because of numerical inaccuracies). The luminance channel of the residual image contains instead the luminance scale factor.

While profile A uses an explicit multiplication operation to scale the base image luminance to the desired target range, encoding and reconstruction are relatively elaborate, and the standard allows simpler alternatives to achieve similar goals in profiles B and C. Both make use of the functional equation of the logarithm to map the addition in Fig. 12.2 that merges the base layer with the extension layer into a multiplication or division — that is,

$\begin{array}{ll}\hfill \text{HDR}{(x,y)}_i& =exp(log(C\Phi {(\text{LDR}(x,y))}_i)+log{(\text{RES}(x,y))}_i-2^{R_{\text{b}}-1})\hfill \\ \hfill & =exp(-2^{R_{\text{b}}-1})C\Phi {(\text{LDR}(x,y))}_i\cdot R\text{RES}{(x,y)}_i(i=0,1,2),\hfill \end{array}$

where in Eq. (12.3) $exp$ and $log$ are applied component-wise on the components of the vectorial image data. Similarly, a subtraction in the logarithmic domain becomes a division for the HDR image signal. The reader may want to compare Eq. (12.3) with Eq. (12.1) used for IDR reconstruction in part 6.

This transformation of a multiplication into an addition in the logarithmic domain is the motivating idea for the secondary base and secondary residual nonlinear point transformations in Fig. 12.2; the logarithm can here also be understood as a transformation into an approximately perceptually uniform domain based on Weber’s law.

The Trellis Management XDepth proposal reconstructs the HDR image as the quotient of an inversely gamma-corrected LDR signal reconstructed from the legacy JPEG stream and a residual included in a side channel. This quotient can be rewritten as the difference of two terms in the logarithmic domain, where the $log$ and $exp$ functions necessary for conversion between the linear and the logarithmic domain are expressed in the standard decoder design shown in Fig. 12.1 by the secondary base and secondary residual nonlinear transformations and the output conversion:

$\begin{array}{ll}\hfill \text{HDR}{(x,y)}_i& =\sigma exp(log(C\Phi {(\text{LDR}(x,y))}_i)-log{(\Psi (\rho (R\text{RES}(x,y)))+ϵ)}_i)\hfill \\ \hfill & =\sigma \frac{C\Phi {(\text{LDR}(x,y))}_i}{\Psi (\rho (R\text{RES}{(x,y)}_i))+ϵ}(i=0,1,2).\hfill \end{array}$

The “Output Conversion” in Fig. 12.2 includes here, compared with Eq. (12.3), an additional scale factor σ whose purpose will be described below. As in profile A, the base layer nonlinearity Φ maps the gamma-corrected LDR image back into the linear domain, and Ψ is an additional power map applied in the residual to compress its dynamic range–that is, Ψ(x) = x^β, where β is a data-dependent exponent selected by the encoder. As in profile A, R is a decorrelation transformation from YCbCr to RGB, although the luma signal is this time not split off but is included in the transformation. Finally, ρ represented in the standard decoder architecture by the “Intermediate NLT Point Trafo” is an affine scaling transformation that maps the RGB signals of the extension layer into a suitable range. It is described next.

Encoding in profile B is much simpler than in profile A as it does neither require prescaling by ν nor postscaling by μ:

$\rho (R\text{RES}{(x,y)}_i)={\Psi }^{-1}\left(\sigma \frac{C\Phi {(\text{LDR}(x,y))}_i}{\text{HDR}(x,y)}\right).$

In general, the extension layer computed by the right hand will not be in range of the native JPEG standard. The transformation R⁻¹, however, is there already present as the RGB to YCbCr decorrelation transformation. The purpose of ρ⁻¹ is now to perform the necessary scaling and to map the output of Ψ⁻¹ into this interval. It is a data-dependent transformation that must be computed by the encoder.

While the profile B algorithm allows for arbitrary LDR images and hence for arbitrary tone mapping of the HDR content, the original XDepth proposal considered only one special choice for the map Φ, the color transformation C, and the LDR image — namely,

$\text{LDR}{(x,y)}_i=min({\Phi }^{-1}{({\sigma }^{-1}\text{HDR}(x,y))}_i,1),$

where Φ⁻¹ is the gamma correction and C is the identity. Hence, the HDR image is first inversely scaled with σ, then gamma-corrected and clamped to the range [0,1]. One observes now that this has the following consequences for the residual. As long as the sample value of the HDR image is below σ, no clamping occurs and the quotient defining the extension layer

$\Psi (\rho (R\text{RES}{(x,y)}_i))=\sigma \frac{\Phi {({\Phi }^{-1}({\sigma }^{-1}\text{HDR}(x,y)))}_i}{\text{HDR}{(x,y)}_i}=1$

becomes the identity. For larger HDR sample values, the LDR image is saturated, and the residual carries all the image information of the overexposed region. The purpose of σ becomes now obvious: It determines which image luminances go into the LDR image and which become part of the extension layer. Using the analog printing process as metaphor, σ controls the “exposure” of the (now digital) negative “HDR(x,y)” into the print “LDR(x,y)”; in this process, the overexposed regions are preserved in the extension layer. Hence the name “exposure value” for the inverse of σ.

While profile A and profile B decoding algorithms — with or without prescaling and postscaling transformation — are clearly invertible as mathematical operations and hence suitable encoding algorithms are readily derived, they are not invertible without loss as numerical algorithms. This is because both of them depend on floating point representations and use multiplications or transcendental functions, which all introduce (numerical) loss when the (mathematical) inverse is taken. Profile C avoids these problems and extends the decoding algorithm from part 6 (Eq. 12.1) to HDR without depending on floating point arithmetic.

To understand its design, recall the IEEE encoding of floating point numbers (IEEE Computer Society, 2008). In its binary encoding, an IEEE floating point number consists of a sign bit s, a number of k exponent bits e, and a number l of mantissa bits m. The floating point number f is then given by

$f={(-1)}^s\cdot 2^{e-b}\cdot \left(1+2^{-l}m\right),$

where b is an exponent bias that depends on the precision of the floating point number. This number f is now stored in memory as the concatenation of the s, e, and m bits; the very same bit pattern can also be interpreted as integer number i:

$i=2^{k+l}s+2^{l}e+m.$

The trick is now that the reinterpretation of f as i, which is a no operation for a computer since it changes only the interpretation of the bits rather than their values, is for s = 0 an approximation of a logarithm to the base 2, here written as $\psi log$:

$\begin{array}{l}\hfill i=\psi log(f)=2^l({log}_{2}f+b)-2^l{log}_2(1+2^{-l}m)+m\approx 2^l({log}_{2}f+b),\end{array}$

where one uses that ${log}_2(1+x)\approx x$ for x between 0 and 1. In fact, the error

$ϵ(x)=x-{log}_2(1+x)$

vanishes at the edges x = 0 and x = 1.

This observation can now be applied to Eq. (12.3), the multiplicative decoding algorithm, by replacement of the logarithm by $\psi log$, defined by re-interpretation of bit patterns, and its inverse, denoted by $\psi exp$. This change implements the reconstruction algorithm as

$\begin{array}{ll}\hfill \text{HDR}{(x,y)}_i& =\psi exp(\psi log(C\Phi {(\text{LDR}(x,y))}_i)+\psi log{(\text{RES}(x,y))}_i-2^{R_{\text{b}}-1})\hfill \\ \hfill & \approx \psi exp(-2^{R_{\text{b}}-1})\Phi {(C\text{LDR}(x,y))}_i\cdot R\text{RES}{(x,y)}_i(i=0,1,2),\hfill \end{array}$

which carries over to lossless compression and is a natural extension of the coding algorithm of part 6. Since $\psi exp$ and $\psi log$ require only a reinterpretation of bits, this coding algorithm is also of very low complexity.

Eq. (12.6) can be simplified even further. First of all, if we ignore the color transformation matrix C for a moment, the base layer contributes to the overall HDR image through

$\psi log(\Phi (\text{LDR}(x,y))).$

Now, both operations — the nonlinearity Φ and the pseudologarithm mapping floating points to integers — can be merged into a single operation that consists only of a table lookup process as the combined operation maps integers to integers. Second, since the residual image is never displayed on a monitor for observation, it does not matter whether RES or $\psi log\text{RES}$ is encoded. The latter has the advantage that it already consists of integers suitable for JPEG encoding. With these changes, one gets a complete integer-based and numerically invertible reconstruction algorithm:

$\begin{array}{l}\hfill \text{HDR}{(x,y)}_i=\psi exp(\widehat{\Phi }(\text{LDR}(x,y))+R\text{RES}(x,y)-2^{R_{\text{b}}-1})(i=0,1,2).\end{array}$

An additional color transformation can now either be applied directly on the LDR image in the gamma-corrected space or be applied outside in the logarithmic space after inverse tone mapping and pseudologarithmic mapping by $\widehat{\Phi }$. Since Φ and $\widehat{\Phi }$ are nonlinear in general, neither of these operations is identical to the color transformation C in the linear space, and hence additional errors are introduced that can be compensated by the residual image. Hence, coding gain is lost by transformation to an alternative color space. A second strategy, outlined in Section 12.4.2, leaves the color space untouched, but uses negative sample values to represent out-of-gamut colors. Clearly, the original logarithmic and exponential map of Eq. (12.3) cannot express negative values, although an extension of $\psi log$ and $\psi exp$ to the negative axis is straightforward by extension of both functions in an (almost) point-symmetric way around the origin x = 0:

$\begin{array}{l}\hfill \psi log(-x):=-\psi log(x)-1\text{for}x\ge 0.\end{array}$

The additional subtraction of 1 seems curious, but it allows one to distinguish between the two representations of zero the IEEE floating point format has to offer — namely, + 0 and − 0. Note that one cannot implement this extension of the pseudologarithm by reinterpreting the bit pattern of IEEE floating point numbers, and its implementation requires a conversion from a sign-magnitude representation to a two’s complement representation. This conversion, however, is also lossless.

12.6.1 IDR Coding Performance

IDR image representations are based on integer samples and thus evaluation by established indices such as the peak signal to noise ratio (PSNR) is, at least in principle, possible. Nevertheless, some caution in interpreting such simple indices is certainly required, even if IDR imaging does not cover the same dynamic range as HDR imaging. This is because IDR images are typically not gamma-corrected, unlike LDR images; while gamma correction in the LDR regime creates at least an approximate perceptually uniform space, this is no longer given for IDR and HDR coding. For simplicity, and because of the lack of any better known alternative, only PSNR and multiscale structural similarity (SSIM) (Wang et al., 2003) figures will be presented, despite their known limited correlation to subjective quality. Multiscale SSIM is an objective “top-down” quality index that first separates reference and distorted images into several scales, then segments each scale into small blocks and measures luminance, contrast, and structure deviations within each block before pooling all scores together. It was found that the correlation of SSIM to subjective scores is better than that of PSNR, despite its relatively low complexity. For details, see Wang et al. (2003).

One typically obtains IDR images by demosaicing raw sensor data and either keeping them in the camera color space or transforming them into a scene-referred reference color space. Unlike HDR images, IDR images are typically obtained in a single exposure, although because of improved sensor technology, in bit depths beyond the eight bits per sample. The relation of IDR sample values to physical intensities is here given by the response curve of the camera, which is, however, typically unknown. The test image test in Fig. 12.4 was obtained in such a way (ie, by the demosaicing of “raw” camera streams and their transformation into the 16-bit linear scRGB color space). While this color space defines the same primaries as sRGB, it still allows an extended gamut by including an additive offset in the sample representation. Out-of-gamut colors are hence represented by sample values below the offset, and in-gamut colors are represented by coordinates larger than the offset. Thus, the offset shifts color coordinates which would otherwise be negative into the positive semiaxis.

The images in Fig. 12.4 were then compressed by JPEG XT part 6, then expanded again, and the PSNR and multiscale SSIM (Wang et al., 2003) were measured in the IDR regime and plotted against the total bit rate (Fig. 12.5). Nothing new can be learned about the quality of the base layer alone as this is simply defined by the legacy JPEG standard.

Two additional codecs have been included in the benchmark — namely, JPEG 2000 and JPEG XR, the former in PSNR optimal and visually optimal mode. JPEG XT part 6 also allows various configurations, three of which have been tested here. For that, recall that JPEG XT requires two images as input, an IDR image and an LDR image. In the first experiment, the corresponding LDR images forming the base layer were obtained by conversion of the scRGB images with a color management system to sRGB. When used in its “perceptual” setting, the color management system used here — namely TinyCMS — includes a simple automatic tone mapping to generate perceptually “convincing” results. The base and extension layer quality settings of the encoder were then such that the target quality was maximized for a given bit rate constraint. The resulting curve is denoted as “JPEG XT” in Fig. 12.5.

Leaving the quality parameter of the legacy JPEG codestream unconstrained as done above may, however, result in unpleasing LDR images, an effect that was avoided in the second experiment by the disallowance of quality parameters for the LDR codestream below 75. This shrinks the parameter space for optimization, creates additional overhead, and hence lowers the corresponding PSNR and SSIM scores of the HDR image. This curve is denoted by “JPEG XT > 75” in Fig. 12.5.

For the third experiment, the simple scRGB to sRGB map was replaced by the global Zhang et al. (2011) operator to study the dependency of the quality on the tone-mapping algorithm. Base quality is here again unconstrained. As seen from the curve “JPEG XT (Reinhard),” this has only little impact on the overall performance.

In comparison with more advanced technologies such as JPEG 2000 or JPEG XR, JPEG XT shows, of course, a disadvantage coming from its simpler coding technology and its inclusion of a base channel. Additionally, one should keep in mind that the two former codecs are usually run in PSNR-optimized mode, whereas the JPEG quantization matrix already implements a form of visual weighting by quantizing higher-frequency bands more aggressively than the more visible lower-frequency bands. SSIM scores of JPEG XT are therefore usually closer to those of more advanced technology, although the difference is also image dependent.

The dependency of the overall quality on the base and extension layer quality parameter can be seen much better in a three-dimensional plot (Fig. 12.6, where PSNR is plotted against base layer and extension layer rate). The tone mapper is here again the one defined by TinyCMS. The plots show a curious difference between the two images. For the DSC_218 picture, the base and the extension layer contribute approximately equally to the overall PSNR, whereas the bit rate for DSC_532 has to be invested mostly into the extension layer and the overall quality almost does not depend on the base layer rate. The difference lies in the nature of the images. DSC_532 is an almost bitonal image with very light and dark image regions, requiring a very extreme tone mapping to convert it to LDR, whereas DSC_218 is comparably moderate in nature.

12.6.2 Coding Performance on HDR Images

Even when we leave the relevance of such quality indices aside, PSNR and SSIM cannot be directly carried over to the HDR regime as was possible for IDR; this is simply because the definitions of both indices depend on a natural maximum value or maximum brightness which does not exist for HDR. Furthermore, both are not scale independent as required for relative radiance formats (ie, multiplication of all sample values by the same scale alters the value of the index).

A seemingly obvious replacement for the PSNR is the signal-to-noise ratio (SNR), defined as

$\begin{array}{l}\hfill \text{SNR}:=10{log}_{10}\frac{\sum _i{x}_i^2}{\sum _i{(x_i-y_i)}^2},\end{array}$

because it does not depend on a scale. In Eq. (12.9), x_i is the sample value (grayscale value) of the original image and y_i is the sample value of the reconstructed (distorted) image. Unfortunately, this definition comes with a lot of problems of its own. While the PSNR has some known cases of failure to evaluate image defects correctly according to their visibility, it is even easier to construct defects in HDR images that are misclassified by the SNR. A small relative error in a very light image region will cause an almost invisible defect, but such an error will lower the SNR significantly even though the defect is not obvious to human observers. Similarly, a small defect in a dark image patch may give rise to a significant image defect that is obvious to human observers; the SNR will, however, not differ significantly because the defect is low in amplitude and does not change the denominator much.

A better option to make traditional metrics available for HDR image coding is to first map the absolute luminances, as given by the sample values of the original and the reconstructed image, into a perceptually uniform space, and then apply known metrics in this space instead. If the input scene is described in relative radiance, a suitable scale factor to derive absolute radiance values must be estimated though. A candidate for this is the PU-Map of Aydin et al. (2008); an even simpler map is given by the logarithm, followed by clamping. The motivation for the latter is Weber’s law (Netravali and Haskell, 1989), which states that the sensitivity of the human visual system is approximately inversely proportional to the amplitude of the input stimulus.

While this approach seems somewhat ad hoc, one should recall that a similar map is, implicitly, also applied in LDR imaging. Image data here are usually gamma-corrected and hence recorded in a nonlinear space. The gamma correction, even though historically motivated differently, can here be understood as a similar approximation of the map from the physical to an approximately uniform perceptual space.

Despite these attempts to apply known expressions in the HDR regime, a couple of quality indices have been designed explicitly for HDR imaging, and the area of an HDR image and video quality metrics are discussed in much more detail in Chapter 17. For the purpose of this chapter, three quality indices will be used: one of these is Daly’s visual difference predictor (Daly, 1992; Mantiuk et al., 2005), extended and enhanced in a later version by Mantiuk et al. (2011), and another is the mean relative square error (Richter, 2009) (MRSE). Unfortunately, currently only limited data exist to assess the performance of these indices themselves — that is, there is only limited evidence that they correlate well to image quality as observed in subjective experiments. The reason is that subjective evaluation of HDR image quality is a relatively young field and not much is known about suitable test conditions and test protocols, partially because equipment to reproduce such images has not been available until recently. Unlike the evaluation of LDR data, the response curve and the properties of the particular monitor used for image reproduction must be taken into consideration as they are another source of nonlinearities. Calibration of the overall signal path from sample values to observable radiance is nontrivial and is beyond the scope of this section.

As in Section 12.6.1, the HDR image quality gained after JPEG XT compression and as estimated by an appropriate index is a function of two parameters, the base layer bit rate and the extension layer bit rate. The bit rate itself is controlled by a JPEG “quality” parameter which acts as a scaling factor of a suggested default quantization matrix found as an example in the JPEG standard. This choice of the quantization matrix is not necessarily the best for all profiles, although research results for better alternatives are not yet available.

Three objective quality indices have been tried here — namely, PU-PSNR, MRSE, and HDR-VDP2. PU-PSNR first maps the data with the PU2-Map defined in Aydin et al. (2008) into a perceptually uniform space and then measures the PSNR there. The second measures the MRSE in the image domain, and is defined as

$\begin{array}{l}\hfill \text{MRSE}:=-10{log}_{10}\sum _i\frac{{(x_i-y_i)}^2}{x_i^2+y_i^2}.\end{array}$

As one can see by a Taylor expansion, this is approximately identical to measurement of the PSNR after application of a logarithmic mapping. The third quality index is HDR-VDP2, defined in Mantiuk et al. (2011), which includes an extensive model of the human visual system, including a map to a perceptual luminance space, separation into perceptually defined frequency bands, masking, and error pooling. More details on HDR-VDP2 are given in Daly (1992) and Mantiuk et al. (2005, 2011).

The HDR images for this study were taken from the HDR Photographic Survey (Fairchild, 2015), which contains 106 images obtained through a multiple-exposure process. Photographs of the same scene were taken multiple times with different exposure times to cover a large range of luminances, and then were computationally merged into a single image per scene. Such computational methods are beyond the scope of this chapter, although the outputs are, in the absence of additional calibration, relative radiance images; that is, the floating point sample values obtained are (approximately) proportional to the luminance of the original scene. For details, see Reinhard et al. (2010). Some (tone-mapped) example images from this test set are shown in Fig. 12.7.

Comparison with subjective quality scores, even though not done in this chapter, requires an additional step — namely, the conversion of relative or absolute radiance data into an absolute radiance display-referred color space that describes the characteristics of the monitor used in the test sessions. Such monitors are already available on the market as professional equipment. In this particular evaluation, a maximum luminance of 4000 cd/m² was assumed to define the mapping.

The performance of a JPEG XT encoder depends not only on the HDR input image but also on the selection of the LDR image encoded in the base layer. Here, the base images were taken from the JPEG XT internal “validation test” which was run to check the suitability and correctness of the various proposals. In particular, the tone-mapped LDR images were prepared with Photoshop by a semiautomatic process where parameters were adjusted by a human observer to generate “pleasing” LDR images. While somewhat arbitrary, this procedure mimics the typical HDR workflow for which JPEG XT has been designed. The difference between the test here and the validation test of JPEG XT is that this test uses absolute radiance output-referred images, whereas the latter used the original relative radiance scene-referred images.

Each HDR-LDR image pair was compressed with each of the three profiles of part 7 (profiles A, B, and C), with variation of both base layer and extension layer quality, and measurement of both the extension layer and the base layer bit rate and the image quality, as approximated by PU-PSNR, MRSE, and HDR-VDP2. Fig. 12.8 shows plots of HDR-VDP2 against the base layer and the extension layer bit rate for four images. BloomingGorse2 is a structurally very complex image with a medium dynamic range, 507 contains many flat areas, but is almost bitonal and has a higher dynamic range than BloomingGorse2. McKeesPub and MtRushmore2 are somewhat between these extremes. Figs. 12.9 and 12.10 show the performance of the same profiles on the very same images, but measured by the MRSE. As can be seen from the figures, profile A performs quite well on complex images and shows an advantage for medium to low bit rates, whereas profile C performs better in the high-quality regime for relatively flat images. What cannot be observed from the plots is that the profile C quality does not saturate toward high bit rates (ie, profile C implements a scalable lossy to lossless compression). Profiles A and B level out toward higher bit rates, and increasing the rate beyond a threshold does not improve the performance anymore.

As can be seen, the relative performance of the three profiles is quite consistent between the metrics except for the complex BloomingGorse2 image, where the advantage is either for profile A or profile C, depending on the quality index used. Subjective tests have shown that the correlation to subjective scores of HDR-VDP2 is usually better than that of the far simpler indices.

A somewhat more conventional one-dimensional plot of quality as measured by HDR-VDP2 versus the bit rate is given in Fig. 12.11. In this case, an extensive search on the two-dimensional base/extension layer rate parameter space has been made to obtain the best possible quality for a given rate constraint. The same trend as in Fig. 12.8 can be observed, showing advantages for profiles A and B for low bit rates and profile C for high bit rates. In this test, the base rate is unconstrained and hence possibly very low; hence, the LDR quality is uncontrolled and may be too low to provide an image of reasonable quality.

Fig. 12.12 shows the results if the base layer quality is constrained to be at least 75, resulting in a good-quality LDR preview. Here, profile C falls even more behind for low bit rates. However, it is the only profile whose quality does not saturate for increasing bit rate. It remains the best choice if scalable lossy to lossless coding is required or really high qualities need to be obtained.