4.2.2.1 Camera versus scene movement

Misalignment can be classified into different categories according to the kind of movement of the camera and the objects in the scene. Table 4.1 shows the types of possible misalignment present in multiple exposures for HDR generation.

A static camera refers to a camera fixed to a tripod or any other support that keeps the objective still during the acquisition. The free path classification considers camera movement because the camera is either handheld or supported by an articulated device, therefore following an arbitrary trajectory. Stereo or multiscopic acquisition includes stereo pairs of cameras or camera rigs composed of three or more positions of one or more cameras horizontally aligned, to capture respectively two views or more of the same scene.

The scene is classified as dynamic if any object moves during the acquisition, no matter what the amount of movement or the size of the object is. Otherwise the scene is considered as static.

In every case it is possible to produce an HDR video. For static images (first row in Table 4.1), it is possible to repeat the acquisition for a given time step and combine the HDR video into a time-lapse video. Time lapse (also known as slow motion) is a technique for capturing frames significantly slower than video frame rates. When played at video frame rates, time appears to be faster. This is often used to capture a natural phenomenon such sunrise or sunset, or in the animation industry. Camera- and scene-constrained sequences are, in general, easier to align than the general case since only one kind of misalignment takes place. The three cases (second, third, and fourth rows in Table 4.1) are used in film production. The last case (fifth row in Table 4.1), concerning stereo and multiscopic cameras, has become very popular since stereo and autostereoscopic displays have appeared on the market.

4.2.2.2 Local and/or global misalignment

Misalignments as defined in the previous section can be categorized in three different types:

1. Global misalignment is the consequence of camera motion (change in position) and affects all pixels in the image (third and fifth rows in Table 4.1). It is common in exposure sequences acquired with handheld cameras, although it is possible to find misalignment even for still sequences acquired with tripods (camera shaking with the mechanism activation). Between consecutive pairs of images, it is generally a small movement corresponding to translations or rotations. It may cause ghosting in the resulting HDR image, but some efficient techniques have been proposed to correct this misalignment. However, even for small movements, problems of object occlusion and parallax could be difficult to solve.

2. Local misalignment is produced by dynamic objects in the scene and affects only certain areas inside the image (second row in Table 4.1). Capturing a set of LDR images takes at least the sum of the shutter speeds of each picture. This time is enough to introduce differences in the positions of a dynamic object in the scene. In this case some areas occluded in some images may be visible in others. Depending on the speed of the dynamic object and the kind of the movement, it may produce important differences between the inputs.

3. Local and global misalignment combines the two previous types and concerns the fourth row in Table 4.1. When a camera follows a free path to record a dynamic scene, each frame contains both local and global misalignment. Pixels in the image are affected by transformations of different nature.

Fig. 4.1 shows an example of movement in a common multiexposure sequence. Notice that in stereo pairs (Fig. 4.1A and B), both images were acquired at the same time by two different sensors. Even if the scene is dynamic, both images correspond to the same time and no local misalignment is possible. The only misalignment possible is global, due to changes in the perspective from the two points of view.

4.2.2.3 Generated HDR content

If both the scene and the camera remain static during the acquisition, the multiple exposures are aligned. The only possible result is an HDR image. However, even in this case, time-lapsed HDR video can be produced combining HDR frames acquired from static multiexposed LDR images. In such a case, any of the existing techniques for static images (Mann and Picard, 1995; Debevec and Malik, 1997; Mitsunaga and Nayar, 1999) can be used to recover radiance values and merge them into HDR images.

In cases where the camera remains static recording a dynamic scene, it is possible to detect the areas affected by dynamic objects in the scene and treat them locally. Several techniques have been presented for motion detection in the context of HDR image generation (Khan et al., 2006; Jacobs et al., 2008; Gallo et al., 2009; Pece and Kautz, 2010; Heo et al., 2011; Orozco et al., 2012, 2013, 2014), many of them already discussed in Chapter 3.

In the opposite case (static scene and dynamic camera), the movement between consecutive frames is very small. Some computationally efficient methods were proposed to solve such misalignment (Ward, 2003; Grosch, 2006; Skala, 2007). The most difficult case is when both the camera and the scene move. In such a case, dense correspondences between frames are required.

In a handheld video sequence (Fig. 4.1C and D), the images correspond to different time instants and every pixel in the image is affected by global misalignment due to changes in the position of the camera, and some pixels are also affected by local misalignment due to dynamic objects like the boat in this figure.

The rest of this chapter is dedicated to analyzing the most important approaches to solve the misalignment described in Table 4.1. Section 4.3 focuses on cases corresponding to the second to fourth, whereas Section 4.4 analyzes existing solutions for the case described in the fifth row.

4.4.3 Multiple-Exposure Stereo Matching

4.4.3.1 Problem formulation

The use of stereo matching or disparity estimation for pixel matching on differently exposed stereo/multiview images is not straightforward. Stereo matching or disparity estimation is the process of finding the pixels in the multiscopic views that correspond to the same 3D point in the scene. The rectified epipolar geometry simplifies this process of finding correspondences on the same epipolar line. It is not necessary to calculate the 3D point coordinates to find the corresponding pixel on the same row of the other image. The disparity is the distance d between a pixel and its horizontal match in the other image. Akhavan et al. (2013, 2014) compared the different ways of obtaining disparity maps from HDR, LDR, and tone-mapped stereo images. A useful comparison among them is offered, illustrating that HDR input can have a significant impact on the quality of the result.

Fig. 4.4 shows an example of a differently exposed multiview set corresponding to one frame in a multiscopic system of three views. The main goal of stereo matching is to find the correspondences between pixels to generate one HDR image per view for each frame.

Correspondence methods rely on matching cost functions for computing the similarity of images. It is important to consider that even when one uses radiance space images, there might be brightness differences. Such differences may be introduced by the camera because of image noise or slightly different settings or vignetting. For good analysis and comparison between existing matching costs and their properties, see (Scharstein and Szeliski, 2002; Hirschmuller and Scharstein, 2009; Bonnard et al., 2014).

Many approaches have been presented to recover HDR from multiview and multiexposed sets of images. Some of them (Troccoli et al., 2006; Lin and Chang, 2009; Sun et al., 2010) share the same pipeline as in Fig. 4.5. All the work mentioned takes as input a set of images with different exposures acquired with a camera with unknown response function. In such cases, the disparity maps need to be calculated in the first instance with use of LDR pixel values. Matching images under important differences of brightness is still a big challenge in computer vision.

4.4.3.2 Per frame CRF recovery methods

To our knowledge, Troccoli et al. (2006) proposed the first technique for HDR recovery from multiscopic images of different exposures. They observed that the normalized cross-correlation (NCC) is approximately invariant to exposure changes when the camera has a gamma response function. Under such an assumption, they used the algorithm described by Kang and Szeliski (2004) to compute the depth maps that maximize the correspondence between one pixel and its projection in the other image. The original approach proposed by Kang and Szeliski (2004) used the SSD but it was replaced by the NCC. Eqs. (4.1) and (4.2) show how to calculate the SSD and NCC, respectively, for two image patches of N pixels centered in p and q of images I_L and I_R.

Images are warped to the same viewpoint with the depth map. Once pixels have been aligned, the CRF is calculated by the method proposed by Grossberg and Nayar (2003) over a selected set of matches. The problem that arises from the use of the NCC is that this introduces ambiguity within the equivalence gamma response. With the CRF and the exposures, all images are transformed to radiance space and the matching process is repeated, this time with the SSD. The new depth map improves the previous one and helps to correct artifacts. The warping is updated and HDR values are calculated by a weighted average function.

$\begin{array}{ll}\hfill \text{SSD}& =\sum _{q,p\in N}{(I_{\mathrm{L}}(q)-I_{\mathrm{R}}(p))}^2,\hfill \end{array}$

$\begin{array}{ll}\hfill \text{NCC}& =\frac{\sum _{q,p\in N}{I}_{\mathrm{L}}(q)\cdot I_{\mathrm{R}}(p)}{\sqrt{\sum _{q,p\in N}{I}_{\mathrm{L}}{(q)}^2\cdot \sum _{q,p\in N}{I}_{\mathrm{R}}{(p)}^2}}.\hfill \end{array}$

The same problem was addressed by Lin and Chang (2009). Instead of the NCC, they use scale-invariant feature transform (SIFT) descriptors to find matches between LDR stereo images. SIFT is not robust for different-exposure images. Only the matches that are coherent with the epipolar and exposure constraints are selected for the next step. The selected pixels are used to calculate the CRF.

The stereo matching algorithm they propose is based on previous work (Sun et al., 2003). Belief propagation is used to calculate the disparity maps. The stereo HDR images are calculated by means of a weighted average function. Even with the best results, SIFT is not robust enough if there are significant exposure variations.

A ghost removal technique is used afterward to treat the artifacts due to noise or stereo mismatches. The HDR image is exposed to the best exposure. The difference between them is calculated and pixels over a threshold are rejected, considering them like mismatches. This is risky because HDR values in areas underexposed and overexposed in the best exposure may be rejected. In this case the problem of ghosting would be solved but LDR values may be introduced in the resulting HDR image.

Sun et al. (2010) (inspired by Troccoli et al., 2006) also follow the pipeline described in Fig. 4.5. They assume that the disparity map between two rectified stereo images can be modeled as a Markov random field. The matching problem is presented like a Bayesian labeling problem. The optimal label (disparity) values are obtained by minimization of an energy function. The energy function they use is composed of a pixel dissimilarity term (NCC in their solution) and a disparity smoothness term. It is minimized by the graph cut algorithm to produce initial disparities. The best disparities are selected to calculate the CRF with the algorithm proposed by Mitsunaga and Nayar (1999). Images are converted to radiance space and another energy minimization is performed to remove artifacts. This time the pixel dissimilarity cost is computed with the Hamming distance between candidates.

The methods presented so far have a high computational cost. Calculating the CRF from nonaligned images may introduce errors because the matching between them may not be robust. Two exposures are not enough to obtain a robust CRF with existing techniques. Some of them execute two passes of the stereo matching algorithm, the first one to detect matches for the CRF recovery and the second one to refine the matching results. One might avoid this by calculating the CRF in a previous step using multiple exposures of static scenes. Any of the available techniques (Mann and Picard, 1995; Debevec and Malik, 1997; Mitsunaga and Nayar, 1999; Grossberg and Nayar, 2003) can be used to get the CRF corresponding to each camera. The curves help to transform pixel values into radiance for each image, and the matching process is executed in radiance space images. This avoids one stereo matching step and prevents errors introduced by disparity estimation and image warping.

4.4.3.3 Offline CRF recovery methods

Bonnard et al. (2012) proposed a method to create content that combines depth and HDR video for autostereoscopic displays. Instead of varying the exposure times, they use neutral density filters to capture different exposures. A camera with eight synchronized objectives and three pairs of 0.3, 0.6, and 0.9 filters plus two nonfiltered views provides eight views with four different exposures of the scene stored in 10-bit RAW files. They used a geometry-based approach to recover depth information from epipolar geometry. Depth maps drive the pixel matching procedure.

Bätz et al. (2014) presented a workflow for disparity estimation divided into the following steps:

• Cost initialization consists in evaluating the cost function for all values within a disparity search range. They use zero-normalized cross correlation (ZNCC), defined in Eq. (4.3):

$\begin{array}{l}\hfill \text{ZNCC}=\frac{\sum _{q,p\in N}(I_{\mathrm{L}}(q)-{\stackrel{-}{I}}_{\mathrm{L}}(q))\cdot (I_{\mathrm{R}}(p)-{\stackrel{-}{I}}_{\mathrm{R}}(p))}{\sqrt{\sum _{q,p\in N}{(I_{\mathrm{L}}(q)-{\stackrel{-}{I}}_{\mathrm{L}}(q))}^2\cdot \sum _{q,p\in N}{(I_{\mathrm{R}}(p)-{\stackrel{-}{I}}_{\mathrm{R}}(p))}^2}}.\end{array}$

The matching is performed on the luminance channel of a radiance space image with patches of 9 × 9 pixels. The result of searching for disparities is the disparity space image (DSI), a matrix of m × n × d + 1 for an image of m × n pixels, with d + 1 being the disparity search range.

• Cost aggregation is done to smooth the DSI and find the actual disparity of each pixel in the image. They use an improved version of the cross-based aggregation method described by Mei et al. (2011). This step is performed over the actual RGB images, not in the luminance channel as in the previous step.

• Image warping is responsible for actually shifting all pixels according to their disparities. How to deal with occluded areas between the images is the main challenge. Bätz et al. (2014) propose doing the warping in the original LDR images, which adds a new challenge: dealing with underexposed and overexposed areas. A backward image warping is chosen to implicitly ignore the saturation problems. The algorithm produces a new warped image with the appearance of the reference one by using the target image and the corresponding disparity map. Bilinear interpolation is used to retrieve values at subpixel precision.

Selmanovic et al. (2014) propose generating stereo HDR video from a pair of HDR and LDR videos, using an HDR camera (Chalmers et al., 2009) and a traditional digital camera (Canon 1Ds Mark II) in a stereo configuration. Their work is an extension to video of previous work (Selmanović et al., 2013) focused only on stereo HDR images. In this case, one HDR view needs to be reconstructed from two completely different sources.

Their method proposes three different approaches to generate the HDR video:

1. Stereo correspondence is computed to recover the disparity map between the HDR and LDR images. The disparity map allows the HDR values to be transferred to the LDR image. The SAD (Eq. 4.4) is used as a matching cost function. Both images are transformed to Lab color space, which is perceptually more accurate than RGB.

$\begin{array}{l}\hfill \text{SAD}=\sum _{q,p\in N}|I_{\mathrm{L}}(q)-I_{\mathrm{R}}(p)|.\end{array}$

The selection of the best disparity value for each pixel is based on the winner takes all technique. The lower SAD is selected in each case. An image warping step based on the work of Fehn (2004) is used to generate a new HDR image corresponding to the LDR view. The SAD stereo matcher can be implemented to run in real time but the resulting disparity maps could be noisy and not accurate. The overexposed and underexposed pixels may end up in the wrong position. In large areas of the same color and hence same SAD cost, the disparity will be constant. Occlusions and reflective or specular objects may cause some artifacts.

2. The expansion operator could be used to produce an HDR image from the LDR view. Detailed state-of-the-art reports on LDR expansion have been presented by Banterle et al. (2009) and Hirakawa and Simon (2011). However, in this case, we need the expanded HDR to remain coherent with the original LDR. Inverse tone mappers are not suitable because the resulting HDR image may be very different from the acquired one, producing results that are not possible to fuse through a common binocular vision.
Selmanovic et al. (2014) propose an expansion operator based on a mapping between the HDR and the LDR image using the first one as a reference. A reconstruction function maps LDR to HDR values (Eq. 4.5) based on an HDR histogram with 256 bins putting the same number of HDR values in each bin as there are in the LDR histogram.

$\begin{array}{l}\hfill \text{RF}=\frac{1}{\text{Card}({\Omega }_c)}\sum _{i=M(c)}^{M(c)+\mathrm{Card}({\Omega }_c)}{C}_{\mathrm{hdr}}(i).\end{array}$

In Eq. (4.5), Ω_c = { j = i. .N : c_ldr( j) = c}, c = 0. .255 is the index if a bin Ω_c, Card(⋅) returns the number of elements in the bin, N is the number of pixels in the image, c_ldr(j) are the intensity values for pixel j, $M(c)={\sum }_0^c\text{Card}({\Omega }_c)$ is the number of pixels in the previous bin, and c_hdr are the intensities of all HDR pixels sorted in ascending order. RF is used to calculate the look-up table and afterward expansion can be performed directly, assigning the corresponding HDR value to each LDR pixel.
The expansion runs in real time, is not view dependent, and avoids stereo matching. The main limitation is again on saturated regions.

3. The hybrid method combines the two previous methods. Two HDR images are generated by the previous approaches (stereo matching and expansion operator). Pixels in well-exposed regions are expanded by the first method (expansion operator), whereas matches for pixels in underexposed or overexposed regions are found by SAD stereo matching with the addition of a correction step. A mask of undersaturated and oversaturated regions is created by use of a threshold for pixels over 250 or below 5. The areas out of the mask are filled in with the expansion operator, and the underexpose or overexposed regions are filled in with an adapted version of the SAD stereo matching to recover more accurate values in overexposed or underexposed regions.

Instead of having the same disparity over the whole underexposed or overexposed region, this variant interpolates disparities from well-exposed edges. Edges are detected by a fast morphological edge detection technique described by Lee et al. (1987). Nevertheless, some small artifacts may still be produced by the SAD stereo matching in such areas.

Orozco et al. (2015) presented a method to generate multiscopic HDR images from LDR multiexposure images. They adapted a patch match approach to find matches between stereo images using epipolar geometry constrains. This method reduces the search space in the matching process and improves the incoherence problem of the patch match. Each image in the set of multiexposed images is used as a reference, looking for matches in all the remaining images. These accurate matches allow one to synthesize images corresponding to each view which are merged into one HDR per view that can be used in autostereoscopic displays.