1.1 Stereoscopic Imaging

Apart from a huge amount of colors and fine details, our visual system is able to perceive depth and 3D shape of objects. Digital images offer a representation of reality projected in 2D arrays. We can guess the distribution of objects in depth because of monoscopic cues like perspective, but we cannot actually perceive depth in 2D images. Our brain needs to receive two slightly different projections of the scene to actually perceive depth.

Stereoscopy is any imaging technique which enhances or enables depth perception using the binocular vision cues [15]. Stereo images refer to a pair of images horizontally aligned and separated at a scalable distance similar to the average distance between human eyes. The different available stereo display systems project them in a way such that each eye perceives only one of the images. In recent years, technologies like stereoscopic cameras and displays have become available to consumers [16–18]. Stereo images require recording at minimum, two views of a scene, one for each eye. However, depending on the display technology, it could be more. Some auto-stereoscopic displays render more than nine different views for an optimal viewing experience [1].

Some prototypes were proposed to acquire stereo HDR content from two or more differently exposed views. Most approaches [2–5, 8, 19, 20] are based on a rig of two cameras placed like a conventional stereo configuration that captures different exposed images. The next sections offer a background of the geometry of stereo systems (Section 1.2) as well as a survey on the different existing approaches for multiscopic HDR acquisition (Section 2).

1.2 Epipolar Geometry

One of the most popular topics of research in computer vision is stereo matching, which refers to the correspondence between pixels of stereo images. The geometry that relates 3D objects to their 2D projection in stereo vision is known as epipolar geometry. It explains how the stereo images are related and how depth can mathematically be retrieved from a pair of images.

Fig. 2 describes the main components of the epipolar geometry. A point x in the 3D world coordinates is projected onto the left and right images I_L and I_R, respectively. c_L and c_R are the two centers of projection of the cameras, the plane formed by them, and the point x is known as the epipolar plane. x_L and x_R are the projections of x in I_L and I_R, respectively.

For any point x_L in the left image, the distance to x is unknown. According to the epipolar geometry, the corresponding point x_R is located somewhere on the right epipolar line. Epipolar geometry does not mean direct correspondence between pixels. However, it reduces the search for a matching pixel to a single epipolar line. The accurate position of a point in the space requires the correct matches between the two images, the focal length, and the distance between the two cameras. Otherwise, only relative measures can be approximated.

If the image planes are aligned and their optical axes are parallel, the two epipolar lines (left and right) converge. In such a case, correspondent pixels rely on the same epipolar line in both images, which simplifies the matching process. Aligning the cameras to force this configuration might be difficult, but images can be aligned.

This alignment process is known as rectification. After the images are rectified, the search space for a pixel match is reduced to the same row, which is the epipolar line. To the best of our knowledge, all methods in stereo HDR are based on rectified images and they take advantage of the epipolar constrain during the matching process. Rectified image sets are available on the Internet for testing purposes, like Middlebury [21].

If corresponding pixels (matches) are on the same row on both images, it is possible to define the difference between the images by the horizontal distance between matches in the two images. The image that stores all the horizontal shifts between stereo pairs is called disparity maps.

Despite epipolar geometry simplifying the problem, it is far from being solved. Determining pixel matches in regions of similar color is a difficult problem. Moreover, the two views correspond to different projections of the scene, which means that occlusion takes place between objects. The HDR context adds the fact that different views might be differently exposed, reducing the possibilities of finding color consistent matches.

2.1 Per Frame CRF Recovery Methods

To our knowledge, Troccoli et al. [3] introduced the first technique for HDR recovery from multiscopic images of different exposures. They observed that normalized cross correlation (NCC) is approximately invariant to exposure changes when the camera has a gamma response function. Under such an assumption, they use the algorithm described by Kang and Szeliski [25] to compute the depth maps that maximizes the correspondence between 1 pixel and its projection in the other image. The original approach [25] used sum of squared differences (SSD), but it was substituted by NCC in this work.

Images are warped to the same viewpoint using the depth map. Once pixels are aligned, the CRF is calculated using the method proposed by Grossberg and Nayar [26] over a selected set of matches. With the CRF and the exposure values, all images are transformed to radiance space and the matching process is repeated, this time using SSD. The new depth map improves the previous one and helps to correct artifacts. The warping is updated and HDR values are calculated using a weighted average function.

The same problem was addressed by Lin and Chang [4]. Instead of NCC, they use scale-invariant feature transform (SIFT) descriptors to find matches between LDR stereo images. SIFT is not robust under different exposure images. Only the matches that are coherent with the epipolar and exposure constraints are selected for the next step. The selected pixels are used to calculate the CRF.

The stereo matching algorithm they propose is based on a previous work [27]. Belief propagation is used to calculate the disparity maps. The stereo HDR images are calculated by means of a weighted average function. Even using the best results only, SIFT is not robust enough under significant exposure variations.

A ghost removal technique is used afterward to tackle the artifacts, due to noise or stereo mismatches. The HDR image is exposed to the best exposure of the sequence. The difference between them is calculated and pixels over a threshold are rejected considering them like mismatches. This is risky because HDR values in areas under- and overexposed in the best exposure may be rejected. In this case ghosting would be solved, but LDR values may be introduced in the resulting HDR image.

Sun et al. [5] (inspired by Troccoli et al. [3]) follow the pipeline described in Fig. 4 too. They assume that the disparity map between two rectified stereo images can be modeled as a Markov random field. The matching problem is presented like a Bayesian labeling problem. The optimal disparity values are obtained by minimizing an energy function.

The energy function they use is composed of a pixel dissimilarity term (NCC in their solution) and a disparity smoothness term. It is minimized using the graph cut algorithm to produce initial disparities. The best disparities are selected to calculate the CRF with the algorithm proposed by Mitsunaga and Nayar [28].

Images are converted to radiance space and then another energy minimization is executed to remove artifacts. This time the pixel dissimilarity cost is computed using the Hamming distance between candidates.

The methods presented up to this point have a high computational cost. Calculating the CRF from nonaligned images may introduce errors because the matching between them may not be robust. Two exposures are not enough to obtain a robust CRF with existing techniques. Some of them execute two passes of the stereo matching algorithm, the first one to detect matches for the CRF recovery and a second one to refine the matching results. This might be avoided by calculating the CRF in a previous step using multiple exposures of static scenes. Any of the available techniques [26, 28–30] can be used to get the CRF corresponding to each camera. The curves help to transform pixel values into radiance for each image and the matching process is executed in radiance space images. This avoids one stereo matching step and prevents errors introduced by disparity estimation and image warping.

Rufenacht [19] proposes two different ways to obtain stereoscopic HDR video content. The first, a temporal approach, where exposures are captured by temporally changing the exposure time of two synchronized cameras to get two frames of the same exposure per shot. And the second, a spatial approach, where cameras have different exposure times for all the shots so the two frames of the same shot are exposed differently.

2.2 Offline CRF Recovery Methods

Bonnard et al. [7] propose a methodology to create content that combines depth and HDR video for auto-stereoscopic displays. Instead of varying the exposure times, they use neutral density filters to capture different exposures. A camera with eight synchronized objectives and three pairs of 0.3, 0.6, and 0.9 filters, plus two nonfiltered views provide eight views with four different exposures of the scene stored in 10-bit RAW files. They use a geometry-based approach to recover depth information from epipolar geometry. Depth maps drive the pixel match procedure.

Bätz et al. [2] present a work flow for disparity estimation, divided in the following steps:

• Cost initialization consists of evaluating the cost function, Zero NCC in this case, for all values within a disparity search range.
The matching is performed on the luminance channel of radiance space image using patches of 9 × 9 pixels. The result of searching for disparities is the disparity space image (DSI), a matrix of m × n × d + 1 for an images of m × n pixels with d + 1 being the disparity search range.

• Cost aggregation smoothes the DSI and finds the final disparity of each pixel in the image. They use an improved version of the cross-based aggregation method described by Mei et al. [31]. This step is performed, not in the luminance channel like in the previous step, but in the actual RGB images.

• Image warping is in charge of actually shifting all pixels according to their disparities. Dealing with occluded areas between the images is the main challenge in this step. The authors propose to do the warping in the original LDR images which adds a new challenge: dealing with under- and overexposed areas. A backward image warping is chosen to implicitly ignore the saturation problems. The algorithm produces a new warped image with the appearance of the reference one by using the target image and the corresponding disparity map. Bilinear interpolation is used to retrieve values at subpixel precision.

Selmanovic et al. [10] propose to generate stereo HDR video from a pair of HDR and LDR videos, using an HDR camera [32] and a traditional digital camera (Canon 1Ds Mark II) in stereo configuration. This chapter is an extension to video of a previous one [15] focused only on stereo HDR images. In this case, one HDR view needs to be reconstructed from two different sources.

Their method proposes three different approaches to generate the HDR:

1. Stereo correspondence: It is computed to recover the disparity map between the HDR and the LDR images. The disparity map allows us to transfer the HDR values to the LDR image. The sum of absolute differences (SAD) is used as a matching cost function. Both images are transformed to lab color space, which is perceptually more accurate than RGB.
The selection of the best disparity value for each pixel is based on winner takes all technique. The lower SAD value is selected in each case. An image warping step based on Fehn [33] is used to generate a new HDR image corresponding to the LDR view. The SAD stereo matcher can be implemented to run in real time, but the resulting disparity maps could be noisy and not accurate. The over- and underexposed pixels may end up in a wrong position. In large areas of the same color and hence same SAD cost, the disparity will be constant. Occlusions, reflective or specular objects may cause some artifacts.

2. Expansion operator: It could be used to produce an HDR image from the LDR view. Detailed state-of-the-art reports on LDR expansion were previously published [34, 35]. However, in this case, we need the expanded HDR to remain coherent with the original LDR. Inverse tone mappers are not suitable because the resulting HDR image may be different from the acquired one, producing results not possible to fuse through a common binocular vision.
They propose an expansion operator based on a mapping between the HDR and the LDR image, using the first one as reference. A reconstruction function maps LDR to HDR values (Eq. 1) based on an HDR histogram with 256 bins, putting the same number of HDR values in each bin, as there are in the LDR histogram.

$\begin{array}{l}\hfill RF=\frac{1}{Card({\mathrm{\Omega }}_c)}\sum _{i=M(c)}^{M(c)+Card({\mathrm{\Omega }}_c)}{C}_{\mathrm{hdr}}(i)\end{array}$

In Eq. (1), Ω_c = {j = i…N : c_ldr(j) = c}, c = 0.255 is the index if a bin Ω_c, Card(⋅) returns the number of elements in the bin, N is the number of pixels in the image, c_ldr(j) are the intensity values for the pixel j, $M(c)={\sum }_0^{c}Card({\mathrm{\Omega }}_c)$ is the number of pixels in the previous bin, and c_hdr are the intensities of all HDR pixels sorted ascending. RF is used to calculate the look-up table and afterward expansion can be performed directly assigning the corresponding HDR value to each LDR pixel.
The expansion runs in real time, is not view dependent, and avoids stereo matching. The main limitation is again on saturated regions.

3. Hybrid method combines the two previous ones. Two HDR images are generated using the previous approaches (stereo matching and expansion operator). Pixels in well-exposed regions are expanded using the first method (expansion operator), while matches for pixels in under- or overexposed regions are found using SAD stereo matching adding a correction step. A mask of under- and oversaturated regions is created, using a threshold for pixels over 250 or below 5. The areas out of the mask are filled in with the expansion operator while the under- or overexposed regions are filled in with an adapted version of the SAD stereo matching to recover more accurate values in over- or underexposed regions.

Instead of having the same disparity over the whole under- or overexposed region, this variant interpolates disparities from well-exposed edges. Edges are detected using a fast morphological edge detection technique described by Lee et al. [36]. Even though some small artifacts may still be produced by the SAD stereo matching in such areas.

Orozco et al. [37] presented a method to generate multiscopic HDR images from LDR multiexposure images. They adapted a patch-match approach [13] to find matches between stereo images using epipolar geometry constrains. This method reduces the search space in the matching process, and includes an improvement of the incoherence problem described for the patch-match algorithm. Each image in the set of multiexposed images is used as a reference, looking for matches in all the remaining images. These accurate matches allow us to synthesize images corresponding to each view, which are merged into one HDR per view that can be used in auto-stereoscopic displays.

3.1 Nearest Neighbor Search

For a pair of images I_r and I_s, we compute a nearest neighbor field (NNF) from I_r to I_s using an improved version of the method presented by Barnes et al. [12]. NNF is defined over patches around every pixel coordinate in image I_r for a cost function D between two patches of images I_r and I_s. Given a patch coordinate r ∈ I_r and its corresponding nearest neighbor s ∈ I_s, NNF(r) = s. The values of NNF for all coordinates are stored in an array with the same dimensions as I_r.

We start initializing the NNFs using random transformation values within a maximal disparity range on the same epipolar line. Consequently the NNF is improved by minimizing D until convergence or a maximum number of iterations is reached. Two candidate sets are used in the search phase as suggested by Barnes et al. [12]:

1. Propagation uses the known adjacent nearest neighbor patches to improve NNF. It quickly converges, but it may fall in a local minimum.

2. Random search introduces a second set of random candidates that are used to avoid local minimums. For each patch centered in pixel v₀, the candidates u_i are sampled at an exponentially decreasing distance v_i previously defined by Barnes et al.:

$\begin{array}{l}\hfill u_i=v_0+w{\alpha }^i{R}_i\end{array}$

where R_i is a uniform random value in the interval [−1, 1], w is the maximum value for disparity search, and α is a fixed ratio (1/2 is suggested).

Taking advantage of the epipolar geometry, both search accuracy and computational performance are improved. Geometrically calibrated images allow us to reduce the search space from 2D to 1D domain, consequently reducing the search domain. The random search of matches only operates in the range of maximum disparity in the same epipolar line (1D domain), avoiding a search in 2D space. This significantly reduces the number of samples to find a valid match.

However, the original NNFs approach [12] used in the patch-match algorithm has two main disadvantages, the lack of completeness and coherency. These problems are illustrated in Fig. 7 and the produced artifact in Fig. 8. The lack of coherency refers to the fact that two neighbor pixels in the reference image may match two separated pixels in the source image, like in Fig. 7A. Completeness issues refer to more than 1 pixel in the reference image matching the same correspondence in the source image, as shown in Fig. 7B.

To overcome this drawback, we propose a new distance cost function D by incorporating a coherence term to penalize matches that are not coherent with the transformation of their neighbors. Both Barnes et al. [12] and Sen et al. [13] use the SSD described in Eq. (4) where T represents the transformation between patches of N pixels in images I_r and I_s. We propose to penalize matches with transformations that differ significantly from it neighbors by adding the coherence term C defined in Eq. (5). The variable d_c represents the Euclidean distance to the closest neighbor’s match and Max_disp is the maximum disparity value. This new cost function forces pixels to preserve coherent transformations with their neighbors.

$\begin{array}{ll}\hfill D& =SSD(r,s)/C(r,s)\hfill \end{array}$

$\begin{array}{ll}\hfill SSD& =\sum _{n=1}^N{\left(I_{\mathrm{r}}-T(I_{\mathrm{s}})\right)}^2\hfill \end{array}$

$\begin{array}{ll}\hfill C(r,s)& =1-d_c(r,s)/Ma{x}_{\mathrm{disp}}\hfill \end{array}$

Fig. 8D and F corresponds to the results including the improvements presented in this section. Fig. 8C and D shows a color mentioned of the NNFs using HSV color space. The magnitude of the transformation vector is visualized in the saturation channel and the angle in the hue channel. Areas represented with the same color in the NNF color mentioned mean similar transformation. Objects in the same depth may have similar transformation. Notice that the original Patch Match [12] finds very different transformations for neighboring pixels of the same objects and produces artifacts in the synthesized image.

3.2 Image Alignment and HDR Generation

The nearest neighbor search step finds correspondences among all the different views. The matches are stored in a set of n² − n NNFs. This information allows to generate n − 1 images with different exposures realigned on each view. The set of aligned multiple exposures per view feeds the HDR generation algorithm to produce an HDR image for every view (see Fig. 9).

Despite the improvements in the cost function presented in the previous section, NNF may not be coherent in occluded or saturated areas. However, even in such cases, a match to a similar color is found between each pair of images I_r;I_s. This makes it possible to synthesize images for each exposure corresponding to each view.

Direct warping from the NNFs is an option, but it may generate visible artifacts as shown in Fig. 10. We use Bidirectional Similarity Measure (BDSM) (Eq. 6), proposed by Simakov et al. [38] and used by Barnes et al. [12], which measures similarity between pairs of images. The warped images are generated as an average of the patches that contribute to a certain pixel. It is defined in Eq. (6) for every patch Q$\subset I_{\mathrm{r}}$ and P$\subset I_{\mathrm{s}}$, and a number N of patches in each image, respectively. It consists of two terms: coherence that ensures that the output is geometrically coherent with the reference and completeness that ensures that the output image maximizes the amount of information from the source image:

$\begin{array}{l}\hfill d(I_{\mathrm{r}},I_{\mathrm{s}})=\stackrel{d_{\mathrm{completeness}}}{\overbrace{\frac{1}{N_{I_{\mathrm{r}}}}\sum _{Q\subset I_{\mathrm{r}}}\underset{P\subset I_{\mathrm{s}}}{min}D(Q,P)}}+\stackrel{d_{\mathrm{coherence}}}{\overbrace{\frac{1}{N_{I_{\mathrm{s}}}}\sum _{P\subset I_{\mathrm{s}}}\underset{Q\subset I_{\mathrm{r}}}{min}D(P,Q)}}\end{array}$

This improves the results by using bidirectional NNFs ($I_{\mathrm{r}}\to I_{\mathrm{s}}$ and backward, $I_{\mathrm{r}}\leftarrow I_{\mathrm{s}}$). It is more accurate to generate images using only two iterations of nearest neighbor search and bidirectional similarity, than four iterations of neighbor search and direct warping. Table 1 shows some values of mean squared error (MSE) and peak signal-to-noise ratio (PSNR) of images warped like the ones in Fig. 10 comparing to the reference LDR image. The values in the table corresponds to the average MSE and PSNR calculated per each channel of the images in L*a*b* color space, using Eqs. (7) and (8), respectively.

$\begin{array}{ll}\hfill MSE(I,I^{\prime })& =\frac{1}{N}\sum _{1=0}^N{\left(I(i)-I^{\prime }(i)\right)}^2\hfill \end{array}$

$\begin{array}{ll}\hfill PSNR(I,I^{\prime })& =10{log}_{10}\left(\frac{MA{X}^2}{MSE(I,I^{\prime })}\right)\hfill \end{array}$

Because the matching is totally independent for pairs of images, it was implemented in parallel. Each image matches the remaining other views. This produces n − 1 NNFs for each view. The NNFs are, in fact, the two components of the BDSM of Eq. (6). The new image is the result of accumulating pixel colors of each overlapping neighbor patch and averaging them.

$\begin{array}{ll}\hfill E(i,j)& =\frac{\sum _{n=1}^{N}w(I_n(i,j))\left(\frac{f^{-1}(I_n(i,j))}{\mathrm{\Delta }{t}_n}\right)}{\sum _{n=1}^{N}w(I_n(i,j))}\hfill \end{array}$

$\begin{array}{ll}\hfill w(I_n)& =1-{\left(2\frac{I_n}{255}-1\right)}^{12}\hfill \end{array}$

The HDR images (one HDR per view) are generated using a standard weighted average [28–30] as defined in Eq. (9) and the weighting function of Eq. (10) proposed by Khan et al. [39] where I_n represents each image in the sequence, w corresponds to the weight, f is the CRF, Δt_n is the exposure time for the Ith image of the sequence.