4.1 Single Aperture Multiple Sensor Solutions

These groups of algorithms record the scene from a single aperture. The light that enters the aperture is then projected onto multiple sensors through a beam-splitting element. Each sensor is set to a different exposure value, which allows it to capture a different range of illumination. The sensor outputs are then combined to produce an HDR frame. This form of image generation is also called split-aperture imaging [2]. In general, this group of algorithms entirely avoids alignment and ghosting problems, as all sensors receive the same input.

A pioneering algorithm of this kind, in fact, utilizes a single sensor placed behind a transparency mask with multiple bands [89]. In other words, although there is a single sensor, different parts of the sensor receive different amounts of light depending on the transparency of the mask that is adjacent to it. The camera is designed as a panoramic camera, which rotates around its central axis. This allows the same scene point to be projected on different parts of the sensor with different exposures. As such, each scene point will be captured under multiple exposures. As the angular step size of the motor is known, one can compute the corresponding sensor positions of the same scene points, allowing them to be merged into a panoramic HDR image. This work does not propose a solution for dealing with potential changes in the scene during the capture process and therefore, it is suitable to capture only static scenes. A later work by the same authors extends this method to capture images at high frame rates, which are suitable to capture HDR videos, by using a beam-splitting mirror [67].

Wang and Raskar use a split-aperture camera to capture HDR videos [68]. The corner of a cube is used to generate a three-face pyramid, which transmits light toward three CCD sensors. Each sensor is adjacent to an ND filter with transmittances of 1, 0.5, and 0.25. The sensors are carefully aligned with respect to the pyramid such that the sensors are normal to the optical axis.

The beam-splitting approaches discussed earlier are wasteful of light in that they admit only a fraction of the light toward sensors. For example, if three-way beam-splitting is used, each sensor receives only one-third of the incident light. This light is further attenuated by the ND filters adjacent to the sensors. This reduction in light compels the photographer to use longer exposure times, which in turn may introduce problems such as thermal noise and motion blur. To address this problem, Tocci et al. propose a novel system that uses two beam-splitters which allow reusing the optical path to improve light efficiency [69]. Furthermore, they modify the HDR merging equation to consider the neighbors of a pixel rather than treating each pixel individually. This mitigates the cross-talk artifacts that may occur, for example, if a neighboring pixel is saturated, but the pixel itself not. Furthermore, it allows a smoother transition between pixels taken from different exposures.

The methods discussed earlier assume that sensors are perfectly aligned such that the same scene point is projected to the same locations in different sensors. However, this may be difficult to achieve in practice. Kronander et al. address this problem by assuming there exists an affine transform relating the coordinate system of each sensor to a virtual reference [70]. This transform is found in an offline stage by matching the corners of a checkerboard pattern. This method is later extended to capture HDR videos that perform demosaicking, denoising, alignment, and HDR assembly within a unified approach [71].

4.2 Multiple Camera Solutions

These group of algorithms utilize two or more cameras installed on a custom rig. Each camera captures the scene from its own viewpoint and with a different exposure time. The recorded frames must be registered prior to combining them to create HDR frames.

The first work that utilizes a multicamera solution was proposed by Ramachandra et al. [72]. In their system, the authors utilize a two-camera setup, whereby one camera is set to a short exposure time and the other to a long exposure time. As the longer exposed frames may be subject to motion-blur, the first step of their algorithm is deblurring of the long exposure. To this end, first, the histogram of the short exposure is matched to that of the long exposure to normalize exposure differences. Then for each 128 × 128 image patch for the blurry image (called target patch), a corresponding reference patch is found from the sharp exposure. Next, both the target and reference patches are subjected to a multiscale multiorientation pyramid decomposition. Then several features extracted from the reference subbands (such as the moments and the correlation of the coefficients) are applied as constraints on the corresponding features of the target subbands. The collapsed target subbands then produce the deblurred long exposure. However, because there could be local (smaller than the 128 × 128 patch size) variations between the patches, this process may produce ringing artifacts. These artifacts are fixed by applying a deringing filter. Once the long exposure is sharpened by applying this process and optical-flow based registration is performed, as in Kang et al. [80], the HDR frame is reconstructed using Debevec and Malik’s algorithm [4].

Bonnard et al., on the other hand, propose a system composed of eight cameras [73]. These cameras are installed in a camera box with aligned and synchronized objectives [90]. Normally, this setup is designed to produce 3D content for autostereoscopic displays. The authors furnish this system with HDR capabilities by placing ND filters in front of the camera objectives. This produces eight differently exposed frames that are captured simultaneously. The disparity maps between these frames are computed by using Niquin et al.’s algorithm [91]. Before the application of this algorithm, differently exposed frames are normalized to the same average intensity level, using the light blockage ratios of the ND filters. After the disparity map calculation, each pixel is assigned to a set of matching pixels from the other views. This information is used to generate eight autostereoscopic HDR frames using the HDR reconstruction algorithm of Debevec and Malik [4].

The system proposed by Bätz et al. [75] is comprised of two cameras and two workflows. In the offline calibration workflow, the response curves of both cameras are estimated. This has to be done only once for a given pair of cameras. The main workflow is comprised of several stages. In the first one, stereo rectification is performed, which results in geometrically transformed images such that a given point in the left view resides on the same scanline as in the right view (see Fig. 3). This simplifies determining correspondences between pixel values, which are performed in the second stage. For measuring similarity between pixel values, zero mean normalized cross correlation (ZNCC) is used as it is resistant to exposure variations. The determined disparity maps are then smoothed to avoid abrupt changes between neighboring pixels.

In the third stage, the disparity map is used to warp the source exposure onto the target exposure. To this end, the target exposure is selected as the one with the fewest number of under- and over-exposed pixels. Backward warping is used which, for every pixel in the target exposure, computes a sampling position from the source exposure. As this is likely to produce noninteger sampling coordinates, bilinear interpolation is used to compute the final color value. Finally, the HDR frame is computed by merging the target exposure with the warped one. However, merging is only performed for pixels that are under- or overexposed in the target exposure, which limits the influence of potential errors that may be introduced in the earlier stages.

Similar to Bonnard et al. [73], the system proposed Selmanovic et al. aspires to generate a stereoscopic HDR video [74]. Their system is comprised of two video cameras with one camera being an HDR camera [92] and the other an LDR one. The goal is to generate the missing HDR frame by using the information from the existing HDR and LDR frames. To this end, three methods are proposed. The first one is based on warping the HDR frame toward the LDR frame based on a disparity map, similar to the method of Bätz et al. [75]. The second method involves directly expanding the dynamic range of the LDR frame using a novel inverse tone mapping approach. The third method is a hybrid of the two in which the well-exposed regions of the LDR frame are expanded and the under- and overexposed regions are warped from the HDR frame. The authors conclude that the hybrid approach produces the highest quality results.

4.3 HDR Sensor Solutions

Various algorithms discussed earlier have been necessitated due to the LDR nature of image sensors. Ideally, using an HDR image sensor would solve the problems of the aforementioned algorithms. While not being the primary focus of this chapter, here we will briefly review HDR sensor solutions for the sake of completeness.

There are several ways to design an HDR image sensor. In general, an image sensor can be considered as an array of photodetectors with each photodetector connected to a readout circuit. There may be an optional layer of optical mask in front of the photodetectors as well. An HDR sensor can be produced by either spatially varying the transmittance of the optical mask [76, 93], by modifying the readout circuitry [94], or by using HDR photodetectors [95].

The idea of using a spatially varying filter resembles that of using a Bayer color filter array to produce color images. However, instead of being selective of the wavelength, spatially varying transmittance filters transmit different amounts of light onto each photodetector. A pioneering work in this direction was proposed by Nayar and Mitsunaga [93], who used a regular array of optical filter elements aligned with photodetectors. The final HDR pixel values are obtained by computing the average of the neighboring pixel values with different exposures. However, this approach results in a slight reduction in resolution, especially in regions that contain sharp edges. Schöberl et al. show that the resolution can be improved if one uses a nonregular pattern for the optical mask due to the sparsity characteristics of natural images in the Fourier domain [76]. In a more recent work, it has been shown that high quality HDR images can be obtained from a single coded exposure using convolutional sparse coding [96].

As for modifying the readout circuitry, several alternatives have been proposed. These are (1) measuring time-to-saturation, (2) performing multiple nondestructive readouts during the capture process, (3) readout with asynchronous self-reset, and (4) synchronous self-reset with residue readout. A quantitative evaluation of these four groups of techniques was conducted by Kavusi and Gamal, who conclude that each technique has different advantages and disadvantages [94].

Finally, an HDR image sensor can be obtained by directly extending the dynamic range of the photodetecting elements. As a reference, we refer the reader to Zhou et al. for a design that involves punchthrough enhanced phototransistors [95].

5.1 View Dependent Enhancement of the Dynamic Range of Video [77]

Among the first global approaches to HDR video deghosting problems, Niskanen presented a global alignment method that produces results that are suitable to be used by computer vision algorithms rather than to be viewed by human observers [77]. Deghosting is achieved by aligning SIFT features [97] with RANSAC sampling [98] to compensate for global motion. Therefore, this technique is only applicable when local motion between the frames is negligible.

The video is generated by combining sequential LDR frames with varying exposure times. A CCD camera was used with linear or known camera response. The acquisition process has been augmented by two techniques that control exposure times adaptively. They are based on the information content. The first technique involves maximizing the entropy of observations;

$\begin{array}{l}\hfill (\mathrm{\Delta }{t}_1,\dots ,\mathrm{\Delta }{t}_N)=\underset{\mathrm{\Delta }{t}_1,\dots ,\mathrm{\Delta }{t}_N}{argmax}H(O_1,\dots ,O_N),\end{array}$

where O₁, …, O_N are the real scene (S) observations, H(.) is the entropy of observations, and Δt₁, …, Δt_N are the exposure times to be selected.

The second technique maximizes the average SNR for all I pixels in the resulting HDR image:

$\begin{array}{l}\hfill (\mathrm{\Delta }{t}_1,\dots ,\mathrm{\Delta }{t}_N)=\underset{\mathrm{\Delta }{t}_1,\dots ,\mathrm{\Delta }{t}_N}{argmax}\frac{\sum _{x,y}{S}^2(x,y)}{\sum _{x,y}{\mathcal{N}}^2(x,y)},\end{array}$

where $S(x,y)=r{\sum }_{i=1}^N{I}_i(x,y)$ with I_i(x, y) being the exposure time normalized and unsaturated signal level for a given pixel in the ith exposure and r is the number of photons required to gain one level of intensity. $\mathcal{N}(x,y)$, on the other hand, represents the noise level and it is modeled as a sum of the photon shot and quantization noise:

$\begin{array}{l}\hfill \mathcal{N}(x,y)=\sqrt{r\sum _{i=1}^N{I}_i+N\frac{r^2}{12}}.\end{array}$

Using either technique, the optimal exposure times can be computed from frame histograms. It is shown by the author that this technique can select suitable exposure times at 30 FPS among 500 different candidate values.

Once the frames are captured, they are registered using RANSAC sampled SIFT features [97]. This is followed by the construction of HDR frames that are based on modeling light radiation as a Poisson process. In the evaluation of this approach, it was noticed that for small motion, to a certain extent, using the multiple frames without alignment gives better performance than using fewer frames with motion correction.

5.2 Histogram-Based Image Registration for Real-Time HDR Videos [78]

The motivation of this method is to capture and display HDR video in real-time. However, it only considers global translational motion between the frames to be registered. This is done by creating row and column histograms that contain counts of dark and bright pixels in a row or column, and maximizing the correlation between the histograms of two consecutive frames. The two-dimensional search problem is reduced to two one-dimensional searches. The reduction in computation time enables real-time HDR video recording. The robustness of the estimation is increased by Kalman filter application [99].

The image capturing is based on the method of [100]. It achieves efficiency by doing only partial reexposures of a single frame. At first, a full resolution frame is captured at optimal exposure (shutter and gain) setting. This frame is then analyzed to indicate which regions of the frame need reexposures. Usually these regions are frame partials. Additional higher or lower exposures are then captured for these regions only, which reduces the capture and processing times. The reexposures are analyzed and, if needed, additional partial reexposures are captured. The captured images are converted to Y xy color space and all subsequent computations are performed on the Y (luminance) channel.

The process of registration creates a two-dimensional integer translation vector ${\overrightarrow{v}}_i$ between a parent exposure and each reexposure. Summing up all translation vectors along the path from an exposure to the base frame, one can calculate an absolute shift between each exposure and the base frame.

Median threshold bitmaps (MTBs) as described in [10] are used for estimation of translation vectors for a pair of images. As MTBs create an image that contains roughly 50% white and 50% black pixels, it produces similar bitmaps for different exposures. Once the MTBs are computed, the number of black pixels in each column is counted and these counts are collected in column histograms for each MTB.

The MTB algorithm starts with building a regular histogram with 256 bins over the brightness values of I_i−1 and its reexposure I_i. Then the median brightness value m_i for reexposure I_i to be used as threshold is determined. Fifty percent of pixels greater than this threshold will be white, while the remaining will be black. The near median pixels are ignored for improved robustness.

The exposure values e_i−1 and e_i of the images as well as the camera response function f are assumed to be known. To save histogram computation over I_i−1, m_i−1 is calculated as follows:

$\begin{array}{l}\hfill m_{i-1}=f\left(f^{-1}(m_i)\frac{e_{i-1}}{e_i}\right).\end{array}$

For the purpose of this method, it is sufficient to determine the thresholds m_i−1 and m_i, and the MTBs are actually not created.

Next, the horizontal and vertical components of the translation vector ${\overrightarrow{v}}_i=(x_i,y_i)$ between a parent exposure and each reexposure are computed. The horizontal (vertical) component is estimated by computing normalized cross correlation (NCC) between the two column (row) histograms.

Assuming that the width and height of image I_i are w_i and h_i, respectively, for each column j = 1, …, w_i, the column histogram $B_i^x(j)$ of exposure I_i is the number of black pixels and is defined as:

$\begin{array}{l}\hfill B_i^x(j)=|\{I_i(j,k)<m_i-T;k=1,\dots ,h_i\}|,\end{array}$

where I_i(j, k) is the pixel value at position (j, k) and |.| denotes set cardinality. Analogously, histogram $W_i^x$ counts the number of white pixels. The two histograms are created for I_i−1 in the same fashion. These four histograms are used to estimate the horizontal shift component, x_i. This is a search problem that maximizes the NCC between I_i−1 and I_i histograms:

$\begin{array}{l}\hfill \mathrm{NCC}(s)=\frac{C}{\sqrt{N_1{N}_2}},\end{array}$

where C, N₁, and N₂ are computed as follows:

$\begin{array}{ll}\hfill C& =\sum _{j=1}^{w_i}(W_i^x(j)W_{i-1}^x(j-s)+B_i^x(j)B_{i-1}^x(j-s)),\hfill \end{array}$

$\begin{array}{ll}\hfill N_1& =\sum _{j=1}^{w_i}(W_i^x{(j)}^2+B_i^x{(j)}^2),\hfill \end{array}$

$\begin{array}{ll}\hfill N_2& =\sum _{j=1}^{w_i}(W_{i-1}^x{(j-s)}^2+B_{i-1}^x{(j-s)}^2).\hfill \end{array}$

The search range for s can be, for example, ± 64 pixels and for the estimation of x_i, the s value that maximizes Eq. (9) is used. Using row histograms, the value of y_i is estimated analogously, which results in the final translation vector of ${\overrightarrow{v}}_i=(x_i,y_i)$.

To incorporate temporal motion estimation Kalman filter is used [99]. To determine the weight of the current translation computation and the preceding trajectory, a novel heuristic is applied. The mean μ and standard deviation σ of the distances d between a pair of consecutive motion vectors (MVs) ($d=|{\overrightarrow{v}}_{i-1}-{\overrightarrow{v}}_i|$) were precomputed for manually registered images. Assuming Gaussian distribution and near zero mean, over 99% of the MVs are found to be within 3σ from the previous vector. A likely indicator of incorrect measurement is d > 3σ and, in this case, the corresponding shift vector is discarded. On the contrary, if d ≤ 3σ, the state of the Kalman filter is updated using ${\overrightarrow{v}}_i$ as the current state and d as the variance of the measurement. In both cases, the current state of the filter is used as the shift vector.

5.3 A Real-Time System for Capturing HDR Videos [79]

In a more recent work by Guthier et al., a unified real-time system for capturing, registering, stitching, and tone mapping HDR videos is presented [79]. Fig. 4 shows an overview of this HDR video processing system.

The aim of the first step is to only use the exposure times with the highest contribution of information to the generated HDR frame. By minimizing the number of captured images, the processing time is reduced, allowing higher frame rates. This is achieved by defining a contribution function c_Δt(E) for each LDR image captured at exposure time Δt. The function computes the contribution of each image captured at Δt to the estimation of radiance E. It is defined as:

$\begin{array}{l}\hfill c_{\mathrm{\Delta }t}(E)=w(f(E\mathrm{\Delta }t)),\end{array}$

where E is the scene radiance, f is the camera response function, and w is a weighting function that measures the well-exposedness of pixel values.

The method computes the optimal exposure time sequence by using radiance histograms, which are generated as a by-product of tone mapping the previous frames. The details of using log-radiance histograms to compute optimal exposure times can be found in [101]. Briefly, the logarithm of the scene radiance histogram contains M bins, (j = 1, …, M), where each index j corresponds to the logarithm of a discrete radiance value: $b_j=log(E_j)$. Each bin j contains H(j) number of pixels in the HDR image having a log-radiance close to b_j. The bins have even spacing in the log domain, thus, Δb = b_j+1 − b_j. Nonlogarithmic radiance values corresponding to two consecutive bins differ by a constant factor $exp(\mathrm{\Delta }b)=exp(b_{j+1})/exp(b_j)=E_{j+1}/E_j$. The optimal exposure times are selected such that the most frequent radiance values are well-exposed in at least one LDR image. This is obtained by ensuring that the peaks of the contribution functions in each exposure coincide with the histogram peaks as shown in Fig. 5. For a different exposure time Δt′, the corresponding contribution function is computed by simply shifting the original function to another position in the histogram. Therefore, the corresponding exposure times are computed by moving the contribution function over the histogram peaks and deriving the corresponding exposure time by using the following formula:

$\begin{array}{l}\hfill \mathrm{\Delta }{t}^{\prime }=exp{(\mathrm{\Delta }b)}^s\mathrm{\Delta }t,\end{array}$

where the contribution vector is shifted by a number of s bins. The computation of new exposure times is repeated until the whole histogram is covered. With this approach, only those LDR images which accurately measure the scene radiance are captured.

The described algorithm for computing exposure times has two problems. The first problem is that the algorithm assumes perfect scene histograms. However, perfect histograms are not available for real-time videos. The problem of imperfect histograms is handled by choosing the first exposure time such that its contribution peak covers the highest radiance bin of the histogram. The reason for choosing the first exposure time in such a way is the fact that underexposed images have more accurate information than overexposed ones. This is because in underexposed images, dark pixels are a noisy estimate of the radiance in the scene, and this noise is unbiased. However, overexposed pixels always have a maximum pixel value, regardless of how bright the scene actually is.

The second problem is flicker that occurs due to the changes in exposure time sequence over time. Furthermore, if the camera is running in a sequence mode (i.e., a sequence of exposure parameters are sent only once and are then repeatedly used throughout the capture), any change in exposure time sequence requires an expensive retransmission of the parameters to the camera. To avoid this, a stability criterion is enforced. This criterion ensures that the new exposure time sequence is only retransmitted when the computed sequence is different from the previous sequence for a number of successive frames. With this approach, temporal stability is also achieved.

The second stage of the proposed system is the alignment of the successive frames, which utilizes the algorithm described in Section 5.2. Once the frames are aligned, HDR stitching is performed using the radiance map recovery technique of Debevec and Malik [4].

The last stage of the system is tone mapping of generated HDR frames. The aim is to apply a tone mapping operator designed for still images while taking measures to avoid flicker. To this end, the average brightness of tone mapped frames is adjusted to be closer to the preceding frames.

6.1 High Dynamic Range Video [80]

This algorithm is considered to be the first approach to generate HDR videos using off-the-shelf cameras by alternating frames between short and long exposures. It is comprised of the following three steps: (1) frame capture using automatic exposure control, (2) HDR stitching across neighboring frames, and (3) temporal tone mapping for display.

6.2 Video Matching [82]

Sand and Teller introduced an algorithm for spatiotemporal alignment of a pair of videos [82]. The input of the proposed algorithm is two videos, which may have been recorded with different exposures. Of these two videos, one is marked as the primary (reference) and the other as the secondary. The output is a modified secondary video whose pixels have been brought to temporal and spatial alignment with the primary video. The proposed method is not specialized to HDR video, but it has been shown that it can be used to create HDR videos as well.

6.3 HDR Video Through Fusion of Exposure Controlled Frames [110]

Youm et al. [110] proposed a technique that employs a novel exposure control method in addition to ghost removal. The proposed method adaptively selects the exposure times of the individual frames during the acquisition of an LDR video with alternating long and short exposures. After the acquisition, the pyramid-based image fusion method of Burt and Adelson [111] is used for the fusion of the captured frames.

The exposure control method adaptively selects the exposure time of the next frame, using the information available in the current frame, and it is based on the adaptive dynamic range imaging method of Nayar and Branzoi [112]. The algorithm uses two user-set parameters: the saturation level Z^sat and the desired intensity level range [Z^des −ΔZ, Z^des + ΔZ]. The exposure time of the current frame, Δt_i, is determined from the exposure time of the previous frame with the similar exposure, Δt_i−2, using the following set of rules assuming that the current frame is a short exposure:

$\begin{array}{l}\hfill \mathrm{\Delta }{t}_i=\left\{\begin{array}{ll}0.5\mathrm{\Delta }{t}_{i-2},& \text{if}{M}^{\text{avg}}\ge Z^{\text{sat}},\\ \mathrm{\Delta }{t}_{i-2}\frac{Z^{\text{des}}}{M^{\text{avg}}},& \text{if}{Z}^{\text{sat}}>M^{\text{avg}}\ge Z^{\text{des}}+\mathrm{\Delta }Z,\\ \mathrm{\Delta }{t}_{i-2},& \text{if}{Z}^{\text{des}}+\mathrm{\Delta }Z>M^{\text{avg}}\ge Z^{\text{des}}-\mathrm{\Delta }Z,\\ \mathrm{\Delta }{t}_{i-2}\left(0.5\frac{Z^{\text{des}}}{M^{\text{avg}}}+0.5\right),& \text{if}{Z}^{\text{des}}-\mathrm{\Delta }Z>M^{\text{avg}},\end{array}\right.\end{array}$

where M^avg is the average intensity in the bright regions of the scene for short-exposure frames. For long exposures, an analogous set of rules is applied.

In the exposure fusion step, the contributions of a pair of long- and short-exposure frames to the output frame depend on the well-exposedness mask M^we and the ghost mask M^g. Given a pixel value Z_i(x, y) from the long exposure and Z_i−1(x, y) from the short exposure, the well-exposedness mask is defined as:

$\begin{array}{l}\hfill M_i^{\text{we}}=\left\{\begin{array}{ll}1,& \text{if}{Z}_i(x,y)<Z^{\text{cut}},\\ \frac{255-Z_i(x,y)}{255-Z^{\text{cut}}},& \text{if}{Z}_i(x,y)\ge Z^{\text{cut}},\end{array}\right.\end{array}$

where Z^cut is the cut-off intensity for the well-exposedness map. The ghost mask M^g is calculated by comparing the current exposure with the previous corresponding exposure. It takes the value of zero if the absolute pixel intensity difference is larger than 20 for 8-bit input frames and one otherwise.

The output frame is a weighted pyramid-based blending [111] of short and long exposures. The blending weight of the long exposure is $M_i^{\text{we}}\cdot M_i^{\text{g}}$ and the blending weight of the short exposure is $1-M_i^{\text{we}}\cdot M_i^{\text{g}}$.

6.4 High Dynamic Range Video With Ghost Removal [86]

Recognizing that dense optical-flow based HDR video [80] cannot cope well with occlusions and fast-moving objects, Mangiat and Gibson proposed a novel method that uses block-based motion estimation instead [86]. Their method is comprised of four stages: (1) block-based motion estimation, (2) MV refinement in saturated blocks, (3) HDR reconstruction, and (4) artifact removal. The details of each stage is explained as follows.

6.4.2 MV Refinement in Saturated Blocks

It is possible that the previous matching algorithm produces incorrect correspondences between blocks. This problem is most pronounced for saturated blocks as for these blocks, the information from the current frame is not used at all. However, although these saturated blocks are not informative, one can still estimate the minimum (or maximum) pixel values that could exist in a matching block. Assuming that the current image is the long exposure and the reference is either the previous or the next short exposure, a saturated pixel of the current exposure should not have been matched with a reference pixel that is too dark to saturate in the current exposure. If this happens, this match is likely to be erroneous.

Therefore, in this stage of the algorithm, the authors determine the minimum pixel value that is valid as:

$\begin{array}{l}\hfill Z_s^{*}=g^{-1}(g(Z_{\text{max}})-ln\mathrm{\Delta }{t}_l+ln\mathrm{\Delta }{t}_s).\end{array}$

  (26)

Any pixel that is smaller than this value in the reference exposure cannot be the correct match for the saturated pixel in the current exposure. Similarly, if the current exposure is a short exposure, a maximum valid value is computed as:

$\begin{array}{l}\hfill Z_l^{*}=g^{-1}(g(Z_{\text{min}})-ln\mathrm{\Delta }{t}_s+ln\mathrm{\Delta }{t}_l).\end{array}$

  (27)

The erroneous pixels are identified to be those that are smaller than $Z_s^{*}$ if the current frame is a long exposure and greater than $Z_l^{*}$ if it is a short exposure. Once these pixels are located, their MVs are updated using their eight immediate neighbors and eight nearest neighbors from the adjacent blocks. This amounts to choosing an MV that maximizes the following weight function:

$\begin{array}{l}\hfill w(p,q_k)=r(p,q_k)c(p,q_k)=\frac{1}{\parallel p-q_k\parallel }\frac{1}{|I_r(p+M{V}_k)-I_r(q_k+M{V}_k)|},\end{array}$

  (28)

where I_r represents the reference exposure and q_k the 16 candidate pixels from which one of the MVs will be selected. The r(p, q_k) term represents the spatial distance and therefore, nearby pixels are favored. The c(p, q_k) term, on the other hand, is a color similarity term based on the idea that pixels that are similar in color should have similar MVs. The refinement process is illustrated by an example in Fig. 6.

6.5 Spatially Adaptive Filtering for Registration Artifact Removal in HDR Video [88]

A follow-up work by Mangiat and Gibson proposes to improve the artifact removal stage of their previous algorithm. As discussed in the previous section, the artifact removal stage involved applying a CBF to fix the block artifacts:

$\begin{array}{l}\hfill \text{CBF}{[J]}_p=\frac{1}{W_p}\sum _{q\in \mathcal{S}}{G}_{{\sigma }_s}(\parallel p-q\parallel )G_{{\sigma }_r}(|I_p-I_q|)J_q,\end{array}$

where $\mathcal{S}$ is a neighborhood around pixel p. Here, the tone mapped image J is filtered by using the edge information in frame I. $G_{{\sigma }_s}$ denotes a Gaussian function with standard deviation σ_s and likewise for $G_{{\sigma }_r}$. W_p represent the sum of all weights. This process blurs across the edges of image J if those edges are not present in image I, e.g., the edges caused by blocking artifacts.

Recognizing that artifacts are most likely to occur in large motion regions, the authors propose the modify the standard deviation of the range kernel as:

$\begin{array}{l}\hfill {\sigma }_r(p)=\alpha \underset{q\in \mathcal{S}}{max}(\parallel M{V}_q\parallel ),\end{array}$

where α is a user parameter used to adjust the amount of smoothing.

However, this modification cannot fix registration errors that may occur in saturated regions as, in these regions, the current frame does not have any edge information. Therefore, the authors propose a second modification to use the edge information in the tone mapped HDR frame directly for the saturated regions of the current frame. This is formulated as follows:

$\begin{array}{l}\hfill \text{HDR}{[J,I]}_p=\left\{\begin{array}{ll}\frac{1}{W_p}\sum _{q\in \mathcal{S}}{G}_{{\sigma }_s}(\parallel p-q\parallel )G_{{\sigma }_I(p)}(|I_p-I_q|)J_q,& \text{if}10<I_p<230,\\ \frac{1}{W_p}\sum _{q\in \mathcal{S}}{G}_{{\sigma }_s}(\parallel p-q\parallel )G_{{\sigma }_I(p)}(|I_p-I_q|)G_{{\sigma }_J(p)}(|J_p-J_q|)J_q,& \text{otherwise.}\end{array}\right.\end{array}$

In other words, in regions where the current frame is well-exposed, the tone mapped frame has no influence on the filter. In these regions, the adaptive version of the CBF is used. However, in the saturated regions of the current frame, the tone mapped frame itself is used to smooth out potential registration errors. An additional benefit of this approach is that it obviates the need to perform perceptual color similarity or SSIM computations as the final tone mapped frame becomes the filtered frame itself.

6.6 Filter-Based Deghosting for Exposure Fusion Video [83]

In their work, Chapiro et al. [83] present a method to deal with ghost artifacts based on the exposure fusion approach of Mertens et al. [116]. The proposed method addresses small-scale movements rather than fast object and camera motions. For improving the quality of the input video by eliminating ghost artifacts on object boundaries, a ghosting term is included in addition to the contrast, saturation, and well-exposedness terms of the exposure fusion algorithm. This term reduces the contribution of the motion-affected pixels.

The proposed method begins with applying a multiresolution global registration to each pair of consecutive input frames. In order to detect the pixels that may cause ghost artifacts, low-pass Gaussian filters are applied to the input frames, followed by high-pass Laplacian filters. The ghost term for a specific pixel is calculated by subtracting the SAD from one around a rectangular region in each pair of input frames.

We note that this method does not generate an HDR video, but rather, a better exposed LDR video by directly merging the input frames similar to Youm et al. [110]. Therefore, it does not require information, such as the camera response and exposure times.

6.7 Toward Mobile HDR Video [81]

Castro et al. [81] presented a method of reconstruction of HDR video for hand-held cameras as an extension of a similar earlier work [84]. Because the method is based on histograms, it is of low computational cost and thus, suitable for less powerful processors, such as mobile phones. The proposed algorithm initially captures a multiexposure sequence on Nokia N900. Then the captured exposures are transferred to a desktop computer to generate HDR videos. Finally, the constructed HDR videos are tone mapped to enable visualization on LDR displays. The proposed method of generating HDR video has three steps: (1) photometric calibration, (2) multiresolution alignment, and (3) radiance map estimation.

For photometric calibration, the method takes a sequence of three (or possibly more) exposures Fⁱ as input, where $F^i=\{F_1^i,F_2^i,F_3^i\}.$ Exposure $F_1^i$ is constant for all i and is determined before capture, using the auto-exposure feature of the camera. $F_2^i$ and $F_3^i$ are exposures of twice and half the exposure value of $F_1^i$, respectively, for all i. Photometric calibration relies on pixel correspondences between pixels from different frames. The authors assume that exposure changes preserve monotonicity of pixel values and define the radiance mapping M_{P, Q} between two consecutive frames, P and Q as:

$\begin{array}{l}\hfill M_{P,Q}(p_i)=q_i,\forall i,\end{array}$

where p_i and q_i represent the corresponding pixels between two frames. Radiance map is then reconstructed by using the approach described in Mitsunaga and Nayar [104].

However, prior to constructing the radiance map, the exposures must be first brought into global alignment. This global alignment is performed by applying Ward’s MTB registration method [10].

Finally, the removal of ghost artifacts is achieved by analyzing the variance of radiance values over the corresponding pixels of aligned images in Fⁱ. Four radiance maps, ${\widehat{F}}_j^i$ for j = 1, 2, 3, 4 are generated for each Fⁱ. The first three radiance maps are simply the original frames converted to the radiance domain, and the last one is the reconstructed HDR image after global alignment. The first three radiance maps are then updated based on the magnitude of the variance. If the pixel variance is low, the HDR frame is given more weight, whereas for high-variance pixels, the original pixel values are prioritized.

6.8 Patch-Based High Dynamic Range Video [85]

The main contribution of this method, which arguably produces the highest quality deghosting results, is a judicious combination of optical-flow [80] and patch-based [62] HDR image synthesis algorithms. For the capture process, the input is assumed to be N LDR images alternating between M different exposures (L_{ref, 1}, L_{ref, 2}, …, L_{ref, N}). The task is to generate the N HDR frames (H_n, n ∈ 1, …N). For each frame, additional (M − 1) missing frames are synthesized (for all but the reference frame).

Although [62] works very well for still images, it is not suitable for HDR video due to lack of temporal coherency. Also, this algorithm fails if a large region of the image is under- or over-exposed. Thus, the direct application of [62] is not suitable. Therefore, the following modified energy function is proposed:

$\begin{array}{ll}\hfill E(\text{all}{L}_{m,n})& =\sum _{n=1}^N\sum _{p\in \text{pixels}}[({\alpha }_{\text{ref},n_{(p)}}\cdot {(h{(L_{\text{ref},n})}_{(p)}-H_{n(p)})}^2\hfill \\ \hfill & +(1-{\alpha }_{\text{ref},n_{(p)}})\cdot \sum _{m=1,m\ne \text{ref}}^M\mathrm{\Lambda }(L_{m,n}){(h{(L_{m,n})}_{(p)}-H_{n(p)})}^2\hfill \\ \hfill & +(1-{\alpha }_{\text{ref},n_{(p)}})\cdot \sum _{m=1}^M\mathrm{TBDS}(L_{m,n},L_{m,n-1},L_{m,n+1}))],\hfill \end{array}$

where h(.) converts LDR images to the linear radiance domain, α is an approximation of how well a pixel is exposed, and H_n is an HDR frame. Λ in the second term is weighted triangle function used for merging [4]. The third term, temporal bidirectional similarity (TBDS), adds temporal coherence between the estimated exposures:

$\begin{array}{l}\hfill \mathrm{TBDS}(L_{m,n},L_{m,n-1},L_{m,n+1})=\mathrm{BDS}(L_{m,n},L_{m,n-1})+\mathrm{BDS}(L_{m,n},L_{m,n+1}),\end{array}$

with BDS is proposed in [117]:

$\begin{array}{l}\hfill \mathrm{BDS}(T,S)=\frac{1}{|S|}\sum _{p\in \text{pixels}}\underset{i\subset f_S^T(p)\pm w_S^T(p)}{min}D(s(p),t(i))+\frac{1}{|T|}\sum _{p\in \text{pixels}}\underset{i\subset f_T^S(p)\pm w_T^S(p)}{min}D(t(p),s(i)),\end{array}$

where patch center in pixel p is marked by s(p) (source) and t(p) (target) and function D is the sum of the squared differences (SSD) between s and t. Here, $f_S^T(p)$ approximates the optical flow at pixel p from the source to target and $w_S^T(p)$ scales the search window. Note that for improved efficiency, the search is conducted around the regions indicated by an initial optical-flow estimation. In summary, the proposed method contains the following three main steps:

Initial motion estimation. During this stage, motion is roughly estimated in both directions for consecutive frames ($f_S^T(p)$ and $f_T^S(p)$). For global motion estimation, a planar similarity model and for local motion, optical-flow is used. The accuracy of this step influences the size of the search window in the next step.

Search window map computation. A window size of $w_S^T(p)$ and $w_T^S(p)$ is computed for every flow vector. This window is used to refine the initial motion estimate.

HDR video reconstruction. A two-stage iterative method is utilized in which the first stage minimizes the last term in Eq. (34), and the second stage minimizes the first two terms. These two stages are performed in succession until convergence. This produces all of the missing frames and the corresponding HDR image at each time instance.

6.9 A MAP Estimation Framework for HDR Video Synthesis [87]

One of the main difficulties in HDR video synthesis is estimation of accurate MVs between a sequence of multiexposures captured at different shutter speeds. Li et al. [87] proposed a maximum a posteriori (MAP) estimation framework for HDR video synthesis that does not use optical flow calculation for motion estimation. The proposed algorithm is an extension of the rank minimization-based HDR imaging (RM-HDR) [42, 50, 118].

The framework is based on an image acquisition model defined as follows: It is assumed that a pixel value z is defined as z = g(aΔt) + n, where a is the radiance value, Δt is the exposure time, g(⋅) is the response function, and n is a Gaussian random noise. The conditional probability of z given a is defined as:

$\begin{array}{l}\hfill \begin{array}{cc}f(z|a)& =\frac{d}{dz}Pr\{g(a)+n<z\}\\ & =\frac{d}{dz}\varphi \left(\frac{z-g(a)}{{\sigma }_n}\right)\alpha \text{exp}\left(-\frac{{(z-g(a))}^2}{2{\sigma }_n^2}\right),\end{array}\end{array}$

where σ_n is the standard deviation of n, and ϕ is the cumulative distribution function of the standard Gaussian distribution.

The model generates an aligned background and computes the foreground radiance simultaneously. To solve the problem of scene motion, each exposure is separated into foreground and background regions. Initially, a binary matrix S ∈{0, 1}^KxN, where K is the number of pixels in each frame and N is the number of exposures is constructed as:

$\begin{array}{l}\hfill S_{i,j}=\left\{\begin{array}{ll}1,& \text{if pixel}(i,j)\text{is in the foreground,}\\ 0,& \text{if pixel}(i,j)\text{is in the background,}\\ \end{array}\right.\end{array}$

where (i, j) corresponds to the ith pixel in the jth frame.

A probabilistic model for S is then constructed as a Markov random field under the assumption that interactions of a given pixel with all others are entirely described by its neighbors. The probability distribution function that can be considered as the prior probability of S is then defined as:

$\begin{array}{l}\hfill f(\mathbf{S})\alpha \text{exp}\left(\sum _{(i,j)\in \mathcal{E}}{\lambda }_{ij}{l}_i{l}_j+\sum _{i\in \mathcal{V}}{\lambda }_i{l}_i\right),\end{array}$

where $\mathcal{V}$ and $\mathcal{E}$ are the set of vertices and edges of the graphical representation of S, respectively, λ_ij and λ_i are edge interaction costs and vertex weights, and l_i ∈{−1, 1} is the latent variable corresponding to the ith pixel in S. We refer the reader to Lee et al. [119] for a more detailed description of S and setting the values of parameters λ_ij and λ_i.

Next, the radiance matrix A for the synthesized HDR video is defined as a combination of the foreground C and background B matrices:

$\begin{array}{l}\hfill \mathbf{A}={\mathcal{P}}_{\mathbf{s}}(\mathbf{C})+{\mathcal{P}}_{{\mathbf{s}}^{\mathbf{c}}}(\mathbf{B}),\end{array}$

where ${\mathcal{P}}_{\mathbf{x}}(\mathbf{Y})$ is a sampling operator defined as:

$\begin{array}{l}\hfill {[{\mathcal{P}}_{\mathbf{x}}(\mathbf{Y})]}_{ij}=\left\{\begin{array}{ll}{Y}_{ij},& \text{if}{X}_{ij}=1,\\ 0,& \text{otherwise.}\end{array}\right.\end{array}$

Since B, C, and S are unknown, they need to be estimated. Their estimation is defined as a MAP estimation problem:

$\begin{array}{l}\hfill (\widehat{\mathbf{S}},\widehat{\mathbf{B}},\widehat{\mathbf{C}})=\underset{\mathbf{S},\mathbf{B},\mathbf{C}}{argmax}f(\mathbf{S},\mathbf{B},\mathbf{C}|\mathbf{D}),\end{array}$

where D := [vec(I₁), vec(I₂), …, vec(I_N)] corresponds to the scene observation matrix, where each I_k denotes the observed radiance map of the kth frame obtained by using the inverse camera response function as defined in Debevec and Malik [4], and f(S, B, C|D) corresponds to the joint probability of S, B and C, given D. The authors rewrite the estimation problem defined in Eq. (41) as:

$\begin{array}{l}\hfill \begin{array}{cc}(\widehat{\mathbf{S}},\widehat{\mathbf{B}},\widehat{\mathbf{C}})=& \underset{\mathbf{S},\mathbf{B},\mathbf{C}}{argmin}\frac{1}{2}{‖{\mathcal{P}}_{\mathrm{\Omega }}(\mathbf{D}-\mathbf{B})‖}_F^2+\alpha {‖\mathbf{B}‖}_{*}+\beta {‖\mathbf{S}‖}_1\\ & +\gamma {‖\mathbf{W}\text{vec}(\mathbf{S})‖}_1+{‖\mathbf{V}\text{vec}({\mathcal{P}}_S(\mathbf{C}-\mathbf{E}))‖}^2,\end{array}\end{array}$

where Ω is the support of the well-exposed background, W is the weighting matrix responsible for interactions between neighboring pixels, and α, β, γ > 0 are constant parameters used to control the relative importance between each term. To solve Eq. (42), an iterative optimization over each variable, S, B, and C, is performed until convergence. More detailed description on the derivation and solution of Eq. (42) can be found in [87].

6.10 Motion Aware Exposure Bracketing for HDR Video [120]

Gryaditskaya et al. [120] introduced a real-time adaptive selection of exposure settings and an off-line algorithm for the reconstruction of HDR video, which is specifically tailored for processing the output of the proposed adaptive exposure approach.

First, the authors present the results of a psychophysical experiment for comparing the relative importance of limited dynamic range vs. ghosting artifacts, because there is a trade-off between the two in multiexposure HDR acquisition. As a result of this experiment, the authors find that the effect of ghosting artifacts is much more significant than that of the limited dynamic range on the perceived quality.

Next, the authors introduce their metering algorithm for the detection of optimal exposure times for minimizing ghosting, while maximizing dynamic range. The first two frames with long and short exposures are captured with the built-in metering system of the camera. The pixel intensities in the frame with shorter exposure are adjusted to the longer exposure. Then, the Hierarchical Diamond Search of Urban et al. [121] is used to estimate the motion between frames.

After calculating the motion field, the dominant MV for a frame is found by using the block-based recursive weighted least-squares method. The blocks which follow the dominant MV are marked as dominant motion regions (DMRs), while other blocks are marked as local motion regions (LMRs).

Placing priority on deghosting while extending the dynamic range, a requirement found from the psychophysical experiment, is satisfied by fulfilling the following two conditions while adaptively selecting the best estimate $\mathrm{\Delta }{\widehat{t}}_{i+1}$ for the exposure time of the next frame Δt_i+1:

1. A specific pixel must be properly exposed in at least one of the subsequent exposures.

2. At least a certain percentage of pixels must be properly exposed in both of the subsequent frames.

When these conditions are met, the output HDR is expected to have a dynamic range as high as possible without causing severe ghosting artifacts. Because in order to recover a poorly exposed region from other exposures, at least one of them must be well-exposed (condition 1) and in order to find a reliable correspondence to prevent ghosting between different exposures, at least some of the pixels must be well-exposed in multiple frames (condition 2) because image registration algorithms mostly fail in poorly exposed regions.

The first condition is satisfied by establishing an upper limit on Δt_i+1:

$\begin{array}{ll}\hfill f^{-1}(Z^{\text{low}})/\mathrm{\Delta }{t}_i& \le f^{-1}(Z^{\text{high}})/\mathrm{\Delta }{t}_{i+1},\hfill \end{array}$

$\begin{array}{ll}\hfill \mathrm{\Delta }{t}_{i+1}\le \mathrm{\Delta }{\stackrel{-}{t}}_{i+1}=& f^{-1}(Z^{\text{high}})\mathrm{\Delta }{t}_i/f^{-1}(Z^{\text{low}}),\hfill \end{array}$

where [Z^low, Z^high] is the user-defined pixel intensity range for well-exposedness, such that the pixels whose intensities are lower than Z^low are obtained from longer exposures, and the pixels whose intensities are higher than Z^high are obtained from shorter exposures.

The second condition is satisfied by applying refinements to $\mathrm{\Delta }{\widehat{t}}_{i+1}$ for achieving a desired upper limit on the percentage of overexposed pixels in LMRs and DMRs. The expected number of overexposed pixels in the next frame corresponds to the number of pixels in the current frame whose intensities are larger than $\widehat{Z}$:

$\begin{array}{l}\hfill \widehat{Z}=f(f^{-1}(Z^{\text{high}})\mathrm{\Delta }{t}_i/\mathrm{\Delta }{\widehat{t}}_{i+1}).\end{array}$

The percentage of overexposed pixels in LMRs and DMRs is calculated by counting the number of pixels in the current frame whose intensities are larger than $\widehat{Z}$. This operation is performed for different exposure settings provided by the camera. The exposure setting which satisfies the upper limit on Δt_i+1 and the predefined limit on the percentage of the overexposed pixels in DMRs and LMRs is selected as the value of Δt_i+1. This process is used for searching for a longer exposure time and searching for a shorter exposure time is an analogous operation.

Once the whole sequence of frames is captured using the real-time adaptive exposure selection method described earlier, it is processed to produce an HDR video. Because the exposure settings of the captured frames are not periodic and the exposure times are not regularly spaced, available HDR reconstruction algorithms cannot be used without modifications. For this purpose, the authors use the method of Kalantari et al. [85] (Section 6.8) after making necessary modifications to their algorithm.

The method of Kalantari et al. [85] is extended by synthesizing a set of exposures for each frame in the sequence. The exposures in the set are found by recursively searching for the frames with shorter and longer exposure times, which have sufficient spatial overlap with the current frame. Once the exposure stack of each frame is formed, they are merged into HDR frames. The method of Debevec and Malik [4] is modified by introducing a probability of correct match term for the patch-based search of Kalantari et al. [85]. The results of the proposed algorithm are found to be better than the original method of Kalantari et al. [85], when compared using the mean structural similarity index (SSIM) [115] score.