3.1.1 Aliasing (Jaggy)

When the sampling frequency applied to generate the digital image is lower than the highest spatial frequency of the natural image under concern, sampling theorem (Section 1.1) tells us that the obtained digital image will be susceptible to aliasing noise. The aliasing distorted digital image is observed to have undesirable high frequency oscillation around the high spatial frequency region of the image. The aliasing noise is not only observed in the under‐sampled digital image but also observed in the interpolated image. This is because the frequency response of the interpolation kernel will almost likely not to be an ideal low‐pass filter, and hence the high frequency components of the aliasing component in the up‐sampling process will remain in the interpolated image. These high frequency noises will have the same effect in the interpolation process as that of the high frequency noises in the down‐sampling operation. An example of aliasing distorted image is shown in Figure 3.2 a, where the aliasing noise is observed as staircase‐like features and is therefore also known as “jaggy” artifact. This observation closely resembles the effect of aliasing noise in the down‐sampling process. Transitional pixels are required to smooth the sharp changes between the grayscales on the two sides of a sharp edge to make it to appear to be pleasant to human observation. Such transitional pixels occur naturally in images captured by digital camera. However, when such transitional pixels are lost in the down‐sampling process, they may not be reproduced in the interpolation process, thus generating an interpolated image corrupted by aliasing noise.

3.1.2 Smoothing (Blurring)

Smoothing or blurring is observed when the high frequency components are lost, which can happen in the texture‐rich regions or along/across edges. An example of “blurred” image is shown in Figure 3.2 b, where the letter A has a “washed‐out” appearance. In some cases, the smoothing is localized, thus producing undesirable piecewise constant or blocky regions in the interpolated image as shown in Figure 3.2 b. In particular, when the edges of the interpolated image are over‐smoothed, the interpolated image will appear to be out of focus, which can also be observed in Figure 3.2 b. While the smoothing problem can be the result of a number of operations, the most common cause is due to the application of an interpolation kernel that is low‐pass in nature. Such high frequency lossy interpolation process is vivid from the linear interpolation process of a sampled 1D step curve by linear interpolation as shown in Figure 3.3 , where the details will be discussed in Chapter . Figure 3.3 b shows the interpolation of a low‐resolution step function. The linear interpolation result obtained by averaging the two nearest‐neighbor pixels is shown in Figure 3.3 c, which shows the step is dispersed and becomes a ramp function when compared with the high‐resolution step function in Figure 3.3 a. We can easily conjecture from the linear interpolation process shown in Figure 3.3 that smoothing/blurring will occur in any interpolation process, which involves the estimation of unknown pixel by averaging the neighboring known pixels. The blurring problem worsens with increased interpolation kernel size, which will cause the averaging effect spans over a larger number of pixels.

3.1.3 Edge Halo

The edge halo can be considered as a visual artifact that is opposite to smoothing. An image corrupted with edge halo artifact is shown in Figure 3.2 c. It is vivid from Figure 3.2 c that the edges are observed to be over‐sharpened where white tracks are formed around the edges of the images, which creates an impression of an additional false edge and hence its name “halo.” The halo is more apparent than the contrast between the two sides of the edges, which creates the illusion of enhanced sharpness. But edge halos are undesirable in natural image interpolation, especially in the case where it creates ghost images around natural objects.

3.1.4 Ringing

Besides the artifacts caused by nonideal frequency response of the interpolation kernel, the visual quality of the interpolated image is also affected by the spatial properties of the interpolation kernel. Ringing or oscillating wavelike artifacts can be observed in the interpolated image because most good interpolation kernels are functions of oscillating waves (see Figure 3.2 c). The extent of the ringing artifact is proportional to the length of the interpolation kernel. Furthermore, ringing often happens around step edges, where the oscillating waves are the natural results of the Gibbs phenomenon (both intensity and spatial occupancy) [43 ]. Note that the discontinuity at the image block edges is also considered to be a kind of step edges and hence will cause ringing noise too. An appropriate interpolation kernel (smooth and non‐oscillating spatial function) or high sampling rate can help to reduce the ringing artifacts.

3.1.5 Blocking

Besides the aforementioned artifacts, there is another artifact known as blocking artifact (also known as “zigzag”), which has a similar outlook as that of jaggies, where there are discontinuities within or along the image features. However, the discontinuity looks more like that of a repetitive block of image feature copied from nearby regions, and this kind of artifacts is also known as the blocking effect. Moreover, the origin of the blocking artifact, in the frequency response aspect, is different from that forming the jaggy artifacts. The blocking artifact is strengthened by the finite kernel size in spatial domain interpolation where the size of the kernel is smaller than the entire feature size. It can also be caused by cropping the high frequency components of the image due to finite block size of signal processing tool (also known as the kernel size). It becomes severe when the interpolation magnifies the image for several times. More details will be discussed in Chapter 5.

3.2.1 Mean Squares Error

Intuitively, the interpolated image can be regarded as the sum of the high‐resolution reference image and an error signal (also known as error image). Therefore, the MSE is one of the most traditional similarity measures. Starting with the computation of the error image (also known as the difference image) between the interpolated image array and the high‐resolution reference image array , both of size by

(3.1)

The MATLAB source code in MATLAB 3.2.1 implements the function to compute the error image.

With the availability of the error image, the total error between the two images is given by . However, the elements in have both positive and negative values. Therefore, it is more reasonable to consider the magnitude of . The mean absolute error (MAE) (also known as mean absolute difference) provides such a quality factor between the interpolated image array and the high‐resolution reference image array , both of size .

(3.2)

The MATLAB source code in MATLAB 3.2.2 implements the function to compute the error image.

In particular, the most popular quality factor is the MSE, which is equivalent to the computation of the square power of the error signal . The MSE is defined as

(3.3)

The MATLAB source code listed in MATLAB 3.2.3 implements the MSE function.

It is also common to give the MSE, mse_value, through the square root operation to generate a value that resembles the meaning of average pixel error of the two images, which is known as the root mean squares error (RMSE).

(3.4)

The MATLAB source code listed in MATLAB 3.2.4 computes the RMSE by means of the function mse.

It should be noted that g and r should have the same array size to avoid runtime error.

3.2.2 Peak Signal‐to‐Noise Ratio

The MSE does not consider the dynamic range of the image but only the absolute error in between two images and is therefore biased. Such bias can be removed by normalization. The PSNR is the most commonly used normalized objective quality metric for interpolated image quality assessment. The denominator of the PSNR is the MSE, while the numerator is the highest dynamic range achievable by the image function under consideration, which is also known as the ratio between the maximal power of the reference image and the noise power of interpolated image. It is represented in the logarithmic domain in decibels (dB) because the powers of signals usually have a wide dynamic range. An example for PSNR computed for an ‐bit grayscale image is given by

(3.5)

For example, an 8‐bit/pixel grayscale image will have as the numerator of the PSNR. The MATLAB code 3.2.5 will compute the PSNR with the assumption that the input image array in “uint8” datatype and thus .

The above discussed quality metrics can be easily extended to color images by treating each color channel independently as a grayscale image. In the case of color images in RGB domain, the PSNR of the three color channels are first computed and then recombined to give the final PSNR by averaging as

(3.6)

where , , and are the PSNR values for the red, green, and blue channels of the color image computed with Eq. (3.5 ), respectively. Without loss of generality, the rest of the book will use PSNR to imply both the PSNR in Eq. ( 3.5 ) for grayscale images and in Eq. (3.6 ) for color images depending on the context.

The PSNR is widely used because it is simple to calculate, has clear physical meanings, and is mathematically easy to deal with for optimization purposes. High PSNR value of the interpolated image is more favorable because it implies less distortion. However, the PSNR measure is not ideal. Its main shortcoming is that the signal strength is estimated by the highest dynamic range of the image that can be possibly achieved, which is , rather than the actual signal strength of the image. Furthermore, PSNR does not take the HVS into consideration. It has been widely criticized for not correlating well with subjective quality measurement. One of such quality is the preservation of edges in the interpolated image. Otherwise, the PSNR is considered to be able to provide an acceptable measure for comparing interpolation results.

3.2.3 Edge PSNR

A critical shortcoming of MSE and PSNR is that they are not compliant to HVS. This problem is vivid in the interpolated images in Figure 3.5 . In this example, the down‐sampled Cat image is interpolated by two different algorithms to produce (a) and (b). It is vivid that the image (a) has better visual quality, while (b) is visually observed to be seriously degraded; however, the PSNR of the image in (a) and (b) are both close to 23.04 dB. This is because the HVS perceives pixels differently and depends on their visual features, while PSNR considers all pixels to be the same. As a result, although being an objective and simple measure, the PSNR might lead to a totally wrong quality measurement result.

A simple step to improve the correlation between PSNR and visual quality of the interpolated image is to incorporate the differentiation of pixels perceived by the HVS. This can be achieved by assigning different weights to the edge and non‐edge pixels in the error image when computing the PSNR to simulate the relative importance of different pixels perceived by the HVS. The edge pixels can be located by the edge extraction algorithm presented in Section 2.5. It should be noted that applying different edge detection algorithms will lead to minor differences in the result. The Sobel edge detector is being adopted by the International Telecommunication Union (ITU) [1 ] for the EPSNR. Without loss of generality, assume the weight is assigned to the edge pixels and to the non‐edge pixels. The error image should be modified as

(3.7)

where is the edge map of the interpolated image with 0 being the value assigned to edge pixels and 1 be assigned to non‐edge pixels.

Besides the consideration of the HVS sensitivity differences toward edge and non‐edge pixels in the interpolated image, the actual contrast of the interpolated image should also be addressed by applying the peak intensity pixel value in the computation of the objective metric instead of the highest possible pixel value as in Eq. ( 3.5 ). The MATLAB source code 3.2.6 computes the EPSNR.

where edge is a MATLAB built‐in function that returns the binary edge map of the input image. The Sobel filter and the sensitivity threshold t have been chosen as the input parameters for edge in Listing 3.2.6. The matrix eedge is the error image . As you may have noticed from the MATLAB function epsnr, the mean squares edge error is normalized not by the image size, but by the number of pixels that are declared as edge pixels in . Similar to PSNR, the higher the EPSNR, the less the distortion will be observed on the image edges, and thus the better in perceived image quality. Finally, it must be pointed out that the EPSNR result is deeply affected by the threshold t, which is the threshold value applied to the gradient results obtained by the Sobel filter to decide each pixel locations to be edge or non‐edge pixels. The threshold value should be determined by the local contrast of the image, and therefore, a global threshold might not produce good edge detection results as discussed in Section 2.5 and hence biased the EPSNR. The following will discuss the structure similarity metric that applies localized analysis to evaluate the difference between the two images.

3.3.1 Luminance

The mean luminance can be used to compare the luminance of two images. A simple comparison metric can be formed by considering the ratio between the geometric means and the arithmetic means of the two luminance means as

(3.15)

such that and equals to 1 if and only if . The factor is added to the computation of to ensure the robustness of . Otherwise, with and both , the metric will be undefined with . Among all the possible values, SSIM selected

(3.16)

where is the dynamic range of the pixel intensity. In an 8‐bit grayscale image, and the squares are the result of considering a two‐dimensional image. As a result, is totally controlled by and should be chosen to avoid the luminance component to dominate SSIM. has been suggested in [63 ], which has shown to provide a useful SSIM metric. Incidentally, this definition is also compatible with the Weber's law of just‐noticeable luminance change, which states that the just‐noticeable luminance change within a local area in an image depends on the relative change in mean luminance with the localized area under concern with . The equivalent between Weber's luminance quality index and can be established by

(3.17)

To incorporate the local statistical property of the nonstationary image signal into the metric, a window function is applied to preprocess a localized image block. The applied window function has to have unit gain and be circular symmetric to avoid spatial bias. One of such windows is the Gaussian window. In this book, a Gaussian window of size with standard deviation of 1.5 will be applied to preprocess the image signal. The mean of the image block will be replaced with the Gaussian weighted mean, which can be conveniently implemented by convolution operation as

(3.18)

In MATLAB, this can be implemented with the filter2 operation as

which is implemented in MATLAB Listing 3.3.1 to generate a map of localized mean weighted by a Gaussian window.

3.3.2 Contrast

The contrast component in SSIM is computed in a similar manner as that of the luminance component with being replaced by as

(3.19)

such that and it equals to 1 if and only if . The factor has a similar function as that of , and thus it is also chosen to be equal to

(3.20)

Similar to , is totally controlled by and should be chosen to avoid the luminance component to dominate the SSIM. has been suggested in [63 ], which has shown to provide useful SSIM metric for interpolation algorithm performance comparison. Eq. (3.19 ) has shown that the metric depends on the relative contrast changes, which is consistent with the contrast masking property of the HVS.

3.3.3 Structural

The structural similarity between two random variables is best investigated by the Pearson correlation [35 ], and thus the structure component in SSIM is given by

(3.21)

such that . If we discard , the Pearson correlation factor when the two images and are not related. If the two images are associated with each other, . In particular when and can form a linear relationship, with constants and . This relationship implies that the two images are an exact copy of each other structurally with difference in lighting condition only. The factor is similar to and . The overall SSIM is given by the product of these three metrics, , and as

(3.22)

It is vivid from Eqs. ( 3.19 ) and (3.21 ) that the numerator of and the denominator of share the same factor except the constants and . Therefore, there are two factors that can be eliminated from SSIM. Furthermore, and are unified to form a single constant and hence obtained the SSIM in Eq. ( 3.8 ).

Readers should also take note that when and are both equal to zero, the metric will be the same as the universal quality index (UQI).

3.3.4 Sensitivity of SSIM

In the above sections, we have discussed how the luminance, contrast, and structure of an image are considered in the SSIM. Eq. ( 3.8 ) tells us that SSIM is a function of image parameters ( , , , , and ) and user‐defined functions ( and ). These two user‐defined functions adjust the impact of luminance, contrast, and structure of a natural image toward the SSIM computation. The values of and are controlled by two user input parameters, and , respectively. In the following sections, we shall explore the sensitivity of SSIM toward the parameters and ( and ) and find out the appropriate range of and that should be chosen such that a fair SSIM index could be generated that provides meaningful comparison among wide range of natural images.

3.3.4.1 Sensitivity

To understand the sensitivity of SSIM toward , a sensitivity analysis can be performed by rewriting the SSIM function as depicted in Eq. ( 3.8 ) as

(3.24)

The sensitivity of SSIM toward is the first derivative of Eq. (3.6 ) with respect to with , , , , , and considered to be constant, such that we have

(3.24)

with , and . It is vivid that the sensitivity of SSIM toward depends on , , and , but to what extent?

It should be noted that for a given image, has to be a constant depending on the data type of the image. For example, an image in uint8, . To simplify and without affecting the discussions, we generalize to use to represent and , unless otherwise specified. With being the mean intensity of an image, it is vivid that the dynamic range of is 0–255. In other words, no matter which interpolation method has been applied to the interpolated image, . To focus our study to the impact of the choice of toward the SSIM sensitivity to , we consider , , and , and the SSIM sensitivity to as a function of at , 0.03, and 0.05 are plotted in Figure 3.6 (solid lines). It should be noted that it is rare in natural image to have equal 0 or 255, as implies that all pixels within the localized image block to be 0, while implies all pixels within the localized image block to be 255. It is because there is always a background noise generated from the capture device, and hence a large region of pure color seldom happens in the captured images. Therefore, we can ignore the part of the SSIM sensitivity to curves at the two ends. In this case, the remaining curves are all fairly flat and close to zero, which allows us to conclude that the SSIM sensitivity to is independent to the choice . We can further conclude that the SSIM of a natural image is insensitive to for a fixed . Figure 3.7 shows the sensitivity of SSIM as a function of (see the solid line), where the curve is obtained by considering and to be 10 greater than and , respectively. The curve is almost independent to , which further confirms the above conjecture.

3.3.4.2 Sensitivity

Similar to Section 3.3.4.1 , the sensitivity of SSIM toward can be analyzed by rewriting Eq. ( 3.8 ) as

(3.25)

The sensitivity of SSIM to is the first derivative of SSIM with respect to and is given by

(3.26)

with and . It is vivid from Eq. (3.26 ) that the sensitivity of SSIM toward depends on , , and . To simplify and without affecting the discussions, we decided to use to represent and , unless otherwise specified. To focus our study to the impact of the choice of toward the SSIM sensitivity to , we consider , , and . The SSIM sensitivity to as a function of at , 0.03, and 0.05 is plotted in Figure 3.6 (dashed lines) with . It is vivid from Figure 3.6 that the SSIM sensitivity to with respect to is sensitive to . The is more sensitive to when is small (less than ). It can also observe from Figure 3.6 that the sensitivity increases with large . Figure 3.7 shows the sensitivity of SSIM toward as a function of (see the dashed line), where the curve is obtained by considering and to be 10 greater than and , respectively. Both and have chosen to be below 15, such that is the most sensitive with respect to , but the curve in Figure 3.7 exhibits a much less magnitude when compared with that shown in Figure 3.6 . It shows that is also not sensitive to the variation in . In conclusion, it is more appropriate to choose both and to be small such that both the sensitivity of and are kept to minimal. Therefore, Wang et al. [63] proposed to use and in the SSIM analysis, which is generally suitable for wide range of natural images.

1. 3.1 Modify the MATLAB function mse such that it will verify the size of the input images matrices to have the same size. Otherwise, it will output an error message of images are not the same size and set mse=NaN.
2. 3.2 Besides the image interpolation quality computation method shown in Figure 3.4 , it has been proposed in literature that the objective quality measures can be obtained by computing the quality metrics between the original image and an image obtained by:
   1. First interpolating the original image and then down‐sample it back to the original image size.
   2. Twelve successive rotation of of the original image.

   Please comment on the applicability of the above quality metric evaluation methods.

3. 3.3 Down‐sample the Cat image by directds in Section 2.7.2.1 to generate f. Interpolate the down‐sampled Cat image by bilinear interpolation method using MATLAB built‐in function interp2(f,2).
   1. Compute the error image e of the interpolated image f and the original image. Rescale the error image to make it span the numerical range of [0, 255]. Plot the scaled error image.
   2. Compute the SSIM of f with the default and used in Section 3.3 . Rescale the SSIM map to span the numerical range of [0, 255]. Plot the scaled SSIM map.
   3. Compute the edge image of f of the interpolated image using MATLAB built‐in Sobel edge detection function edge(f,‘Sobel’,t) with several different threshold value t. Rescale the obtained Sobel edge map to make it span the numerical range of [0, 255]. Plot the scaled edge map.

   Observe and comment the following:

   1. The similarity and disagreement between the three plotted images.
   2. Derive a method to combine the obtained edge maps under different threshold values to generate an image that looks more similar to the
      1. (a) Error image.
      2. (b) SSIM map.