6.1 Sampling and Sampling Theorem

A specific example used here is space sampling. Sensors integrate signal charges generated by incident light^* that enters the sensor part formed in each pixel arranged in a two-dimensional area. The output signal at each pixel during one exposure period is only one, and the signal value is the incident light intensity information at the coordinate point. Therefore, if the number of pixels is smaller, or the periodicity of sampling or the space frequency are lower, then the image quality based on space information is low because of coarse sampling, as shown in Figure 6.1.

Figure 6.2 explains how the spatial frequency in the obtained image information is restricted. The solid line in the top frame indicates three kinds of frequency and four input signals. A sine wave curve whose frequency is sampling frequency, f_s, is shown in the bottom frame. The sampling pitch p is expressed as 1/f_s. Sampling operations are carried out at the positions of the maximum point, as indicated by the up arrows. In the case that the input signal frequency is sufficiently low compared with the sampling frequency, f_s, as shown in Figure 6.2d, a broken curve obtained by tracing the sampling point, which is indicated by filled circles, accurately shows the same wave as the input signal. Both the amplitude and the frequency are maintained.

Then how is it possible to reproduce high frequency? The case when the frequency is half the sampling frequency, that is, f = f_s/2, is shown in Figure 6.2c. In this figure, the positions of the peaks and troughs (corresponding to white and black in the images) fit with that of the sampling points shown by filled circles. The amplitude and frequency of the reproduced curve shown by a broken line are retained, although the shape is a triangular waveform. Because this is the condition in which peaks and troughs fit with the sampling points, it is easily understood that any input signal with a higher frequency than f_s/2 cannot be reproduced accurately by the sampling frequency f_s. Therefore, the maximum frequency that can be reproduced is just half the sampling frequency. This frequency is called the Nyquist frequency, and the relation is called the Nyquist theorem or the sampling theorem.

Denoting the sampling pitch and the Nyquist frequency as p and f_N, respectively, we obtain the following relations:

However, an input signal whose frequency is f_s/2 is not always reproduced accurately. Figure 6.2b shows an input signal whose frequency is f_s/2, which is the same as Figure 6.2c, the only difference being that the phase is a quarter cycle late. The positions of the sampling points are in the middle of the peaks and troughs (corresponding to gray in the images). The signal curve that is obtained by tracing the sampled points is flat with no amplitude, as shown in Figure 6.2b. In this case, neither the amplitude nor the frequency is retained. Thus, this phase is also important, especially around the Nyquist frequency, as will be seen.

An input signal whose frequency is higher than the Nyquist frequency is shown in Figure 6.2a. The reproduced signal obtained by tracing the sampled points is very different from the original input signal, as shown in the figure. This false signal is called aliasing or folding noise. Thus, a higher sampling frequency is necessary to obtain accurate signal information for a higher Nyquist frequency.

In the above description, the sampling width is assumed to be infinitesimal. However, the actual sampling operation cannot be realized with a zero sampling width, but requires some finite width. Figure 6.3 shows the impact of the sampling width on the sampled result.

In the case of an infinitesimal sampling width, the maximum and minimum values of the input signal are reflected in the sampled result. However, in the case of sampling with a finite width, the sampling is carried out during the sampling period by integration or averaging. Thus, the maximum and minimum values cannot be directly reflected as sampled points. Accordingly, a larger amplitude of the sampled signal is obtained by sampling with a narrower sampling width.

6.2 Sampling in Space Domain

A schematic diagram of spatial sampling is shown in Figure 6.4a. Pixels are periodically arrayed with sampling pitch p and aperture a in real space. Since only light that passes through an aperture can reach the sensor parts, the sampling operation is only carried out in the aperture area; that is, the aperture width is the same as the sampling width. As the sampling pitch is p, the sampling frequency f_s equals 1/p and the sampling width is a. Under this condition, the frequency dependency of the sampled signal amplitude of sine wave input signals is shown in Figure 6.4b. The frequency is normalized by the sampling frequency. The three states of aperture pitch, a/p, which are 1, 0.5, and 0.2, normalized by the sampling pitch p are shown. As mentioned in Section 6.1, a wider aperture indicates a lower amplitude, especially in the higher-frequency region. This signal amplitude shows the spatial frequency response characteristics in transfer systems and is called the modulation transfer function (MTF). To obtain higher-frequency information, a higher Nyquist frequency, that is, a shorter pixel pitch, is required, as shown in Figure 6.1. Although a narrower aperture gives a higher amplitude, amplitude performance is not emphasized in usual applications because the narrower aperture brings about lower sensitivity, while higher sensitivity is the first preference for imaging systems.

Next, the images that are obtained by spatial sampling are confirmed by simple simulations using spreadsheet software. Using the inputs, the signal is observed using a circular zone plate (CZP) chart, which is often used to check the frequency of false signals. A calculated drawing in mathematical form is shown in Figure 6.5, indicating in a concentric fashion that the spatial frequency is in proportion with the square of the distance from the center. The waveform from the center to the right edge along the arrow indicated in Figure 6.5 is shown in Figure 6.6a.

Figure 6.6b shows aperture periodicity, that is, the sampling frequency. To emphasize the effects, a coarser pitch is chosen, and the aperture ratio a/p is set at 0.5. As the waveform in Figure 6.6a is sampled with the pitch of Figure 6.6b, the mathematical forms of Figure 6.6a and b are multiplied using spreadsheet software. All values except those for the aperture areas in Figure 6.6b are set to zero. The calculated results are shown in Figure 6.6c and d. That the amplitude modulation is seen even in a lower-frequency region than the Nyquist frequency indicates that the amplitude is not reproduced accurately according to the sampling conditions. Figure 6.6c and d in a higher area than the Nyquist frequency show that the waveforms are false signals completely different from the input signals. Specifically, Figure 6.6c shows symmetry with the Nyquist frequency at the axis, as the name “folding noise” suggests.

In actual imaging systems, the signal component at and over the Nyquist frequency is removed or reduced by using an optical low-pass filter (OLPF) to avoid impacts caused by false signals, as indicated in Figure 6.6a. The difference between Figure 6.6c and d is the sampling phase. While the amplitude at the Nyquist frequency is retained in Figure 6.6c, it is zero in Figure 6.6d. This result is due to the difference between the sampling phases, as explained in Figure 6.2. The sampling phases in Figure 6.6c and d correspond to those of Figure 6.2b and c, respectively.

Figure 6.7a and b show real pictures of a CZP chart taken by image sensors. In the CZP chart, the resolutions at the right and left edges of the line passing through the center mean 600 television (TV) lines,^* while the top and bottom edges correspond to 450 TV lines.

Figure 6.7a shows an emphasized picture of a CZP chart taken with a CCD with a 4.1 μm pitch square pixel with 955(H) × 550(V) numbers without an OLPF. Figure 6.7a shows many false signals of concentric circles, especially the strong signal observed at the Nyquist frequency of 550 TV lines in both the picture and the measured signal amplitude. Figure 6.7b is a picture taken with the OLPF set just in front of the sensor to suppress the false signal. The false signal is suppressed to an unobservable level in the picture and the amplitude shows almost zero at the Nyquist frequency. Comparing the amplitude graphs of Figure 6.7a and b, it can be seen in Figure 6.7b that the false signal at the Nyquist frequency is removed completely and the amplitude decreases with the frequency, especially at areas higher than the Nyquist frequency by the effect of OLPF.

An OLPF is a low-pass filter of spatial frequency, as its name indicates.^1,2 The most commonly used base material for OLPFs is crystalline quartz. Using birefringence of the crystal, the incident light beam is split into two parts, an ordinary ray and an extraordinary ray, as shown in Figure 6.8. While the ordinary ray propagates to the pixel directly underneath, the extraordinary ray is one-pixel pitch shifted^† through the crystal and, accordingly, reaches the pixel next to the pixel that the ordinary beam arrives at, as shown.

Because one cycle of the Nyquist frequency input is a two-pixel pitch, it has the spatial frequency shown by the black and white bars in Figure 6.8. Each intensity of the bars is divided in half: one half is underneath the pixel and the other adjoins the pixel. (The OLPF thickness is adjusted so that the separation distance equals the pixel pitch.) Therefore, each pixel receives a light intensity of half a black bar and half a white bar of the Nyquist frequency component of input, shown at the top of the figure. This means that the intensity of this frequency component is distributed equally to each pixel, or no amplitude. As the component of the Nyquist frequency vanishes through the OLPF in this way, the false signal is greatly reduced, as shown in Figure 6.7b.

As can be understood from the mechanism, the frequency whose amplitude is deleted perfectly is only one point, the effect remains around the target frequency to reduce the signal amplitude, as shown in the bottom graph in Figure 6.7b. Thus, the OLPF deletes the Nyquist frequency component, which causes a false signal and reduces the amplitude around it and the higher-frequency component of it.

The case of a smaller pixel pitch is discussed. Figure 6.9 shows pictures of a CZP chart taken by an image sensor at 1.8 μm pitch and 3096(H) × 2328(V) pixels without an OLPF. Since the resolution of the CZP with a full angle of view is only 600 TV lines, it was taken with an adjusted angle of view so that the resolution at the horizontal edge of the CZP is 2350 TV lines, as shown in Figure 6.9a; an expanded picture is shown in Figure 6.9b. Despite no OLPF, the false signal due to aliasing at the Nyquist frequency is only slightly observed in the especially emphasized image. While it is observed in the amplitude distribution, the level is quite light compared with that of the 4.1 μm pixel in Figure 6.7. Thus, it seems that the Nyquist frequency has moved to a higher-frequency region where the MTF of the lens is not high, resulting from the achievement of higher-resolution sensors based on the progress of pixel shrinkage technology. Since the level of the false signal is light, it tends to be processed using a digital signal processor (DSP) without an OLPF. Although it depends on the application, two OLPFs are necessary for vertical and transversal directions or more than two for diagonal directions. Because one OLPF works for only one direction, it is necessary for the number of OLPF plates to be in accordance with the number of directions. Additionally, as the thickness of an OLPF is in the order of hundreds of micrometers, it is effective in achieving thinner imaging systems, thereby avoiding OLPF usage, since the trend is to reduce the size of the system.

6.3 Sampling in Time Domain

In this section, sampling in the time domain is described. Since information concerning time is only image blurring in the case of still images, only cases of moving pictures are considered here. As already mentioned in Chapter 1, still images are repeatedly taken at a constant time interval in capturing moving images. As described in Chapter 4 on electronic shutters, images are picked up during some part of the interval time, that is, the exposure period.

This event is shown in Figure 6.10 as a schematic diagram along a time axis by the repetition of a frame time involving the exposure period. This sampling structure is exactly the same as that of the space sampling in Figure 6.4. Therefore, a shorter sampling pitch means a higher frame rate that can take higher-frequency information, that is, more accurate high-speed images can be obtained. A shorter exposure period means a narrower sampling width providing less blurred images of moving objects, that is, a higher MTF or amplitude as well as space sampling. False signals exist due to aliasing during sampling in the time domain as well as space sampling. For example, on TV, the phenomenon of a rotating wheel appearing not to rotate or, inversely, appearing to rotate slowly is caused by this mechanism based on the synchronization of periodic motion and exposure timing.

6.4 Sampling in Wavelength Domain and Color Information

If the same approach as in the case of space and temporal information is adopted to obtain wavelength information, sampling by dividing the wavelength domain is considered. For higher-quality wavelength information, a higher sampling frequency and a narrower sampling width are required. Specifically, multiband cameras^* capture images at each divided wavelength region, as shown in Figure 6.11, and synthesize them.

Figure 6.11 shows the case of 16 bands as the wavelength area is divided into 16 parts. In actual methods, 16 color filters, each with a spectral response that corresponds to each divided wavelength area, are prepared, and 16 still pictures are captured by using each color filter. Then, 16 pictures are synthesized to 1 color still picture. From the procedure, it is clearly understood that this system can only apply to still objects. Therefore, its application is restricted to particular kinds of tasks, such as digital archive development of art objects. And for practical reasons, the number of bands range from around 4 to 8.^† Thus, a multiband camera system is inadequate for general application.

Almost all of the camera systems that are actually used are single-chip color cameras represented by the Bayer color filter array shown in Figure 1.7. For high-quality imaging, such as for broadcasting, professional, and high-end consumer use, the three-chip color camera^‡ is used, in which the wavelength region is separated into three parts by a prism, leading to the use of each corresponding sensor.

An example of the spectral response of red, green, and blue in the Bayer color filter is shown in Figure 6.12. If it were thought of as one of a sampling means, it is quite different from that of space and time because of the very small sampling point number of only three and the very wide sampling widths that overlap each other. The reason is that this method in not a kind of sampling, but a method utilizing the human eye and brain’s perception of color. Here, the color perception of the human eye is briefly mentioned. As is well known, the human eye processes light through the retina. The retina contains two kinds of photoreceptor cells: rods and cones. Rods only detect light intensity at a very low light level, but do not sense color. Readers might have experienced the ability to discern the shape of a body but not color under very low illumination. On the other hand, cones detect both light intensity and color in relatively bright illuminance. There are three types of cones, referred to as S, M, and L after their size.

Figure 6.13 shows the wavelength dependence of the cones’ spectral response³: the highly sensitive wavelength ranges of S, M, and L are 400–500, 500–600, and 550–650 nm, respectively. In other words, S, M, and L are sensors that detect the ranges of violet to blue, green to orange, and yellow-green to red, respectively. Because of the overlap of the highly sensitive range, each of the three types of cones is excited by the incidence of any wavelength in visible light and responds to generate a reference stimulus.

A set of reference color stimuli generated by each of the three types of cones by light absorption facilitates color perception in the human brain. For example, if yellow, whose wavelength is around 580 nm, comes into focus at the retina, the stimulus occurs at the same level in cones L and M and at a lower level in cone S. This stimuli signal is transmitted to the brain, facilitating color perception as yellow. However, if the same intensity of green and red light is focused at the same point on the retina at the same time, the same level of stimuli occurs at cones M and L, which is transmitted to the brain and facilitates the color perception. This is the the same as in the case of yellow incident light, as the brain accepts only the stimuli. Therefore, the brain perceives yellow in both cases.

When colors A and B are focused at the same point on the retina, the human color vision senses a different color C. It is not sensed as a chord-like sound. The human eye and brain detect a stimulus of a wavelength and sense as a color based on a set of stimuli at cones S, M, and L. Therefore, overlapping of the spectral response is necessary to reproduce a hue.

Display devices made up of only three primary colors can depicted as a wide range of colors because of the mechanisms of the human eye and brain.

Actually used systems are real time, such as the single-sensor camera equipped with color filters of primary or complementary colors, and the three-sensor camera. Thus, what occurs in nature is “wavelength,” not “color.” Color cannot be discussed exclusive of human perception. At the stage that the parameter that is a physical quantity, “wavelength,” is substituted by human perception, “color,” physically objective affirmation of information accuracy becomes impossible. Indeed, the colors used in filters are not completely the same, such as the case of red, green, and blue based on the primary colors and the other case of cyan, magenta, yellow, and green based on the complementary colors⁴. Various kinds of light sources can be captured, such as natural sunlight, fluorescent light, incandescent light, and light-emitting diodes (LEDs). For these reasons, there is great difficulty in physically reproducing precise color. Therefore, what is aspired to is inevitably subjective color reproduction such as perceptually equivalent color, memory color, and preferred color.

The color information of the images obtained by the method utilizing human perception has limited effectiveness for applications that human eyes do not view.

Thus, a precision signal ratio of R, G, and B is necessary for precision color information. A less-accurate ratio signal causes color error at the pixel level. Figure 6.14 shows the impact on color error caused by a random noise change according to the light intensity. The light volume increases from left to right. The signal electron numbers at the highlighted parts (forehead of doll) are 90, 180, and 300 electrons, respectively. The bottom images are expanded portions of the darker areas of the above images. While strong color error is seen in the image of 90 signal electrons, the impact of the color error decreases by signal-to-noise ratio (SNR) improvement with increasing illuminance.

Since space, color, and time are the coordinate points built in ⟨r, c, t⟩ space, no margin of noise occurs as a coordinate point itself. Therefore, error and defection of information of the coordinate point occur in light intensity S at the coordinate point as a false signal, such as moiré, spurious resolution, false color, color noise, blurring, and lag.

References

1. S. Nagahara, Y. Kobayashi, S. Nobutoki, T. Takagi, Development of a single pickup tube color television camera by frequency multiplexing, Journal of the Institute of Television Engineers of Japan, 26(2), 104–110, 1972.

2. S. Nagahara, http://www.ieice-hbkb.org/files/08/08gun_04hen_02-04.pdf, pp. 116–118, October 25, 2011 (accessed January 10, 2014).

3. B. Wandell, Foundations of color vision: Retina and brain, in ISSCC 2006 Imaging Forum—Color Imaging, pp. 1–23, February 9, San Francisco, CA, 2006, http://white.stanford.edu/~brian/papers/ise/ISSCC-2006-Wandell-ColorForum.pdf (accessed January 10, 2014).

4. Y. Sone, K. Ishikawa, S. Hashimoto, T. Kuroda, Y. Ohkubo, A single chip CCD color camera system using field integration mode, Journal of the Institute of Television Engineers of Japan, 37(10), 855–862, 1983.