There has been an enormous amount of research behind a small name—image sensor. From developing CMOS sensors, which can take crystal-clear images, to employing enhancement of images using image processing, there are so many efforts involved. Making different types of cameras suitable for different applications and processing algorithms is not easy at all.
Image sensors are used in cameras, and they are very difficult to use in unpackaged conditions, as they are fragile and sensitive to electrostatic discharge. We will be using a camera (basically a lens with the protective case and an interface cable) to demonstrate image capturing using a RasPi. Performance of a camera with a RasPi is always rated low in discussions because of its poor performance with image processing and video processing, with comments that image processing and video capturing is not the RasPi's strong point. Somehow, these comments are true when we run computation-intensive algorithms with a huge amount of processing, and the RasPi gives a delayed response. But it all really depends on the hardware hookups at the time of camera interfacing. There are many factors that could affect the performance of the RasPi when you are using the camera. This was the issue with the older models of the RasPi, such as Raspberry Pi 1 model B, B+, and A+. But now, with Raspberry Pi 2 model B, this will not be the case. RasPi 2 model B is six times more powerful in performance than the older RasPi models.
In this chapter, we will start off by giving a brief introduction to image processing and OpenCV. Then we will go through the camera interfacing ports and look at the different kinds of cameras that can be interfaced with the RasPi. Next, we will do a long and tiring (but well-worth the time) installation of the OpenCV library on our RasPi. Once the libraries are installed, we will write a C program to capture an image from a camera. Moving one step ahead, we will experiment with the live streaming of a video over the same network as the one in your house. In a further step, a project will be prepared to detect human motion or movement in a particular area, capture the image, and alert you immediately.
Have you ever tried to look at an image by zooming it to a maximum level? It just looks like a floor with organized tiles and colorful patterns on it. These square tiles in the image are known as pixels. Basically, an image is a group of such pixels, with each pixel containing a particular value of color, which forms the recognizable patterns by providing information to the human eye. It all depends on how humans perceive the image by observing shapes and colors. Each pixel in an image contains information that can be generated from a byte (8 bits) or a couple of bytes, which defines the depth of an image. Depth of an image is nothing but the number of bits present in a single pixel. Current display monitors and graphics engines support up to 64-bit depth of images. Basic types of images are binary, grayscale, and RGB, and many more such as HSV, HLS, and YCC are known types. A grayscale image does have the range of values from 0 to 255 in the 8-bit mode, while a binary image has them in the range of 0 to 1. As the name itself suggests, a binary image has only black and white colors possible on the image plane. The difference between a grayscale and a binary image can be observed in the following images:
An RGB image has a total of three planes to access, as it has R (red) values varying from 0 to 255; and the same applies for G (green) and B (blue) colors for 8-bit depth. A grayscale image has shades from white to black, with a total of 256 discrete values (0 to 255) for an 8-bit deep image.
Image processing is performed to enhance an image to a certain levels, to extract maximum information. Just as we perform computations in signal processing, we do them in almost the same way in image processing. Here, the plane is a two-dimensional array of information in the form of pixels, whereas in signal processing, it is a one-dimensional array of signal streams. In complex image processing applications, there can be more than 20 or 30 algorithms and functions to be applied to an image to retrieve the distorted image or enhance the quality of it. In image processing algorithms, images are used as two-dimensional arrays. The number of X and Y columns of the image represents the image resolution. When algorithms are applied to an image, they individually process every pixel of it. Changes in the pixels occur according to the algorithm applied. Image processing can also be performed on a video, which is nothing but a group of images shown at a particular rate to create an illusion to the human eye, to perceive it as motion.
In a motion picture, there are multiple images (frames) showed at a time. There should be at least 22 to 25 frames per second (FPS) to create an illusion of motion for the eyes. These frame rates have been in practice since long. Mostly, cartoons and silent films in the 1920s started using a picture frame rolled in front of a concentric light to project it on a screen. Visual perception of every human is different. It is observed that less than 16 frames per second doesn't give better visual experience, as there will be a brief disruption of darkness. These days, the latest technologies, such as full HD (1080p), use frame rates of 50 to 72 FPS. Some applications require higher frame rates—up to 20,000 FPS—compared to our digital camera (30 FPS) or 3 FPS CCTV cameras. High-speed cameras for ultra-slow-motion videography require 20,000 FPS. All of these applications require a huge amount of computational power. Image processing on live video is mostly performed using individual frames.
Digital image processing is a vast field. Learning image processing can help you get employed in space agencies such as ISRO, NASA, or JAXA. Image processing is extensively used by space agencies to filter distorted images received from space telescopes and satellites. Image processing is used not only in space agencies but also in medical treatments, robotics, agriculture, automation, and stitching of Google maps images. Google sphere is also one of the best examples of image processing.
Understanding image processing requires a lot more studies of spatial domains and filtering algorithms. Currently, several algorithms have been developed to enhance the image quality. Fields such as artificial neural networks and pattern recognition are trending topics in research.