Color Features

Detection of image regions with similar colors is not a trivial task due to inhomogeneous illumination, shadows, and reflections. Color-based feature tracking is therefore normally employed only in the environments where controlled illumination conditions can be established or the color of the tracked object is distinctive enough from all the other objects in the environment.

Color in a digital image is normally represented with three color components: red, green, and blue; this is known as the RGB color model. The color is restored from a combination of red, green, and blue color components that are filtered through red, green, and blue color filters, respectively. Two other color spaces that are seldom used in machine vision are HSL (hue-saturation-lightness) and HSV (hue-saturation-value). HSV color space is seldom used since it enables a more natural description of colors and better color segmentation than can be achieved in RGB space. The values from the RGB color space given in the range [0, 255] can be converted to HSV color space according to the following:

$\begin{array}{l}\hfill \begin{array}{cc}H& =\left\{\begin{array}{ll}0& \text{if}M-m=0,\\ 60\frac{G-B}{M-m}mod360& \text{if}M=R,\\ 60\frac{B-R}{M-m}& \text{if}M=G,\\ 60\frac{R-G}{M-m}& \text{if}M=B,\end{array}\right.\\ S& =\left\{\begin{array}{ll}0& \text{if}M=0,\\ \frac{M-m}{M}& \text{otherwise},\end{array}\right.\\ V& =\frac{M}{255},\end{array}\end{array}$

where $M=max\{R,G,B\}$ and $m=min\{R,G,B\}$. In Eq. (5.87) the saturation S and value V are in the range [0, 1], and the hue H is in the range [0, 360) that corresponds to angular degrees and wraps around. In Fig. 5.23 RGB and HSV components of the color image in Fig. 5.22A are shown.

Color histograms can be used to segment regions of a particular color. HSV histograms (with 32 bins) of several templates with similar color are shown in Fig. 5.24 (the color regions have been extracted from the image in Fig. 5.22A). Histograms of each template can be back-projected to the original image, that is, for every pixel in the image the frequency of the bin that belongs to the value of the pixel in the corresponding color channel in the image is set. The resulting back-projected grayscale images can be merged together as a linear combination of color channels. The results of blue, orange, red yellow, and wood color HSV histogram back-projection is shown in the first column of Fig. 5.25. In the resulting image the grayscale levels represent the measure of similarity of the pixels to the color template. Before the image is thresholded (see the third column in Fig. 5.25) some additional image filtering can be applied. The image can be smoothed with a 2D Gaussian filter (see the second column in Fig. 5.25) to remove some noisy peaks. The thresholded binary image can be filtered with, for example, some morphological or other binary image processing operations to remove or fill some connected regions. The resulting image mask (see the fourth column in Fig. 5.25) can be used to filter out the detected regions in the original image (see the last column in Fig. 5.25). More on image processing algorithms can be found, among others, in [10]. In practical implementations of color-based image segmentation some filtering processes are skipped in order to speed-up algorithms’ performance for the price of accuracy. Sometimes every pixel in the input image is considered part of the object if the color values of all components are within some lower and upper bounds that correspond to some color. In Fig. 5.24 it can be observed that hue and saturation values enable simple segmentation of colors. However, it should be pointed out that color-based segmentation is sensitive to changing illumination conditions in the environment. Although some approaches can compensate for inhomogeneous illumination conditions [11], the color-based image segmentation is the most commonly used only in situations where appropriate illumination conditions can be achieved.

The results of image segmentation is a binary image where image pixels that belong to the object are set to a nonzero value and all the other points are set to zero. There may be several or only a single connected region. If there are several connected regions an appropriate algorithm needs to be applied in order to find the positions and shapes of these objects [11]. To make searching of connected regions more robust some constraints can be introduced that reject areas with too small or too big areas, or some other region property. If the observed object is significantly different from the surrounding environment in a way that it can be robustly detected as a single region in a segmented image, the position and shape of this image region can be described with image moments.

The definition of a raw moment of a (binary) digital image I(x, y) is the following:

$\begin{array}{l}\hfill m_{p,q}=\sum _x\sum _y{x}^p{y}^{q}I(x,y)\end{array}$

where p and q are positive integers (p + q is the order of the moment). The moment m_{0, 0} represents object mass that is equal to the area of the object in the case of a binary image. Zero- and first-order raw moments can be used in order to find the center of the object (x₀, y₀) in the binary image:

$\begin{array}{l}\hfill x_0=\frac{m_{10}}{m_{00}},y_0=\frac{m_{01}}{m_{00}}\end{array}$

To describe object orientation and shape central moments can be used:

$\begin{array}{l}\hfill {\mu }_{p,q}=\sum _x\sum _y{(x-x_0)}^p{(y-y_0)}^{q}I(x,y)\end{array}$

which are invariant to in-image translation. Central moments can be used to fit an ellipsoid to the detected object in the image. The ellipse’s semimajor axis a and semiminor axis b can be obtained from eigenvalues λ_a and λ_b (λ_a ≥ λ_b) of the matrix Q, which consists of second-order central moments:

$\begin{array}{l}\hfill \mathit{Q}=\left[\begin{array}{ll}\hfill {\mu }_{2,0}\hfill & \hfill {\mu }_{1,1}\hfill \\ \hfill {\mu }_{1,1}\hfill & \hfill {\mu }_{0,2}\hfill \end{array}\right]\end{array}$

The ellipse axes are

$\begin{array}{l}\hfill a=\frac{2\sqrt{{\lambda }_a}}{m_{00}},b=\frac{2\sqrt{{\lambda }_b}}{m_{00}}\end{array}$

The orientation of the semimajor ellipse axis is given by the angle θ:

$\begin{array}{l}\hfill \theta =\frac{1}{2}arctan\frac{2{\mu }_{1,1}}{{\mu }_{2,0}-{\mu }_{0,2}}\end{array}$

In Fig. 5.26 raw and central moments have been used in order to fit an ellipsoid over the binary patch (the result of orange color segmentation from Fig. 5.25). The way ellipsoid parameters can be obtained from image moments is shown in Listing 5.14, where central moments were derived from raw moments. As it can be seen, image moments can be used to describe simple image features and determine locations of the features in an image. In [12] another set of moments is defined that are invariant to scale, translation, and rotation.

Artificial Pattern Markers

The introduction of artificial markers presents a minimal customization of the environment that can significantly simplify some vision-based tasks like, for example, object tracking, camera pose estimation, etc. An artificial pattern marker is a very distinctive pattern that can be robustly detected. A sample set of three artificial markers is shown in Fig. 5.27. Marker patterns are normally designed in a way that the markers can be reliably and precisely detected. Moreover, in the pattern of the marker, the marker ID is encoded; therefore, multiple markers can be tracked and identified through a sequence of images. One of the popular artificial marker detection algorithms is ArUco [13, 14]. Fig. 5.28 shows the detected artificial markers from Fig. 5.27 in two camera views.

The algorithms for detection of artificial markers normally consist of a marker localization step, in which the position, orientation, and size of the marker is determined. This procedure needs to robustly filter out only true markers from the rest of the objects in the environment. Once the locations of markers in the image are known, the projection between the detected marker and marker’s local coordinate frame can be established. Based on the pattern of the marker in the transformed local marker coordinate frame the ID of the marker is determined. Some artificial marker patterns are designed in a way that enable pattern correction that makes identification of patterns more reliable and also robust to noise and occlusions.

Natural Local Image Features

Images contain features that are inherently present in the environment. Over the years many machine vision algorithms have been developed that enable automatic detection of local features in images. Some of the important local feature algorithms are SIFT (Scale Invariant Feature Transform) [15], SURF (Speeded-Up Robust Features) [16], MSER (Maximally Stable Extremal Regions) [17], FAST (Features from Accelerated Segment Test) [18], and AGAST (Adaptive and Generic Accelerated Segment Test) [19]. Most of these algorithms are included in the open source computer vision library OpenCV [20, 21]. In robotic applications an important property of description of images with features is the algorithm efficiency to enable real-time performance. Different approaches have been developed for local feature extraction (localization of features), local feature description (representation of feature properties), and feature matching.

The goal of feature extraction is to detect and localize points of interest (local regions), normally in order to reduce the dimensionality of the raw image. Most commonly the features are extracted from grayscale images. Feature extractors can detect different types of image patterns: edges (e.g., Canny filter, Sobel filter, and Roberts cross), corners (e.g., Harris and Hessian operator, FAST), blobs (e.g., Laplacian of Gaussian, difference of Gaussians, MSER). To enable detection of features of different sizes, the raw image is seldom represented in scale-space [15]. Feature extraction should be invariant to some class of image transformations and distortions to enable repeatable detection of features across multiple views of the same scene. Normally it is desired that features are invariant to image translation, rotation, scaling, blur, illumination, and noise. Features should also be robust to some partial occlusions. The desirable property of the features is therefore locality. In mobile robotic applications it is normally required that features are also detected accurately and that the quantity of the features is sufficiently large, since this is a common prerequisite for accurate and robust implementation of algorithms that depend of extracted features (e.g., feature-based mobile robot pose estimation). An example of detected features in two camera views of the same scene is shown in Fig. 5.29. Every feature is given with position, orientation, and size.

The purpose of feature description is to describe each feature (based on the properties of the image pattern around the feature) in a way that enables identification of the same features across multiple images if the features reappear in the other images. The local pattern around the feature (e.g., the region marked with a square in Fig. 5.29) is used in order to determine the appropriate feature descriptor, which is normally represented as a feature vector. Local feature descriptors should be distinctive in a way that accurate feature identification is enabled regardless of different variations in the environment (e.g., illuminations changes, occlusions). Although local image features could be directly compared with, for example, convolution of local feature regions, this kind of approach does not have appropriate distinctiveness, and it is also quite computationally demanding. Feature descriptor vectors tend to be a minimal representation of features that enable an adequate level of feature distinctiveness for reliable feature matching. Which feature extractor and feature is used depends on the particular application. An experimental comparison of various feature detectors can be found in, for example, [22]. An example of two sets of feature description vectors of features extracted from the left and right image in Fig. 5.29 is graphically represented in Fig. 5.30.

Feature matching is one of the basic problems in machine vision. Many machine vision algorithms—like scene depth estimation, image-based 3D scene reconstruction, and camera pose estimation depend on appropriate feature matches between multiple images. Features can be matched by comparing the distances between the feature descriptors (vectors). Depending on the type of feature descriptor different distance measures can be used. Normally Euclidean or Manhattan distance is used for real-valued feature descriptors, and Hamming distance is used for binary feature descriptors. Due to imprecise feature localization, feature distortions, and repeated patterns, feature description matching does not always provide appropriate matches. Therefore an appropriate feature matching technique should be used that can eliminate spurious matches.

Given two sets of feature descriptors $\mathcal{A}={\{a_i\}}_{i=1,2,\dots ,N_A}$ and $\mathcal{B}={\{b_j\}}_{j=1,2,\dots ,N_B}$, the matching process should find the appropriate feature matches between the two sets. For every feature in the set $\mathcal{A}$ the closest feature in the set $\mathcal{B}$ (according to selected feature distance measure) can be selected as a possible match candidate. This is in general a surjective mapping, since some features in set $\mathcal{B}$ can have more than only one match from set $\mathcal{A}$. In normal conditions a feature in one set should have at most a single match in the other set. To overcome this issue, normally a two-way search is performed (i.e., for every feature in set $\mathcal{A}$ the closest feature in set $\mathcal{B}$ is found and vice versa) and only matches that are the same in both searches are retained. Additionally, if the distance between the best and the second best match is too small (below a selected threshold), the matched pair of features should be rejected, since there is a high probability of matching error. Also the pair of features should be rejected, if the distance between the matched feature descriptors is above a certain threshold, since this implies too high feature dissimilarity. In Fig. 5.30 the matches between two sets of feature descriptors are shown. Although the aforementioned filtering techniques were applied, not all incorrect matches were removed, as it is seen in Figs. 5.31 and 5.32 that there are still some incorrect matches. It should be noted that even if there are only a few outliers, these can have a significant influence on the results of the estimation algorithms that assume the features are matched correctly.

Some matched features may not satisfy system constraints: for example, if two matched features do not satisfy the epipolar constraint or if the reconstructed 3D point from the matched features would appear behind the camera, the match should be rejected. More constraints can be found in [23]. In wheeled mobile robotics additional constraints can be introduced based on the predicted robot states from known actions and the kinematic model. Incorporating a model of some geometric constraints into the feature matching process can be used to filter out the matched feature pairs that are not physically possible. For this purpose some robust model-fitting approach should be used. Commonly a RANSAC (Random Sample Consensus) method [24] is used that is able to fit the model even if there are many outliers.

The RANSAC algorithm can be explained on a line fitting problem. In the initial step the minimum number N_min of points that are required to fit the model are selected at random from the data. In the case of a line only two points are required (N_min = 2). The model (line) is then fitted to the selected (two) points. Around the fitted model (line) some confidence region is selected and the number of all data points in this region is counted. Then another N_min points are selected from the data, and the process is repeated. After some iterations the model with the largest number of points in the confidence region is selected as the best model. In the final step the model is refitted to all the points in the corresponding confidence region using a least squares method.