6.3.1 Set 1: Background Variation

With this set, we aim to study the impact of background. The set was captured in indoor controlled illumination with both white background and natural backgrounds. The white background was created using an A4 sheet of paper while the natural background included a variety of objects that are typically found in the indoor environment. The subset Natural Indoor (NI) was collected to study the impact of changing backgrounds with a constant controlled illumination indoors. Both White Indoor (WI) and Natural Indoor (NI) subsets have 8 images per subject for each finger (right index and right middle). Hence, the total number of images in this set are: 64 subjects × 2 background types × 2 fingers × 8 samples = 2048 images. Fig. 6.3A and Fig. 6.3C show some sample images of this set.

6.3.2 Set 2: Illumination Variation

With this set, we aim to study the effect of illumination. This set was captured in a completely uncontrolled outdoor illumination environment with both white and natural background. The subset Natural Outdoor (NO) incorporated both challenges simultaneously, that is, changing background and outdoor uncontrolled illumination. Both White Outdoor (WO) and Natural Outdoor (NO) dataset individually contain 1024 images, making a total of 2048 images. Some sample images from this set are shown in Fig. 6.3B and Fig. 6.3D.

6.3.3 Set 3: Live-Scan Fingerprints

This set is created to evaluate the performance of matching systems where the gallery consists of live-scan fingerprints while the probe images are camera captured fingerphotos. We capture live-scan images using a Lumidigm Venus IP65 sensor. The number of images in this set is 64 subjects × 2 fingers × 8 samples = 1024 images. Some example images of this set are shown in Fig. 6.3E.

6.4.1 Fingerphoto Segmentation

The aim of fingerphoto segmentation is to segment out the relevant foreground information containing the finger ridge patterns. The major challenge for segmentation is the varying background noise, along with varying distance from the finger to the camera. It was observed that the skin color of the finger remained mostly consistent despite having varying background and illumination. Following this observation, the fingerphoto is segmented from the background using an adaptive skin color thresholding algorithm. Given an input image, we covert it to the CMYK color space with the magenta channel preserving most of the skin color. Using Otsu's thresholding method [11], an adaptive binary threshold is applied on the magenta (M) color channel, post which a binary mask of fingerphoto is obtained. However, the binary mask contains both false negative and false positive errors, thus leaving out certain finger regions as well as adding some background noise as a foreground region. We utilize the shape information of the finger to reduce the amount of noise by applying morphological operations on the obtained binary mask. A morphological opening operation using a square structuring element is applied twice to reduce the amount of false positives, and further the largest connected component is obtained using run-length encoding to minimize the false negative error. Thus, as a result of this coarse segmentation, a single connected binary mask corresponding to the finger region is obtained as the foreground. To find out a rectangular region of interest (ROI) of the binary mask which is invariant to pose variation, a boundary trace of the binary mask is performed. One assumption that is made for boundary tracing is that the finger is approximately present around the center of the image. In the binary mask, we find the leftmost true pixel from the center row of the image. We traverse in both upward and downward direction, till the point where we find a true pixel that is more than 10 pixels away from the previous true pixel. This final true pixel in both directions will be the leftmost extreme point of the ROI. We repeat the same procedure for the right most true middle pixel to find the boundary on the right side. Using these four points, a rectangular ROI is segmented around the finger. Only the pixels inside the rectangular region are considered as true pixels and retained in the RGB image, while the rest of the image is blacked out.

6.4.2 Fingerphoto Enhancement (Enh#1)

It is essential to build a robust enhancement algorithm to cater challenges such as illumination variation and focus variation. The aim of this step is to improve the contrast between the ridges and the valleys in the finger. The RGB segmented image obtained from the previous step is converted to grayscale and a median filter is applied to reduce the effect of focus blur introduced during capturing. Further, histogram equalization is performed to normalize the illumination variation and increase contrast between ridge and valleys. To make high frequency components such as ridges more prominent and reduce the impact of low frequency components, we perform image sharpening. This is done by subtracting the Gaussian blurred image (with $\sigma =2$) from the original image. In the rest of this chapter, this enhancement approach is abbreviated as Enh#1, and a sample output after segmentation and enhancement is shown in Fig. 6.6.

6.4.3 LBP Based Enhancement (Enh#2)

Local Binary Pattern (LBP) is an effective texture descriptor for images which thresholds the neighboring pixels based on the value of the current pixel [12]. LBP descriptors efficiently capture the local spatial patterns and the gray scale contrast in an image. It can be observed from the segmented fingerphoto image in Fig. 6.6 that, in order to trace the ridge lines, it is important to make use of the ridge–valley intensity contrast. The edge lines on the ridges have a higher intensity compared to their spatial neighborhood valleys. LBP embeds this spatial structure into its descriptor, thereby tracing the ridge lines in a fingerphoto image. LBP has been widely used in many computer vision applications.¹ However, it is to be noted that LBP is popularly proposed as a feature descriptor, while we propose to use LBP as an image enhancement technique. Computation of the LBP descriptor from an image is a four-step process and is explained below.

1. For every pixel $(x,y)$ in an image, I, choose P neighboring pixels at a radius R.

2. Calculate the intensity difference of the current pixel $(x,y)$ with the P neighboring pixels.

3. Threshold the intensity difference, such that all the negative differences are assigned 0 and all the positive differences are assigned 1, forming a bit vector.

4. Convert the P-bit vector to its corresponding decimal value and replace the intensity value at $(x,y)$ with this decimal value.

Thus, the LBP descriptor for every pixel is given as

$LBP(P,R)=\sum _{p=0}^{P-1}f(g_p-g_c)2^p$

where $g_c$ and $g_p$ denote the intensity of the current and neighboring pixel, respectively. P is the number of neighboring pixels chosen at a radius R. Fig. 6.7 shows some sample LBP enhanced fingerphoto images.

6.4.4 Scattering Network Based Feature Representation

Fingerphoto images captured using smartphone camera are prone to geometric and resolution variations. Due to these challenges, minutiae based matching tends to give poor matching performance in case of finger-photos. There is a need for a feature representation technique which is not only translation and rotation invariant but also keeps high frequency information intact since we aim to keep the ridge pattern of the fingerphoto. To cater all these challenges, we propose a novel feature representation technique using Scattering Networks (ScatNet) [13], which quite efficiently extracts texture patterns of the image. ScatNet is a rotation and translation invariant feature representation which is made from a filter bank of wavelets [14]. To obtain a locally affine invariant representation, a low pass averaging filter ${\varphi }_J(u)=2^{-2J}\varphi (2^{-J}u)$ is applied on the signal $R^2$ to obtain the zeroth order ScatNet features as follows:

$S_{0}x(u)=x\star {\varphi }_J(u).$

Though these zeroth order features are translation invariant, they lack high frequency information. To extract the high frequency information, a filter ${\psi }_{\theta ,j}(u)$ is constructed using a bank of wavelets by varying the scale $2^j$ and the rotation parameter θ. To obtain the first order ScatNet coefficients, a low pass filter is applied to the high frequency components obtained using the high frequency filter:

$S_{1}x(u,{\lambda }_1)=|x\star {\psi }_{{\lambda }_1}(u)|\star {\varphi }_J(u).$

Higher order ScatNet coefficients can be extracted by recursively constructing wavelet filter banks. As we generate higher order coefficients, more high frequency components are preserved, but at the same time we increase the dimension of the feature representation vector exponentially. As a trade-off between high frequency information and computation time for a smartphone, we choose second order coefficients, and they are a concatenation of all responses, i.e., $\{S_0,S_1,S_2\}$. Mathematically, the second order coefficients are given as

$S_{2}x(u,{\lambda }_1)=(||x\star {\psi }_{{\lambda }_1}(u)|\star {\psi }_{{\lambda }_1}|\star \varphi ).$

6.4.5 Matching Techniques

We have performed verification to determine whether the presented fingerprint matches the claimed identity. The aim of this experiment is to find out if a pair of fingerphoto images are a genuine or an impostor pair. To achieve this, we have explored both distance based matching as well as learning based matching techniques. Also, the features generated using ScatNet are typically of very high dimension. Thus, to reduce the matching/training time and also to ensure the removal of co-linearity in the features, dimensionality reduction becomes inevitable. We used the Principal Component Analysis (PCA) as a dimensionality reduction technique. To get a compact representation of features extracted using ScatNet, we preserved $99\text{\%}$ of eigenenergy and reduced the dimensionality of our data.

1. Distance based matching. Let P and G be the $1\times N$ vectorized ScatNet representation of the probe and the gallery image, respectively. An L2-distance based matching, M, is given as follows:

$M(P,G)=\{\begin{array}{cc}0\hfill & d_{L2}(P,G)⩾t,\hfill \\ 1\hfill & d_{L2}(P,G)<t\hfill \end{array}$

where t is a threshold hyperparameter and

$d_{L2}(P,G)=\frac{\sqrt{{\sum }_{i=1}^N{(P_i-G_i)}^2}}{\sqrt{{\sum }_{i=1}^N{G}_i^2}}.$

2. Learning based matching. In this approach, the aim is to train a supervised binary classifier $f_{\theta }:<P,G>\to \{\text{Match},\text{nonMatch}\}$ to verify a pair of fingerphotos as a match or a non-match pair. The classifier can be considered as a nonlinear function learnt over the L2-distance between the probe and the gallery image and given as follows:

$M(P,G)=\{\begin{array}{cc}1\hfill & f_{\theta }(d_{L2}(P,G))⩾t,\hfill \\ 0\hfill & f_{\theta }(d_{L2}(P,G))<t.\hfill \end{array}$

In our proposed approach, a Random Decision Forest (RDF)² is used as the nonlinear binary classifier [15,16]. In the presence of highly uncorrelated features, discriminative classifier such as RDF are known to perform competitively [17].

6.5.1 Performance of the Proposed Matching Pipeline

The proposed pipeline includes the proposed adaptive segmentation algorithm, followed by the image based enhancement technique (Enh#1). Level-2 scattering network features are extracted followed by a PCA based dimensionality reduction technique to represent the fingerphoto image. A random decision forest (RDF) based matching algorithm is used to match the gallery and probe images. Equal Error Rate (EER) is used as the evaluation metric, and the results of the proposed matching pipeline are tabulated in Table 6.3 and Table 6.4. The corresponding Receiver Operating Characteristic (ROC) curve is plotted in Fig. 6.8.

The major observations made from the results are as follows:

• The results from the same-domain image matching in Expt1 provides, in general, better performance with (3.6–7.4)% error. While matching fingerphoto images to fingerprint images in Expt2, the error rates are a little higher in the range of (5.5–10.5)%.

• In Expt1, with WI as the gallery, using fingerphoto images captured in an outdoor uncontrolled environment with a natural background provides the best performance. Though this result is counter-intuitive, it can be justified by the presence of a balanced diffused lighting in the provided outdoor environment, which makes enhancement easier. Also, in the outdoor environment the objects in the background are typically farther away from the finger, making it easy for the adaptive segmentation algorithm to segment out the foreground.

• The worst performing experiment in Expt1 is the probe NI subset. In the indoor conditions, the focused lighting created due to the ceiling lights create a shadowing effect on the fingerphoto images. Thus, certain parts of the finger are either blacked out or over-saturated, leading to a loss of captured information, and hence to degraded performance.

• In Expt2, as the gallery fingerprint images do not have any background or illumination challenges, much more intuitive results are obtained as seen in Table 6.4. The WI probe subset provides the best matching performance, while NO subset provides the worst performance.

6.5.2 Comparison of Matching Algorithms

To study the effectiveness of the proposed ScatNet + RDF based matching algorithm, four different matching techniques are discussed as follows: (i) ScatNet + NN, which is also a learning based matching algorithm with Neural Network as a nonlinear classifier and ScatNet as features, (ii) ScatNet + L2, which matches the ScatNet features using an L2-distance metric and linear threshold, (iii) CompCode [18], which is one of the most successful methods in the literature for matching touchless fingerprint images, and (iv) Minutiae Cylinder Code (MCC) based matching [19], which is a robust fixed length descriptor over the traditional minutia features. CompCode and MCC based matching methods are explained in detail.

Minutiae Based Feature Representation. Minutiae Cylinder Code (MCC) is found to be one of the most successful minutiae based matching algorithm in the literature [20]. MCC uses minutiae for generating a feature representation and it is a computationally efficient matching algorithm. A fixed radius neighborhood is defined around each minutiae where a local structure cylinder is created. The cylinders in MCC are built from angles and invariant distances around each minutia extracted. Given a set of minutiae $M=\{M_1,M_2,M_3,\dots ,M_n\}$, with each minutia defined as a three tuple feature: x coordinate, y coordinate, and the ridge orientation θ at that point. With each minutia acting as the center for the base, a cylinder for each minutiae is made of height 2π and radius R. The cylinder is discretized by enclosing it inside a cube and is divided into cells, with each cell inside the discretized cube being identified using i, j, and k coordinates (k coordinate defines the vertical axis). Each of these cells is then projected on to the cylinder's base at location $(i,j)$. For each of these projected cells, a value $C(i,j,k)$ is calculated, which takes into account the contribution of each minutia inside the neighborhood $N_{ij}$. The neighborhood $N_{ij}$ has a radius of $3{\sigma }_s$. Cappelli et al. [19] showed how the value $C(i,j,k)$ is calculated by summing contributions of each minutia in the neighborhood $N_{ij}$. This value signifies the likelihood of finding a minutia near the projected center $(i,j)$ for the cell identified using the coordinate $(i,j,k)$. The minutia points are extracted using a commercial ten-print matcher called VeriFinger [21].

CompCode Based Features Representation. Competitive coding (CompCode) is one of the successful non-minutiae based feature representation techniques in the literature. CompCode uses Gabor filters for feature extraction. CompCode uses J different orientations with intervals of $\frac{\pi }{J}$ of the total available orientations of Gabor filters. A Gabor filter is defined as follows:

$G(x,y)=\frac{1}{2\pi {\sigma }^2}{e}^{\frac{x^2+y^2}{2{\sigma }^2}}{e}^{2(\pi )ifx}$

where the frequency of the sinusoid factor is represented by f and σ denotes the standard deviation of the Gaussian envelope. Let the real part of the above Gabor filter be represented as $G_r$ and let $I(x,y)$ represent the pre-processed image. The CompCode features $C(x,y)$ are extracted by convolution of $I(x,y)$ and $G_r$, and is defined as follows:

$CompCode(x,y)={\mathrm{argmin}}_j\{I(x,y)\star G_r(x,y,\theta )\}.$

The CompCode features obtained above are used for matching. They are typically matched using normalized Hamming Distance.

Tables 6.5 and 6.6 provide the error rates of the various matching algorithms for Expt1 and Expt2, respectively. The corresponding ROC curves are plotted in Figs. 6.9, 6.10, 6.11, and 6.12. The results show that ScatNet + RDF provides the best verification performance under all scenarios. The effectiveness of RDF is demonstrated when compared with neural networks (a highly competitive classifier in the literature) and provides much better performance. In Expt1, while matching fingerphoto with fingerphoto images, CompCode provides comparable performance to the proposed solution as both of them use texture features. However, while matching fingerphoto with fingerprints, our proposed solution remains more robust compared with CompCode, highlighting the generalizing capability of our approach. Further, in both Expt1 and Expt2, minutiae based approach performs poorly, implying that extracting minutiae from fingerphoto images is very challenging and includes lots of spurious minutiae.

6.5.3 Comparison of Distance Metrics

In comparison with the L2 distance metric used previously, various other distance metrics are used for matching fingerphoto images using ScatNet. That is, given a probe ScatNet representation of an image, it is compared with the gallery ScatNet representations of each image using different distance metrics. Table 6.7 summarizes all the distance metrics used for comparison. Minkowski distance with varying p parameter is plotted as ROC in Fig. 6.13B to find the best parameter. It can be observed that the accuracy drops when increasing p, and $p=2$ (Euclidean distance) gives the best performance. Fig. 6.13A shows the performance ROC of various distance metrics, and it can be concluded that, in cases where learning based algorithms cannot be used, L2 distance based ScatNet coefficient matching for fingerphotos yields best results.

Table 6.7

Various distance metrics used in our experiments for matching fingerphotos

Distance metric	Equation
Chebyshev	$d_{st}=ma{x}_j\left\{|x_{sj}-y_{tj}|\right\}$
Correlation	$d_{st}=1-\frac{(x_s-\overline{x_s})(y_s-\overline{y_s})}{\sqrt{(x_s-\overline{x_s}){(x_s-\overline{x_s})}^{\prime }}\sqrt{(y_t-\overline{y_t}){(y_t-\overline{y_t})}^{\prime }}}$
Cosine	$d_{st}=1-\frac{x_s{y}_t^{\prime }}{\sqrt{\left(x_s{x}_s^{\prime }\right)\left(y_t{y}_t^{\prime }\right)}}$
Minkowski	$d_{st}={\left({\sum }_{i=1}^n{|x_i-y_i|}^p\right)}^{\frac{1}{p}}$
L2 distance	$d_{st}={\left({\sum }_{i=1}^n{|x_i-y_i|}^2\right)}^{\frac{1}{2}}$
Spearman's coefficient	$d_{st}=1-\frac{(r_s-\overline{r_s})(r_s-\overline{r_s})}{\sqrt{(r_s-\overline{r_s}){(r_s-\overline{r_s})}^{\prime }}\sqrt{(r_t-\overline{r_t}){(r_t-\overline{r_t})}^{\prime }}}$ where $r_s=\frac{1}{n}{\sum }_j{r}_{sj}=\frac{(n+1)}{2}$$r_t=\frac{1}{n}{\sum }_j{r}_{tj}=\frac{(n+1)}{2}$

6.5.4 Effect of Enhancement

As explained in Section 6.4.3, LBP based enhancement technique (referred to as Enh#2) can also be used to improve the ridge–valley contrast in camera captured fingerphoto images. A matching pipeline with adaptive background segmentation followed by LBP based enhancement is used to preprocess the image. This enhanced image is further subjected to multiple feature extraction and matching algorithms, and the results obtained are compared with the results produced by Enh#1 enhancement. Tables 6.8 and 6.9 show the performance error of various matching algorithms using LBP based enhancement. The corresponding ROC plots are plotted in Figs. 6.14, 6.15, 6.16, 6.17, and 6.18. It can be observed that the results produced by Enh#2 + ScatNet + RDF are comparable to the Enh#1 + ScatNet + RDF. Also, while matching same subset images such as WI/2 with WI/2, LBP based enhancement is found to provide very high verification performance. One of the most promising observations in using LBP for enhancement can be observed in Expt2, when all the probe subsets provide almost the same performance for the proposed matching pipeline. ROC plotted in Fig. 6.14 suggests the same, namely when the gallery contains fingerprint images, the effect of various illumination and background environments on the probe fingerphoto images can be neutralized using LBP based enhancement. Thus, LBP based enhancement shows very promising results in (i) matching same subset fingerphoto images, and (ii) while matching fingerphoto images to fingerprint images using the proposed ScatNet + RDF matching pipeline.