7.3.1 Datasets

GC in restricted datasets, i.e., those that have been collected under well-constrained environments, has reported very high classification rates that unfortunately have not been generalized to other independent datasets. This conclusion is evidenced by the GC results achieved by the Face Recognition Technology (FERET) dataset [102], a database containing 2413 high-resolution images of 856 individuals, see Fig. 7.2. Even though FERET was created to evaluate facial recognition, it has also been used to evaluate facial GC, as suggested by the Mäkinen et al. survey [93].

Although several tests have achieved almost perfect performance for FERET, the resulting classifiers are not able to maintain a similar performance in unrestricted image collections, evidencing that the dataset bias advantage produces an optimistic GC performance [9]. In fact, different authors have suggested that correct classification rates of such classifiers trained with FERET drop substantially when tested with wild datasets, see, for example, Erdogmus et al. [43] who showed that after almost perfect GC on FERET, only 65.2% was achieved on LFW.

In this sense, recent GC work has shifted towards more challenging and unconstrained viewing conditions, following a similar development in facial recognition research, suggesting the community interest in evaluating GC in the wild, i.e., uncontrolled conditions related to capture (illuminations, sensor, etc.), and subject. This situation has also been remarked in the FRVT report, where large in-the-wild datasets mean greater variability in terms of (i) identity, age, and ethnicity, (ii) pose and illumination conditions, and (iii) image resolution.

In addition to the LFW and GROUPS in the wild datasets considered in the FRVT report, we will also briefly describe PubFig and Adience, and later mention other datasets referred to in the literature:

• Labeled Faces in the Wild (LFW) [64]. As mentioned above, this dataset was created collecting photographs to study the problem of unconstrained face recognition, which nowadays is the standard benchmark for this purpose [84]. The dataset contains more than 13,000 images of 5749 individuals collected from the web. Among them, 1680 have two or more photos in the dataset. Related to GC, we can mention some characteristics: (i) it contains several samples per individual, (ii) both classes are not balanced, and (iii) the inclusion of public figures introduces a selection bias [124]. As argued by Baluja and Rowley [14], GC results are biased when the same individual is present in both the training and test sets.

• Public Figures Face Database (PubFig) [72]. The dataset contains 58,797 images of 200 individuals taken in completely uncontrolled situations with non-cooperative subjects, presenting large variation in pose, lighting, expression, scene, camera, imaging conditions and parameters, etc. However, we are unaware of any previous work which used this set for GC, probably due to the low number of different identities.

• The Images of Groups (GROUPS) [51]. This dataset was created to study social interaction with images present in Flickr with more than one individual. The complete dataset contains 5080 images with 28,231 annotated faces in terms of gender and age group. According to the FRVT report [97], this database is currently the hardest for GC.

• Adience [41]. The authors claim to have included a far wider range of challenging real-world imaging conditions, compared to other datasets such as LFW or PubFig. The dataset comprises changes in appearance, noise, pose, lighting and more, without careful preparation or posing. Once again the sources are Flickr albums. The dataset includes 26,580 samples belonging to 2284 individuals.

The various dataset statistics are summarized in Table 7.1, and some dataset images are shown in Fig. 7.3, excluding PubFig, that, as far as we know, has no GC results reported.

As well as to the 2015 NIST report, the recent survey by Santarcangelo et al. [112] included the following alternatives in their list of unconstrained datasets for GC: ClothesDB [50], Genki-4K [53], and KinFace [122]. The limited number of samples contained in them, 931, 3000 and 600, respectively, compared to GROUPS, LFW, and Adience, led us to exclude them in the proposals summarized below.

We would also like to mention the dataset created by Satta et al. [114] to evaluate GC in children. Their aim was to tackle the difficulties, even for humans, to classify gender among children due to their lack of many gender-specific adult face traits. They have collected a dataset making use of standard face detection techniques, integrating contextual features to boost classification accuracy.

A final remark may be made on the influence of ethnicity and age in GC [10,52,56]. The former may be analogous to the human “other-race” effect, which is explained by psychologists with the “contact hypothesis”, that highlights the difficulties in extracting demographics from faces of other races [31].

Similarly, automatic systems might be positively affected by an ethnicity-balanced training set, or the alternative would be to create ethnicity-specific gender classifiers [48]. In any case, balanced datasets in terms of ethnicity have rarely been considered in the literature for GC. We should mention the EGA (Ethnicity, Gender, and Age) dataset [109], originally created for face recognition purposes, which has recently been evaluated for GC [25,26]. EGA has been created with images from other known face datasets. These samples have been chosen to reduce the distortions due to pose, illumination, and expression (PIE), to concentrate more on demographics. Unfortunately, these PIE restrictions and number of samples do not qualify the dataset as being in-the-wild.

7.3.2 Proposals Summary

As mentioned above, current state-of-the-art GC approaches achieve high precision based simply on visual facial features in controlled scenarios. This fact has also been proven with commercial solutions, as stated in the 2015 FRVT report on Performance of Automated Gender Classification Algorithms, which focused for the first time on GC [97]. This evaluation reported an accuracy of around 96.5% with an independent dataset containing roughly one million facial samples acquired under constrained conditions.

However, these results were not reproduced in datasets with unconstrained capture conditions or in the wild. Two of the above mentioned datasets were evaluated by Ngan et al. [97]: (i) LFW and (ii) GROUPS. In these datasets, available commercial solutions were not able to maintain similar classification rates in both cases. On the one hand, LFW seems to be a simpler dataset as the best accuracy reported by standard commercial solutions was 95.2%, certainly quite close to the numbers reported for constrained datasets. On the other hand, the accuracy achieved for GROUPS was significantly worse, hardly reaching 90.4%. This evidence confirms the difficult scenario represented by this particular dataset. This conclusion has already been highlighted by recent surveys and experimental evaluations [23,96].

Below we summarize GC results for three of the above mentioned in-the-wild datasets: LFW, GROUPS, and Adience. Observe that PubFig is not included due to the absence of experimental evaluations. Two scenarios are considered, depending on if the experimental evaluation includes a single or double datasets. When a single dataset is used, training and test sets are built based on the same dataset, a fact that may be affected by the dataset gathering protocol, or even include the same identity in both training and test sets. These results are commonly referred to in the literature as single/in/intra/within-database GC. A summary of recent results is reported in Table 7.2.

However, single dataset results may be overly optimistic. Indeed, the community knows that achieving high GC rates for a dataset does not generalize to all scenarios due to the influence of dataset bias [124]. Close to perfect accuracies in FERET [102] have been achieved. The work by Moghaddam and Yang [95] reported an accuracy of 96.62%, making use of pixel intensities and support vector machines (SVM). However, even more recent developments, trained with FERET, have not been able to reach significant accuracies on in-the-wild datasets, as, for example, those reported by Erdogmus et al. [43], who achieved an accuracy of just 65.2% on LFW. Therefore, a more challenging evaluation is referred to as cross-database GC where independent datasets are used for training and testing. As observed in Table 7.3, summarizing cross-database results, the achieved accuracies for the test dataset are commonly lower.

Added to the training and test data used to evaluate the different approaches, the main differences among them are related to the features used and classification design adopted [96]. Assuming the state-of-the-art appearance-based methods, approaches are based on features such as intensity raw values, or local descriptors, which may later be filtered or not, while the classification may cover SVMs, linear discriminant, nearest neighbor, or neural networks among others.

Among the three datasets, it can be observed that GROUPS and LFW have received far more attention, with the former apparently being the most challenging of them. The recently arrived Adience has attracted interest in Convolutional Neural Networks (CNN) based solutions. Below, we present a rough chronological summarized description of the different approaches evaluated in the wild, considering single and cross-database results.

Prior to Dago et al. [36], GC evaluations mainly focused on single datasets such as FERET, covering a relative large population containing a significant number of individuals. However, the dataset high quality images and controlled capture conditions are rather different from the image variety of uncontrolled or real-world scenarios. Dago et al. presented results for LFW making use of the BEFIT protocol,⁷ and configured an experimental protocol for GROUPS using a subset with larger faces.⁸ In the study, they evaluated the use of LBP and Gabor features using LDA or SVM for classification. For single dataset GC, both features reached similar results, slightly better using Gabor jets, achieving an accuracy of 86.61% for GROUPS and 94.01% for LFW. They also included additional cross-database results, i.e., results after training with one dataset and testing with a different one to avoid dataset bias [124]. They were probably the first researchers interested in performing cross-database evaluations considering in the wild databases. When training with GROUPS and testing with LFW, they reported an accuracy of 89.77% using LBP and SVM, while the opposite combination reached only 81.02%.

Bekios et al. [9] also reported cross-database GC results, but their experiments were restricted to PAL, UCN, and FERET. Their approach combined LDA/PCA features with a Bayesian classifier, focusing on reaching similar to state-of-the-art GC rates with linear classification. On those datasets, they succeeded in achieving quite similar performances while reducing computational requirements needed by SVM based systems. A major conclusion was the proof that cross-database experiments are closer to real-world scenarios and demonstrated that single database experiments are optimistically biased.

In a subsequent study, Bekios et al. [10] tackled the in-the-wild scenario. Their study evaluated dependencies among gender, age, and pose. The authors integrated pose information in the classification, avoiding the initial need for face alignment. Their evaluation using a five-fold strategy reported an accuracy of 80.5% for single database GC in GROUPS.

More recently, other single dataset GC experiments have been carried out on LFW. Shan [115] extracted LBP features that were later classified using SVM to obtain an accuracy of 94.8% in LFW. Subsequently, Shafey et al. [118] reported results for FERET and LFW, making use of Total Variability (i-vectors) and Inter-Session Variability (ISV) modeling techniques, reaching an accuracy of 94.6% for LFW. Tapia et al. [129] compared the fusion of different LBP-based features, scales, and mutual information measures, reporting for LFW an accuracy of 98%. Ren and Li [106] evaluated two types of local descriptors (gradient features and Gabor wavelets), introducing a later selection step based on RealAdaBoost. A linear SVM was applied as a classifier, and results for FERET, KinFace, and LFW were presented. For LFW, they reported 98%, claiming it to be faster than other previous approaches with similar accuracy. Erdogmus et al. [43] explored the best grid setup for the LBP-based features extraction with BANCA and MOBIO databases. Later, they evaluated on FERET, MORPH, and LFW, the latter with the BEFIT protocol and achieved an accuracy of 98%.

Reducing the working image resolution, El Din et al. [40] designed a two-stage system that combined appearance and shape features in the first stage, activating the second stage only for images with low confidence. Their reported accuracy for small, normalized thumbnails, $16\times 16$ pixels, of LFW reached an accuracy of 91.5%.

Specifically using cross-database results for LFW, Jia and Cristianini [66] focused on assembling and labeling large datasets automatically, avoiding the workload of human annotation. They evaluated their approach with LFW after gathering four million images and achieved an accuracy of 96.86%. Antipov et al. [2] applied CNN to GC on LFW, the authors claimed to use 10 times less training samples than Jia and Cristianini [66], i.e., 400,000, for identical experimental setup. They reported an accuracy of $97.1\text{\%}$ using three CNNs.

Chen and Gallagher [22] built a facial appearance representation based on the 100 most common names in the USA. Each pairwise name classifier provided a score, the whole collection was used to create the feature vector. The voting of the top five names is used to assign a gender to a test image. The achieved accuracy for GROUPS reached 90.4%. They claimed to beat any previous evaluation on GROUPS, including the application of the gender classifier designed by Kumar et al. [73] integrated into a face verification approach based on describable visual attributes, which using GROUPS gave rise to an accuracy of 86.4%.

Han and Jain [61] applied pose and photometric normalizations before extracting biologically inspired features (BIF), which are later classified with SVM to get age, gender, and race. Their experimental results on GROUPS and LFW reported 87.14% and 95.4% accuracy, respectively. Also for GROUPS, Fazl-Ersi et al. [46] reached an accuracy of 91.4% using a combination of different visual information sources. They combined LBP, SIFT, and color histograms after a feature selection stage. More recently, Mery and Bowyer [89] evaluated a technique based on the sparse representation of random patches achieving in a GROUPS subset containing roughly 2000 samples, an accuracy of 93.3%.

Danisman et al. [34] presented different cross-database results, making use of pixel intensities and SVM classification. Training with their dataset, WebDB, they reported an accuracy for LFW and GROUPS of 91.87% and 88.16%, respectively. In a more recent work [35], they have integrated a Fuzzy Inference System (FIS), combining inner and outer facial features training with GROUPS. The system achieved a performance of 93.35% when testing with LFW.

Eidinger et al. [41] reported results on GROUPS and Adience. Their approach based on LBP-like features and SVM reported accuracies of 87.5% and 76.1%, respectively. From these results, they claimed that Adience represents better real world conditions. More recently, Hassner et al. [60] focused on the frontalization problem to better synthesize the frontal face from a given view. The GC evaluations in Adience boosted accuracy up to 79.3%.

In relation to our work on GC in the wild, we initially proposed, similar to other authors, the integration of facial and non-facial features. More specifically, we focused on the face, and head and shoulder patterns (F and HS, see Fig. 7.4) [23,27,103], extracting features based on LBP and HOG. The best reported in-dataset accuracies for LFW and the GROUPS Dago's protocol were 95% and 91.65%, respectively, though the latter increased to 94.28% when considering only adults for training and testing. The exploration of a larger collection of local descriptors, grid resolutions and the combination with more densely extracted features from the periocular area (P) [30], and the mouth area (M) [28,29] (see Fig. 7.4) have more recently led to an improvement in performance. For the GROUPS Dago's protocol, the accuracy integrating P increased up to 93.54%, including both P and M areas achieved 94.04%.

Deep learning has recently achieved significant results in different computer vision tasks. Facial analysis is a problem, where the pattern variation complexity is well suited for applications in the wild. Liu et al. [83] adopted this focus to extract up to 40 facial attributes from CelebFaces and LFW, they achieved an accuracy of 94% for GC in LFW. Levi and Hassner [79] apparently reported the first GC results based on Deep CNN for a hard in the wild dataset, particularly for Adience. Using the in-plane aligned version of faces, they increased performance to 86.8%, suggesting the need to work on better alignment and frontalization to boost performance further. Another conclusion was the evidence of a larger number of classification errors in babies and children, an aspect also observed in our studies related to GROUPS [23].

More recently, different studies have suggested combining CNN outputs and local descriptors, also called handcrafted features, for GC. Wolfshaar et al. [132] proposed the fusion of Deep-CNN and SVM for GC, reporting results from FERET and Adience datasets. The latter achieved an accuracy of 87.25%. Mansanet et al. [88] weighted local features and CNN outputs achieving single dataset GC for LFW and GROUPS of 96.25% and 90.58%, respectively. Their cross-database experiments reported 94.48% training with GROUPS and testing with LFW, and 83.03% for the reverse.

Considering this trend, we have also performed a study to compare GC performance based on the fusion of local descriptors versus CNN [23]. The rather similar accuracies achieved for LFW, GROUPS, and MORPH motivated us to evaluate the integration of the CNN outputs in the fusion approach. The outcome is an evident increase in GC performance, reporting state-of-the-art accuracies in both in-database and cross-database GC performance. For full in-database cross-validation, accuracies reached over $97\text{\%}$ ($98\text{\%}$ in adults) and $99\text{\%}$ for GROUPS and LFW, respectively. Related to cross-database results [23], training with LFW and testing with GROUPS reported 90.14% and, while for the reverse combination, accuracy was 98%, considering full datasets in both cases. These cross-database accuracies beat most literature in-dataset evaluations for both datasets.

7.3.3 Discussion

Observing the summarized results for single database experiments in Table 7.2, the first impression suggests that LFW accuracies are typically over 90%, achieving up to 98%, thus suggesting little room for improvement. LFW is the lightest in the wild datasets studied as its performance is similar to constrained datasets, and quite different from the typical results achieved for Adience and GROUPS, which clearly exhibit lower accuracies. Thus, both datasets seem to be closer to real world situations, including larger variations in terms of pose, background and resolution. The former is a recent dataset with relatively few experimental evaluations so far, with accuracies under 88%. The latter is, according to the 2015 NIST report, currently the most challenging dataset with difficulties to achieve results over 92%, similar to commercial systems. It is worth highlighting our own recent approach that combines handcrafted features and CNN reaching over 97.2% for the GROUPS dataset.

This conclusion is also confirmed by observing the cross-database results in Table 7.3, where, with the exception of testing with the biased FERET or LFW, accuracy hardly reaches 85%, showing a lack of accuracy of state-of-the-art solutions in cross-database GC in most cases. This is probably due to the special test dataset characteristics, which reduce the complexity. We therefore agree that high accuracy can be achieved for homogeneous, biased and/or reduced datasets of good quality, etc.

Another conclusion is the current challenge scenario is to provide high precision with cross-database classification, i.e., testing with a completely independent in the wild dataset. Cross-database GC is closer to real world applications and avoids any dataset bias. In realistic applications, a gender classifier is first trained with a set of images, and then deployed under different scenarios that may certainly differ from those of the training dataset.

This fact has been demonstrated several times in recent GC literature, single dataset experiments with FERET have reported very high GC rates [7,9,93,132], but the generalization of the classifier is poor if tested in the wild [132], with a significant drop in accuracy [6]. This observation is also confirmed in Table 7.3, when LFW is used for training, in which accuracy testing with GROUPS is remarkably lower than for the reverse situation. GROUPS dataset again highlights the difficulties. Unfortunately, we are aware of just one reported cross-database results testing with Adience, reducing the relevance of further conclusions.

The latest results that integrate CNNs suggest their utility. The characteristic of CNNs as end-to-end classifier reduces the need for human design to select the features to use. This was suggested by Perlin et al. [100] for GC using full or almost full body images. Specifically, ViPer [49] and HATdb [116] extract information from the upper and lower body added to GC performance. Under this approach, the feature extraction task is mainly performed by the CNN. However, several recently reported results combining handcrafted features and CNN [23,88,132] suggest an improvement in the overall performance, when CNN and features are combined, indicating a promising research avenue.

To explore future improvements in GC, we would like to mention the results presented by Klare et al. [74] and the proven connection between age and gender by Bekios et al. [10]. Two aspects are highlighted: (i) the need for training on datasets that are evenly distributed across demographics, to manage different demographic groups; and (ii) the interest in designing solutions for different demographic groups to increase performance on such groups.

The former is evidenced by cross-database results, where training with the hardest datasets, reported higher accuracies than in lighter ones. It is also likely to be affected by the demographic variations included. The latter agrees with the results reported in [23], removing under 20-year-old individuals from the Dago's protocol, and reaching significantly higher GC accuracy, up to 94.2%. Table 7.4 presents the GROUPS average faces per age range and gender, with some evident appearance differences.

Given existing in the wild datasets, whose compilations have not taken into consideration the balance in terms of variables such as ethnicity, age, etc., it is not straightforward to obtain a general conclusion about the current leading solutions. Fig. 7.5 illustrates a subset of the classification errors obtained when evaluating GROUPS with the state-of-the-art classifier proposed in [23]. The upper and bottom rows present, respectively, females and males wrongly classified in the samples. For both classes only the five most distant to the classification border samples are shown, presenting them according to their output score. For females, two samples produced a score significantly farther from the class border. Both belong to elderly ladies, and is likely to be due to a poor representation of that demographic group within the training set. For male samples, age seems to be again a misclassification factor, but also annotation errors might be present. In any case, it is clear that there is still room for improvement.

A final comment can be made related to difficult or ambiguous samples [25], El Din et al. [40] made use of a second classifier specialized in difficult or borderline samples, which may be an approach to be considered.