8.1 Machine Vision Approach

The Machine Visions approach to gait recognition entails the acquisition of gait signals using one or more video-cameras from a distance. Therefore, it requires an ambient set-up. As a first common step, systems in this category use techniques for video and image processing to detect the user's image in a scene, to track the user's walk, and to extract gait features for user recognition. In the most common design, a preprocessing phase includes background subtraction and body silhouette extraction, eventually identifying the Degree of Freedom (DOF) points [1] (generally corresponding to body joints) in order to track user's gait. What generally changes from one system to another is the possible further preprocessing used to improve the quality of the matchable extracted data, and/or the kind of matching strategy used in order to find the correct identity. The majority of Machine Vision-based works in the state-of-the-art convert preprocessed data into a Gait Energy Image (GEI), and use these images as the base for feature extraction. (See Fig. 8.1.)

Among the other possible differences in the state-of-the-art proposals, one should mention the use of different technologies for data acquisition, such as different kinds of cameras (fixed or Pan-Tilt-Zoom) that work in different conditions (visible light, infrared, or thermal).

In addition, there are systems that exploit the fusion of the data acquired by more cameras, in any combination. In the case of multiple sources, it is obviously necessary to synchronize those signals, generally using a stereo calibration procedure which requires additional computational costs but in general tends to improve performance.

The state-of-the-art proposals that work in visible light setting may eventually suffer from pose, illumination, and occlusion problems, especially if in outdoor environments. Moreover, another aspect to take into account is the perspective with respect to the camera. In this case, when the user is not consistently aligned with the camera, this create anomalies and distortions, for example, in the extracted GEI.

Due to the mentioned problems, Machine Vision-based techniques can not be used as the only information source in uncontrolled scenarios, unless some enhancement strategy is devised. Some possible solutions are proposed in the literature in order to strengthen machine vision-based systems. For instance, the problem of pose can be resolved combining data from different cameras or choosing as video source only the frames in which the highest number of DOF points can be extracted from the images. The problem of illumination can be reduced introducing infrared cameras that allow a more accurate silhouette extraction, especially in dark scenarios, even if they have problems with strong illumination sources if not combined with a visible light camera. The possible occlusion of elements in the body silhouette represents a delicate aspect because the recognition algorithms significantly decrease their performance if the subject holds an object or carries a backpack, due to an erroneous silhouette extraction. A thermal camera can be a suitable solution to solve this problem, as shown in [2], because it can help in the identification of the subject body, ignoring the eventually carried objects. Finally, the problem of perspective can be attenuated by geometric transformations, but this would increase the computational costs and it is not always possible to project data in a reliable way, to reconstruct an aligned position.

It is worth mentioning that, notwithstanding the exploited technology, machine vision algorithms can be divided into two main groups: model-free and model-based. The model-free techniques are also often referred as “silhouette-based” techniques. The first common step is to separate the human silhouette from the background on a frame-by-frame basis. Classifiers are designed to consider the observed motion of the silhouette. Model-based techniques rely on a precise model of the human movement that is built by limbs and joins. Such features are extracted from images and matched against those in the model.

Interested readers can find a more complete survey on machine vision-based gait recognition in [3], while [4] provides a description of model-free machine vision approaches only.

8.2 Floor Sensor Approach

The Floor Sensor-based approach relies on the use of a specially equipped floor able to record pressure variations. This allows a data acquisition that is not afflicted by the well-known and above mentioned machine vision problems, i.e., pose, illumination, and occlusion. Moreover, the preprocessing algorithms, working generally on linear signals, have a very little impact in terms of computational costs. On the other hand, as machine vision-based systems, also those implementing this approach suffer from the lack of ubiquitousness because, even in this case, the control of one or multiple zones requires equipment set-up and possibly duplication. Besides this, the performances are generally lower with respect to those of the machine vision. There are very few works about this approach used for recognition, and the research in this field is probably being definitively phased out by the new and more practical wearable sensors.

Fig. 8.2 shows an example of a floor equipped with sensors.

For more details on this kind of techniques, interested readers can see [5–7].

As a further note, even if this approach is nowadays rarely used for the recognition of persons, the interest is still alive in the Biomedical field. In that field, equipped floors are used for gait pattern analysis in the diagnosis of particular pathologies and as a rehabilitation support. Two examples can be found in [8] and [9]. In the first, data from equipped floors are used for the diagnosis of Cerebral Palsy (permanent movement disorders that appear in early childhood) and the evaluation of the outcomes from treatments, while in the second they are used in the study of Parkinson's disease.

8.3 Wearable Sensor Approach

This section gives a more extensive description of the latest line of research in gait recognition: the one based on wearable sensor devices. The most used one in this category is, of course, the accelerometer, so it will be further discussed in more detail in the following. As for now, it is sufficient to say that it records acceleration values along three orthogonal axes. Another kind of sensor sometimes used in gait recognition is the gyroscope. The gyroscope is a sensor made up by a spinning wheel or disk, rotating around its axis. When the disk is rotating, the axis tends to maintain an orientation always parallel to itself and to oppose any attempt to change such an orientation, according to the law of conservation of angular momentum. For this reason, gyroscopes are generally useful for measuring or maintaining orientation. Regarding gyroscopes applied to gait recognition, state-of-the-art works report discordant results. When used, the gyroscope is mostly considered as an additional source of information to support recognition by accelerometer. In some cases, it seems to improve the global performances of recognition, but in some others it lowers them instead. In addition, differently from the accelerometer, that is a standard equipment of smart devices, e.g., smartphones and tablets, gyroscope is sometimes missing. For sake of completeness, it is worth mentioning the magnetometer too, because it is often another standard equipment of smart devices. For instance, this sensor is the one that allows geolocalization. It is used to measure magnetization, the strength and possibly the direction of the magnetic field at a certain point. For this reason it acts as a compass in consumer devices. To the best of our knowledge, this sensor is barely used in gait recognition because it merely contributes to detect walking direction. Moreover, it can be negatively affected by external magnetic fields beyond the Earth one.

8.3.1 The Accelerometer Sensor

In this section, readers can find a brief description of the accelerometer, one of the most used equipment in wearable sensor-based gait recognition. Section 8.3.1.1 sketches a general description of this kind of sensor, showing the principle underlying its functioning, and how the most recent models are built by adapting this principle to miniaturization. Section 8.3.1.2 describes in details the most important and useful parameters to be taken into account when working with this type of sensor. Section 8.3.1.3 introduces some problems in the use of accelerometers and proposes some solutions, when possible. Finally, Section 8.3.1.4 discusses some problems related to data acquisition.

8.3.1.1 General Description

The accelerometer is a sensor able to register acceleration variations in time, reporting them in terms of $\frac{m}{s^2}$ or g. Even if it is possible to find accelerometers with only one or two axes, the most common models have three. Nowadays, the widespread use of smartphones has significantly increased their diffusion. In fact, modern cellular phones always have a built-in tri-axial accelerometer sensor, and many of them have a gyroscope and a magnetometer, too. There are a lot of different accelerometers, but this chapter will further discuss only those built in smart devices. In any case, the general principle is always the same: a mass is taken hang up by some force, e.g., the one produced by direct attachment to an elastic element, such as a spring, and, when an external force moves the sensor (and consequently the mass), the device measures the movement. Taking into account the direct proportionality among movement and acceleration, it is possible to coherently convert the variation in position into an electric signal. This signal will so contain the acceleration variation during time. It is worth noticing that this sensor can reveal a different acceleration per each axis, so it is possible to access three different measurements. Fig. 8.3 shows a simplified schema of the accelerometer functioning: it is possible to see a spherical mass hang up by three springs, representing the three axes, which pass through it. Moving the cube, the mass will change its position, compressing and extending the spring lengths. These compressions and extensions allow revealing the physical acceleration on each axis and its direction.

Even if the majority of accelerometers use this kind of schema, nowadays in order to reduce dimensions, a micro-manufactured silicon structure is adopted, as highlighted in [10]. In this case, the mass is not spherical and can be substituted by a mobile plate in a capacitor, hanged up between two other plates fixed in the structure in a way that avoids any contact between them. The sensor measures the mass movement exploiting the electric capacity variation in the capacitor, which directly depends on the plate distance.

Finally, the wearable commercial devices nowadays generally use accelerometer sensors made of a single silicon chip with an integrated electronic circuit. These chips are microscopic, with dimensions comparable with a match tip, and they are included in the MEMS (Micro-Electro-Mechanical-Systems) category. A complete description of all MEMS characteristics can be found in [11]. This kind of sensor, in addition to its microscopic size, has generally high sensibility (see below for a definition of this characteristic), is little influenced by temperature variations, provides good accuracy, is able to reveal relatively small acceleration variations, and last but not least, it has very low power consumption and is very cheap. For these reasons, such sensors are perfect to be integrated in everyday usable devices such as smartphones and tablets.

8.3.1.2 Accelerometer Characteristics

It is worth reminding an important accelerometer property, i.e., the fact that it is a linear sensor, because its response is directly proportional to the physical acceleration it is intended to measure. This is a very useful characteristic that can be used in various ways. When working with an accelerometer sensor, it is worth taking into account some important parameters that define its further physical characteristics and help better exploit its functionality. In the following, the most relevant ones are introduced.

The maximum range parameter describes the range of acceleration values that can be measured by the sensor: if a collected value is over the maximum of this range, the accelerometer will lose the linearity property, and this happens symmetrically for values below the minimum. This parameter is normally expressed in terms of g (gravitational force, 9.81 m/s²). Common built-in accelerometers have a range that varies from $\pm 2g$ to $\pm 8g$.

The bandwidth expresses the maximum frequency of detectable variations, and is better known as the sampling rate. This value is measured in Hz (1/s), and for accelerometers built in mobile devices it is generally about 100 Hz, while it is possible to find high quality accelerometers with a sampling rate of more than 500 Hz.

The sensitivity, sometimes denoted as resolution, describes the minimum detectable acceleration variation. This value is generally expressed in terms of LSB (Least Significant Bit)/g. This means that if an accelerometer has x as sensibility value, it can provide only measurements that are multiples of x.

Another important parameter of accelerometer sensors is the Offset (often referred as Zero-g Offset or Zero-g Bias). This value describes the difference between the real output and the ideal output when there is no acceleration applied to the sensor. Considering sensors built in smartphones, it is assumed here that the X axis is the one co-planar with the screen, parallel to the short side and with positive direction rightwards, the Y axis is the one co-planar with the screen, parallel to the long side and with positive direction upwards, and the Z axis is orthogonal to the screen with positive direction frontwards (see Fig. 8.4). In an ideal scenario, when an accelerometer sensor is placed on a horizontal flat surface with the frontal part facing up, the Offset value should be 0g on the X and Y axes and 1g for the Z axis. Inverting the sensor by 180^∘, the values for X and Y axes would remain unchanged while the value for Z would change to $-1g$. Table 8.1 reports the ideal values in all of the six “flat” positions, which are shown in Fig. 8.4 when the sensor is embedded in a mobile device, a smartphone in this case.

Table 8.1

Ideal value for device position

8.3.1.3 Pros and Cons of Using Accelerometer Sensors

The major advantage of using an accelerometer sensor in biometrics is, of course, its ubiquitous presence in everyday life. This is due to the wide spread of smartphones, which nowadays always have these sensors built-in. Their normal usage in this setting is just for very simple tasks, such as shake motion, screen orientation changes, and so on. Nevertheless, it is important to consider that, being sensors of MEMS type, they can support much more complex and demanding tasks. For instance, the use of a smartphone for biometric gait recognition is increasing in popularity. A number of state-of-the-art proposals in this field show how the smartphone itself can be “trained” to recognize its owner (1:1 matching) by walking pattern, exploiting the internal accelerometer and eventually the gyroscope. Moreover, the smartphones can send data to a remote server, and this can be useful for recognition in identification modality (1:N matching), e.g., to grant access to a reserved place. In this case, the user does not claim an identity and must be recognized among a number of enrolled ones, or rejected as unknown. It is worth noticing that the simple identification of the device does not ensure that the person carrying it is actually the intended one. A further possibility is the use in multibiometric settings. In this case, the remote server collects different biometric templates (e.g., face or gait visual pattern, via video cameras) and fuses recognition results. It is to notice that accelerometer data acquisition does not suffer from well-known problems of computer vision-based approaches, such as pose, illumination, and occlusion. These sensors can follow the user everywhere, eliminating the need for further devices or equipment duplication, as well as any environment modification, like in floor sensor-based techniques. In addition, this approach, differently from the other two mentioned before, can allow the recognition of multiple users without any further dedicated algorithm because there is no overlapping of data. In this way, it can be possible to separately recognize more users that contemporaneously walk in the same controlled zone, without increasing neither the complexity nor the accuracy of the system. Last but not least, of course, the cost is fairly near to zero.

Notwithstanding the above mentioned positive aspects, it is worth pointing out limitations, too. For the cons, it is possible to mention all the problems related to gait biometrics described in Section 8 (e.g., possibly noisy signals, influence of shoes characteristics and ground slopes, influence of walking speed), which may be more evident with respect to machine vision techniques because the input sources are directly attached to the user. Moreover, the accelerometers suffer from inter-device differences. Even in the case of the same accelerometer model from the same production line put in identical conditions, there can be relevant differences in the captured accelerometer signals. This is due to calibration and systematic errors that likely happen, especially when the sensor is built in a smartphone (even because in this case, their usual role is just for gaming or other applications that do not require high accuracy). However, as shown in [12], it is possible to significantly reduce this latter problem by a simple ad hoc procedure.

8.3.1.4 Denoising, Interpolation, and Device Rotations

As for all signals that are collected by a physical sensor, even the acceleration data are naturally affected by noise. Moreover, in the case of an embedded accelerometer, it is worth considering that the data might be not read with a constant frequency but only after “significant” value changes, as for Android standard. In addition, the acceleration values are not independent from the sensor orientation and this creates significant problems when the accelerometer (or the device in which it is built-in) can rotate, e.g., if it is in a bag or in a large pocket. Different kinds of approaches are studied in the literature in order to reduce these problems.

In the case of noise, it is necessary to consider it from two opposite perspectives. On the one hand, it can complicate and negatively affect the eventual preprocessing phases, such as the step segmentation procedures. On the other hand, applying a heavy denoising algorithm can significantly modify the original signal, therefore reducing its discriminative power. A possible solution is to use some “heavy” denoising filter in the preprocessing phases, e.g., for a more reliable step detection, and to use the derived information, e.g., the detected start and stop points of steps, to analyze the original signal, or a version obtained with a “soft” denoising filter. For instance, as for step segmentation, a preliminary of step boundaries can be obtained from the denoised signal, and then refined over the original one.

To avoid the problems arising from the possibly varying frequency of data capture, a lot of state-of-the-art works perform time normalization on the gait signals, in order to have data with a constant frequency. In some of them, an additional goal is to obtain signals with the same number of samples. These goals can be achieved by setting up the appropriate parameters in advance, so to capture a signal with the desired characteristics. As an alternative, this can be obtained after capture, via a post processing. For example, this often means than a missing point in the time sequence is approximated by interpolation from the neighboring ones. As it happens with denoising, this could reduce the discriminative power of signals. For example, the results achieved in [13] over an in-house dataset (no signal preprocessing is applied to the collected raw data) and those achieved with the same algorithms over the dataset available from [14] (that provides interpolated walk signals) are somehow contradictory. Some of the proposed algorithms seem to get advantage from this kind of preprocessing while some others, instead, seem to reduce their performances. So it could be interesting to conduct a more extensive and deeper study in this direction, in order to understand the reasons of such differences.

The last problem, namely the signal distortions caused by sensor rotations, is probably the most difficult to tackle. All state-of-the-art proposals, except the ones which specially focus on this issue (for example, the proposals in [15,16]), fix the accelerometer/device, e.g., to the user body. In this way, they avoid any external rotation that is not strictly dependent on gait. Another, much simpler way to avoid casual rotations could be using the magnitude vector (given for each sample i by the formula $\sqrt{x_i^2+y_i^2+z_i^2}$) instead of the individual value for each of the three axes. This would create a 1D vector that is rotation invariant, but would also cause a total loss of the correlation between axes. Unfortunately, in gait analysis by an accelerometer signal, there will always be a dominant axis (depending of the position of accelerometer/device) that has a higher impact on recognition. It is important to maintain reference to such axis. Therefore, this strategy, though apparently viable, is not suitable to solve this problem.

8.4 Datasets Available for Experiments

The already mentioned work in [14] has also introduced one of the largest freely accessible datasets for wearable sensor gait recognition. It contains more than 1800 walking signals from 175 subjects, with accelerometers set at 5 different body locations, during two different sessions. This dataset provides significantly long walking signals, but the accelerometer samples are interpolated, and it is not possible to get access to the raw data. Another very large dataset is the one released by Ngo et al. in [17], which contains signals from 744 subjects collected by 4 devices put in the hip and waist zones. Nevertheless, even if it has a higher number of data and users than that in [14], this dataset is made of very short walking signals, collected in a single session, and derived by manually segmenting a single overall one at positions where the ground slope changes.

Is worth noticing that very few of the state-of-the-art proposals face the problem of identification. As a matter of fact, smartphone is still used as an authentication device only to confirm the identity of the owner, so that most results are reported in the literature in terms of verification operations.

8.5 An Example of a Complete System for Gait Recognition

This section shortly describes the main design elements of a possible completely automatic gait recognition system, feasible for use in a real scenario setting. The example stems from the extension of the works proposed in [18] and in [13]. The prototypical system is composed of two different modules, one running on an Android mobile device, e.g., a smartphone, and the other on a remote server.

Fig. 8.5 shows a sketch of the system.

One of the components of the system is a pair of beacons. The beacon devices are very small Bluetooth emitting sources that use the low energy protocol (i.e., the standard Bluetooth 4.0), and, in general, their only function is to broadcast their IDs.

The mobile module running on the personal mobile device is an App with two possible uses. The first purpose is data acquisition during both subject enrollment and normal operation. In order to allow a completely transparent, non-intrusive and non-invasive automatic procedure, the system will exploit the two mentioned beacons, which will be used to trigger the start and stop of the recording. In practice, in the setup of the described system, beacons will only transmit their IDs to all other listening devices at settable intervals: when a mobile device receives that signal, it will start or stop the gait recording, according to the received IDs. In this way, no cooperation is required by the user except for turning on Bluetooth on the mobile device and walking to the area of interest. After recording termination, the complete signal will be finally sent to the server (the second module) which handles either the storing of the new enrolled identity, or the recognition of the identity of the incoming user, and eventually grants the authentication. So, there will be only a data transfer between the mobile device and the server, in order to reduce the energy consumption.

In the second possible modality of use, the mobile App does not go through all the above mentioned steps, and can be useful for mobile device security. In other words, the App can be used to authenticate the owner of the mobile device in order to unlock it. In this case, the mobile App will implement a local recognition system that will work in verification modality, using the owner identity as the implicitly claimed identity. This is possible because the recognition algorithms used for gait are generally not so computationally demanding, especially when carrying out a 1:1 matching.

As mentioned above, the second module is a server. The use of an external server is mainly due to security/privacy issues. A 1:1 verification is better carried out in a totally local way, therefore avoiding unnecessary network traffic. On the contrary, if working in identification modality, for example, to grant to a subset of users access to a critical service/area, a mobile device should not contain data from all other authorized persons. Moreover, the decision of acceptance or denial would be preferably assigned to an external part arbiter. Finally, another advantage of using an external server is the possibility of combining gait with other biometrics, or with gait itself in a machine vision setting. This would create a multibiometric system with higher robustness and accuracy.

The server will wait until a new gait signal file is received by the mobile App and then will start the recognition according to the configured matching strategy. In the literature it is possible to find a significant number of works regarding this topic. Among these works it is possible to identify two main categories. Since a complete review of gait recognition techniques is out of the scope of this chapter, only some examples are mentioned. Proposals in the first category rely on a preliminary step/cycle detection, so that “chunks” of signal are matched instead of the full one; proposals in the second one do not carry out this preliminary phase and generally uses machine learning techniques. Some examples of the first category are [19,21,22] and [20]. The first two exploit the well-known Dynamic Time Warping (DTW) algorithm (as for other examples in this group), while the third uses signature points and neighbor search, and the last one exploits the minimum Euclidean distance between each pair of steps. Some examples of the second category are [23–26] and [14]. The first of these works uses Support Vector Machine (SVM) technique, the second exploits Hidden Markov Model (HMM), the third and the fourth use k-NN algorithm. The last one, as an evolution of the system proposed in [22], uses again the signature points but with a preliminary clustering phase that increases the final performance.

The recognition methods tested with the system in [18] and [13] fall into both categories and exploit the DTW algorithm, as the works in [20,21]. It relies on a novel step segmentation algorithm, and in the mentioned works it is compared with some other state-of-the-art proposals, showing the advantages of this new approach. Table 8.2 reports the most relevant achieved results, among the plenty of tests performed, both in verification and in identification modality. As for verification, settings with a single or multiple gallery template per subject have been tested, with results in terms of Equal Error Rate – EER. As for identification, both open set and closed set settings have been tested, according to the presence or not of all incoming subjects in the gallery, with results in terms of EER again for the former and Recognition Rate – RR for the latter. The in-house dataset (BWR) and the two large ones described in Section 8.4 (ZJU-gaitacc and OU-ISIR) have been used as benchmarks.

The results cover both matching of entire signals, and different strategies entailing step segmentation. The label Walk in the table refers to entire signal matching. The label Best Step refers to matching the centroids of the two sets of segmented steps (the centroid is defined as the step with the minimum average distance from the others, measured in terms of DTW). The label Best Step vs. All refers to the matching of all steps in the probe with the centroid of a gallery walk. For each probe, the average distance of its steps from the gallery centroid is measured, and taken as the matching result. The label All Steps vs. All refers to a similar matching, but this time each step in the segmented probe is compared with each step in the gallery walk. The label Step Sliding Window refers to a strategy that attempts to overcome the limitation of the fixed walk length while maintaining the advantages of matching longer walk segments, so preserving inner correlation. The shortest walk signal is used as a sliding window over the longest one, where sliding is guided by segmentation points. The minimum measured distance is taken as the matching result.

The results in Table 8.2 are in line with expectations. As a first element to underline, the presence in the gallery of multiple templates per subject improves recognition performance, since the same subject can be recognized in different situations. This is evident from verification results. The second point concerns the possible segmentation of the gait signal in separate steps. The best results are generally achieved with the unsegmented signal. As a matter of fact, it seems that a kind of co-articulation links the different steps in a characteristic way, as it happens for phonemes in speech, so that breaking such a link causes some loss of information. On the other hand, using the entire signals causes a constraint concerning the same or comparable length of the signals to be matched, which is a limitation in uncontrolled or under-controlled settings. The kind of segmentation-based strategy that achieves the best results depends on the dataset. In general, All Steps vs. All tends to be the second best matching. It is to notice that all methods show very poor performance on OU-ISIR dataset. However, matched chunks of signals are very short in that case, and this may hinder recognition regardless of the used algorithm. Regarding ZJU-gaitacc dataset, results achieved on these data are not consistent with the others, as mentioned in Section 8.3.1.4. For example, Best Step vs. All and Best Step provide poor accuracy on the other datasets, while achieving good results with this one. This might happen due to the interpolation of the source signals.

Research on the described is ongoing, further focusing attention on the multi-device normalization problem, in order to improve interoperability and flexibility.

8.6 Conclusions

This chapter discussed different approaches to gait recognition suggested up to now. It mainly focused on the wearable sensor-based one that is increasing in popularity during the last years. For this reason, some information was provided about the most used sensors in this field, as well as the pros and cons of their usage with respect to the other approaches in the literature, namely Machine Vision- and Floor Sensor-based methods. Some possible solutions to problems raised by this technology were discussed. A fully automatic system was sketched, with a brief description of its functioning and some references to other state-of-the-art works. Finally, some results from an ongoing study were presented. The approach seems very promising, and results seem to testify that gait recognition can be a useful strategy for biometric applications, especially in a multi-biometric perspective.