2.1: Early and commercial data gloves

The first data glove was developed in 1977 by researchers in MIT (Massachusetts Institute of Technology). It was called the “Sayre Glove” and utilized elementary fiber-optic sensors (Sturman and Zeltzer, 1994). These sensors consisted of tubes that transmitted light between their two ends. The intensity of the light passing through the tubes decreased as the tubes were bent by the flexion at the finger joints. The light intensity was measured by the voltage of a photocell placed at one end of the tube. It was observed that there was a strong correlation between the angle of flexion and the voltage of the photocell. Other examples of early data gloves made in the early 1980s include the “Digital Entry Data” and the “Super Glove,” which used bend sensors and printed resistive inks respectively (Dipietro et al., 2008).

Recent commercial data gloves include the “Cyberglove,” “5DT Data Glove,” and “Didjiglove.” The Cyberglove developed by Stanford University consists of 18 or 22 piezoresistive sensors. The model with 18 sensors only measures the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joints, while the model with 22 sensors measures the MCP, PIP, and distal interphalangeal (DIP) joints (CyberGlove II, 2020). In addition, both models measure the abduction, adduction, and wrist movements. The 5DT glove measures movement at the joints using fiber-optic sensors (5DT, 2020). These sensors measure the angle of flexion by its correlation to the weakening of light. It utilizes only one sensor per finger. In particular, the overall flexion at the MP and IP of the thumb is measured by a single sensor, while for other fingers, the overall flexion at the MCP and PIP joints are measured by a single sensor. An upgraded version of the glove uses more sensors to measure abduction and adduction between the fingers. The Didjiglove employs capacitive sensors to measure the flexion at the MCP and PIP joint (Dipietro et al., 2008). The capacitive sensors comprise of two comb-shaped conductive layers that are separated by a dielectric. Although recent data gloves have improved the accuracy of early data gloves, the core design of embedding a strain sensor in a textile glove has been retained. Therefore, it is imperative that we discuss the sensing mechanism of the popular strain sensors used in these data gloves.

2.2: Sensing mechanism in data gloves

Data gloves can be categorized based on their sensing mechanism. The three main types of sensors used in data gloves are fiber-optic sensors, conductive strain sensors, and inertial sensors. This section reviews their operating principles, advantages, and disadvantages.

4.1: Classical machine learning algorithms

Machine learning algorithm can be differentiated by their type of learning. Supervised learning occurs when correct input-output pairs are provided for the algorithm during training, while unsupervised learning requires the algorithm to determine clusters of similar input data as no target output is provided (Rautaray and Agrawal, 2015). In this section, we describe a summarized theoretical background of the popular classical machine learning algorithms used in glove-based gesture classification.

4.1.4: Artificial neural network (ANN)

ANN is a biologically inspired machine learning algorithm. It consists of input, hidden, and output layers that comprise of neurons. These artificial neurons simulate neurons in the brain by receiving an input, processing it using an activation function and producing an output. The output of ith neuron can be calculated as:

$y_i=f_i\left.\sum _{j=1}^n{\mathit{w}}_{\mathit{ij}}{x}_j-{\theta }_i\right),$

  (5)

where f_i is the transfer function, y_i is the output of the neuron i, x_j is the jth input to the neuron, w_ij is the connection weight between the neurons, and θ_i is the bias of the neuron (Neto et al., 2013). Traditionally, the transfer function is either Gaussian, sigmoid, or Heaviside. Moreover, ANNs are trained by adjusting the connection weights. This can be achieved by algorithms such as backpropagation or reinforcement learning. The key factor in the operating principle of the learning algorithms is their weight-adjustment rules such as the Hebbian rule and the delta rule. Fig. 4 shows a feedforward neural network (FFNN), the simplest form of an ANN. It never feeds the output back to the output because it operates in a single forward direction.

4.2: Glove-based gesture classification with classical machine learning algorithms

Several classical machine learning algorithms have been explored in the classification of gestures from data acquired by a data glove. Initially, ANN was used to classify five gestures from the American sign language (ASL) (Beale and Edwards, 1990). Thereafter, a modified version of ANN using backpropagation was implemented to recognize a taxonomy comprising of forty-two gestures from Japanese Kana (Murakami and Taguchi, 1991) acquired by a VPL data glove. Back propagation is a feedback algorithm that improves the classification results by using gradient descent to adjust the learning weights of the network. The network consisted of 16 nodes in the input layer representing 10 bend sensors and 6 positional sensors, 150 nodes in the hidden layer, and 10 nodes in the output layer representing the 10 dynamic gestures. Moreover, the input data was augmented and filtered to improve the accuracy of the network. The results show a 96% accuracy when the data was filtered and augmented; and 80% accuracy without augmentation and filtering.

Furthermore, a radial basis function (RBF) network were employed to classify twenty static gestures from five users of a Cyberglove (Weissmann and Salomon, 1999). The network was trained with four users and validated with the last user. The results showed that the average accuracy of the network was 88% during cross validation. In contrast when the validation user was included in the training set, the average accuracy was 98.3%. This study illustrates the difference in the accuracy of machine learning algorithms between “unseen” experiments and “seen” experiments. In unseen experiments, the data in the validation set is not included in the training set, while in seen experiments, some or all of the validation data is included in the training data set. We observed that machine learning algorithms are less accurate in classifying unseen data. Particularly, in glove-based gesture classification, the reduced accuracy of the machine learning algorithms in unseen experiments can be attributed to the difference in the hand dimensions of the unseen users in the validation data set and the users in the training data set. However, the difference in accuracies between unseen and seen experiments can be reduced by utilizing a large number of users in training the algorithm.

Furthermore, a feedforward ANN was utilized in classifying gestures for a VR driving application (Xu, 2006). Three hundred gestures were acquired from five participants with the Cyberglove. The gestures were split into 200 gestures for the training set and 100 gestures for the validation set. The average accuracy was 98%. This high accuracy was obtained because this was a seen experiment. In contrast, when data from three new (unseen) participants were used as a validation set, the recognition accuracy reduced to 92%.

In Luzanin’s study (Luzanin and Plancak, 2014), PNN was used to classify twelve static gestures acquired with the 5DT Data Glove. Clustering algorithms were implemented to reduce the training data without affecting the performance of PNN and to maintain the representation of the actual input data. These clustering algorithms were K-means, X-means, and Expected Maximization (EM) algorithms. The classification accuracies for the seen experiment were 93.4%, 96.18%, and 95.98% for K-means, X-means, and EM algorithms, respectively. Furthermore, for an unseen validation user, the results were 63.05%, 52.48%, and 77.14% for K-means, X-means, and EM algorithms, respectively.

Two ANNs connected in series were employed in gesture classification for human-robot control (Neto et al., 2013). As depicted in Fig. 5, the first ANN was used to classify static communicative gestures, while the second ANN was used to classify noncommunicative gestures that occurred within the transition between the communicative gestures in the continuous data. The data was acquired with Cyberglove II that contains 22 sensors. Therefore, in both ANNs, the input layer and hidden layer each comprised of 44 neurons that represent two frames per sensor. Classification accuracy was up to 99.8% for 10 gestures and 96.3% for a taxonomy of 30 gestures. The aim of the study was to accurately recognize static gestures within continuous data; therefore, the authors limited the validation data set to only seen data, hence the high accuracy.

A multiclassifier approach was undertaken in (Ibarguren et al., 2010) to classify gestures acquired by a 5DT Data Glove. The gestures were eighteen ASL alphabets that do not require positional measurements of the wrist. The gestures were classified using a combination of decision tree and k-nn algorithms as shown in Fig. 6. A clustering method based on Euclidean distance generated a decision tree. Thereafter, k-nn (k = 1) was used to classify letters at the lowest level nodes. These letters such as a/e or f/o are very similar, and the 1-nn classifier aids in providing a more accurate classification. In addition, a segmentation layer is utilized before classification to separate the recorded gestures from the real-time continuous data. Experimental results show a 99.49% segmentation accuracy and a 94.61% classification accuracy.

A self-organizing map (SOM) was used to classify 10 static gestures acquired with a 5DT Data Glove (Jin et al., 2011). SOM is an unsupervised machine learning algorithm that aims to model the input data into a discretized lower dimension map. Training data was acquired from six participants, and the algorithm was validated with 10 participants that included two participants from the training data. The algorithm performed well with a 94.29% accuracy.

In addition, a custom IMU data was used alongside an Extreme Machine Learning (ELM) algorithm to classify 10 static gestures (Lu et al., 2016). Two sets of 44 and 45 features were extracted from the input gesture data set. The 45 feature set comprised of yaw, pitch, and roll angles of the five fingers, while the 44 feature set comprised of the 45 features of the fingers; and the yaw, pitch, and roll angles of the palm, forearm, and upper arm. ELM algorithm was utilized because of its low computational burden and reduced human reliance as its input weights and hidden layer neurons and biases are generated randomly. A modified version of ELM algorithm proposed by Huang et al. (Huang et al., 2011) that employs a kernel method was also used in the study. The original ELM algorithm, the ELM-kernel algorithm, and SVM were compared using both sets of extracted features. The classification accuracy using the 44 feature set were 68.05%, 89.59%, and 83.65% for the ELM, ELM-kernel, and SVM algorithms, respectively, while for the 45 feature set, the accuracy for ELM, ELM-kernel, and SVM algorithms were 84.40%, 85.51%, and 81.09%, respectively, thereby highlighting the superiority of the ELM-kernel algorithm for gesture classification.

A more comprehensive comparison of classical machine learning algorithms in gesture classification was illustrated in the study cited herein (Heumer et al., 2007). A Cyberglove was employed in acquiring grasp types based on Schlesinger’s taxonomy. Subsequently, several classical ML algorithms were used to classify the data in six classification scenarios comprising of seen and unseen experiments. These 28 algorithms were obtained from a software package (Witten et al., 2005) and were grouped based into five categories: rule sets, trees, function approximators, lazy learners, and probabilistic methods. Rule algorithms classify the gestures by a set of logical rules, while tree algorithms (such as decision tree) classify gestures based on a pyramid of binary decisions. In addition, function approximators are supervised learning algorithms that derive an approximate function between the input data and the output class. Probabilistic algorithms produce probability models of each class and then determine (using a method such as Bayes theorem) the probability of each input data belonging to a specific class. Lazy learners delay classification of an input data until a request is received. Thereafter, the class of the data item is determined from the class of the closest data items based on the specified distance metric. The results depict that function approximating classifiers performed well with a minimum and maximum accuracy of 81.41% and 86.8%, respectively. Although the best classifier was a Lazy classifier at an accuracy of 87.61%, the average accuracy of Lazy classifiers was 78.77%. However, Bayesian, tree-based, and rule-based classifiers were poor performers. Particularly, Bayesian classifiers had a maximum and minimum accuracy of 75.31% and 61.02%, respectively. Tree-based classifiers had a maximum accuracy of 83.44% and a minimum accuracy of 31.06%, while rules-based classifiers had a maximum accuracy of 78.13% and a minimum accuracy of 30.88%. The best and worst classifier in each category is highlighted in Table 1.

4.3: Deep learning

Deep learning (DL) is a class of machine learning whose algorithms comprise of neural networks with several hidden layers. Examples of popular deep learning algorithms are Deep Belief Network (DBN), Deep Boltzmann Machine (DBM), Recurrent Neural Network (RNN), and Convolutional Neural Network (CNN) (LeCun et al., 2015). The advantage of deep learning algorithms over traditional machine learning algorithm is the ability of DL algorithms to automatically extract features from the input data without the bias that comes with manual feature extraction in classical machine learning algorithms as illustrated in Fig. 7. However, DL algorithms require significantly more data and computation resources than classical machine learning algorithms.

4.4: Glove-based gesture classification using deep learning

In this section, we review the applications that have utilized deep learning in glove-based gesture classification. Notably, a simple CNN algorithm was used to classify dynamic sign language gestures from data obtained with an IMU data glove (Fang et al., 2019). The CNN comprised of a convolutional layer, a batch normalization layer and a fully connected layer. A pooling layer was noticeably absent as the authors felt it was redundant in this architecture. The performance of the algorithm was compared to an LSTM method and a PCA-SVM method. The CNN algorithm had the highest accuracy at 99.6%, while the PCA-SVM and LSTM algorithms had accuracies of 82% and 80.8% respectively.

Furthermore, a light CNN architecture shown in Fig. 8 was implemented in a real time gesture recognition using a custom IMU data glove (Diliberti et al., 2019). The CNN was implemented using a very similar configuration to AlexNet. However, the authors performed a series of experiments to determine the optimal implementation of the network for their application. They achieved this by reducing the depth and width of the network till the set goal of at least 98% accuracy was met. The depth of the network was reduced by removing some layers in the network while reducing the width of the network is reducing the number of neurons in the layers. The depth percentage was measured as a percentage of the original number of layers, while the width factor, WF, was expressed as:

${\mathit{Neu}}_{\mathit{new}}=2^{-\mathit{WF}}{\mathit{Neu}}_{\mathit{old}},$

where Neu_old and Neu_new are the original and new number of neurons, respectively, in the layers of the network. The optimal architecture with an accuracy of 98.03% was found to have a 20% depth percentage and a width reduction factor of 4.

However, CNNs were outperformed by other deep learning algorithms. In particular, an LSTM algorithm was seen to perform better than a CNN algorithm in the classification of dynamic gestures acquired in a data glove (Simão et al., 2019). The LSTM classification accuracy was 96.5% for seen users and 89.1% for unseen users, while CNN achieved an accuracy of 81.9% and 54.7% for seen and unseen users, respectively. However, when a smaller percentage of the test users are used, CNN performs comparatively with LSTM and even outperforms it in some of these scenarios. This may have occurred because as the number of test users increases, the advantage of the memory properties of LSTM materializes.

Furthermore, a deep neural network (DNN) was utilized in classifying hand gestures acquired by a passive RFID data glove (Kantareddy et al., 2019). The DNN comprised of three fully connected hidden layers with 64, 128, and 32 neurons, respectively. Subsequently, its performance was compared with CNN, SVM, and random forest classifier (RFC). The CNN consisted of three 1D convolutional layers with 64, 128, and 64 filters, respectively, while the RFC had 10 trees, an average depth of 14.1 and an average number of 244.6 nodes. The DNN algorithm achieved an accuracy of 99%, while RFC, CNN, and SVM achieved an accuracy of 98%, 97%, and 86%, respectively. The CNN was outperformed by DNN because the DNN algorithm converged the global information, while the CNN algorithm only extracted the local information.

In addition, a deep learning algorithm was employed in the prediction of hand gestures. This involves predicting the next gesture to be performed by the user within a specific time frame. It helps to improve human-computer interactions by increasing the speed of gesture classification. Notably, RNN was used to predict hand gestures because of its ability to learn the temporal properties of the continuous data (Kanokoda et al., 2019). The performance of the RNN shown in Fig. 9 was compared to a time-delay neural network (TDNN) and a multiple linear regression (MLR) algorithm. Although TDNNs are proven algorithms in gesture prediction, they are limited to learning short-range dependencies and can only operate within fixed-size temporal windows (Sak et al., 2014). The results showed that the deep learning algorithm, RNN, outperformed both TDNN and MLR with a classification accuracy of 90.8% and 74.0% in predicting the next 100 ms and 300 ms of gestures, respectively.