11.1 Introduction

Deep learning is a sub-field of artificial intelligence which endeavors to learn significant layerwise abstracted patterns and information by using deep network models. It is an arising approach and has been broadly applied in conventional decision reasoning and machine learning areas, for example, semantic parsing [9], natural language processing [39], transfer learning [13, 46], computer vision [33], and some more. Three most significant explanations behind the booming of deep learning today: the significantly expanding and high processor abilities (e.g., GPU units), the low expense of computing hardware, and the extensive advancement of machine learning algorithms [17]. Recently, diverse deep learning strategies have been broadly implemented and examined in many application areas such as self-driving cars [25], news aggregation and fraud news detection [57], natural language processing [31], virtual assistants entertainment [30], visual recognition [50], fraud detection [47], healthcare [21], and many more. Deep learning is widely being accepted for computer vision tasks as it has shown near-human or even better capabilities to perform numerous tasks, such as object detection and sequence learning. As opposed to deep learning, traditional machine learning approaches for classification tasks require hand-crafted discriminating features which essentially contribute to design a system, for example, linear discriminant analysis (LDA), or support vector machine (SVM). The hand-crafted feature extraction techniques require detailed information of the problem domain for discovering exceptionally expressive features in complex tasks, for example, computer vision and natural language processing. Additionally, the large dimension of extracted features also prompts the curse of dimensionality. Deep learning is the best approach to move toward these complex issues. The different layers in a deep learning model reveals the complex features by progressively learning theoretical portrayals in the first few layers and more specific qualities in the accompanying layers [22]. Deep networks have been exposed to be effective for computer vision tasks such as visual recognition, object recognition [18], motion tracking [19], action recognition [36], human pose estimation [53], and semantic segmentation [37]. Since, they can separate suitable features while simultaneously performing classification or detection tasks. Through the ImageNet LargeScale Visual Recognition Challenge (ILSVRC) competitions [1], deep learning techniques have been generally embraced by various analysts and accomplished highest accuracy scores. Recently, deep learning strategies have found applications in more innovative and promising areas of research dealing with human perception. As vision is one of the main component in the human perception system, it has great potential to find applications based on deep learning techniques. At the point when the eyes get visual instigation, neural spikes are conveyed to the brain. The analysis of these neural spikes is turning into a fascinating research subject in the field of computer vision. Even though deep learning mechanism [35], for example, CNN, has accomplished a great improvement in the image classification tasks [26, 43], computer vision is as yet unapproachable with of the extent of the actual human vision. Alternatively, extricating discriminating component relying upon the train dataset, the visual stimuli evoked recognition in a human’s mind includes intellectual hypotheses as well as perceptive process [7], i.e., how the visual stimulus such as images and video invigorate the human’s brain with respect to color shading and shape, and which parts of mind visual cortex can offer reaction to them. Some intellectual neuroscience researches investigated that while recording the brain’s electrical activity when a person is seeing an image, there are some explicit pattern about visual evoked image categories produced in the brain [45]. The human cerebrum has the capacity to arrange visual improvements depending on the common and frequent patterns. This grouping and categorizations of these frequent patterns is fast and shows up in millisecond time scales [10]. Because of the splendid temporal resolutions of the electroencephalogram (EEG) signals, the visual object’s images can be successfully perceived utilizing explicit patterns generated from EEG signals that are captured when images are shown on screen before the subjects [56]. EEG signals have been used to investigate and analyze the complexity of EEG signal for object detection and for classification tasks using machine learning methods [6].

Recently, the accessibility of huge EEG datasets and advancement in machine learning techniques have both motivated the implementation of deep learning tools, particularly in analyzing EEG signals and in perceiving the brain’s functionality. At present, deep learning has likewise attracted lot of consideration in the field of EEG classification research. A few researchers have attempted to apply the deep learning strategy to EEG in order to accomplish good accuracy [15]. But deep learning strategies ar still challenging when applied to the EEG-based BCI framework because of different affecting factors, for example, the relationship among channels, artifacts, and noise while EEG recording and the high-dimensional EEG information. To address all these challenges mentioned above, the effective implementation of deep learning strategies for the characterization of EEG signals is a serious accomplishment.

This paper aims at providing a CNN- and LSTM-based framework for classification of visually evoked stimuli. Figure 11.1 describes the overall workflow of the proposed architecture. As a very initial step, we have explored the utilization of CNNs for multi-class classification of EEG signals recorded while a subject is viewing image from animal (i.e., cat, dog, panda, capuchin, and elephant) and object (i.e., mug, bike, airliner, broom, and phone) category. In the second step, the acquired EEG signals are then pre-processed using basic filtering process in order to remove artifacts, followed by a Short-Time Fourier Transform (STFT) implementation so as to get the spectrogram images of the EEG signals. Finally, these images are then used as input to the proposed CNN model for a binary classification task representing the two categories, i.e., animal and object. The paper also presents accuracy results of the implemented CNN network designed to classify stimuli evoked EEG recordings along with a comparative study of some past outcomes from CNN and combinational CNN+LSTM-based classifiers. The result shows that the proposed CNN+LSTM-based model provides better accuracy compared to some of the existing work. The following sections of the paper presents an extensive literature review, followed by the methodology section including dataset description, the proposed architecture and the implementation of the model. At last, the results are documented followed by the conclusion section.

11.2 Literature Review

There are various innovative researches based on feature extraction and machine learning techniques [14]. Priorly, the time-recurrence examination [23], nonlinear elements, unpredictability, synchronization, and additions of collected vitality [28] techniques were utilized as feature extraction strategies. Additionally, the machine learning classifier also incorporates naive Bayes, conventional neural network system, SVM, random forest classifiers [24]. Indeed, feature extraction strategies have been utilized effectively in EEG classification tasks. In past studies, different feature extraction techniques for retrieving EEG complex information have been proposed [5]. Existing strategies for the discovery of complex pattern uses hand-crafted procedures for extraction of prominent features from EEG signals, then the chosen features must be able to characterize EEG signals by utilizing a wide range of classifiers, for example, SVM, KNN, principal component analysis, independent component analysis, and LDA [51] using FFT-based dual tree complex wavelet transform [11].

Most of the literature on the traditional approaches extract features from EEG for different application domains. This results in a 1D vector representation of the extracted features and this representation may loose some of the discriminating and prominent features. Therefore, few researcher have used methods to represent an EEG signal using a 2D representation before applying a deep learning framework. In [16], STFT was used to transform time-frequency features of EEG data into 2D images. In some previous researches, feature extraction was not executed and the deep learning models were rather prepared with raw EEG signals [4, 29].

These images were then used as input neurons for two class motor imagery classifications by combining CNN and VAE (Variational Auto-Encoder) framework. Multiple motor imagery classification has also been implemented using two variant of CNN models, namely, monolithic (single CNN) and modular (four CNN) architecture by using Bayesian optimization technique. These two procedures uses features extracted by a variation of Discriminative Filter Bank Common Spatial Pattern (DFBCSP) [44].

Deep Learning techniques such as DBN (Dynamic Bayesian Network), RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), SAE (Stacked Autoencoder), and hybrid deep networks have made noteworthy leaps in seizure and emotion detection, sleep scoring, motor imagery, event related potential detection, and visual stimuli–related image classification problem [15] and pattern recognition tasks [20, 32]. Recently, EEG information used in conjugation with facial expressions has been applied in persistent emotion recognition [49].

In [55], a new 3D CNN architecture to predict different stages of EEG data for epilepsy classification from multichannel EEG signals was discussed. Each EEG channel is converted into 2D images and are then combined to form 3D images which serve as input to a 3D CNN structure for classification. According to [59], frequency-based signals have greater potential than temporal signals for CNN applications; therefore, both frequency and time domain features for EEG classification were tested using the same CNN classification architecture. The EEG signal is also utilized to screen the brain’s electrical activity to analyze diverse sleep disorders. Deep learning has been also applied for sleep disorder detection using single channel EEG [40] and speech recognition [42]. Decoding visual stimulus from EEG captured brain’s activities is an interdisciplinary investigation of neuroscience and computer vision. Novel EEG-driven automatic visual classification and regeneration techniques have likewise been proposed. Ran Manor and Amir B. Geva have presented a CNN model for single-trail EEG classification of visually evoked potential tasks [38]. Similar tasks were performed for object classification using EEG signals evoked by stimulus-related potentials but with different CNN architecture [48]. Nicholas Waytowich proposed compact CNN to extract features from raw EEG signals automatically which can be used to decode signals from a 12 class visual evoked potential dataset [54]. The combination of CNN and LSTM has also been implemented for visual evoked classification. An automated model for visual classification in which LSTM is used to extract prominent feature representation from EEG signal and further that is fed to ResNet that leads to the improvement of classification performance [58]. A deep neural network (DNN) with RNN and CNN is trained for the EEG classification task by using video as visual stimuli [52]. In [8], authors proposed a deep learning model that uses a hybrid architecture based on CNN and RNN to classify visual evoked EEG signals and achieved better accuracy than CNN only as LSTM works well with temporal data as it helps to find the prominent features, and then, CNN architecture uses those features for improved classification. Despite many examples of impressive progress of deep network for visual stimuli evoked EEG signal classification, there is still room for considerable improved performance accuracy with hybrid deep structure.

11.3 Methodology

11.3.4 Classification of EEG Spectrogram Images With Proposed Combinational CNN+LSTM Model

The proposed network consists of two different modules: a CNN module which uses the Xception model, pretrained on ImageNet dataset implemented by Keras and another independent module is LSTM module. Each one of these modules runs parallel to each other. The first module is a CNN-based architecture, i.e., Xception, which stands for “Extreme Inception”. This is an extension of the Inception architecture which replaces the standard Inception modules with depthwise separable convolutions. The Xception architecture has 36 convolutional layers comprising the feature extraction unit of the network. The two spatial dimensions of width and height and the channel dimension helps the convolution layer to learn filters in a 3D space. Thus, the task of simultaneously mapping cross-channel features and spatial features rests with a single convolution kernel [12].

The EEG signals contain the temporal, frequential, and spatial features. These features should be extracted from the EEG signals as it consist important information regarding EEG signals. The conventional CNN do not perform convolution across all channels. That means the number of connections are fewer and the model is lighter and is unable to extract depthwise features. Therefore, we have implemented Xception network which can extract spatial features as it uses depthwise separable convolution. Initially, the entire network takes a RGB spectrogram image whose shape is 299 × 299 × 3 and passed through the pretrained Xception model until it reaches the final convolution block which has the bottleneck features, which is of batch size 2,048. The second module, i.e., LSTM layer is used to extract the temporal relationship of the input spectrogram images generated from the input EEG signals.

On the LSTM module, the 299 × 299 × 3 image is converted to grayscale image of size 299 × 299 × 1 to be able to properly split it into chunks to feed it into the LSTM. Afterward, this 299 × 299 image is reshaped into (23, 3887), where 23 is the time-step and 3887 is the dimension of each time-step. These values were chosen because 23 × 3887 = 299 × 299. The reshaped image is then passed through two LSTM layers, each of which are of (batch size, 2,048) output. The Xception network yields the depthwise features using multiple feature extraction layers from EEG signals and the other independent LSTM model process the EEG signals to yield temporal features which when combined can help to recognize the overall pattern of EEG spectogram and can improve the classification results. Next, now that we have (batch size, 2,048) from both the CNN and LSTM modules, these two outputs are merged using elementwise multiplication. The output of this multiplication is then provided to the classification layer which consists of two nodes (two classes, i.e., object and animal) and a softmax activation. Figure 11.2 shows the architecture of proposed hybrid CNN+LSTM network, and Figure 11.3 depicts the overall architecture of original Xception network [12].

11.4 Result and Discussion

As discussed in the previous section, two different models, i.e., a modified CNN model and a hybrid CNN+LSTM model is implemented to classify the EEG-based spectrogam images. As described in dataset description section, two datasets, namely, MindBig and Perceive datasets, are used to train and test the proposed model. As clarified in the above sections, the raw EEG signals are converted into spectrogram images using STFT, which then serve as an input data for the proposed CNN- and CNN+LSTM-based techniques. Additionally, this method was processed in mini-batches of size 32 and Adam optimizer has been used for both datasets for 50 epochs. The overall accuracy for both models on both datasets are provided in Table 11.3. The proposed CNN model achieved 69.84% and 71.45% classification accuracy, and combinational CNN+LSTM model has performed better with classification accuracy of 74.57% and 76.05% on MindBig and Percieve datasets, respectively. In this current work, firstly, a CNN model was implemented but as the acquired accuracy was not very good, depthwise CNN architecture (Xception Network) with LSTM model was implemented to extract multilevel spatial and channelwise features as well as temporal and frequential features from LSTM layer which can help to improve classification accuracy. Figures 11.4a and 11.4b show the accuracy graph of training and testing of CNN model and Figures 11.5a and 11.5b present the proposed combinational CNN+LSTM model’s training and testing accuracy.

11.5 Conclusion

In this chapter, a visual classification system is proposed utilizing EEG signals, evoked from visual stimuli while seeing an image. A CNN-based model and combinational CNN+LSTM model has been proposed to classify the images based on the recorded EEG signals. The contribution of this work is in two essential viewpoints. The first is the utilization of the STFT strategy to transform the visually evoked 1D EEG signals into 2D spectrogram images for obtaining better results. Second, this paper provides the deep learning–based network model to accomplish the problem. This paper proposes a CNN-based model and a combinational CNN+LSTM model, which is used to classify the visual evoked EEG signal and recognizes the image an individual is seeing at the time of EEG recording procedure. The proposed CNN model was able to classify the spectrogram images as inputs extracted from the MindBigData to identify two different classes (animal and object) the subject was viewing while the EEG was captured. The CNN+LSTM-based model efficiency is evident from its accuracy in comparison to CNN model. A future extension to the work may include testing the model on other datasets acquired through different visual stimulus. This architecture can also be tested for multi classification using more image classes. Application wise, this technique can find varied implementations in the field of brain fingerprinting and also early abnormality detection in children by asking them to visualize any object that is shown to them on screen.