Dataset

The two devices that are used for data collection are the NeuroSky MindWave Mobile device and a Raspberry Pi 3 board. The NeuroSky MindWave device can be used to safely record the EEG signals. The device consists of a headset, an ear clip, and a sensor (electrode) arm. The headset’s ground electrodes are available on the ear clip, whereas the EEG electrode is on the sensor arm that will rest on the forehead above the eye after putting on the headset. The device uses a single AAA battery, which can last for eight hours.

This device is connected to a Raspberry Pi 3 board via Bluetooth, as shown in Figure 9-1. It is a third-generation Raspberry Pi model that comes with a quad-core processor, 1GB of RAM, and a number of ports for connecting various devices. It also comes with wireless LAN and Bluetooth support, which can help to connect wireless devices like our MindWave Mobile. The software provided by the NeuroSky device vendor is installed on the Pi board to acquire the serial data from the device.

Interfacing the Raspberry Pi with MindWave Mobile via Bluetooth

There are two ways to connect the MindWave Mobile with the Raspberry Pi. The first one is to connect the MindWave Mobile with the Raspberry Pi desktop. Initially switch on the Raspberry Pi, boot into the Raspberry Pi operating system, and then switch on the MindWave Mobile Bluetooth. Then click the Bluetooth symbol in the Raspberry Pi OS, which will show the devices that are ready to pair with the Raspberry Pi. In the list, the MindWave Mobile can be selected, and the pairing password 0000 as prescribed by the vendor can be used. Now, the MindWave Mobile device is paired with the Pi, as shown in Figure 9-2.

The signals from the MindWave Mobile device can be extracted via this Bluetooth connection. Another way to connect the Raspberry with MindWave is by using Pypi 0.1.0. The steps are explained at https://github.com/cttoronto/python-MindWave-mobile. This link provides the data about alpha, beta, and gamma values of the brainwave signals. However, in this work, the dataset is developed from the EEG signals.

Data Collection Process

The participants are seated in a small, darkened room, which is also radio silent to prevent them from acoustic and visual disturbances. The terms and conditions are explained prior to the experiment, and they are instructed to stop the test if they have any discomfort. A manual score sheet was also provided to the participants to rate their emotions during each picture. There was a total of 15 participants, and 15 signals spread across three different emotions were recorded, thereby making a total of 15 × 3 = 45 EEG signals. The emotions were happy, afraid, and sad.

Initially, raw EEG signals were acquired from the user using a NeuroSky device. The raw EEG signal extracted from the brain cannot be directly used for further processing. As the subject is exposed to emotion stimulation based on the visual inputs for a specific duration, the resulting emotional reaction would be a time-varying one. It is essential therefore to identify the duration of peak activity of the brain and extract the features only for that duration so as to enhance the classification results. To achieve this, the recording is started exactly one minute after the start of experiment, which gives enough time to simulate the emotions of the participants using the image slides corresponding to the particular emotion. Also, to avoid dealing with large data, only 15 seconds of data with 512 samples per second are considered, thereby reducing the data size to just 15 × 512 =7680 samples, as illustrated in Figure 9-3. Figure 9-3 shows the signal for the entire duration of recording with the signal in the peak period of brain activity indicated in red, and Figure 9-4 shows this part separately.

Features Taken from the Brain Wave Signal

EEG signals are a rich source of brain function information. To get meaningful information from EEG signals, different attributes of the signals need to be extracted. A total of 9 different time domain attributes are extracted from the EEG signals, and these features are illustrated as follows.

The latency to amplitude ratio (LAR) is defined as the ratio of the maximum signal time to the maximum signal amplitude; see Equation 9-1.

(9-1)

Here, t_smax={t|s(t)=s_max} is the time where the maximum signal value occurs, and s_max=max{s(t)} is the maximum signal value.

The peak to peak signal value (PP) is defined as the difference between the maximum signal value and the minimum signal value and is shown in Equation 9-2.

s_pp = s_max − s_min (9-2)

Here, s_max and s_min are the signal maximum and minimum values, respectively.

The peak to peak time window (PPT) is defined as the difference between the maximum signal time and the minimum signal time and is shown in Equation 9-3.

t_pp = t_s max − t_s min (9-3)

Here, t_s max and t_s min are the times at which the maximum and minimum signal values occur.

The peak to peak slope (PPS) is defined as the ratio of peak to peak signal value (PP) to the peak to peak time value (PPT) and is shown in Equation 9-4.

(9-4)

Here, s_pp is the peak to peak signal value, and t_pp is the peak to peak time window.

The signal power (P) is defined as the signal that exists for infinite time for constant amplitude. The signal power is shown in Equation 9-5.

(9-5)

The mean value of signal (μ) is defined as the average of data samples between the end points of the selected area and displays the average value. The mean value of signal is given in Equation 9-6.

(9-6)

where N is total number of samples in signals.

Kurtosis (K) is the sharpness of the peak of a frequency-distribution curve and is given in Equation 9-7.

(9-7)

Here, m₄ and m₂ is the fourth moment and variance of signal.

Mobility (M) is defined as the ratio of first-order variance of signal to the variance of the signal and is given in Equation 9-8.

(9-8)

Complexity (C) is defined as the first derivative of mobility divided by mobility and is given in Equation 9-9.

(9-9)

Unstructured Data to Structured Dataset

Exploratory Data Analysis from the EEG Data

The modifications in the dataset include dropping the columns and changing the data using the exploratory data analysis section in Chapter 8.

Figure 9-5 shows the visualization of a histogram of the mean data in the emotion dataset.

Classifying the Emotion Using Learning Models

Let’s first use the k-NN algorithm to classify the emotions based on the data. k-NN is a simple supervised machine learning algorithm that categorizes the available data and assigns new data to a particular category based on a similarity score. The k-NN algorithm works by finding the distance between the test data and the training data. After finding the distance to each training data, the training data is sorted in ascending order of the distance values. In this ordered data, the first k data is selected, and the algorithm will assign the most frequent label occurring in this to the test data. The Euclidean distance is the most commonly used distance measure for the k-NN algorithm, and the distance between two data points, x_i and y_i, is given by the following expression:

In the confusion matrix, the row headers can be treated as inputs, and column headers can be treated as outputs. For instance, if we consider the first row, only one of the three EEG signals corresponding to the “happy” emotion is identified correctly, and the remaining two signals are wrongly classified as “sad” emotion. Similarly, in the second row, two signals corresponding to the “fear” emotion are classified correctly, and in the third row, one signal corresponding to the “sad” emotion is identified correctly. To understand better, the diagonal elements in the confusion matrix represent the data that is classified correctly, and the remaining elements indicate misclassification. In total, four out of the nine test signals are classified correctly. giving the system an accuracy of 44.44 percent.

Exploratory Image Data Analysis

The isnull( ) function can be used to detect the presence of missing values along the rows and columns of the image. The sum( ) function can be used to count the number of missing values in the dataframe along rows and columns. If the result of the sum( ) function is not equal to zero, then the image consists of missing values, and they can be treated using the ffill( ) function, which replaces each missing value with the pixel above it. This method of forward filling or backward filling will not cause any visible changes in the image because pixel values are often closely placed in an image except at edges in the image. As shown from the previous code, the number of missing values is 0; i.e., there are no missing values in the image. Once the image is checked and treated for missing values, the dataframe can be converted back to a NumPy array using to_numpy( ) in Pandas. Since the pixel values are closely placed, there may be repetition of same pixel values at many regions in the image. Because of this property, identification of duplicate values is irrelevant in the case of the image data.

Using a USB webcam or the Pi camera in natural lighting may often result in poor-quality images. So, the next step after treating missing values is to plot the histogram of the image. The histogram plot will give an idea about the contrast of the image, as shown in Figure 9-11. This is illustrated in the following code:

plt.hist(gray.ravel( ),bins=256)

plt.xlabel('bins')

plt.ylabel('No of pixels')

plt.show( )

The pixel values in a grayscale image range from 0 (representing black) to 255 (representing white). The hist( ) function in the previous code plots a bar chart of the count of each pixel value in this range. This plot gives insight about the contrast of the image that we are dealing with. Figure 9-7 shows the histogram of our grayscale image. It can be seen that the majority of the pixels are in the range (120,160). If the spread of pixels is concentrated in the lower bins, then we have a low-contrast image, and vice versa. So, depending on this plot, a decision can be made as to whether the image needs contrast adjustment.

The other cause for the poor quality of images may be the presence of noise induced by various factors. These noises can be visually perceived, while observing the captured images, in the form of grains. In such cases, these noises have to be removed before going for further processing. There are many different kinds of noises such as Gaussian noise, salt and pepper noise, etc., and there are many different types of filters that can be used to remove those noises that are beyond the scope of this book. Let’s just look at one particular filter used often in image processing called the averaging filter . It is a low-pass filter that can be used to remove high-frequency content from a digital image. This filtering works by passing a kernel of particular size, say 3×3, across the dimensions of the image, taking the average of all the pixels under the kernel area and replacing the central element with this average. The overall effect is to create a blurring effect. The following code illustrates the implementation of averaging filter to our image. Figure 9-12 shows the image obtained after filtering.

blur=cv2.blur(gray,(3,3))

plt.imshow(blur)

plt.axis('off')

plt.show( )

Preparing the Image Data for Model

Once the preprocessing steps are completed, the next step is to analyze or prepare the image for a learning model. This can be done in two ways. The first way is to extract features that represent useful information and use them for modeling. The features extracted may be another transformed image, or they may be attributes extracted from the original image. There are numerous features that can be extracted from an image, and the selection of a particular feature depends on the nature of our application. A discussion of these numerous features is beyond the scope of this book. Instead, we will discuss one particular feature: edge detection.

Edges represent the high-frequency content in an image. Canny edge detection is an algorithm that uses a multistage approach to detect a wide range of edges in images. It can be implemented in Python by using the Canny( ) function in OpenCV, as illustrated in the following code. Figure 9-13 shows the image after the edge detection process.

edge_img=cv2.Canny(gray,100,200)

plt.imshow(edge_img,cmap='gray')

plt.axis('off')

plt.show( )

The second way is to directly feed the image to a deep learning model. Deep learning is a popular machine learning approach that is being increasingly used for analyzing and learning from images. This approach can directly learn the useful information from the image and does not require any feature extraction. The image may be resized to a different shape and then fed to the learning model, or the image array may be converted to a one-dimensional vector and then fed to the model.

Object Detection Using a Deep Neural Network

Figure 9-14 shows the architecture of a single-shot multibox detector.

In the architecture, the dimension of the input image is considered as 300×300×3. The VGG-16 architecture is used as a base network, and the fully connected networks are discarded. The VGG-16 architecture is popular and has a strong classification ability with the transfer learning technique. Here, a part of the convolutional layers of the VGG-16 architecture is used in the earlier stages. A detailed explanation of SSD is available at https://towardsdatascience.com/understanding-ssd-multibox-real-time-object-detection-in-deep-learning-495ef744fab.

The multibox architecture is a technique for identifying the bounding box coordinates and is based on two loss functions such as confidence loss and location loss. Confidence loss uses a categorical entropy for measuring the confidence level of identifying the objects for the bounding box. Location loss measures the distance of the bounding box, which is away from the object in the image. For measuring the distance, the L2 norm is used. The multibox loss can be measured with the help of the following equation:

Multi-box loss=confidence Loss+α* Location Loss

This gives information about how far the bounding box landed from the predicted objects. The following code implements the SSD configure file with the DNN weights for detecting the objects in COCO names. The SSD configure file (i.e., ssd_mobilenet_v3_large_coco_2020_01_14.pbtxt) with the DNN weights (i.e., frozen_inference_graph.pb) for detecting the objects in COCO names can be downloaded from https://github.com/AlekhyaBhupati/Object_Detection_Using_openCV.

COCO names are called common objects in this context, and the dataset for the COCO names is available at the official website: https://cocodataset.org/#home. COCO has segmented common objects such as chair, car, animals, humans, etc., and these segmented images can be used to train the deep neural network. See Figure 9-15 and Figure 9-16.

Here’s the code:

import cv2

thres = 0.5# Threshold to detect object

cap = cv2.VideoCapture(0)

cap.set(3,1280)

cap.set(4,720)

cap.set(10,70)

classNames= []

classFile = 'coco.names'

with open(classFile,'rt') as f:

     classNames = f.read().rstrip(' ').split(' ')

configPath = 'ssd_mobilenet_v3_large_coco_2020_01_14.pbtxt'

weightsPath = 'frozen_inference_graph.pb'

net = cv2.dnn_DetectionModel(weightsPath,configPath)

net.setInputSize(320,320)

net.setInputScale(1.0/ 127.5)

net.setInputMean((127.5, 127.5, 127.5))

net.setInputSwapRB(True)

print('1st done')

while True:

     success, img = cap.read()

     classIds, confs, bbox = net.detect(img, confThreshold=thres)

     print(classIds, bbox)

      if len(classIds) != 0:

         for classId, confidence,box in zip(classIds.flatten(),confs.flatten(),bbox):

            cv2.rectangle(img,box,color=(0,255,0),thickness=2)

            cv2.putText(img,classNames[classId-1].upper(),(box[0]+10,box[1]+30),

                        cv2.FONT_HERSHEY_COMPLEX,1,(0,255,0),2)

            cv2.putText(img,str(round(confidence*100,2)),(box[0]+200,box[1]+30),

                         cv2.FONT_HERSHEY_COMPLEX,1,(0,255,0),2)

     cv2.imshow("Output",img)

     # Hit 'q' on the keyboard to quit!

When the code is executed, the frames in the video from the webcam are captured using the OpenCV capture functions. Then, each and every frame is inserted into the already trained SSD-DNN model for identifying the objects. The SSD-DNN model classifies the objects based on the COCO names and creates a bounding box on the detected images with a COCO name label and accuracy. The video file of Figure 9-15 was fed as the input to the previous program. The figure has the objects such as a chair, a book, and a mouse. From Figure 9-16, it can clearly be concluded that the SSD-based DNN model identifies the three objects with an accuracy of 72.53 percent for the chair, 67.41 percent for the book, and 81.52 percent for the mouse.

Raspberry Pi as a Localized Cloud for Industry 4.0

To implement Industry 4.0, a sophisticated computer is required to connect the devices, collect the data, and process the data. The collected data can be stored in a cloud service for further processing. However, these days, subscriptions of cloud services are costlier and suitable for highly profitable companies. Small-scale companies will want to implement a localized cloud for real-time processing. Further, a localized cloud approach can provide data security because it’s on-site and attackers are not able to invade via remote access.

As discussed in Chapter 3, the Raspberry Pi can act as a localized cloud that can connect sensors, IoT devices, other nearby computers, and mobile phones, as shown in Figure 9-18. Sophisticated computers also can act as a localized cloud, but they occupy a large space. Also, it is difficult to implement the computers in remote areas. The Raspberry Pi has the advantage of occupying less space and can be implemented in remote areas. Based on this, the Raspberry Pi is used as a localized cloud for the Industry 4.0 framework, as shown in Figure 9-19.

There are three modules available in the Industry 4.0 framework with the Raspberry Pi. The modules are collecting the data from the sensors, collecting the information using cameras, and connecting the Raspberry Pi with other computers.

Collecting Data from Sensors

Preparing the Industry Data in the Raspberry Pi

It can be seen from the previous illustration that there are two outliers corresponding to the row indices of 47 and 57. Rather than removing the outliers that correspond to the data points with a Z-score greater than 3, we retain all the data points with a Z-score less than 3.

Exploratory Data Analysis for the Real-Time Sensor Data

Visualizing the Real-Time Sensor Data

In Figure 9-20, the first two plots show the distribution of temperature and humidity where every sample of the data is plotted along the time axis, which is indicated in hours. We can see that the temperature and humidity are inversely proportional as expected. But the distribution over time is better expressed by taking the average of the samples every four hours and plotting the data in a bar chart, as shown in the third and fourth figures. The fifth and sixth figures show pie charts that focus on the distribution of temperature and humidity rather than their variation over time. Since the sensor data is recorded for only 28 hours, there will not be large variations in the data, and hence only four bins are used to plot the distribution. From these two figures, we can see that the temperature is mostly in the range of 15 to 20 during those 28 hours, and the humidity is mostly in the range of 19 to 25, respectively.

Report Generation by Reading Bar Codes Using Vision Cameras

Today many industries have documented their products with the help of barcodes and QR codes. Information about the product can be printed on the product for easy identification and documentation. Dedicative bar/QR code scanners are available on the market, but it requires human effort to scan the bar/QR code on the products. This may decrease productivity on the assembly line. Nowadays, vision systems are employed to automatically scan the bar/QR code on the products. This will improve productivity by eliminating the human effort and by reducing the time on the assembly line. Hence, a camera can be interfaced with the Raspberry Pi to scan the bar/QR code of the products on the assembly line.

We already discussed how to enable cameras on the Raspberry Pi in case study 2 of this chapter (refer to case study 2 for the steps to interface a webcam with the Raspberry Pi). The following code [30] continuously collects the images of the product on the assembly line, identifies the bar/QR code in the image, decodes the information in the bar/QR code, and displays the decoded information on the image screen.

# import the required packages

from imutils.video import VideoStream

from pyzbar import pyzbar

import argparse

import datetime

import imutils

import time

import cv2

# construct the argument parser and parse the arguments

ap = argparse.ArgumentParser()

ap.add_argument("-o", "--output", type=str, default="barcodes.csv",

     help="path to output CSV file containing barcodes")

args = vars(ap.parse_args())

# initialize the video stream and allow the camera sensor to warm up

print("[INFO] starting video stream...")

vs = VideoStream(src=0).start()

#vs = VideoStream(usePiCamera=True).start()

time.sleep(2.0)

# open the output CSV file for writing and initialize the set of

# barcodes found thus far

csv = open(args["output"], "w")

found = set()

# loop over the frames from the video stream

while True:

     # grab the frame from the threaded video stream and resize it to

     # have a maximum width of 400 pixels

     frame = vs.read()

     frame = imutils.resize(frame, width=400)

     # find the barcodes in the frame and decode each of the barcodes

     barcodes = pyzbar.decode(frame)

     # loop over the detected barcodes

     for barcode in barcodes:

          # extract the bounding box location of the barcode and draw

          # the bounding box surrounding the barcode on the image

          (x, y, w, h) = barcode.rect

          cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 0, 255), 2)

          # the barcode data is a bytes object so if we want to draw it

          # on our output image we need to convert it to a string first

          barcodeData = barcode.data.decode("utf-8")

          barcodeType = barcode.type

          # draw the barcode data and barcode type on the image

          text = "{} ({})".format(barcodeData, barcodeType)

          cv2.putText(frame, text, (x, y - 10),

               cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)

          # if the barcode text is currently not in our CSV file, write

          # the timestamp + barcode to disk and update the set

          if barcodeData not in found:

               csv.write("{},{} ".format(datetime.datetime.now(),

                    barcodeData))

               csv.flush()

               found.add(barcodeData)

               # show the output frame

     cv2.imshow("Barcode Scanner", frame)

     key = cv2.waitKey(1) & 0xFF

     # if the `q` key was pressed, break from the loop

     if key == ord("q"):

          break

# close the output CSV file do a bit of cleanup

print("[INFO] cleaning up...")

csv.close()

cv2.destroyAllWindows()

vs.stop()

The previous code acquires an image using a webcam and captures each and every frame using a while loop. Further, the frames are displayed continuously with the help of an infinite while loop. The 'q' key is used to break the infinite while loop. Then, the image acquisition can be released with the help of cap.release. In the program, each acquired frame is fed to the pyzbar module to identify the bar/QR code in the image and also to decode the data in the bar/QR code [30]. The decoded information is displayed in the corresponding frame. Figure 9-21 shows the output of the program.

Transmitting Files or Data from the Raspberry Pi to the Computer

In some scenarios, the data in the Raspberry Pi needs to be shared with nearby computers. Also, if the Raspberry Pi is somewhere else, it needs to be accessed via remote access. Many ways are available to transfer the data from the Raspberry Pi to other computers. One of the easiest and more efficient ways is to use the VNC viewer for sharing data and for remote access. VNC is the graphical desktop sharing application that allows you to control one system (i.e., the Raspberry Pi) from another system via remote access. This section discusses the installation procedure and usage of the VNC viewer for sharing files and controlling the Raspberry Pi from a remote desktop computer using VNC.

To install the VNC in Pi, the following code is used in the command window in the Raspberry Pi, as shown in Figure 9-22:

sudo apt update

sudo apt install realvnc-vnc-server realvnc-vnc-viewer

Meanwhile, VNC viewers need to be installed on a remote desktop computer. If the remote desktop computer has a different operating system (OS), VNC is compatible with all the OSs. After installing the VNC on the Pi, we have to enable the VNC server in the Raspberry Pi. The VNC server can be enabled graphically in the Raspberry Pi by following these steps:

1. 1.

   Go to the Raspberry Pi graphical desktop, and select Menu ➤ Preferences ➤ Raspberry Pi Configuration. The Raspberry Pi Configuration window will open, as shown in Figure 9-23.

    

These procedures are for creating a private connection between a remote desktop with the Raspberry Pi. To create a private connection, both the remote desktop and the Raspberry Pi are connected in the same network. This will create a connection only within the campus of the company. If the user wants to upload the data to the cloud, then the user needs to sign in to the VNC viewer for connecting the Pi with the remote desktop, which can be anywhere in the world.

By opening the VNC viewer in another remote desktop, as shown in Figure 9-24, the IP address of the Raspberry Pi is entered at the space provided, and the VNC server establishes the connection between the computer and the Raspberry Pi. The login window will open, as shown in Figure 9-25, and ask for the username and password.

Typically, the username and password for the Raspberry Pi is pi. Enter pi for the username and password, and the Raspberry Pi desktop will appear on the remote desktop computer, as shown in Figure 9-26.

Now, the Raspberry Pi desktop can access other computers remotely. Also, the files and data in the Raspberry Pi can be shared by using the file sharing option in the VNC viewer, as shown in Figure 9-27.