Enable GPU Training

Training this model should be very quick, but it will be even faster if we use a GPU runtime. To enable this option, go to Colab’s Runtime menu and choose “Change runtime type,” as illustrated in Figure 12-2.

This opens the “Notebook settings” dialog box shown in Figure 12-3.

From the “Hardware accelerator” drop-down list, select GPU, as depicted in Figure 12-4, and then click SAVE.

You’re now ready to run the notebook.

Figure 12-2. The “Change runtime type” option in Colab

Figure 12-3. The “Notebook settings” dialog box

Figure 12-4. The “Hardware accelerator” drop-down list

Install dependencies

The first step is to install the required dependencies. In the “Install dependencies” section, run the cells to install the correct versions of TensorFlow and grab a copy of the training scripts.

Prepare the data

Next, in the “Prepare the data” section, run the cells to download the dataset and split it into training, validation, and test sets.

The first cell downloads and extracts the dataset into the training scripts’ directory. The dataset consists of four directories, one for each gesture (“wing,” “ring,” and “slope”) plus a “negative” directory for data that represents no distinct gesture. Each directory contains files that represent raw data resulting from the capture process for the gesture being performed:

data/
├── slope
│   ├── output_slope_dengyl.txt
│   ├── output_slope_hyw.txt
│   └── ...
├── ring
│   ├── output_ring_dengyl.txt
│   ├── output_ring_hyw.txt
│   └── ...
├── negative
│   ├── output_negative_1.txt
│   └── ...
└── wing
    ├── output_wing_dengyl.txt
    ├── output_wing_hyw.txt
    └── ...

There are 10 files for each gesture, which we’ll walk through later on. Each file contains a gesture being demonstrated by a named individual, with the last part of the filename corresponding to their user ID. For example, the file output_slope_dengyl.txt contains data for the “slope” gesture being demonstrated by a user whose ID is dengyl.

There are approximately 15 individual performances of a given gesture in each file, one accelerometer reading per row, with each performance being prefixed by the row -,-,-:

 -,-,-
-766.0,132.0,709.0
-751.0,249.0,659.0
-714.0,314.0,630.0
-709.0,244.0,623.0
-707.0,230.0,659.0

Each performance consists of a log of up to a few seconds’ worth of data, with 25 rows per second. The gesture itself occurs at some point within that window, with the device being held still for the remainder of the time.

Due to the way the measurements were captured, the files also contain some garbage characters. Our first training script, data_prepare.py, which is run in our second training cell, will clean up this dirty data:

# Prepare the data
!python data_prepare.py

This script is designed to read the raw data files from their folders, ignore any garbage characters, and write them in a sanitized form to another location within the training scripts’ directory (data/complete_data). Cleaning up messy data sources is a common task when training machine learning models given that it’s very common for errors, corruption, and other issues to creep into large datasets.

In addition to cleaning the data, the script generates some synthetic data. This is a term for data that is generated algorithmically, rather than being captured from the real world. In this case, the generate_negative_data() function in data_prepare.py creates synthetic data that is equivalent to movement of the accelerometer that doesn’t correspond to any particular gesture. This data is used to train our “unknown” category.

Because creating synthetic data is much faster than capturing real-world data, it’s useful to help augment our training process. However, real-world variation is unpredictable, so it’s not often possible to create an entire dataset from synthetic data. In our case, it’s helpful for making our “unknown” category more robust, but it wouldn’t be helpful for classifying the known gestures.

The next script to run in the second cell is data_split_person.py:

# Split the data by person
!python data_split_person.py

This script splits the data into training, validation, and test sets. Because our data is labeled with the person who created it, we’re able to use one set of people’s data for training, another set for validation, and a final set for test. The data is split as follows:

train_names = [
    "hyw", "shiyun", "tangsy", "dengyl", "jiangyh", "xunkai", "negative3",
    "negative4", "negative5", "negative6"
]
valid_names = ["lsj", "pengxl", "negative2", "negative7"]
test_names = ["liucx", "zhangxy", "negative1", "negative8"]

We use six people’s data for training, two for validation, and two for testing. In addition, we mix in our negative data, which isn’t associated with a particular user. Our total data is split between the three sets at a ratio of roughly 60%/20%/20%, which is pretty standard for machine learning.

In splitting by person, we’re trying to ensure that our model will be able to generalize to new data. Because the model will be validated and tested on data from individuals who were not included in the training dataset, the model will need to be robust against individual variations in how each person performs each gesture.

It’s also possible to split the data randomly, instead of by person. In this case, the training, validation, and testing datasets would each contain some samples of each gesture from every single individual. The resulting model will have been trained on data from every single person rather than just six, so it will have had more exposure to people’s varying gesturing styles.

However, because the validation and training sets also contain data from every individual, we’d have no way of testing whether the model is able to generalize to new gesturing styles that it has not seen before. A model developed in this way might report higher accuracy during validation and testing, but it would not be guaranteed to work as well with new data.

Make sure you’ve run both cells in the “Prepare the data” section before continuing.

Load TensorBoard

After the data has been prepared, we can run the next cell to load TensorBoard, which will help us monitor the training process:

# Load TensorBoard
%load_ext tensorboard
%tensorboard --logdir logs/scalars

Training logs will be written to the logs/scalars subdirectory of the training scripts’ directory, so we pass this in to TensorBoard.

Begin training

After TensorBoard has loaded, it’s time to begin training. Run the following cell:

!python train.py --model CNN --person true

The script train.py sets up the model architecture, loads the data using data_load.py, and begins the training process.

As the data is loaded, load_data.py also performs data augmentation using code defined in data_augmentation.py. The function augment_data() takes data representing a gesture and creates a number of new versions of it, each modified slightly from the original. The modifications include shifting and warping the datapoints in time, adding random noise, and increasing the amount of acceleration. This augmented data is used alongside the original data to train the model, helping make the most of our small dataset.

As training ramps up, you’ll see some output appearing below the cell you just ran. There’s a lot there, so let’s pick out the most noteworthy parts. First, Keras generates a nice table that shows the architecture of our model:

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 128, 3, 8)         104
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 42, 1, 8)          0
_________________________________________________________________
dropout (Dropout)            (None, 42, 1, 8)          0
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 42, 1, 16)         528
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 1, 16)         0
_________________________________________________________________
dropout_1 (Dropout)          (None, 14, 1, 16)         0
_________________________________________________________________
flatten (Flatten)            (None, 224)               0
_________________________________________________________________
dense (Dense)                (None, 16)                3600
_________________________________________________________________
dropout_2 (Dropout)          (None, 16)                0
_________________________________________________________________
dense_1 (Dense)              (None, 4)                 68
=================================================================

It tells us all the layers that are used, along with their shapes and their numbers of parameters—which is another term for weights and biases. You can see that our model uses Conv2D layers, as it’s a convolutional model. Not shown in this table is the fact that our model’s input shape is (None, 128, 3). We’ll look more closely at the model’s architecture later.

The output will also show us an estimate of the model’s size:

Model size: 16.796875 KB

This represents the amount of memory that will be taken up by the model’s trainable parameters. It doesn’t include the extra space required to store the model’s execution graph, so our actual model file will be slightly larger, but it gives us an idea of the correct order of magnitude. This will definitely qualify as a tiny model!

You’ll eventually see the training process itself begin:

1000/1000 [==============================] - 12s 12ms/step - loss: 7.6510 - accuracy: 0.5207 - val_loss: 4.5836 - val_accuracy: 0.7206

At this point, you can take a look at TensorBoard to see the training process moving along.

Evaluate the results

When training is complete, we can look at the cell’s output for some useful information. First, we can see that the validation accuracy in our final epoch looks very promising at 0.9743, and the loss is nice and low, too:

Epoch 50/50
1000/1000 [==============================] - 7s 7ms/step - loss: 0.0568 -

accuracy: 0.9835 - val_loss: 0.1185 - val_accuracy: 0.9743

This is great, especially as we’re using a per-person data split, meaning our validation data is from a completely different set of individuals. However, we can’t just rely on our validation accuracy to evaluate our model. Because the model’s hyperparameters and architecture were hand-tuned on the validation dataset, we might have overfit it.

To get a better understanding of our model’s final performance, we can evaluate it against our test dataset by calling Keras’s model.evaluate() function. The next line of output shows the results of this:

6/6 [==============================] - 0s 6ms/step - loss: 0.2888 - accuracy: 0.9323

Although not as amazing as the validation numbers, the model shows a good-enough accuracy of 0.9323, with a loss that is still low. The model will predict the correct class 93% of the time, which should be fine for our purposes.

The next few lines show the confusion matrix for the results, calculated by the tf.math.confusion_matrix() function:

tf.Tensor(
[[ 75   3   0   4]
 [  0  69   0  15]
 [  0   0  85   3]
 [  0   0   1 129]], shape=(4, 4), dtype=int32)

A confusion matrix is a helpful tool for evaluating the performance of classification models. It shows how well the predicted class of each input in the test dataset agrees with its actual value.

Each column of the confusion matrix corresponds to a predicted label, in order (“wing,” “ring,” “slope,” then “unknown”). Each row, from the top down, corresponds to the actual label. From our confusion matrix, we can see that the vast majority of predictions agree with the actual labels. We can also see the specific places where confusion is occurring: most significantly, a fair number of inputs were misclassified as “unknown,” especially those belonging to the “ring” category.

The confusion matrix gives us an idea of where our model’s weak points are. In this case, it informs us that it might be beneficial to obtain more training data for the “ring” gesture in order to help the model better learn the differences between “ring” and “unknown.”

The final thing that train.py does is convert the model to TensorFlow Lite format, in both floating-point and quantized variations. The following output reveals the sizes of each variant:

Basic model is 19544 bytes
Quantized model is 8824 bytes
Difference is 10720 bytes

Our 20 KB model shrinks down to 8.8 KB after quantization. This is a very tiny model, and a great result.

Create a C array

The next cell, in the “Create a C source file” section, transforms this into a C source file. Run this cell to see the output:

# Install xxd if it is not available
!apt-get -qq install xxd
# Save the file as a C source file
!xxd -i model_quantized.tflite > /content/model_quantized.cc
# Print the source file
!cat /content/model_quantized.cc

We can copy and paste the contents of this file into our project so that we can use the newly trained model in our application. Later, you’ll learn how to collect new data and teach the application to understand new gestures. For now, let’s keep moving.

SparkFun Edge

The fastest way to get started is by modifying one of the examples in the SparkFun Edge Board Support Package (BSP). First, follow SparkFun’s “Using SparkFun Edge Board with Ambiq Apollo3 SDK” guide to set up the Ambiq SDK and SparkFun Edge BSP.

After you’ve downloaded the SDK and BSP, you’ll need to tweak the example code so it does what we want.

First, open the file AmbiqSuite-Rel2.2.0/boards/SparkFun_Edge_BSP/examples/example1_edge_test/src/tf_adc/tf_adc.c in your text editor of choice. Find the call to am_hal_adc_samples_read(), on line 61 of the file:

if (AM_HAL_STATUS_SUCCESS != am_hal_adc_samples_read(g_ADCHandle,
                                                     NULL,
                                                     &ui32NumSamples,
                                                     &Sample))

Change its second parameter to true so that the entire function call looks like this:

if (AM_HAL_STATUS_SUCCESS != am_hal_adc_samples_read(g_ADCHandle,
                                                     true,
                                                     &ui32NumSamples,
                                                     &Sample))

Next, you’ll need to modify the file AmbiqSuite-Rel2.2.0/boards/SparkFun_Edge_BSP/examples/example1_edge_test/src/main.c. Find the while loop on line 51:

/*
* Read samples in polling mode (no int)
*/
while(1)
{
    // Use Button 14 to break the loop and shut down
    uint32_t pin14Val = 1;
    am_hal_gpio_state_read( AM_BSP_GPIO_14, AM_HAL_GPIO_INPUT_READ, &pin14Val);

Change the code to add the following extra line:

/*
* Read samples in polling mode (no int)
*/
while(1)
{
    am_util_stdio_printf("-,-,-
");
    // Use Button 14 to break the loop and shut down
    uint32_t pin14Val = 1;
    am_hal_gpio_state_read( AM_BSP_GPIO_14, AM_HAL_GPIO_INPUT_READ, &pin14Val);

Now find this line a little further along in the while loop:

am_util_stdio_printf("Acc [mg] %04.2f x, %04.2f y, %04.2f z,
                     Temp [deg C] %04.2f, MIC0 [counts / 2^14] %drn",
        acceleration_mg[0], acceleration_mg[1], acceleration_mg[2],
        temperature_degC, (audioSample) );

Delete the original line and replace it with the following:

am_util_stdio_printf("%04.2f,%04.2f,%04.2f
", acceleration_mg[0],
                     acceleration_mg[1], acceleration_mg[2]);

The program will now output data in the format expected by the training scripts.

Next, follow the instructions in SparkFun’s guide to build the example1_edge_test example application and flash it to the device.

Logging data

After you’ve built and flashed the example code, follow these instructions to capture some data.

First, open a new terminal window. Then run the following command to begin logging all of the terminal’s output to a file named output.txt:

script output.txt

Next, in the same window, use screen to connect to the device:

screen ${DEVICENAME} 115200

Measurements from the accelerometer will be shown on the screen and saved to output.txt in the same comma-delimited format expected by the training scripts.

You should aim to capture multiple performances of the same gesture in a single file. To start capturing a single performance of a gesture, press the button marked RST. The characters -,-,- will be written to the serial port; this output is used by the training scripts to identify the start of a gesture performance. After you’ve performed the gesture, press the button marked 14 to stop logging data.

When you’ve logged the same gesture a number of times, exit screen by pressing Ctrl- A, immediately followed by the K key, and then the Y key. After you’ve exited screen, enter the following command to stop logging data to output.txt:

exit

You now have a file, output.txt, which contains data for one person performing a single gesture. To train an entirely new model, you should aim to collect a similar amount of data as in the original dataset, which contains around 15 performances of each gesture by 10 people.

If you don’t care about your model working for people other than yourself, you can probably get away with capturing only your own performances. That said, the more variation in performances you can collect, the better.

For compatibility with the training scripts, you should rename your captured data files in the following format:

output_<gesture_name>_<person_name>.txt

For example, data for a hypothetical “triangle” gesture made by “Daniel” would have the following name:

output_triangle_Daniel.txt

The training scripts will expect the data to be organized in directories for each gesture name; for example:

data/
├── triangle
│   ├── output_triangle_Daniel.txt
│   └── ...
├── square
│   ├── output_square_Daniel.txt
│   └── ...
└── star
    ├── output_star_Daniel.txt
    └── ...

You’ll also need to provide data for the “unknown” category, in a directory named negative. In this case, you can just reuse the data files from the original dataset.

Note that because the model architecture is designed to output probabilities for four classes (three gestures plus “unknown”), you should provide three gestures of your own. If you want to train on more or fewer gestures, you’ll need to change the training scripts and adjust the model architecture.