Introducing Our Model

As we mentioned earlier, the model we use in this chapter is trained to recognize the words “yes” and “no,” and is also capable of distinguishing between unknown words and silence or background noise.

The model was trained on a dataset called the Speech Commands dataset. This consists of 65,000 one-second-long utterances of 30 short words, crowdsourced online.

Although the dataset contains 30 different words, the model was trained to distinguish between only four categories: the words “yes” and “no,” “unknown” words (meaning the other 28 words in the dataset), and silence.

The model takes in one second’s worth of data at a time. It outputs four probability scores, one for each of these four classes, predicting how likely it is that the data represented one of them.

However, the model doesn’t take in raw audio sample data. Instead, it works with spectrograms, which are two-dimensional arrays that are made up of slices of frequency information, each taken from a different time window.

Figure 7-1 is a visual representation of a spectrogram generated from a one-second audio clip of someone saying “yes.” Figure 7-2 shows the same thing for the word “no.”

Figure 7-1. Spectrogram for “yes”

Figure 7-2. Spectrogram for “no”

By isolating the frequency information during preprocessing, we make the model’s life easier. During training, it doesn’t need to learn how to interpret raw audio data; instead, it gets to work with a higher-layer abstraction that distills the most useful information.

We’ll look at how the spectrogram is generated later in this chapter. For now, we just need to know that the model takes a spectrogram as input. Because a spectrogram is a two-dimensional array, we feed it into the model as a 2D tensor.

There’s a type of neural network architecture that is specifically designed to work well with multidimensional tensors in which information is contained in the relationships between groups of adjacent values. It’s called a convolutional neural network (CNN).

The most common example of this type of data is images, for which a group of adjacent pixels might represent a shape, pattern, or texture. During training, a CNN is able to identify these features and learn what they represent.

It can learn how simple image features (like lines or edges) fit together into more complex features (like an eye or an ear), and in turn how those features might be combined to form an input image, such as a photo of a human face. This means that a CNN can learn to distinguish between different classes of input image, such as between a photo of a person and a photo of a dog.

Although they’re often applied to images, which are 2D grids of pixels, CNNs can be used with any multidimensional vector input. It turns out they’re very well suited to working with spectrogram data.

In Chapter 8, we’ll look at how this model was trained. Until then, let’s get back to discussing the architecture of our application.

The Basic Flow

The test micro_speech_test.cc follows the same basic flow we’re familiar with from the “hello world” example: we load the model, set up the interpreter, and allocate tensors.

However, there’s a notable difference. In the “hello world” example, we used the AllOpsResolver to pull in all of the deep learning operations that might be necessary to run the model. This is a reliable approach, but it’s wasteful because a given model probably doesn’t use all of the dozens of available operations. When deployed to a device, these unnecessary operations will take up valuable memory, so it’s best if we include only those we need.

To do this, we first define the ops that our model will need, at the top of the test file:

namespace tflite {
namespace ops {
namespace micro {
TfLiteRegistration* Register_DEPTHWISE_CONV_2D();
TfLiteRegistration* Register_FULLY_CONNECTED();
TfLiteRegistration* Register_SOFTMAX();
}  // namespace micro
}  // namespace ops
}  // namespace tflite

Next, we set up logging and load our model, as normal:

// Set up logging.
tflite::MicroErrorReporter micro_error_reporter;
tflite::ErrorReporter* error_reporter = &micro_error_reporter;
// Map the model into a usable data structure. This doesn't involve any
// copying or parsing, it's a very lightweight operation.
const tflite::Model* model =
    ::tflite::GetModel(g_tiny_conv_micro_features_model_data);
if (model->version() != TFLITE_SCHEMA_VERSION) {
  error_reporter->Report(
      "Model provided is schema version %d not equal "
      "to supported version %d.
",
      model->version(), TFLITE_SCHEMA_VERSION);
}

After our model is loaded, we declare a MicroMutableOpResolver and use its method AddBuiltin() to add the ops we listed earlier:

tflite::MicroMutableOpResolver micro_mutable_op_resolver;
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
    tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
micro_mutable_op_resolver.AddBuiltin(
    tflite::BuiltinOperator_FULLY_CONNECTED,
    tflite::ops::micro::Register_FULLY_CONNECTED());
micro_mutable_op_resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,
                                      tflite::ops::micro::Register_SOFTMAX());

You’re probably wondering how we know which ops to include for a given model. One way is to try running the model using a MicroMutableOpResolver, but without calling AddBuiltin() at all. Inference will fail, and the accompanying error messages will inform us which ops are missing and need to be added.

After the MicroMutableOpResolver is set up, we just carry on as usual, setting up our interpreter and its working memory:

// Create an area of memory to use for input, output, and intermediate arrays.
const int tensor_arena_size = 10 * 1024;
uint8_t tensor_arena[tensor_arena_size];
// Build an interpreter to run the model with.
tflite::MicroInterpreter interpreter(model, micro_mutable_op_resolver, tensor_arena,
                                     tensor_arena_size, error_reporter);
interpreter.AllocateTensors();

In our “hello world” application we allocated only 2 * 1,024 bytes for the tensor_arena, given that the model was so small. Our speech model is a lot bigger, and it deals with more complex input and output, so it needs more space (10 1,024). This was determined by trial and error.

Next, we check the input tensor size. However, it’s a little different this time around:

// Get information about the memory area to use for the model's input.
TfLiteTensor* input = interpreter.input(0);
// Make sure the input has the properties we expect.
TF_LITE_MICRO_EXPECT_NE(nullptr, input);
TF_LITE_MICRO_EXPECT_EQ(4, input->dims->size);
TF_LITE_MICRO_EXPECT_EQ(1, input->dims->data[0]);
TF_LITE_MICRO_EXPECT_EQ(49, input->dims->data[1]);
TF_LITE_MICRO_EXPECT_EQ(40, input->dims->data[2]);
TF_LITE_MICRO_EXPECT_EQ(1, input->dims->data[3]);
TF_LITE_MICRO_EXPECT_EQ(kTfLiteUInt8, input->type);

Because we’re dealing with a spectrogram as our input, the input tensor has more dimensions—four, in total. The first dimension is just a wrapper containing a single element. The second and third represent the “rows” and “columns” of our spectrogram, which happens to have 49 rows and 40 columns. The fourth, innermost dimension of the input tensor, which has size 1, holds each individual “pixel” of the spectrogram. We’ll look more at the spectrogram’s structure later on.

Next, we grab a sample spectrogram for a “yes,” stored in the constant g_yes_micro_f2e59fea_nohash_1_data. The constant is defined in the file micro_features/yes_micro_features_data.cc, which was included by this test. The spectrogram exists as a 1D array, and we just iterate through it to copy it into the input tensor:

// Copy a spectrogram created from a .wav audio file of someone saying "Yes"
// into the memory area used for the input.
const uint8_t* yes_features_data = g_yes_micro_f2e59fea_nohash_1_data;
for (int i = 0; i < input->bytes; ++i) {
  input->data.uint8[i] = yes_features_data[i];
}

After the input has been assigned, we run inference and inspect the output tensor’s size and shape:

// Run the model on this input and make sure it succeeds.
TfLiteStatus invoke_status = interpreter.Invoke();
if (invoke_status != kTfLiteOk) {
  error_reporter->Report("Invoke failed
");
}
TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status);

// Get the output from the model, and make sure it's the expected size and
// type.
TfLiteTensor* output = interpreter.output(0);
TF_LITE_MICRO_EXPECT_EQ(2, output->dims->size);
TF_LITE_MICRO_EXPECT_EQ(1, output->dims->data[0]);
TF_LITE_MICRO_EXPECT_EQ(4, output->dims->data[1]);
TF_LITE_MICRO_EXPECT_EQ(kTfLiteUInt8, output->type);

Our output has two dimensions. The first is just a wrapper. The second has four elements. This is the structure that holds the probabilities that each of our four classes (silence, unknown, “yes,” and “no”) were matched.

The next chunk of code checks whether the probabilities were as expected. A given element of the output tensor always represents a certain class, so we know which index to check for each one. The order is defined during training:

// There are four possible classes in the output, each with a score.
const int kSilenceIndex = 0;
const int kUnknownIndex = 1;
const int kYesIndex = 2;
const int kNoIndex = 3;

// Make sure that the expected "Yes" score is higher than the other classes.
uint8_t silence_score = output->data.uint8[kSilenceIndex];
uint8_t unknown_score = output->data.uint8[kUnknownIndex];
uint8_t yes_score = output->data.uint8[kYesIndex];
uint8_t no_score = output->data.uint8[kNoIndex];
TF_LITE_MICRO_EXPECT_GT(yes_score, silence_score);
TF_LITE_MICRO_EXPECT_GT(yes_score, unknown_score);
TF_LITE_MICRO_EXPECT_GT(yes_score, no_score);

We passed in a “yes” spectrogram, so we expect that the variable yes_score contains a higher probability than silence_score, unknown_score, and no_score.

When we’re satisfied with “yes,” we do the same thing with a “no” spectrogram. First, we copy in an input and run inference:

// Now test with a different input, from a recording of "No".
const uint8_t* no_features_data = g_no_micro_f9643d42_nohash_4_data;
for (int i = 0; i < input->bytes; ++i) {
  input->data.uint8[i] = no_features_data[i];
}
// Run the model on this "No" input.
invoke_status = interpreter.Invoke();
if (invoke_status != kTfLiteOk) {
  error_reporter->Report("Invoke failed
");
}
TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, invoke_status);

After inference is done, we confirm that “no” achieved the highest score:

// Make sure that the expected "No" score is higher than the other classes.
silence_score = output->data.uint8[kSilenceIndex];
unknown_score = output->data.uint8[kUnknownIndex];
yes_score = output->data.uint8[kYesIndex];
no_score = output->data.uint8[kNoIndex];
TF_LITE_MICRO_EXPECT_GT(no_score, silence_score);
TF_LITE_MICRO_EXPECT_GT(no_score, unknown_score);
TF_LITE_MICRO_EXPECT_GT(no_score, yes_score);

And we’re done!

To run this test, issue the following command from the root of the TensorFlow repository:

make -f tensorflow/lite/micro/tools/make/Makefile 
  test_micro_speech_test

Next up, let’s look at the source of all our audio data: the audio provider.

The Feature Provider

The feature provider converts raw audio, obtained from the audio provider, into spectrograms that can be fed into our model. It is called during the main loop.

Its interface is defined in feature_provider.h, and looks like this:

class FeatureProvider {
 public:
  // Create the provider, and bind it to an area of memory. This memory should
  // remain accessible for the lifetime of the provider object, since subsequent
  // calls will fill it with feature data. The provider does no memory
  // management of this data.
  FeatureProvider(int feature_size, uint8_t* feature_data);
  ~FeatureProvider();

  // Fills the feature data with information from audio inputs, and returns how
  // many feature slices were updated.
  TfLiteStatus PopulateFeatureData(tflite::ErrorReporter* error_reporter,
                                   int32_t last_time_in_ms, int32_t time_in_ms,
                                   int* how_many_new_slices);

 private:
  int feature_size_;
  uint8_t* feature_data_;
  // Make sure we don't try to use cached information if this is the first call
  // into the provider.
  bool is_first_run_;
};

To see how it’s used, we can take a look at the tests in feature_provider_mock_test.cc.

For there to be audio data for the feature provider to work with, these tests use a special fake version of the audio provider, known as a mock, that is set up to provide audio data. It is defined in audio_provider_mock.cc.

The file feature_provider_mock_test.cc contains two tests. Here’s the first one:

TF_LITE_MICRO_TEST(TestFeatureProviderMockYes) {
  tflite::MicroErrorReporter micro_error_reporter;
  tflite::ErrorReporter* error_reporter = &micro_error_reporter;

  uint8_t feature_data[kFeatureElementCount];
  FeatureProvider feature_provider(kFeatureElementCount, feature_data);

  int how_many_new_slices = 0;
  TfLiteStatus populate_status = feature_provider.PopulateFeatureData(
      error_reporter, /* last_time_in_ms= */ 0, /* time_in_ms= */ 970,
      &how_many_new_slices);
  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, populate_status);
  TF_LITE_MICRO_EXPECT_EQ(kFeatureSliceCount, how_many_new_slices);

  for (int i = 0; i < kFeatureElementCount; ++i) {
    TF_LITE_MICRO_EXPECT_EQ(g_yes_micro_f2e59fea_nohash_1_data[i],
                            feature_data[i]);
  }
}

To create a FeatureProvider, we call its constructor, passing in feature_size and feature_data arguments:

FeatureProvider feature_provider(kFeatureElementCount, feature_data);

The first argument indicates how many total data elements should be in the spectrogram. The second argument is a pointer to an array that we want to be populated with the spectrogram data.

The number of elements in the spectrogram was decided when the model was trained and is defined as kFeatureElementCount in micro_features/micro_model_settings.h.

To obtain features for the past second of audio, feature_provider.PopulateFeatureData() is called:

TfLiteStatus populate_status = feature_provider.PopulateFeatureData(
      error_reporter, /* last_time_in_ms= */ 0, /* time_in_ms= */ 970,
      &how_many_new_slices);

We supply an ErrorReporter instance, an integer representing the last time this method was called (last_time_in_ms), the current time (time_in_ms), and a pointer to an integer that will be updated with how many new feature slices we receive (how_many_new_slices). A slice is just one row of the spectrogram, representing a chunk of time.

Because we always want the last second of audio, the feature provider will compare when it was last called (last_time_in_ms) with the current time (time_in_ms), create spectrogram data from the audio captured during that time, and then update the feature_data array to add any additional slices and drop any that are older than one second.

When PopulateFeatureData() runs, it will request audio from the mock audio provider. The mock will give it audio representing a “yes,” and the feature provider will process it and provide the result.

After calling PopulateFeatureData(), we check whether its result is what we expect. We compare the data it generated to a known spectrogram that is correct for the “yes” input given by the mock audio provider:

TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, populate_status);
TF_LITE_MICRO_EXPECT_EQ(kFeatureSliceCount, how_many_new_slices);
for (int i = 0; i < kFeatureElementCount; ++i) {
  TF_LITE_MICRO_EXPECT_EQ(g_yes_micro_f2e59fea_nohash_1_data[i],
                          feature_data[i]);
}

The mock audio provider can provide audio for a “yes” or a “no” depending on which start and end times are passed into it. The second test in feature_provider_mock_test.cc does exactly the same thing as the first, but for audio representing “no.”

To run the tests, use the following command:

make -f tensorflow/lite/micro/tools/make/Makefile 
  test_feature_provider_mock_test

How the feature provider converts audio to a spectrogram

The feature provider is implemented in feature_provider.cc. Let’s talk through how it works.

As we’ve discussed, its job is to populate an array that represents a spectrogram of one second of audio. It’s designed to be called in a loop, so to avoid unnecessary work, it will generate new features only for the time between now and when it was last called. If it were called less than a second ago, it would keep some of its previous output and generate only the missing parts.

In our code, each spectrogram is represented as a 2D array, with 40 columns and 49 rows, where each row represents a 30-millisecond (ms) sample of audio split into 43 frequency buckets.

To create each row, we run a 30-ms slice of audio input through a fast Fourier transform (FFT) algorithm. This technique analyzes the frequency distribution of audio in the sample and creates an array of 256 frequency buckets, each with a value from 0 to 255. These are averaged together into groups of six, leaving us with 43 buckets.

The code that does this is in the file micro_features/micro_features_generator.cc, and is called by the feature provider.

To build the entire 2D array, we combine the results of running the FFT on 49 consecutive 30-ms slices of audio, with each slice overlapping the last by 10 ms. Figure 7-4 shows how this happens.

You can see how the 30-ms sample window is moved forward by 20 ms each time until it has covered the full one-second sample. The resulting spectrogram is ready to pass into our model.

We can understand how this process happens in feature_provider.cc. First, it determines which slices it actually needs to generate based on the time PopulateFeatureData() was last called:

// Quantize the time into steps as long as each window stride, so we can
// figure out which audio data we need to fetch.
const int last_step = (last_time_in_ms / kFeatureSliceStrideMs);
const int current_step = (time_in_ms / kFeatureSliceStrideMs);

int slices_needed = current_step - last_step;

Figure 7-4. Diagram of audio samples being processed

If it hasn’t run before, or it ran more than one second ago, it will generate the maximum number of slices:

if (is_first_run_) {
  TfLiteStatus init_status = InitializeMicroFeatures(error_reporter);
  if (init_status != kTfLiteOk) {
    return init_status;
  }
  is_first_run_ = false;
  slices_needed = kFeatureSliceCount;
}
if (slices_needed > kFeatureSliceCount) {
  slices_needed = kFeatureSliceCount;
}
*how_many_new_slices = slices_needed;

The resulting number is written to how_many_new_slices.

Next, it calculates how many of any existing slices it should keep, and shifts data in the array around to make room for any new ones:

const int slices_to_keep = kFeatureSliceCount - slices_needed;
const int slices_to_drop = kFeatureSliceCount - slices_to_keep;
// If we can avoid recalculating some slices, just move the existing data
// up in the spectrogram, to perform something like this:
// last time = 80ms          current time = 120ms
// +-----------+             +-----------+
// | data@20ms |         --> | data@60ms |
// +-----------+       --    +-----------+
// | data@40ms |     --  --> | data@80ms |
// +-----------+   --  --    +-----------+
// | data@60ms | --  --      |  <empty>  |
// +-----------+   --        +-----------+
// | data@80ms | --          |  <empty>  |
// +-----------+             +-----------+
if (slices_to_keep > 0) {
  for (int dest_slice = 0; dest_slice < slices_to_keep; ++dest_slice) {
    uint8_t* dest_slice_data =
        feature_data_ + (dest_slice * kFeatureSliceSize);
    const int src_slice = dest_slice + slices_to_drop;
    const uint8_t* src_slice_data =
        feature_data_ + (src_slice * kFeatureSliceSize);
    for (int i = 0; i < kFeatureSliceSize; ++i) {
      dest_slice_data[i] = src_slice_data[i];
    }
  }
}

Note

If you’re a seasoned C++ author, you might wonder why we don’t use standard libraries to do things like copying data around. The reason is that we’re trying to avoid unnecessary dependencies, in an effort to keep our binary size small. Because embedded platforms have very little memory, a smaller application binary means that we have space for a larger and more accurate deep learning model.

After moving data around, it begins a loop that iterates once for each new slice that it needs. In this loop, it first requests audio for that slice from the audio provider using GetAudioSamples():

for (int new_slice = slices_to_keep; new_slice < kFeatureSliceCount;
     ++new_slice) {
  const int new_step = (current_step - kFeatureSliceCount + 1) + new_slice;
  const int32_t slice_start_ms = (new_step * kFeatureSliceStrideMs);
  int16_t* audio_samples = nullptr;
  int audio_samples_size = 0;
  GetAudioSamples(error_reporter, slice_start_ms, kFeatureSliceDurationMs,
                  &audio_samples_size, &audio_samples);
  if (audio_samples_size < kMaxAudioSampleSize) {
    error_reporter->Report("Audio data size %d too small, want %d",
                           audio_samples_size, kMaxAudioSampleSize);
    return kTfLiteError;
  }

To complete the loop iteration, it passes that data into GenerateMicroFeatures(), defined in micro_features/micro_features_generator.h. This is the function that performs the FFT and returns the audio frequency information.

It also passes a pointer, new_slice_data, which points at the memory location where the new data should be written:

  uint8_t* new_slice_data = feature_data_ + (new_slice * kFeatureSliceSize);
  size_t num_samples_read;
  TfLiteStatus generate_status = GenerateMicroFeatures(
      error_reporter, audio_samples, audio_samples_size, kFeatureSliceSize,
      new_slice_data, &num_samples_read);
  if (generate_status != kTfLiteOk) {
    return generate_status;
  }
}

After this process has happened for each slice, we have an entire second’s worth of up-to-date spectrogram.

Tip

The function that generates the FFT is GenerateMicroFeatures(). If you’re interested, you can read its definition in micro_features/micro_features_generator.cc.

If you’re building your own application that uses spectrograms, you can reuse this code as is. You’ll need to use the same code to pre-process data into spectrograms when training your model.

Once we have a spectrogram, we can run inference on it using the model. After this happens, we need to interpret the results. That task belongs to the class we explore next, RecognizeCommands.

The Command Recognizer

After our model outputs a set of probabilities that a known word was spoken in the last second of audio, it’s the job of the RecognizeCommands class to determine whether this indicates a successful detection.

It seems like this would be simple: if the probability in a given category is more than a certain threshold, the word was spoken. However, in the real world, things become a bit more complicated.

As we established earlier, we’re running multiple inferences per second, each on a one-second window of data. This means that we’ll run inference on any given word multiple times, in multiple windows.

In Figure 7-5, you can see a waveform of the word “noted” being spoken, surrounded by a box representing a one-second window being captured.

Figure 7-5. The word “noted” being captured in our window

Our model is trained to detect the word “no,” and it understands that the word “noted” is not the same thing. If we run inference on this one-second window, it will (hopefully) output a low probability for the word “no.” However, what if the window came slightly earlier in the audio stream, as in Figure 7-6?

Figure 7-6. Part of the word “noted” being captured in our window

In this case, the only part of the word “noted” that appears within the window is its first syllable. Because the first syllable of “noted” sounds like “no,” it’s likely that the model will interpret this as having a high probability of being a “no.”

This problem, along with others, means that we can’t rely on a single inference to tell us whether a word was spoken. This is where RecognizeCommands comes in!

The recognizer calculates the average score for each word over the past few inferences, and decides whether it’s high enough to count as a detection. To do this, we feed it each inference result as they roll in.

You can see its interface in recognize_commands.h, partially reproduced here:

class RecognizeCommands {
 public:
  explicit RecognizeCommands(tflite::ErrorReporter* error_reporter,
                             int32_t average_window_duration_ms = 1000,
                             uint8_t detection_threshold = 200,
                             int32_t suppression_ms = 1500,
                             int32_t minimum_count = 3);

  // Call this with the results of running a model on sample data.
  TfLiteStatus ProcessLatestResults(const TfLiteTensor* latest_results,
                                    const int32_t current_time_ms,
                                    const char** found_command, uint8_t* score,
                                    bool* is_new_command);

The class RecognizeCommands is defined, along with a constructor that defines default values for a few things:

• The length of the averaging window (average_window_duration_ms)

• The minimum average score that counts as a detection (detection_threshold)

• The amount of time we’ll wait after hearing a command before recognizing a second one (suppression_ms)

• The minimum number of inferences required in the window for a result to count (3)

The class has one method, ProcessLatestResults(). It accepts a pointer to a TfLiteTensor containing the model’s output (latest_results), and it must be called with the current time (current_time_ms).

In addition, it takes three pointers that it uses for output. First, it gives us the name of any word that was detected (found_command). It also provides the average score of the command (score) and whether the command is new or has been heard in previous inferences within a certain timespan (is_new_command).

Averaging the results of multiple inferences is a useful and common technique when dealing with time-series data. In the next few pages, we’ll walk through the code in recognize_commands.cc and learn a bit about how it works. You don’t need to understand every line, but it’s helpful to get some insight into what might be a helpful tool in your own projects.

First, we make sure the input tensor is the right shape and type:

TfLiteStatus RecognizeCommands::ProcessLatestResults(
    const TfLiteTensor* latest_results, const int32_t current_time_ms,
    const char** found_command, uint8_t* score, bool* is_new_command) {
  if ((latest_results->dims->size != 2) ||
      (latest_results->dims->data[0] != 1) ||
      (latest_results->dims->data[1] != kCategoryCount)) {
    error_reporter_->Report(
        "The results for recognition should contain %d elements, but there are "
        "%d in an %d-dimensional shape",
        kCategoryCount, latest_results->dims->data[1],
        latest_results->dims->size);
    return kTfLiteError;
  }

  if (latest_results->type != kTfLiteUInt8) {
    error_reporter_->Report(
        "The results for recognition should be uint8 elements, but are %d",
        latest_results->type);
    return kTfLiteError;
  }

Next, we check current_time_ms to verify that it is after the most recent result in our averaging window:

if ((!previous_results_.empty()) &&
    (current_time_ms < previous_results_.front().time_)) {
  error_reporter_->Report(
      "Results must be fed in increasing time order, but received a "
      "timestamp of %d that was earlier than the previous one of %d",
      current_time_ms, previous_results_.front().time_);
  return kTfLiteError;
}

After that, we add the latest result to a list of results we’ll be averaging:

// Add the latest results to the head of the queue.
previous_results_.push_back({current_time_ms, latest_results->data.uint8});
// Prune any earlier results that are too old for the averaging window.
const int64_t time_limit = current_time_ms - average_window_duration_ms_;
while ((!previous_results_.empty()) &&
       previous_results_.front().time_ < time_limit) {
  previous_results_.pop_front();

If there are fewer results in our averaging window than the minimum number (defined by minimum_count_, which is 3 by default), we can’t provide a valid average. In this case, we set the output pointers to indicate that found_command is the most recent top command, that the score is 0, and that the command is not a new one:

// If there are too few results, assume the result will be unreliable and
// bail.
const int64_t how_many_results = previous_results_.size();
const int64_t earliest_time = previous_results_.front().time_;
const int64_t samples_duration = current_time_ms - earliest_time;
if ((how_many_results < minimum_count_) ||
    (samples_duration < (average_window_duration_ms_ / 4))) {
  *found_command = previous_top_label_;
  *score = 0;
  *is_new_command = false;
  return kTfLiteOk;
}

Otherwise, we continue by averaging all of the scores in the window:

// Calculate the average score across all the results in the window.
int32_t average_scores[kCategoryCount];
for (int offset = 0; offset < previous_results_.size(); ++offset) {
  PreviousResultsQueue::Result previous_result =
      previous_results_.from_front(offset);
  const uint8_t* scores = previous_result.scores_;
  for (int i = 0; i < kCategoryCount; ++i) {
    if (offset == 0) {
      average_scores[i] = scores[i];
    } else {
      average_scores[i] += scores[i];
    }
  }
}
for (int i = 0; i < kCategoryCount; ++i) {
  average_scores[i] /= how_many_results;
}

We now have enough information to identify which category is our winner. Establishing this is a simple process:

// Find the current highest scoring category.
int current_top_index = 0;
int32_t current_top_score = 0;
for (int i = 0; i < kCategoryCount; ++i) {
  if (average_scores[i] > current_top_score) {
    current_top_score = average_scores[i];
    current_top_index = i;
  }
}
const char* current_top_label = kCategoryLabels[current_top_index];

The final piece of logic determines whether the result was a valid detection. To do this, it ensures that its score is above the detection threshold (200 by default), and that it didn’t happen too quickly after the last valid detection, which can be an indication of a faulty result:

// If we've recently had another label trigger, assume one that occurs too
// soon afterwards is a bad result.
int64_t time_since_last_top;
if ((previous_top_label_ == kCategoryLabels[0]) ||
    (previous_top_label_time_ == std::numeric_limits<int32_t>::min())) {
  time_since_last_top = std::numeric_limits<int32_t>::max();
} else {
  time_since_last_top = current_time_ms - previous_top_label_time_;
}
if ((current_top_score > detection_threshold_) &&
    ((current_top_label != previous_top_label_) ||
     (time_since_last_top > suppression_ms_))) {
  previous_top_label_ = current_top_label;
  previous_top_label_time_ = current_time_ms;
  *is_new_command = true;
} else {
  *is_new_command = false;
}
*found_command = current_top_label;
*score = current_top_score;

If the result was valid, is_new_command is set to true. This is what the caller can use to determine whether a word was genuinely detected.

The tests (in recognize_commands_test.cc) exercise various different combinations of inputs and results that are stored in the averaging window.

Let’s walk through one of the tests, RecognizeCommandsTestBasic, which demonstrates how RecognizeCommands is used. First, we just create an instance of the class:

TF_LITE_MICRO_TEST(RecognizeCommandsTestBasic) {
  tflite::MicroErrorReporter micro_error_reporter;
  tflite::ErrorReporter* error_reporter = &micro_error_reporter;

  RecognizeCommands recognize_commands(error_reporter);

Next, we create a tensor containing some fake inference results, which will be used by ProcessLatestResults() to decide whether a command was heard:

TfLiteTensor results = tflite::testing::CreateQuantizedTensor(
    {255, 0, 0, 0}, tflite::testing::IntArrayFromInitializer({2, 1, 4}),
    "input_tensor", 0.0f, 128.0f);

Then, we set up some variables that will be set with the output of ProcessLatestResults():

const char* found_command;
uint8_t score;
bool is_new_command;

Finally, we call ProcessLatestResults(), providing pointers to these variables along with the tensor containing the results. We assert that the function will return kTfLiteOk, indicating that the input was processed successfully:

TF_LITE_MICRO_EXPECT_EQ(
    kTfLiteOk, recognize_commands.ProcessLatestResults(
                   &results, 0, &found_command, &score, &is_new_command));

The other tests in the file perform some more exhaustive checks to make sure the function is performing correctly. You can read through them to learn more.

To run all of the tests, use the following command:

make -f tensorflow/lite/micro/tools/make/Makefile 
  test_recognize_commands_test

As soon as we’ve determined whether a command was detected, it’s time to share our results with the world (or at least our on-board LEDs). The command responder is what makes this happen.

The Command Responder

The final piece in our puzzle, the command responder, is what produces an output to let us know that a word was detected.

The command responder is designed to be overridden for each type of device. We explore the device-specific implementations later in this chapter.

For now, let’s look at its very simple reference implementation, which just logs detection results as text. You can find it in the file command_responder.cc:

void RespondToCommand(tflite::ErrorReporter* error_reporter,
                      int32_t current_time, const char* found_command,
                      uint8_t score, bool is_new_command) {
  if (is_new_command) {
    error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                           current_time);
  }
}

That’s it! The file implements just one function: RespondToCommand(). As parameters, it expects an error_reporter, the current time (current_time), the command that was last detected (found_command), the score it received (score), and whether the command was newly heard (is_new_command).

It’s important to note that in our program’s main loop, this function will be called every time inference is performed, even if a command was not detected. This means that we should check is_new_command to determine whether anything needs to be done.

The test for this function, in command_responder_test.cc, is equally simple. It just calls the function, given that there’s no way for it to test that it generates the correct output:

TF_LITE_MICRO_TEST(TestCallability) {
  tflite::MicroErrorReporter micro_error_reporter;
  tflite::ErrorReporter* error_reporter = &micro_error_reporter;

  // This will have external side-effects (like printing to the debug console
  // or lighting an LED) that are hard to observe, so the most we can do is
  // make sure the call doesn't crash.
  RespondToCommand(error_reporter, 0, "foo", 0, true);
}

To run this test, enter this in your terminal:

make -f tensorflow/lite/micro/tools/make/Makefile 
  test_command_responder_test

And that’s it! We’ve walked through all of the components of the application. Now, let’s see how they come together in the program itself.

Arduino

As of this writing, the only Arduino board with a built-in microphone is the Arduino Nano 33 BLE Sense, so that’s what we’ll be using for this section. If you’re using a different Arduino board and attaching your own microphone, you’ll need to implement your own audio_provider.cc.

The Arduino Nano 33 BLE Sense also has a built-in LED, which is what we use to indicate that a word has been recognized.

Figure 7-7 shows a picture of the board with its LED highlighted.

Figure 7-7. The Arduino Nano 33 BLE Sense board with the LED highlighted

Now let’s look at how we use this LED to indicate that a word has been detected.

#include "tensorflow/lite/micro/examples/micro_speech/command_responder.h"
#include "Arduino.h"

// Toggles the LED every inference, and keeps it on for 3 seconds if a "yes"
// was heard
void RespondToCommand(tflite::ErrorReporter* error_reporter,
                      int32_t current_time, const char* found_command,
                      uint8_t score, bool is_new_command) {

static bool is_initialized = false;
if (!is_initialized) {
  pinMode(LED_BUILTIN, OUTPUT);
  is_initialized = true;
}

static int32_t last_yes_time = 0;
static int count = 0;

if (is_new_command) {
  error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                         current_time);
  // If we heard a "yes", switch on an LED and store the time.
  if (found_command[0] == 'y') {
    last_yes_time = current_time;
    digitalWrite(LED_BUILTIN, HIGH);
  }
}

// If last_yes_time is non-zero but was >3 seconds ago, zero it
// and switch off the LED.
if (last_yes_time != 0) {
  if (last_yes_time < (current_time - 3000)) {
    last_yes_time = 0;
    digitalWrite(LED_BUILTIN, LOW);
  }
  // If it is non-zero but <3 seconds ago, do nothing.
  return;
}

// Otherwise, toggle the LED every time an inference is performed.
++count;
if (count & 1) {
  digitalWrite(LED_BUILTIN, HIGH);
} else {
  digitalWrite(LED_BUILTIN, LOW);
}

Running the example

To deploy this example, here’s what we’ll need:

• An Arduino Nano 33 BLE Sense board

• A micro-USB cable

• The Arduino IDE

Tip

There’s always a chance that the build process might have changed since this book was written, so check README.md for the latest instructions.

The projects in this book are available as example code in the TensorFlow Lite Arduino library. If you haven’t already installed the library, open the Arduino IDE and select Manage Libraries from the Tools menu. In the window that appears, search for and install the library named Arduino_TensorFlowLite. You should be able to use the latest version, but if you run into issues, the version that was tested with this book is 1.14-ALPHA.

Note

You can also install the library from a .zip file, which you can either download from the TensorFlow Lite team or generate yourself using the TensorFlow Lite for Microcontrollers Makefile. If you’d prefer to do the latter, see Appendix A.

After you’ve installed the library, the micro_speech example will show up in the File menu under Examples→Arduino_TensorFlowLite, as shown in Figure 7-8.

Click “micro_speech” to load the example. It will appear as a new window, with a tab for each of the source files. The file in the first tab, micro_speech, is equivalent to the main_functions.cc we walked through earlier.

Figure 7-8. The Examples menu

Note

“Running the Example” already explained the structure of the Arduino example, so we won’t cover it again here.

To run the example, plug in your Arduino device via USB. Make sure the correct device type is selected from the Board drop-down list in the Tools menu, as shown in Figure 7-9.

Figure 7-9. The Board drop-down list

If your device’s name doesn’t appear in the list, you’ll need to install its support package. To do this, click Boards Manager. In the window that appears, search for your device, and then install the latest version of the corresponding support package. Next, make sure the device’s port is selected in the Port drop-down list, also in the Tools menu, as demonstrated in Figure 7-10.

Figure 7-10. The Port drop-down list

Finally, in the Arduino window, click the upload button (highlighted in white in Figure 7-11) to compile and upload the code to your Arduino device.

Figure 7-11. The upload button, a right-facing arrow

After the upload has successfully completed you should see the LED on your Arduino board begin to flash.

To test the program, try saying “yes.” When it detects a “yes,” the LED will remain lit solidly for around three seconds.

Tip

If you can’t get the program to recognize your “yes,” try saying it a few times in a row.

You can also see the results of inference via the Arduino Serial Monitor. To do this, open the Serial Monitor from the Tools menu. Now, try saying “yes,” “no,” and other words. You should see something like Figure 7-12.

Figure 7-12. The Serial Monitor displaying some matches

Note

The model we’re using is small and imperfect, and you’ll probably notice that it’s better at detecting “yes” than “no.” This is an example of how optimizing for a tiny model size can result in issues with accuracy. We cover this topic in Chapter 8.

ST Microelectronics STM32F746G Discovery Kit

Because the STM32F746G comes with a fancy LCD display, we can use this to show off whichever wake words are detected, as depicted in Figure 7-16.

Figure 7-16. STM32F746G displaying a “no”

#include "tensorflow/lite/micro/examples/micro_speech/command_responder.h"
#include "LCD_DISCO_F746NG.h"

LCD_DISCO_F746NG lcd;

// When a command is detected, write it to the display and log it to the
// serial port.
void RespondToCommand(tflite::ErrorReporter *error_reporter,
                      int32_t current_time, const char *found_command,
                      uint8_t score, bool is_new_command) {
  if (is_new_command) {
    error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                           current_time);

if (*found_command == 'y') {
  lcd.Clear(0xFF0F9D58);
  lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard yes!", CENTER_MODE);

} else if (*found_command == 'n') {
  lcd.Clear(0xFFDB4437);
  lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard no :(", CENTER_MODE);
} else if (*found_command == 'u') {
  lcd.Clear(0xFFF4B400);
  lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard unknown", CENTER_MODE);
} else {
  lcd.Clear(0xFF4285F4);
  lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard silence", CENTER_MODE);
}

make -f tensorflow/lite/micro/tools/make/Makefile 
  TARGET=mbed TAGS="cmsis-nn disco_f746ng" generate_micro_speech_mbed_project

tensorflow/lite/micro/tools/make/gen/mbed_cortex-m4/prj/ 
  micro_speech/mbed

cd tensorflow/lite/micro/tools/make/gen/mbed_cortex-m4/prj/micro_speech/mbed

mbed config root .

mbed deploy

python -c 'import fileinput, glob;
for filename in glob.glob("mbed-os/tools/profiles/*.json"):
  for line in fileinput.input(filename, inplace=True):
    print(line.replace(""-std=gnu++98"",""-std=c++11", "-fpermissive""))'

mbed compile -m DISCO_F746NG -t GCC_ARM

./BUILD/DISCO_F746NG/GCC_ARM/mbed.bin

cp ./BUILD/DISCO_F746NG/GCC_ARM/mbed.bin /Volumes/DIS_F746NG/

ls /dev/tty*

/dev/tty.usbmodem1454203

screen /dev/<tty.devicename 9600>

Heard yes (202) @65536ms