Bag-of-words

Bag-of-words is a model you will probably come across if you decide to do some extra research on NLP as it is quite commonly used.

It is a simplifying representation used to count the occurrence of all words in a piece of text, disregarding grammar.

This approach seems a bit simplistic as it does not take into consideration any semantic meaning and context, but it intends to add some weight to different terms in a text, based on how often they are used. This information is then used as features for training a classifier.

This process is also sometimes referred to as vectorization, as it aims to turn pieces of text into fixed-length vectors.

Concepts without examples can be difficult to understand, so let’s use the following four sentences to see how the Bag-of-words model would apply.

These vectors can then be used as features to train an algorithm.

The two first sentences (“Win millions of dollars” and “Win a Tesla”) could be labelled as “spam” and the two last (“Request for help” and “Help millions of developers”) as “non-spam”.

Considering that similar kinds of words are used in similar documents, the bag-of-words approach can help us determine the likelihood of a sentence being spam or not, based on our previous example dataset.

If the word “win” is often contained in a text labelled as spam, the probability of a new sentence such as “Win the trip of your dreams” being spam is higher than the probability of another sentence like “Feedback on performance”.

It is important to notice that the order of the words in the text does not matter; only the amount of times these words are used.

In real applications, the dataset should be much larger, containing a more diverse corpus, to increase the accuracy of the predictions.

Tokenization

In natural language processing, two common types of tokenization include sentence tokenization and word tokenization.

Sentence tokenization , also called sentence segmentation, is the process of dividing a string into its component sentences. One way to do this is to split sentences whenever we see a full stop (.).

For example, a paragraph such as

“13th is a 2016 American documentary film by director Ava DuVernay. The film explores the “intersection of race, justice, and mass incarceration in the United States;” it is titled after the Thirteenth Amendment to the United States Constitution, adopted in 1865, which abolished slavery throughout the United States and ended involuntary servitude except as a punishment for conviction of a crime.”

Word tokenization , also called word segmentation, is the process of dividing a string into its component words. This can be used by splitting a sentence using a space character as the divider.

Text lemmatization and stemming

Text lemmatization and stemming are techniques to reduce inflectional forms of a word to a common base form.

As related words can have similar meanings, these tools help reduce them to a single one.

The result of this mapping would look something like this:

“They are all eating in front of their computers” => “They are all eat in front of their computer”.

This kind of technique is helpful for use cases where we would want to find results for more than an exact input, for example, with search engines.

When we execute a search, we generally are interested in getting relevant results, not only for the exact search terms that we used but also for other possible forms of these words.

For example, if I run a search using the words “Hike Europe”, I’m also interested in results that will have the word “hikes” and “hiking” or “hikers”, and so on.

Stop words

Stop words are words that can add a lot of noise and are considered irrelevant in a text. They are filtered out before or after processing a text.

Examples of stop words include “and”, “the”, and “a”, but there is no definitive list as it will vary based on language and application.

After filtering out a text for stop words, a sentence like “They are meeting at the station tomorrow at 10am” would end up being [“They”, “are”, “meeting”, “station”, “tomorrow”, “10am”].

As we can see, the sentence did not lose any meaning after removing the stop words.

As machine learning operations can be time-consuming, removing words that do not change the semantic meaning of a text can improve performance without impacting negatively the accuracy of the predictions.

Importing the model

Loading the model

Predictions

Now that we have the main tools we need to run the prediction, we also need to write some helper functions that will turn our input text into vectors, like we talked about in the previous section of this chapter.

Input text cannot be fed to an algorithm as a string, as machine learning models primarily work with numbers, so let’s look into how to execute vectorization in JavaScript.

This piece of code will turn a string like “I learned a lot from this talk” into [“i”, “learned”, “a”, “lot”, “from”, “this”, “talk”].

We start by calling the trim() function to remove potential whitespace at the beginning and end of the string. We then make everything lowercase, remove all punctuation, and split it into an array of substrings using the space character as separator.

Once this is done, we need to turn this array into an array of numbers using the metadata we loaded just previously.

This metadata is a JSON file with the sample data used to train the model we are loading into our application. It contains about 20,000 words.

What this code sample does is loop through the array of substrings we created previously and see if each element exists in the metadata. If it does, it sets the variable wordIndex to the index found in the metadata; otherwise, if the word from our input text was not found in the data used to train the model, it sets the value of wordIndex to 2.

This number 2 is going to be our index for out-of-vocabulary (OOV) words.

These numbers are the indices of each word in our metadata file. As we can see, the number 2 is not present, which means that each word in the sentence “I learned a lot from this talk” was present in the dataset used to train the model.

Before we can use this to run predictions and get the overall sentiment, we need to do one last step of transforming the data.

As explained previously, when training a machine learning model, you need to make sure that the data used has the same shape.

For example, if you were working with images, you could not use images of different sizes (280x280 pixels, then 1200x800 pixels, etc.) in your dataset.

The same principle applies to text data. Our example text “I learned a lot from this talk” has a length of 7; however, our model has been trained with vectors which length were 100.

This value can be found in the metadata.json file as the value of the property max_len.

What this means is that the longest piece of string in the dataset used to train the model contained 100 words.

As we need to work with fixed-length vectors, we need to transform our vector of length 7 to a vector of length 100.

To do this, we need to write a function that will prepend a certain amount of 0s until the length of the vector equals 100.

This function should be used for text data that is less than 100 words long, but what about strings which length exceeds 100?

The process for this case is kind of the opposite, we loop through the array of indices, and when we reach the maximum length, we slice the array and get rid of the rest.

As we can see, we still have our seven indices representing our sentence “I learned a lot from this talk, but prepended with the correct amount of 0s to make this vector have a length of 100.

This score will be a float number between 0 and 1. The closest the score is to 0, the more negative it is predicted to be, and the closest to 1, the more positive.

The score for our input text “I learned a lot from this talk” is 0.9912545084953308, which means that the sentiment predicted for it is “positive”.

A sentence like “This is really bad” produces a score of 0.007734981831163168, which is “negative”.

Complete example

If we want to put this code altogether in an application that would gather input text from users, this is what it could look like.

In this code sample, I am using the user’s input as a single string that will be transformed and used for sentiment analysis. However, if you wanted to split a paragraph into different sentences and run sentiment analysis on each sentence separately, you would need to start by creating a function that breaks the paragraph into an array of sentences.

If you decide to implement this and try sentiment analysis on various pieces of text, you might realize that the accuracy of the prediction is not always the best.

For example, a sentence, like “I really hate this”, has a score of 0.9253836274147034, meaning its sentiment is predicted to be “positive”, which seems incorrect.

This is mainly due to the fact that the data used to train the model is a set of 25,000 movie reviews from IMDB, an online database of information related to films, TV shows, and so on.

Even though, to humans, a sentence like “I really hate this” seems pretty clearly negative, the machine learning model only looks at the occurrence of every word in the sentence, compared to what it has learned from the training data.

If the words “i”, “really”, and “this” were found in more sentences and these were labelled as “positive”, it overweighs the fact that the word “hate” looks to us as a negative sentiment. This word was probably used in less sentences in the dataset.

All this is to remind you to never rely entirely on predictions generated by a machine learning model. Even though algorithms are much better than humans at processing large amounts of data and extracting patterns from it, predictions should be used as a way to augment the way we make decisions, not replace it completely.

If you wanted to improve these predictions, you could look for other open source datasets or pre-trained models used for sentiment analysis.

Importing the model

Predictions

After importing the tools we need, the code to run the prediction on a piece of text is actually pretty small!

For a sentence like the one shown in the preceding code sample, the predictions return as the following array.

At first, you could think that it means the sentence “this is really the most useless talk I have ever watched” has been predicted to belong to the “identity_attack” label; however, that’s not it.

The way the predictions come back with this particular model is as an array of objects that contain data for each label, ordered by alphabetical order.

It can be a bit confusing, as other models used with TensorFlow.js usually produce predictions ordered by score.

To get a better understanding, let’s get a deeper look into the predictions array that we logged earlier.

This screenshot represents the four first entries in the predictions array. As we can see, they are ordered alphabetically by label.

To understand which label is the correct one for the input text, we need to look at the “probabilities” array and the “match” value.

The probabilities array contains two values, the first representing the probability of the label being falsy, meaning the input text not being classified with that label, and the second value representing the probability of the label being truthy, meaning the input text is classified as being this type of toxic content.

The “match” value is a more straightforward representation. If its value is “true”, it means the text corresponds to that label; if “false”, it doesn’t.

Sometimes, the value of “match” is set to “null”. This happens when neither of the predictions exceeds the threshold provided.

This threshold is used to determine the value of the “match” property.

If the first value in the “probabilities” array exceeds the threshold, “match” is set to “false”; if instead the second value exceeds the threshold, “match” is set to true. And, as explained before, if none of the probabilities values exceed the threshold, “match” is set to “null”.

Labels can also be passed as another argument to the model if you only care about certain ones and not all 7.

Providing the labels you are only interested in will return filtered predictions.

Similarly to the sentiment classifier built in the previous section of this chapter, the accuracy of the predictions depends a lot on the data used to train the model.

TensorFlow.js’s toxicity model was trained on a dataset of about 2 million user-generated online news comments published from 2015 to 2017.

An interesting aspect of this is that datasets are often originally labelled by humans, meaning people have to go through all the entries and label each of them with a corresponding label based on a set of labels provided.

In the case of the data used for the toxicity model, it means that people had to go through 2 million pieces of text, and label each of them with the label they thought would be the closest to the type of toxicity the comment belonged to, based on the labels “identity_attack”, “insult”, “obscene”, “severe_toxicity”, “sexual_explicit”, “threat”, and “toxicity”.

One potential issue with this is that different people have different opinions about what they might classify as an “insult” or not, or what they would label as “toxicity” instead of “severe_toxicity”, which adds a certain level of bias into the data that is going to be used to train the model.

So, not only the accuracy of the prediction depends on the quality of the data gathered (the online comments), it also depends on how they were labelled.

If you are in a position where you can inspect the data that was used to train a model you are using in your application, I would really recommend to do it.

Now that we went through a couple of examples of what can be done with sentiment analysis with TensorFlow.js, let’s look into some potential applications.

Importing the model

If you decide to use it as a NPM module, you do not need to import the @tensorflow/tfjs library; however, you will need to make sure you have installed the peer dependencies for tfjs-core and tfjs-converter.

Loading the model

Once the model is imported, we need to load it in our application. Two options are available to do this.

This way of loading the model is the one you should probably use if you want to use Google’s Question Answering model hosted on the Google Cloud Platform (GCP) provider.

These two ways of loading the model will return a model object that we can call methods on to predict our answers.

Predictions

These two parameters need to be strings.

The text property represents the answer to the question the person would have asked. The score is the confidence level for the prediction. The higher it is, the more confident the model is that this answer is correct. The startIndex and endIndex represent the indices of the starting and ending character where the expression was found in the passage.

Complete example

This code will log the following output in your browser’s console.

As we can see in the preceding screenshot, the answers come back as an array of five elements. The one with the highest score is “February 4, 1913”, which is the correct answer to the question “When was Rosa born?”!

Also, if we observe the subsequent predictions, they are all somewhat correct as they contain either a part of the birth date or the birth and death dates.

We could think maybe we just got lucky on that one, so let’s try another question, for example: “Who was Rosa Parks?”.

When asking this question, we get the following answers.

This time, our answers array contains only two objects. However, these predictions are correct!

An interesting aspect of this is that our question mentioned “Rosa Parks”, whereas our passage mentioned her full name “Rosa Louise McCauley Parks”, and the model was still able to understand who our question was about.

This is pretty impressive and has huge potential, but, as with every machine learning model, it has its limits.

For example, when changing the question to “How was Rosa Parks called?”, expecting the answers “the first lady of civil rights” or “the mother of the freedom movement”, the predictions array came back empty, meaning the model could not find answers to this question in the passage.

However, when modifying the question a little bit to be more precise (“What did the United States Congress call Rosa Parks?”), the model managed to provide the right answers – the first one being this.

Unfortunately, asking the question this way would have probably required a certain kind of knowledge from the user. A person unfamiliar with the details of Rosa Parks’ life would have most likely asked the question in a similar format to “What was Rosa Parks called?”, which returned no answer.

Building an interactive education tool

In the preceding code samples, we only used this model to log the results in the browser’s developer tools. However, let’s use this code to build an interactive education tool , in which users would be able to learn about historical figures, using Question Answering.

For this project, to make it simpler, we’re going to store data about a few preselected people in our code base as JSON files. If you are not familiar with it, JSON stands for JavaScript Object Notation and refers to a file format for storing data.

If you wanted to go a step further, you could try to use the MediaWiki API to let users request for any public figure they would want and fetch data dynamically.

The following application is going to have information about three historical figures, let the user decide which one they would like to learn more about, and provide an input field to ask different questions to display the answer with the highest probability, predicted by the machine learning model.

The final state of this project is going to contain three screens and look something like this.

As you may have noticed, I kept the design very minimal because I want to focus on the functionality. The important takeaway from this project is how to create a quick user interface using the Question Answering model.

Now that you have a better idea of what we are about to build, let’s dive into the different features and code samples.

Step 1: Loading page

Loading a pre-trained machine learning model into your application will always have an impact on performance as they tend to be quite heavy. As a result, it is important to think in advance about what kind of user experience you want to create to improve the perceived performance.

This could include loading the model in the background early on, so by the time the user gets to the experience, it is loaded and ready to be used; or providing a loading animation to indicate that the user needs to wait for the interface to be usable.

The latter is not the best option as the user clearly is forced to wait before being able to interact with the experience; however, in the case of this project, this is the method we are going to use, as the only feature of the application is the interaction with the model.

To display this loading state, we have a button element in our HTML file that has a default state of disabled, to prevent the user from trying to run predictions before the model is available.

<section class="intro">

   <h1>Hidden figures</h1>

   <h3>

     Using machine learning, learn about 3 NASA engineers by asking questions about their lives!

   </h3>

   <button disabled>Loading...</button>

 </section>

Listing 4-26

HTML code sample to show some intro text and a button with initial loading state

Then, in the JavaScript file, we can start loading the model as soon as the page loads and change the state of the button once the model is ready.

let model;

const loadModel = async () => await qna.load();

const init = async () => {

  model = await loadModel();

  const startButton = document.querySelector(".intro button");

  startButton.removeAttribute("disabled");

  startButton.innerHTML = "Start";

};

init();

Listing 4-27

JavaScript code sample to load the model and update the state of the button when the model is loaded

The preceding code sample shows that, as soon as the loadModel function has finished loading the model, we remove the disabled state of the button to make it interactive and change its content to “Start” to indicate that the user can start the experience.

At this stage, once the user clicks the start button, we hide this content to show the three options available.

Step 2: Selection page

For the purpose of this small demo, I hard-coded the name of the three engineers in the button elements in the HTML file; however, if you were building something with more entries, or if they were dynamically generated, you could also replace the content of the buttons dynamically in JavaScript.

<section class="selection">

   <h1>Who would you like to learn about?</h1>

   <section class="buttons">

     <button class="figure">Katherine Johnson</button>

     <button class="figure">Dorothy Vaughan</button>

     <button class="figure">Mary Jackson</button>

   </section>

</section>

Listing 4-28

HTML code sample with buttons containing the name of the three NASA engineers

In our JavaScript file, we need to add an event listener for clicks on these buttons and fetch the JSON data.

To avoid loading all content at once, I created a JSON file for each person, so that the code is going to fetch only the data the user has asked for.

As a result, I have three JSON files that I named:

• dorothyVaughan.json

• katherineJohnsons.json

• maryJackson.json

This way, when the user clicks one of the three buttons, the following code will execute the fetching and prepare for the following prediction page.

const engineers = {

  "Katherine Johnson": "katherineJohnson",

  "Dorothy Vaughan": "dorothyVaughan",

  "Mary Jackson": "maryJackson",

};

figureButtons.forEach(button => {

  button.onclick = e => {

    const dataFile = engineers[e.target.textContent];

    fetch(`${window.location.href}${dataFile}.json`)

      .then(response => response.json())

      .then(data => {

        figureData = data;

        const label = document.getElementsByTagName("label")[0];

        label.innerHTML = `What would you like to know about ${e.target.textContent}?`;

      });

  };

});

Listing 4-29

JavaScript code to fetch data when clicking one of the buttons

In the preceding code sample, I started by creating an object to map the names displayed in the UI to the names of the associated JSON file.

Then, for each of the three buttons, I listen for click events, look at the content of the button clicked to get the name of the JSON file, and use the fetch function to get the data and store it in a figureData variable.

I also use the name of the person the user has chosen, to display it in the title “What would you like to know about…”.

There is a bit more code related to showing and hiding HTML elements as we go through the pages, but I will share the complete code for this project at the end of this chapter.

Now that we know which person the user is interested in knowing more about, and we’ve fetched the related data, let’s move on to the prediction page.

Step 3: Predictions

Listing 4-31. JavaScript code sample to get the input string, feed it to the model, and display the output

Our final page has an input field, a button to run the predictions, and a paragraph that outputs the answer.

<section class="question">

  <label for="question"></label>

  <input type="text" name="question" placeholder="Who was she?" />

  <button class="ask">Ask</button>

</section>

<section class="answer-block">

  <p class="input-question"></p>

  <p class="answer"></p>

</section>

Listing 4-30

HTML code sample to display an input field and a section to display the prediction

In our JavaScript file, we need to listen to click events on the button, get the value of the input entered by the user, send it alongside the data fetched as parameters to the method to get the answers from the model, and display the first answer on the page.

The following is the code sample to do this.

Listing 4-31. JavaScript code sample to get the input string, feed it to the model, and display the output

In the first part of the preceding code, we listen to click events, get the input question, and call the findAnswers method with the question and the figureData variable we used in the previous section to store the data fetched from the JSON file.

We then call a function called displayAnswer to which we pass the answers returned by the model, to display it on the page.

As mentioned a bit earlier in this chapter, sometimes, the model is not going to find answers to a question and will return an empty array. We need to handle this case first, and here, we display a generic message.

Then, if the model has found answers, we display the first one in the array as it is the one with the highest probability to be correct.

And that’s it!

In less than 100 lines of JavaScript, you can create a small interactive project allowing users to ask questions and learn about different topics using machine learning!

If you want the complete code sample for this, you can find the HTML and JavaScript in the following listing.

<html lang="en">

  <head>

    <meta charset="UTF-8" />

    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <title>Interactive education tool</title>

    <link rel="stylesheet" href=" styles.css" />

  </head>

  <body>

    <section class="intro">

      <h1>Hidden figures</h1>

      <h3>

        Using machine learning, learn about 3 NASA engineers by asking questions about their lives!

      </h3>

      <button disabled>Loading...</button>

    </section>

    <main>

      <section class="selection">

        <h1>Who would you like to learn about?</h1>

        <section class="buttons">

          <button class="figure">Katherine Johnson</button>

          <button class="figure">Dorothy Vaughan</button>

          <button class="figure">Mary Jackson</button>

        </section>

      </section>

      <section class="question">

        <label for="question"></label>

        <input type="text" name="question" placeholder="Who was she?" />

        <button class="ask">Ask</button>

      </section>

      <section class="answer-block">

        <p class="input-question"></p>

        <p class="answer"></p>

      </section>

    </main>

    <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script>

    <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/qna"></script>

    <script src="index.js"></script>

  </body>

</html>

Listing 4-32

Complete HTML code

"Dorothy Johnson Vaughan (September 20, 1910 – November 10, 2008) was an American mathematician and human computer who worked for the National Advisory Committee for Aeronautics (NACA), and NASA, at Langley Research Center in Hampton, Virginia. In 1949, she became acting supervisor of the West Area Computers, the first African-American woman to supervise a group of staff at the center. She later was promoted officially to the position. During her 28-year career, Vaughan prepared for the introduction of machine computers in the early 1960s by teaching herself and her staff the programming language of Fortran. She later headed the programming section of the Analysis and Computation Division (ACD) at Langley. Vaughan is one of the women featured in Margot Lee Shetterly's history Hidden Figures: The Story of the African-American Women Who Helped Win the Space Race (2016). It was adapted as a biographical film of the same name, also released in 2016. In 2019, Vaughan was awarded the Congressional Gold Medal posthumously."

Listing 4-33

Example JSON data for the file katherineJohnson.json

"Mary Jackson (née Winston, April 9, 1921 – February 11, 2005) was an American mathematician and aerospace engineer at the National Advisory Committee for Aeronautics (NACA), which in 1958 was succeeded by the National Aeronautics and Space Administration (NASA). She worked at Langley Research Center in Hampton, Virginia, for most of her career. She started as a computer at the segregated West Area Computing division in 1951. She took advanced engineering classes and, in 1958, became NASA's first black female engineer. After 34 years at NASA, Jackson had earned the most senior engineering title available. She realized she could not earn further promotions without becoming a supervisor. She accepted a demotion to become a manager of both the Federal Women's Program, in the NASA Office of Equal Opportunity Programs and of the Affirmative Action Program. In this role, she worked to influence the hiring and promotion of women in NASA's science, engineering, and mathematics careers. Jackson's story features in the 2016 non-fiction book Hidden Figures: The American Dream and the Untold Story of the Black Women Who Helped Win the Space Race. She is one of the three protagonists in Hidden Figures, the film adaptation released the same year. In 2019, Jackson was posthumously awarded the Congressional Gold Medal. In 2020 the Washington, D.C. headquarters of NASA was renamed the Mary W. Jackson NASA Headquarters."

Listing 4-34

Example JSON data for the file maryJackson.json

"Dorothy Johnson Vaughan (September 20, 1910 – November 10, 2008) was an American mathematician and human computer who worked for the National Advisory Committee for Aeronautics (NACA), and NASA, at Langley Research Center in Hampton, Virginia. In 1949, she became acting supervisor of the West Area Computers, the first African-American woman to supervise a group of staff at the center. She later was promoted officially to the position. During her 28-year career, Vaughan prepared for the introduction of machine computers in the early 1960s by teaching herself and her staff the programming language of Fortran. She later headed the programming section of the Analysis and Computation Division (ACD) at Langley. Vaughan is one of the women featured in Margot Lee Shetterly's history Hidden Figures: The Story of the African-American Women Who Helped Win the Space Race (2016). It was adapted as a biographical film of the same name, also released in 2016. In 2019, Vaughan was awarded the Congressional Gold Medal posthumously."

Listing 4-35

Example JSON data for the file dorothyVaughan.json

Listing 4-36. Complete JavaScript code

In the code samples we just went through, questions are captured in an input field that users have to type in. However, there are other ways to get the same kind of information using another Web API.

In the code samples we just went through, questions are captured in an input field that users have to type in. However, there are other ways to get the same kind of information using another Web API.

Getting input data from the Web Speech API

A way to make this project even more interactive would be to allow people to ask their questions using their own voice rather than typing on their keyboard.

The Web API that enables speech recognition is the Web Speech API. Using this API, we can add speech-to-text functionality to this project and even text-to-speech with SpeechSynthesis, if we want the answers to also be read out loud.

The amount of code needed to add this feature is relatively small. Considering we already went through the core of the application, it will only impact the part that captures user’s input.

To start using the Web Speech API, you need to include the following lines in your JavaScript file.

var SpeechRecognition = SpeechRecognition || webkitSpeechRecognition;

var SpeechRecognitionEvent = SpeechRecognitionEvent || webkitSpeechRecognitionEvent;

Listing 4-37

Initial setup to use the Web Speech API

These lines will allow you to have access to the speech recognition interface.

Then, to start using this interface, you need to create a speech recognition instance using the following.

const recognition = new SpeechRecognition();

Listing 4-38

Instantiate a new SpeechRecognition instance

Before starting the recognition, you can set a few parameters on your recognition instance. Personally, I set the following ones.

recognition.continuous = false;

recognition.lang = "en-US";

recognition.interimResults = false;

Listing 4-39

Optional parameters

The first one indicates that the listening and recognition is not continuous, meaning you will need a user interaction every time you want to listen to a question from the user. Setting it to false allows you to listen only when the user means to interact with the interface, instead of listening nonstop.

The second setting is the language of the recognition for the request. If unset, it will default to use the language of the HTML document root element, defined in your lang attribute.

At the moment, the list of languages available to use with the SpeechRecognition API includes

Finally, the third setting interimResults set to false indicates I am only interested in getting the last result of the recognition, and not all interim results while I am speaking.

After setting these attributes, we can start the recognition using recognition.start().

If you have not enabled the microphone on the web page already, you will be prompted to do so, but otherwise, the microphone will start listening for inputs.

To have access to the results of the recognition, you need to call the onresult method, like this.

recognition.onresult = function (event) {

   if (event.results[0][0]){

      var result = event.results[0][0].transcript;

   }

   console.log("result", result);

};

Listing 4-40

Getting results from the Web Speech API

Calling onresult returns a callback with an event object of the following shape.

Looking at this output helps make sense of the way we are setting the result variable in the preceding code sample: event.results[0][0].transcript.

Another useful method we can call is onspeechend.

onspeechend is triggered when speech has stopped being detected. To avoid running recognition when not needed, we can use onspeechend to stop the recognition entirely.

recognition.onspeechend = function () {

   recognition.stop();

};

Listing 4-41

Code sample to stop the recognition when the API has detected that the user has stopped talking

Put together, the code sample for speech recognition looks like this:

var SpeechRecognition = SpeechRecognition || webkitSpeechRecognition;

var SpeechRecognitionEvent =

  SpeechRecognitionEvent || webkitSpeechRecognitionEvent;

var recognition = new SpeechRecognition();

const startSpeechRecognition = () => {

  recognition.continuous = false;

  recognition.lang = "en-US";

  recognition.interimResults = false;

  recognition.onspeechend = () => recognition.stop();

  recognition.onresult = function (event) {

    if (event.results[0][0]) var result = event.results[0][0].transcript;

    console.log("result", result);

  };

  recognition.start();

};

startSpeechRecognition();

In only 20 lines of JavaScript, a web application can listen to a user’s voice commands!

If you wanted to add this to the project we built previously, you could implement it where we added the event listener for clicks on the “ask” button.

Instead of using the input written in the input field, you could trigger the speech recognition on click, listen to the user’s voice input, and feed that to the model, the same way we sent our written input string.

Imagining you already added the basic setup of the SpeechRecognition instance from the code sample just earlier, the only change you would need to do to the project code would be in the function listening for clicks on the askButton element.

askButton.onclick = async () => {

  recognition.start();

  recognition.onresult = async (event) => {

    if (event.results[0][0]) {

        var result = event.results[0][0].transcript;

    }

    inputQuestion = result;

    const answers = await model.findAnswers(inputQuestion, figureData);

    displayAnswer(answers);

    document.getElementsByTagName("input")[0].value = "";

  };

};

Listing 4-42

User voice commands with the QNA model

When the user clicks the button, this code starts the speech recognition and, on result, stores the final transcript in the inputQuestion variable. This variable was already sent to the findAnswers method in the original code for this project, but now it contains input from the user’s speech command.

A few changes would need to be done to the UI to reflect this functionality change (e.g., the input field is not necessarily needed anymore), but overall, the change we had to make to go from a written input to a spoken one only took a few lines of code!

Finally, if you wanted to go a little step further and have the output being read out loud to the user on top of being displayed on the screen, you can do so with the following three lines.

let speechSynth = window.speechSynthesis;

var result = new SpeechSynthesisUtterance(answers[0].text);

speechSynth.speak(result);

Listing 4-43

Code sample to get the output prediction read out loud using the SpeechSynthesis Web API

These three lines create a speechSynthesis instance, execute a speech request passing the answer from the model, and call the speak method to make the speech service read the information out loud.

The default voice feels pretty robotic but a few parameters can be set to change this. I will not go into details here, but feel free to look at the MDN Web Docs for the Web Speech API for more information.

One last thing to keep in mind if you want to explore this API further is that the browser support is not at 100%. At the moment, it seems to be well supported on Chrome and Edge, but not Safari and Firefox.

Even though it is still pretty experimental, I would encourage you to look into it if you are interested.

This chapter contained a lot of information.

We looked into the basics of natural language processing, different types of text classification tools, various applications, and also how to implement sentiment analysis, toxicity classification, and question answering using TensorFlow.js, as well as experimenting with voice commands as a way to get text data to run predictions on.

If this is the first time you are diving into machine learning, this amount of information can feel a little bit overwhelming. However, as everything new takes time to understand, feel free to take breaks and come back to this chapter later on if you want.

The following chapter will dive into using other kind of data inputs with machine learning.