Data Prep

To prep a machine, you’ll write the code to supply the input tensors, aka the values [-1, 0, 1, 2, 3, 4] and their corresponding answers [-4, -2, 0, 2, 4, 6]. The index of the question has to match the index of the expected answer, which makes sense when you think about it. Since we are giving the model all the answers to the values, that is what makes this a supervised learning problem.

In this situation, the training set is six examples. Rarely would machine learning be used on such a small amount of data, but the problem is relatively small and straightforward. As you can see, none of the training data has been reserved for testing the model. Fortunately, you can try the model because you know the formula that was used to create the data in the first place. If you’re unfamiliar with the definitions of training and testing datasets, please review “Common AI/ML Terminology” in Chapter 1.

Design a Model

The idea of designing a model might sound tedious, but the honest answer is that it’s a mix of theory, trial, and error. Models can be trained for hours or even weeks before the designers of that model understand the performance of the architecture. An entire field of study could be dedicated to model design. The Layers models you’ll be creating for this book will give you an excellent foundation.

The easiest way to design a model is to use the TensorFlow.js Layers API, which is a high-level API that allows you to define each layer in sequential order. In fact, to start your model, you’ll begin with the code tf.sequential();. You might hear this called the “Keras API” due to the origin of this style of model definition.

The model you’ll create to solve the simple problem you are trying to tackle will have only a single layer and a single neuron. This makes sense when you think about the formula for a line; it’s not a very complex equation.

To add a layer to the model, you will use model.add and then define your layer. With the Layers API, each layer that gets added defines itself and automatically gets connected depending on the order of model.add calls, just like pushing to an array. You’ll define the expected input for your model in the first layer, and the final layer that you add will define the output of your model (see Example 8-1).

model.add(ALayer)
model.add(BLayer)
model.add(CLayer)

// Currently, model is [ALayer, BLayer, CLayer]

The model in Example 8-1 would have three layers. ALayer would be tasked with identifying the expected model input and itself. BLayer doesn’t need to identify its input because it is inferred that the input would be ALayer. Therefore, BLayer only needs to define itself. CLayer would identify itself, and because it is last, this identifies the model’s output.

Let’s get back to the model you’re trying to code. The architected model goal for the current problem has only one layer with one neuron. When you code that single layer, you’ll be defining your input and your output.

// The entire inner workings of the model
model.add(
  tf.layers.dense({
    inputShape: 1, // one value 1D tensor
    units: 1 // one neuron - output tensor
  })
);

The result is a straightforward neural network. When graphed, the network has two nodes (see Figure 8-2).

Figure 8-2. One input and one output

Generally, layers have more artificial neurons (graph nodes) but also are more complicated and have other properties to configure.

Identify Learning Metrics

Next, you’ll need to tell your model how to identify progress and how it can be better. These concepts aren’t foreign; they just seem strange in software.

Every time I try to aim a laser pointer at something, I generally miss it. However, I can see I’m a bit off to the left or right, and I adjust. Machine learning does the same thing. It might start randomly, but the algorithm corrects itself, and it needs to know how you want it to do that. A method that most fits my laser pointer example would be gradient descent. The smoothest iterative way to optimize the laser pointer is a method called stochastic gradient descent. That’s what we’ll use in this case because it works well, and it sounds pretty cool for your next dinner party.

As for measuring error, you might think a simple “right” and “wrong” would work, but there’s a significant difference between being a couple of decimals off versus being wrong by thousands. For this reason, you generally rely on a loss function to help you identify how wrong an AI is with a predicted guess. There are lots of ways to measure error, but in this case, mean squared error (MSE) is a great measurement. For those who need to know the mathematics, MSE is the average squared difference between the estimated values (y) and the actual value (y with a little hat). Feel free to ignore this next bit, as the framework is calculating it for you, but if you’re familiar with common mathematical notation, this can be represented like so:

Why would you like this formula over something simple like distance from the original answer? There are some mathematical benefits baked into MSE that help incorporate variance and bias as positive error scores. Without getting too deep into statistics, it’s one of the most common loss functions for solving for lines that fit data.

When you are ready to tell a model to use specific learning metrics and you’re done adding layers to a model, this is all wrapped up in a .compile call. TensorFlow.js is cool enough to know all about gradient descent and mean squared error. Rather than coding these functions, you can identify them with their approved string equivalents:

model.compile({
  optimizer: "sgd",
  loss: "meanSquaredError"
});

One of the great benefits to using a framework is that as the machine learning world invents new optimizers like “Adagrad” and “Adamax,” they can be tried and invoked by simply changing a string¹ in your model architecture. Switching “sgd” to “adamax” takes relatively no time for you as a developer, and it might significantly improve your model training time without you reading the published paper on stochastic optimization.

Identifying functions without understanding the specifics of a function provides a bittersweet benefit similar to changing file types without having to understand the full structure of each type. A little bit of knowledge of the pros and cons for each goes a long way, but you don’t need to memorize the specification. It’s worth taking a little time when you’re architecting to read up on what’s available.

Don’t worry. You will see the same names being used over and over, so it’s easy to get the hang of them.

At this point, the model is created. It will fail if you ask it to predict anything because it has done zero training. The weights in the architecture are entirely random, but you can review the layers by calling model.summary(). The output goes directly to the console and looks somewhat like Example 8-2.

_________________________________________________________________
Layer (type)                 Output shape              Param #
=================================================================
dense_Dense6 (Dense)         [null,1]                  2
=================================================================
Total params: 2
Trainable params: 2
Non-trainable params: 0
_________________________________________________________________

The layer dense_Dense6 is an automatic ID to reference this layer in the TensorFlow.js backend. Your ID can vary. This model has two trainable parameters, which makes sense since a line is y = mx + b, right? A fun way to think of this visually is to look back to Figure 8-2 and count the lines and nodes. One line and one node means two trainable params. All parameters for the layer are trainable. We’ll cover non-trainable params later. This one-layer model is ready to go.

Task the Model with Training

The final step for training a model is to combine the inputs into the architecture and assign how long it should train. As mentioned earlier, this is often measured in epochs, which is how many times the model will review the flashcards with the right answers, and then when it’s complete, it stops training. The number of epochs you should use depends on the magnitude of the problem, the model, and how correct is “good enough.” In some models, getting another half of a percent is worth hours of training, and in our case, the model is accurate enough to be correct within seconds.

The training set is a 1D tensor with six values. If the epochs were set to 1,000, the model would effectively train 6,000 iterations, which would take any modern computer a few seconds at most. The trivial problem of fitting a line to points is quite simple for a computer.

Put It All Together

Now that you’re familiar with the high-level concepts, you’re probably eager to solve this problem with code. Here’s the code to train a model on the data and then immediately ask the model for an answer for the value 10, as discussed.

// Inputs
const xs = tf.tensor([-1, 0, 1, 2, 3, 4]); 
// Answers we want from inputs
const ys = tf.tensor([-4, -2, 0, 2, 4, 6]);

// Create a model
const model = tf.sequential(); 

model.add( 
  tf.layers.dense({
    inputShape: 1,
    units: 1
  })
);

model.compile({ 
  optimizer: "sgd",
  loss: "meanSquaredError"
});

// Print out the model structure
model.summary();

// Train
model.fit(xs, ys, { epochs: 300 }).then(history => { 
  const inputTensor = tf.tensor([10]);
  const answer = model.predict(inputTensor); 
  console.log(`10 results in ${Math.round(answer.dataSync())}`);
  // cleanup
  tf.dispose([xs, ys, model, answer, inputTensor]); 
});

The data is prepared in tensors with inputs and expected outputs.

A sequential model is started.

Add the only layer with one input and one output, as discussed.

Finish the sequential model with a given optimizer and loss function.

The model is told to train with fit for 300 epochs. This is a trivial amount of time, and when the fit is complete, the promise it returns is resolved.

Ask the trained model to provide an answer for the input tensor 10. You’ll need to round the answer to force an integer result.

Dispose of everything once you’ve got your answer.

Congratulations! You’ve trained a model from scratch in code. The problem you’ve just solved is called a linear regression problem. It has all kinds of uses, and it’s a common tool for predicting things such as housing prices, consumer behavior, sales forecasts, and plenty more. Generally, points don’t perfectly land on a line in the real world, but now you have the ability to turn scattered linear data into a predictive model. So when your data looks like Figure 8-3, you can solve as shown in Figure 8-4.

Figure 8-3. Scattered linear data

Figure 8-4. Predicted line of best fit using TensorFlow.js

Now that you’re familiar with the basics of training, you can expand your process to understanding what it takes to solve more complex models. Training a model is significantly dependent on the architecture and the quality and quantity of the data.

Chapter Challenge: The Model Architect

Now it’s your turn to build a Layers model via specification. What does this model do? No one knows! It’s not going to be trained with any data at all. In this challenge, you’ll be tasked to build a model with all kinds of properties you might not understand, but you should be familiar enough to at least set up the model. This model will be the biggest you’ve created yet. Your model will have five inputs and four outputs with two layers between them. It will look like Figure 8-10.

Figure 8-10. The Chapter Challenge model

Do the following in your model:

• The input layer should have 5 units.

• The next layer should have 10 units and use sigmoid for activation.

• The next layer should have 7 units and use ReLU activation.

• The final layer should have 4 units and use softmax for activation.

• The model should use Adam optimization.

• The model should use the loss function categoricalCrossentropy.

Before building this model and looking at the summary, can you calculate how many trainable parameters the final model will have? That’s the total number of lines and circles from Figure 8-10, not counting the input.

You can find the answer to this challenge in Appendix B.

Review Questions

Let’s review the lessons you’ve learned from the code you’ve written in this chapter. Take a moment to answer the following questions:

1. Why would the Chapter Challenge model not work with the training data from this chapter?

2. What method can you call on a model to log and review its structure?

3. Why do you add activation functions to layers?

4. How do you specify the input shape for the Layers model?

5. What does sgd stand for?

6. What is an epoch?

7. If a model has one input, then a layer of two nodes, and an output of two nodes, how many hidden layers are present?

Solutions to these exercises are available in Appendix A.