Matrix Notation

When dealing with big datasets, the number of features is large (n_x will be big), and so it is better to use a vector notation for the features and the weights, as follows:

where we have indicated the vector with a boldfaced x. For the weights, we use the same notation:

For consistency with formulas that we will use later, to multiply x and w, we will use matrix multiplication notation, and, therefore, we will write

where w^T indicates the transpose of w. z can then be written with this vector notation as

and the neuron output as

Python Implementation Tip: Loops and NumPy

The calculation that we have outlined in the equation (3) can be done in Python by standard lists and with loops, but those tend to be very slow, as the number of variables and observations grows. A good rule of thumb is to avoid loops, when possible, and to use NumPy (or TensorFlow, as we will see later) methods as often as possible.

The actual values are not relevant for our purposes. We are simply interested in how fast Python can multiply two lists, element by element. The times reported were measured on a 2017 Microsoft surface laptop and will vary greatly, depending on the hardware the code runs on. We are not interested in the absolute values, but only on how much faster NumPy is in comparison with standard Python loops. To time Python code in a Jupyter notebook, we can use a “magic command.” Usually, in a Jupyter notebook, these commands start with %% or %. A good idea is to check the official documentation, accessible from http://ipython.readthedocs.io/en/stable/interactive/magics.html , to better understand how they work.

The numpy code needed only 21 ms, or, in other words, was roughly 100 times faster than the code with standard loops. NumPy is faster for two reasons: the underlying routines are written in C, and it uses vectorized code as much as possible to speed up calculations on big amounts of data.

While training deep learning models, you will find yourself doing this kind of operation over and over, and, therefore, such a speed gain will make the difference between having a model that can be trained and one that will never give you a result.

Activation Functions

There are many activation functions at our disposal to change the output of our neuron. Remember: An activation function is simply a mathematical function that transforms z in the output . Let’s have a look at the most used.

Identity Function

This is the most basic function that you can use. Usually, it is indicated by I(z). It returns simply the input value unchanged. Mathematically we have

This simple function will come in handy when I discuss linear regression with one neuron later in the chapter. Figure 2-4 shows what it looks like.

Implementing an identity function in Python with numpy is particularly simple.

def identity(z):

    return z

Sigmoid Function

This is a very commonly used function that gives only values between 0 and 1. It is usually indicated by σ(z).

It is especially used for models in which we must predict the probability as an output (remember that a probability may only assume values between 0 and 1). You can see its shape in Figure 2-5. Note that in Python, if z is big enough, it can happen that the function returns exactly 0 or 1 (depending on the sign of z) for rounding errors. In classification problems , we will calculate logσ(z) or log(1 − σ(z)) very often, and, therefore, this can be a source of errors in Python, because it will try to calculate log 0, which is not defined. For example, you can start seeing nan appearing while calculating the cost function (more on that later). We will see a practical example of this phenomenon later in the chapter.

Note

Although σ(z) should never be exactly 0 or 1, while programming in Python, the reality can be quite different. due to a very big z (positive or negative), Python may round the results to exactly 0 or 1. This could give you errors while calculating the cost function (I will give you a detailed explanation and practical example later in the chapter) for classification, because we will need to calculate log σ(z) and log(1 − σ(z)) and, therefore, Python will try to calculate log0, which is not defined. This may occur, for example, if we don’t normalize our input data correctly, or if we don’t initialize our weights correctly. For the moment, it is important to remember that although mathematically everything seems under control, the reality while programming can be more difficult. It is something that is good to keep in mind while debugging models that, for example, give nan as a result for the cost function.

The behavior with z can be seen in Figure 2-5. The calculation can be written in this form using numpy functions :

s = np.divide(1.0, np.add(1.0, np.exp(-z)))

Note

It is very useful to know that if we have two numpy arrays, A and B, the following are equivalent: A/B is equivalent to np.divide(A,B), A+B is equivalent to np.add(A,B), A-B is equivalent to np.subtract(A,B), and A*B is equivalent to np.multiply(A,B). In case you are familiar with object-oriented programming, we say that in numpy, basic operations, such as /, *, +, and -, are overloaded. Note also that all of these four basic operations in numpy act element by element.

We can write the sigmoid function in a more readable (at least for humans) form as follows:

def sigmoid(z):

    s = 1.0 / (1.0 + np.exp(-z))

    return s

As stated previously, 1.0 + np.exp(-z) is equivalent to np.add(1.0, np.exp(-z)), and 1.0 / (np.add(1.0, np.exp(-z))) to np.divide(1.0, np.add(1.0, np.exp(-z))). I want to draw your attention to another point in the formula. np.exp(-z) will have the dimensions of z (usually a vector that will have a length equal to the number of observations), while 1.0 is a scalar (a one-dimensional entity). How can Python sum the two? What happens is what is called broadcasting.¹ Python, subject to certain constraints, will “broadcast ” the smaller array (in this case, the 1.0) across the larger one, so that at the end, the two have the same dimensions. In this case, the 1.0 becomes an array of the same dimension as z, all filled with 1.0. This is an important concept to understand, as it is very useful. You don’t have to transform numbers in arrays, for example. Python will take care of it for you. The rules on how broadcasting works in other cases are rather complex and beyond the scope of this book. However, it is important to know that Python is doing something in the background.

Tanh (Hyperbolic Tangent Activation) Function

The hyperbolic tangent is also an s-shaped curve that goes from -1 to 1.

In Figure 2-6, you can see its shape. In Python, this can be easily implemented, as follows:

def tanh(z):

    return np.tanh(z)

ReLU (Rectified Linear Unit) Activation Function

The ReLU function (Figure 2-7) has the following formula:

It is useful to spend a few moments exploring how to implement the ReLU function in a smart way in Python. Note that when we will start using TensorFlow, we will have it already implemented for us, but it is very instructive to observe how different Python implementations can make a difference when implementing complex deep-learning models.

In Python, you can implement the ReLU function in several ways. Listed below are four different methods. (Try to understand why they work before proceeding.)

1. 1.

   np.maximum(x, 0, x)

    
2. 2.

   np.maximum(x, 0)

    
3. 3.

   x * (x > 0)

    
4. 4.

   (abs(x) + x) / 2

    

The four methods have very different execution speeds. Let’s generate a numpy array with 10⁸ elements, as follows:

x = np.random.random(10**8)

Now let’s measure the time needed by the four different versions of the ReLU function when applied to it. Let the following code run:

x = np.random.random(10**8)

print("Method 1:")

%timeit -n10 np.maximum(x, 0, x)

print("Method 2:")

%timeit -n10 np.maximum(x, 0)

print("Method 3:")

%timeit -n10 x * (x > 0)

print("Method 4:")

%timeit -n10 (abs(x) + x) / 2

The results follow:

Method 1:

2.66 ms ± 500 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Method 2:

6.35 ms ± 836 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Method 3:

4.37 ms ± 780 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Method 4:

8.33 ms ± 784 μs per loop (mean ± std. dev. of 7 runs, 10 loops each)

The difference is stunning . The Method 1 is four times faster than the Method 4. The numpy library is highly optimized, with many routines written in C. But knowing how to code efficiently still makes a difference and can have a great impact. Why is np.maximum(x, 0, x) faster than np.maximum(x, 0)? The first version updates x in place, without creating a new array. This can save a lot of time, especially when arrays are big. If you don’t want to (or can’t) update the input vector in place, you can still use the np.maximum(x, 0) version.

An implementation could look like this:

def relu(z):

    return np.maximum(z, 0)

Note

Remember: When optimizing your code, even small changes may make a huge difference. In deep-learning programs, the same chunk of code will be repeated millions and billions of times, so even a small improvement will have a huge impact in the long run. Spending time to optimize your code is a necessary step that will pay off.

Leaky ReLU

The Leaky ReLU (also known as a parametric rectified linear unit) is given by the formula

with α a parameter typically of the order of 0.01. In Figure 2-8, you can see an example for α = 0.05. This value has been chosen to make the difference between x > 0 and x < 0 more marked. Usually, smaller values for α are used, but testing with your model is required to find the best value.

In Python, for example, this can be implemented if the relu(z) function has already been defined as

def lrelu(z, alpha):

  return relu(z) - alpha * relu(-z)

Swish Activation Function

Recently, Ramachandran, Zopf, and Le at Google Brain [4] studied a new activation function, called Swish, that shows great promise in the deep-learning world. It is defined as

where β is a learnable parameter. In Figure 2-9, you can see how this activation function looks for three values of the parameter β: 0.1, 0.5, and 10.0. The team’s studies have shown that simply replacing ReLU activation functions with Swish improves classification accuracy on ImageNet by 0.9%. In today’s deep-learning world, that is a lot. You can find more information on ImageNet at www.image-net.org /.

ImageNet is a large database of images that is often used to benchmark new network architectures or algorithms, such as, in this case, networks with a different activation function.

Other Activation Functions

There are many other activation functions, but these are rarely used. As a reference, following are some additional ones. The list is by no means comprehensive but should serve the purposes of giving you an idea of the variety of activation functions that can be used when developing neural networks.

• ArcTan

• Exponential Linear unit (ELU)

• Softplus

Note

Practitioners almost always use only two activation functions: the sigmoid and the ReLU (the ReLU probably most often). With both, you can achieve good results, and, given a complex enough network architecture, both can approximate any nonlinear function [5,6]. Remember that when using tensorflow, you will not have to implement the functions by yourself. tensorflow will offer an efficient implementation for you to use. But it is important to know how each activation function behaves, to understand when to use which one.

Cost Function and Gradient Descent : The Quirks of the Learning Rate

Now that you understand clearly what a neuron is, I will discuss what it means for it (and, in general, for a neural network) to learn. This will allow us to introduce concepts such as hyperparameters and learning rate. In almost all neural network problems, learning simply means finding the weights (remember that a neural network is composed of many neurons, and each neuron will have its own set of weights) and biases of the network that minimize a chosen function, which is usually called the cost function and typically indicated by J.

In calculus, there are several methods for finding the minimum of a given function analytically. Unfortunately, in all neural network applications, the number of weights is so big that it is not possible to use these methods. Numerical methods must be relied on, the most famous being gradient descent. It is the easiest method to understand, and it will give you the perfect basis from which to understand the more complex algorithms that you will see later in the book. Let me give a brief overview on how it works, because it is one of the best algorithms in machine learning to introduce the reader to the concept of learning rate and its quirks.

With ∇J(w), we have indicated the gradient of the cost function, which is a vector whose components are the partial derivatives of the cost function with respect to all the components of the weight vector w, as follows:

To decide when to stop, we could check when the cost function J(w) stops changing too much, or, in other words, you could define a threshold ϵ and stop at any iteration q > k (with k an integer that you have to find) that satisfies | J(w_q + 1) − J(w_q) | < ϵ for all q > k. The problem with this approach is that it is complicated, and this check is very expensive in terms of performance when implemented in Python (remember: you will have to do this step a very large number of times), so, usually, people simply let the algorithm run for a fixed big number of iterations and check the final results. If the result is not what is expected, they increase the fixed big number. How big? Well, that depends on your problem. What you do is choose a certain number of iterations (for example, 10,000 or 1,000,000) and let the algorithm run. At the same time, you plot the cost function vs. the number of iterations, and you check that the number of iterations you have chosen is sensible. Later in this chapter, you will see a practical example in which I will show you how to check if the number you chose was big enough. For the moment, you should know that you simply stop the algorithm after a fixed number of iterations.

We will assume here that the cost function is differentiable. This is not usually the case, but a discussion of this issue goes well beyond the scope of this book. People tend to use a practical approach in this case. The implementations work very well, and so these kinds of theoretical problems are usually ignored by a large number of practitioners. Remember that in deep-learning models, the cost function becomes an incredibly complex function, and studying it is almost impossible.

The series w_n will hopefully converge toward the minimum location, after a reasonable amount of iterations. The parameter γ is called the learning rate and is one of the most important parameters required in the neural network learning process.

The word hopefully, has been chosen for good reason. It is possible that the algorithm will not converge toward the minimum. It is even possible that the series w_n will oscillate between values without converging at all—or diverge outright. Choose γ too big or too small, and your model will not converge (or converge too slowly). To understand why this is the case, let’s consider a practical case and see how the method works while choosing different learning rates.

Learning Rate in a Practical Example

As a cost function , we choose the classical mean squared error (MSE)

where we have indicated with the superscript (i) the ith observation. Remember that with the subscript i (x_i), we have indicated the i^th feature. To recap our notation, we have indicated with the j^th feature and the i^th observation. In the example here, we have just one feature, so we don’t need the subscript j. The cost function can be implemented in Python easily as

np.average((y-hypothesis(x, w0, w1))**2, axis=2)/2

Our goal is to find the values for w₀ and w₁ that minimize J(w₀, w₁).

To apply the gradient descent method , we must calculate the series for w_{0, n} and w_{1, n}. We have the following equations:

Simplifying equations by calculating the partial derivatives gives

Because ∂f(w₀, w₁, x_i)/∂w₀ = 1 and ∂f(w₀, w₁, x_i)/∂w₁ = x_i, the previous equations are the ones that must be implemented in Python, if we want to code the gradient descent algorithm by ourselves.

I have omitted in this book the complete Python implementation of the example, because it would require too much space.

It is instructive to check how the model works, by varying the learning rate. In Figures 2-10, 2-11, and 2-12, the contour lines² of the cost functions have been drawn, and on top of these, the series (w_{0, n}, w_{1, n}) has been plotted, as points to visualize how the series converges (or doesn’t). In the figures, the minimum is indicated by a circle placed approximately at the center. We will consider the values γ = 0.8 (in Figure 2-10), γ = 2 (in Figure 2-11), and γ = 0.05 (in Figure 2-12). The different estimates, w_n, are indicated with points. The minimum is indicated by the circle approximately in the middle of the image.

In the first case (in Figure 2-10), the converging is well behaved, and in just eight steps, the method converges toward the minimum. When γ = 2 (Figure 2-11), the method makes steps that are too big (remember: the steps are given by −γ∇J(w) and therefore the bigger γ the bigger the steps) and unable to get close to the minimum. It keeps oscillating around it, without reaching it. In this case, the model will never converge. In the last case, when γ = 0.05 (Figure 2-12), the learning is so slow that it will take many more steps to get close to the minimum. In some cases, the cost function may be so flat around the minimum that the method takes such a big number of iterations to converge that, practically, you will not get close enough to the real minimum in a reasonable amount of time. In Figure 2-12, 300 iterations are plotted, but the method is not even very close to the minimum.

In deep-learning problems, each iteration will cost time, and you will have to perform this process several times. Choosing the right learning rate is a key part of designing a good model, because it will make training much faster (or make it impossible).

Sometimes it is efficient to change the learning rate during the process. You start with a bigger value to get close to the minimum faster, and then you reduce it progressively, to make sure that you get as close as possible to the real minimum. I will discuss this approach later in the book.

A good way of checking the convergence is to plot the cost function vs. the number of iterations . In this way, you can check its behavior. How the cost function looks in our three learning rates for the preceding example is shown in Figure 2-13. You can clearly see how the case with γ = 0.8 goes to zero rather quickly, indicating that we have reached a minimum. The case with γ = 2 does not even start to go down. It continues to remain at almost the same initial value. And, finally, the case with γ = 0.05 starts to go down, but it is a lot slower than the first case.

Remember also that the absolute value of the learning rate is not relevant. What is important is the behavior. We can multiply our cost function by a constant, and that would not influence our learning at all. Don’t look at the absolute values; check how fast and how the cost function is behaving. Additionally, the cost function will almost never reach zero, so don’t expect it. The value of J at its minimum is almost never zero (it depends on the functions itself). In the section about linear regression, you will see an example in which the cost function will not reach zero.

Now that we have defined the basis, we will use a neuron to solve two simple problems with machine learning: linear and logistic regression.

Example of Linear Regression in tensorflow

The first type of regression will offer an opportunity to understand how to build a model in tensorflow. To explain how to perform linear regression efficiently with one neuron, I must first explain some additional notation. In the previous sections, I discussed inputs . These are the so-called features that describe an observation. Normally, we have many observations. As briefly explained before, we will use an upper index to indicate the different observations between parentheses. Our i^th observation will be indicated with x⁽ⁱ⁾, and the j^th feature of the i^th observations will be indicated as . We will indicate the number of observations with m.

You will remember that I have said many times that numpy is highly optimized to perform several parallel operations at the same time. To get the best performance possible, it is important to write our equations in matrix form and feed the matrices to numpy. In this way, our code will be as efficient as possible. Remember: Avoid loops at all costs whenever possible. Let’s spend some time now in writing all our equations in matrix form. In this way, our Python implementation will be much easier later.

The entire set of inputs (features and observations) can be written in matrix form. We will use the following notation:

where each column is an observation and each row represents a feature in the matrix X, which has dimensions n_x × m. We can also write the output values in matrix form. If you recall our neuron discussion, we have defined a z⁽ⁱ⁾ = w^Tx⁽ⁱ⁾ + b for one observation i. Putting each observation in a column, we can use the following notation:

where we have b = (b b…b). We will define as

where with f (z), we intend the function f be applied element by element to the matrix z.

Now that the formalism is clear, we will prepare the dataset.

Dataset for Our Linear Regression Model

To make things a bit more interesting, let’s use a real dataset. We will use the so-called Boston dataset.³ This contains information collected by the US Census Bureau concerning housing around Boston. Each record in the database describes a Boston suburb or town. The data was drawn from the Boston Standard Metropolitan Statistical Area (SMSA) in 1970. The attributes are defined as follows [3]:

• CRIM: Per capita crime rate by town

• ZN: Proportion of residential land zoned for lots over 25,000 square feet

• INDUS: Proportion of non-retail business acres per town

• CHAS: Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)

• NOX: Nitric oxides concentration (parts per 10 million)

• RM: Average number of rooms per dwelling

• AGE: Proportion of owner-occupied units built prior to 1940

• DIS: Weighted distances to five Boston employment centers

• RAD: Index of accessibility to radial highways

• TAX: Full-value property-tax rate per $10,000

• PTRATIO: Pupil-teacher ratio by town

• B - 1000(Bk - 0.63)^2 - Bk: Proportion of blacks by town

• LSTAT: % lower status of the population

• MEDV: Median value of owner-occupied homes in $1000s

Our target variable MEDV, the one we want to predict, is the median price of the house in $1000s for each suburb. For our example, we don’t have to understand or study the features . My goal here is to show you how to build a linear regression model with what you have learned. Normally, in a machine-learning project, you would first study your input data, check their distribution, quality, missing values, and so on; however, I will skip this part to concentrate on how to implement what you learned with tensorflow.

Note

In machine learning, the variable we want to predict is usually called the target variable.

Let’s import the usual libraries, including sklearn.datasets. Importing the data and getting features and target is very easy with the help of the sklearn.datasets package. You don’t have to download CSV files and import them. Simply run the following code:

import matplotlib.pyplot as plt

%matplotlib inline

import tensorflow as tf

import numpy as np

from sklearn.datasets import load_boston

boston = load_boston()

features = np.array(boston.data)

labels = np.array(boston.target)

Every dataset in the sklearn.datasets package comes with a description. You can check it with the following command:

print(boston["DESCR"])

Now let’s check how many observations and features we have.

n_training_samples = features.shape[0]

n_dim = features.shape[1]

print('The dataset has',n_training_samples,'training samples.')

print('The dataset has',n_dim,'features.')

Linking the mathematical notation with the Python code n_training_samples is m and n_dim is n_x. The code will give the following results:

The dataset has 506 training samples.

The dataset has 13 features.

It is a good idea to normalize each numerical feature defining normalized features according to the formula

where is the average of the j^th feature, and is its standard deviation. This can be easily calculated in numpy with the following function:

def normalize(dataset):

    mu = np.mean(dataset, axis = 0)

    sigma = np.std(dataset, axis = 0)

    return (dataset-mu)/sigma

To normalize our features numpy array, we must simply call the function features_norm = normalize(features). Now each feature contained in the numpy array features_norm will have an average of zero and a standard deviation of one.

Note

It is generally a good idea to normalize the features, so that their average is zero, and the standard deviation is one. Sometimes, some features are much bigger than others and can have a stronger influence on the model, thus bringing wrong predictions. Particular care is needed when the dataset is split into training and test datasets, to have consistent normalizations.

For this chapter, we will simply use all the data for the training, to concentrate on implementation details.

train_x = np.transpose(features_norm)

train_y = np.transpose(labels)

print(train_x.shape)

print(train_y.shape)

The last two prints will give us the dimensions of our new matrices.

(13, 506)

(506,)

The train_x array has dimensions of (13, 506), and that is exactly what we expect. Remember for our discussion that X has dimensions n_x × m.

The training target train_y has dimensions of (506,), which is how numpy describes one-dimensional arrays . tensorflow wants to have dimensions of (1, 506) (remember our previous discussion?), so we must reshape the array in this way:

train_y = train_y.reshape(1,len(train_y))

print(train_y.shape)

and our print statements give us what we need:

(1, 506)

Neuron and Cost Function for Linear Regression

A neuron that can perform linear regression uses the identity activation function. The cost function that needs to be minimized is the MSE (mean square error) that can be written as

where the sum is over all m observations.

The tensorflow code to build this neuron and define the cost function is actually very simple.

tf.reset_default_graph()

X = tf.placeholder(tf.float32, [n_dim, None])

Y = tf.placeholder(tf.float32, [1, None])

learning_rate = tf.placeholder(tf.float32, shape=())

W = tf.Variable(tf.ones([n_dim,1]))

b = tf.Variable(tf.zeros(1))

init = tf.global_variables_initializer()

y_ = tf.matmul(tf.transpose(W),X)+b

cost = tf.reduce_mean(tf.square(y_-Y))

training_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

Note that in tensorflow, you don’t have to explicitly declare the number of observations. You can use None in the code. In this way, you will be able to run the model on any dataset independently of the number of observations, without modifying your code.

In the code, we have indicated the neuron output as y_, because we don’t have a hat in Python. Let me clarify a bit which line of code does what.

• X = tf.placeholder(tf.float32, [n_dim, None]) → contains the matrix X, which must have dimensions n_x × m. Remember that in our code, n_dim is n_x and that m is not declared explicitly in tensorflow. In its place, we use None.

• Y = tf.placeholder(tf.float32, [1, None]) → contains the output values , which must have dimensions 1 × m. Here, this means that instead of m, we use None, because we want to use the same model for different datasets (that will have a different number of observations).

• learning_rate = tf.placeholder(tf.float32, shape=()) → contains the learning rate as a parameter instead of a constant, so that we can run the same model varying it, without creating a new neuron each time.

• W = tf.Variable(tf.zeros([n_dim, 1])) → defines and initializes the weights, w, with zeros. Remember that the weights, w, must have dimensions n_x × 1.

• b = tf.Variable(tf.zeros(1)) → defines and initializes the bias, b, with zero.

Remember that in tensorflow, a placeholder is a tensor that will not change during the learning phase, whereas a variable is one that will change. Weights, w, and bias, b, will be updated during the learning. Now we must define what to do with all those quantities. Remember: We must calculate z. The chosen activation function is the identity function, so z will also be the output of our neuron.

• init = tf.global_variables_initializer() → creates a piece of the graph that initializes the variable and adds it to the graph.

• y_ = tf.matmul(tf.transpose(W),X)+b → calculates the output of the neuron. The output of a neuron is . Because the activation function for linear regression is the identity, the output is . Remember that b being a scalar is not a problem. Python broadcasting will take care of it, expanding it to the right dimensions , to make the sum between a vector w^T X and a scalar b possible.

• cost = tf.reduce_mean(tf.square(y_-Y)) → defines the cost function. tensorflow provides an easy and efficient way of calculating the average—tf.reduce_mean()—that simply performs the sum of all the elements of the tensor and divides it by the number of elements.

• training_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost) → tells tensorflow which algorithm to use to minimize the cost function. In tensorflow language, the algorithms used to minimize the cost function are called optimizers. We now use gradient descent with the given learning rate. Later in the book, other optimizers will be extensively studied.

You will remember from the introduction in Chapter 1 that the previous code will not run any model. It simply defines the computational graph. Let’s define a function that will perform the actual learning and will run our model. It is easier to define it in a function, so that we can rerun it, changing, for example, the learning rate or the number of iterations we want to use.

def run_linear_model(learning_r, training_epochs, train_obs, train_labels, debug = False):

    sess = tf.Session()

    sess.run(init)

    cost_history = np.empty(shape=[0], dtype = float)

    for epoch in range(training_epochs+1):

        sess.run(training_step, feed_dict = {X: train_obs, Y: train_labels, learning_rate: learning_r})

        cost_ = sess.run(cost, feed_dict={ X:train_obs, Y: train_labels, learning_rate: learning_r})

        cost_history = np.append(cost_history, cost_)

        if (epoch % 1000 == 0) & debug:

            print("Reached epoch",epoch,"cost J =", str.format('{0:.6f}', cost_))

    return sess, cost_history

Let’s go through the code again, line by line.

• sess = tf.Session() → creates a tensorflow session.

• sess.run(init) → runs the initialization of the different element of the graphs.

• cost_history = np.empty(shape=[0], dtype = float) → creates an empty vector (for the moment with zero elements) in which the value of our cost function at each iteration is stored.

• for loop... → In this loop, tensorflow performs the gradient descent steps that we have discussed earlier and updates the weights and the bias. In addition, it will save in the array cost_history the value of the cost function each time: cost_history = np.append(cost_history, cost_).

• if (epoch % 1000 == 0)... → Every 1000 epochs we will print the value of the cost function. This is an easy way of checking if the cost function is really decreasing or if nans are appearing. If you perform some initial tests in an interactive environment (such as a Jupyter notebook), you can stop the process if you see that the cost function is not behaving as you expect.

• return sess, cost_history → returns the session (in case you want to calculate something else) and the array containing the cost function values (we will use this array to plot it).

Running the model is as easy as using the call.

sess, cost_history = run_linear_model(learning_r = 0.01,

                                training_epochs = 10000,

                                train_obs = train_x,

                                train_labels = train_y,

                                debug = True)

The output of the command will be the cost function every 1000 epochs (check in the function definition the if, starting with if (epoch % 1000 == 0)).

Reached epoch 0 cost J = 613.947144

Reached epoch 1000 cost J = 22.131165

Reached epoch 2000 cost J = 22.081099

Reached epoch 3000 cost J = 22.076544

Reached epoch 4000 cost J = 22.076109

Reached epoch 5000 cost J = 22.07606

Reached epoch 6000 cost J = 22.076057

Reached epoch 7000 cost J = 22.076059

Reached epoch 8000 cost J = 22.076059

Reached epoch 9000 cost J = 22.076054

Reached epoch 10000 cost J = 22.076054

The cost function clearly decreases and then reaches a value and stays almost constant. You can see a plot of it in Figure 2-14. That is a good sign, indicating that the cost function has reached a minimum. That does not mean that our model is good or that it will give good predictions. This tells us only that the learning has worked efficiently.

It would be nice to be able to visualize graphically how good our fit is. Because we have 13 features, it is not possible to plot the price vs. the other features. However, it is helpful to get a feel of how good the model predicts the observed values. This can be done by plotting our predicted target variable vs. the observed one, as I have done in Figure 2-15 . If we can perfectly predict our target variable, all the points should be on a diagonal line in the plot. The more spread the points are around the line, the worse our model is at predicting. Let’s check how our model is doing.

The points lay reasonably well around the line, so it seems we can predict our price to a certain degree. A more qualitative method for estimating the accuracy of our regression is the MSE itself (which, in our case, is simply our cost function). Whether the value we are obtaining (22.08 in 1000 USD) is good enough depends on the problem you are trying to solve, or the constraint and requirements you have been given.

Satisficing and Optimizing a Metric

We have seen that it is not easy to decide whether a model is good. Figure 2-15 will not allow us to describe quantitively how good (or not good) our model is. For this, we must define a metric.

The easiest way is to set up what is called a single number evaluation metric . That means that you calculate one single number and base your model evaluation on that number. It is easy and very practical. For example, you could use the accuracy or the F1 score, in the case of classification, or the MSE, in the case of regression. Normally, in real life, you will receive goals and constraints for your model. For example, your company may want to predict house prices with an MSE < 20 (in 1000 USD), and your model should be able to run on an iPad, or in less than 1 second. It is useful, therefore, to distinguish between two types of metrics:

• Satisficing metric → Searching through available alternatives until an acceptability threshold is met, for example, code running (RT) time, which minimizes the cost function subject to RT < 1 sec, or choosing among modes the one that has an RT < 1 sec

• Optimizing metric → Searching through available alternatives to maximize a specific metric, for example, choosing the model (or the hyperparameters) that maximize accuracy

Note

If you have several metrics, you should always choose one optimizing and the rest satisficing.

We have written our code to be able to run our model with different parameters. It is very instructive now to do that. Here is how the cost function behaves for three different learning rates: 0.1, 0.01, and 0.001. You can check the different behaviors in Figure 2-16.

As expected for very small learning rates (0.001), the gradient descent algorithm is very slow in finding the minimum, whereas with a bigger value (0.1), the method works quickly. This kind of plot is very useful for giving you an idea of how fast and how good the learning process is going. You will see cases later in the book where the cost function is much less well behaved. For example, when applying dropout regularization, the cost function will not be smooth anymore.

Cost Function

As a cost function, we will use the cross entropy.⁴ The function for one observation is

In the presence of more than one observation, the cost function is the sum over all observations

In Chapter 10, I will provide a complete derivation of logistic regression from scratch, but for the moment, tensorflow will take care of all the details—derivatives, gradient descent implementation, and so on. We only have to build the right neuron, and we will be on our way.

Activation Function

Remember: We want our neuron to output the probability of our observation to be of class 0 or 1. Therefore, we need an activation function that can assume only values between 0 and 1. Otherwise, we cannot regard it as a probability. For our logistic regression, we will use the sigmoid function as the activation function.

The Dataset

To build an interesting model, we will use a modified version of the MNIST dataset. You will find all relevant information from the following link: http://yann.lecun.com/exdb/mnist/ .

The MNIST database is a large database of handwritten digits that we can use to train our model. The MNIST database contains 70,000 images. “The original black and white (bilevel) images from NIST were size normalized to fit in a 20×20 pixel box while preserving their aspect ratio. The resulting images contain grey levels as a result of the anti-aliasing technique used by the normalization algorithm. The images were centered in a 28×28 image by computing the center of mass of the pixels, and translating the image so as to position this point at the center of the 28×28 field” (source: http://yann.lecun.com/exdb/mnist/ ).

For example, we can plot one randomly (see Figure 2-17).

plot_digit(X[36003])

Now comes a very important point. The labels in our dataset as imported will be 1 or 2 (they simply tell you which digit the image represents). However, we will build our cost function with the assumptions that our class’s labels are 0 and 1, so we must rescale our y_train_tr array.

Figure 2-18 shows some of the digits we are dealing with.

tensorflow Implementation

What happened? Suddenly, at some point, our cost function assumes the value nan (not a number). It seems that the model does not do well after a certain point. If the learning rate is too big, or you initialize your weights wrongly, your values for may get very close to zero or one (the sigmoid function assumes values very close to 0 or 1 for very big negative or positive values of z). Remember that in the cost function , you have the two terms tf.log(y_) and tf.log(1-y_), and because the log function is not defined for a value of zero, if y_ is 0 or 1, you will get a nan, because the code will try to evaluate tf.log(0). As an example, we can run the model with a learning rate of 2.0. After only one epoch, you already will get a nan value for the cost function. And it is easy to understand why, if you print out the value for b before and after the first training step. Simply modify your model code and use the following version:

def run_logistic_model(learning_r, training_epochs, train_obs, train_labels, debug = False):

    sess = tf.Session()

    sess.run(init)

    cost_history = np.empty(shape=[0], dtype = float)

    for epoch in range(training_epochs+1):

        print ('epoch: ', epoch)

        print(sess.run(b, feed_dict={X:train_obs, Y: train_labels, learning_rate: learning_r}))

        sess.run(training_step, feed_dict = {X: train_obs, Y: train_labels, learning_rate: learning_r})

        print(sess.run(b, feed_dict={X:train_obs, Y: train_labels, learning_rate: learning_r}))

        cost_ = sess.run(cost, feed_dict={ X:train_obs, Y: train_labels, learning_rate: learning_r})

        cost_history = np.append(cost_history, cost_)

        if (epoch % 500 == 0) & debug:

            print("Reached epoch",epoch,"cost J =", str.format('{0:.6f}', cost_))

    return sess, cost_history

You see how b goes from 0 to -0.05966223 and then to nan? Therefore, z = w^TX + b turns into nan, then y = σ(z) also turns into nan, and, finally, the cost function, being a function of y, will also result in nan. This is simply because the learning rate is way too big.

What is the solution? You should try a different (read: much smaller) learning rate.

No more nan in our output. You can see a plot of the cost function in Figure 2-19. To evaluate our model, we must choose an optimizing metric (as discussed before). For a binary classification problem, a classical metric is the accuracy (which we can indicate with a) that can be understood as a measure of the difference between a result and its “true” value. Mathematically, it can be calculated as

With this model, we reach an accuracy of 98.6%. Not bad for a network with only one neuron.

You could also try to run the previous model (with a learning rate of 0.005) for more epochs . You will discover that at about 7000 epochs, the nan will reappear. The solution here would be to reduce the learning rate with an increasing number of epochs. A simple approach, such as halving the learning rate every 500 epochs, will get rid of the nans. I will discuss a similar approach in more detail later in the book.