API Stack

After the introduction of estimators, the new API stack for TensorFlow can be depicted as shown in Figure 6-1.

So far, you have mostly used mid-level APIs; the use of low-level APIs becomes a rare necessity when you need a finer control over the model development. Now, after learning the estimator API, you may not even use the mid-level APIs for your model developments. But then what happens to the models which you have already developed – can they benefit from using this API? And if so, how to make them use this API? Fortunately, the TensorFlow team has developed an interface that allows you to migrate the existing models to use the estimator interface. Not only that, they have created a few estimators on their own to get you quickly started. These are called premade estimators. They are not just the start-up points. They are fully developed and tested for you to use in your immediate projects. If at all these do not meet your purpose or if you want to migrate existing models to facilitate the benefits offered by the estimator, you can develop custom estimators using the estimator API. Before you learn how to use premade estimators and to build your own ones, let me discuss its benefits in more detail.

Estimator Benefits

These need not be the only benefits, but are certainly the most important ones. I will now discuss each one in further detail. To understand the discussion, keep the estimator interface depicted in Figure 6-2 in your mind.

As you can see in Figure 6-2, the estimator class provides three interface methods for training, evaluation, and predictions. So, once you develop an estimator object, you will be able to call the train, evaluate, and predict methods on the same object. Note that you need to send different datasets for each. This is achieved with the help of an input function. I have explained the structure of this input function in more detail later in this section. It is sufficient to say at this point that the introduction of the Input function simplifies your experimentation with different datasets.

The checkpoints created during training allow you to roll back to a known state and continue the training from this checkpoint. This would save you lots of training time, especially when the errors occur toward the fag end of an epoch. This also makes the debugging quicker. After the training is over, the summary logs created during evaluation can be visualized on TensorBoard, giving you quick insights on how good the model is trained.

After the model is trained to your full satisfaction, next comes the task of deployment. Deploying a trained model on CPU/GPU/TPU, even mobiles and the Web, and a distributed environment previously required several code changes. If you use an estimator, you will deploy the trained model as is or at the most with minimal changes on any of these platforms.

Having said all these benefits, let us first look at the types of estimators.

Estimator Types

The classification can be visualized in the diagram shown in Figure 6-3.

The premade estimators are like a model in a box where the model functionality is already written by the TensorFlow team. On the other hand, in custom estimators, you are supposed to provide this model functionality. In both cases, the estimator object that you create will have a common interface for training, evaluation, and prediction. Both can be exported for serving in a similar way.

The TensorFlow library provides a few premade estimators for your immediate go; if these do not meet your purpose and/or you want to migrate your existing models to get the benefits of estimators, you would create your own estimator class. All estimators are subclasses of the tf.estimator.Estimator base class.

The DNNClassifier, LinearClassifier, and LinearRegressor are the few examples of premade estimators. The DNNClassifier is useful for creating classification models based on dense neural networks, while the LinearRegressor is for handling linear regression problems. You will learn how to use both these classes in the upcoming sections in this chapter.

As part of building a custom estimator, you will be converting an existing Keras model to an estimator. Doing so will enable you to take advantage of the several benefits offered by estimators, which you have seen earlier. You will be building a custom estimator for the wine quality regression model that you developed in the previous chapter. Finally, I will also show you how to build a custom estimator based on tfhub modules.

To work with the estimators, you need to understand two new concepts – Input functions and Feature columns. The input function creates a data pipeline based on tf.data.dataset that feeds the data to the model in batches for both training and evaluation. You may also create a data pipeline for inference. I will show you how to do this in the DNNClassifier project. Feature columns specify how the data is to be interpreted by the estimator. Before I discuss these Input function and Feature column concepts, I will give you an overview of the development of an estimator-based project.

Workflow for Estimator-Based Projects

As part of the learning that you have done in all previous chapters, you are definitely familiar with many steps in the preceding workflow. What needs attention is defining Features columns and the Input function. I will now describe these requirements.

Features Column

Features column provide a bridge between raw data and an estimator. It transforms a diverse range of raw data into formats that estimators need. You use the tf.feature_column module to build a list of Feature columns. This then becomes an input to the estimator constructor. The estimator object uses this list while interpreting the data coming from the Input function. The whole process can be visualized as shown in Figure 6-4.

As seen in the central block of Figure 6-4, the list of Feature columns is as follows:

• Fixed acidity (numeric)

• Volatile acidity (numeric)

• Citric acid (numeric)

• …

These are the features of our wine quality model that you developed in the previous chapter. Each element in the list is of type tf.feature_column.numeric_column. The following code snippet tells you how to build such a list:

# build numeric features array

numeric_feature = []

for col in numeric_columns:

 numeric_feature.append

    (tf.feature_column.numeric_column(key=col))

Sometimes, your dataset may contain categorical fields which you want to use as the features in your model building. The following code snippet tells you how to build a Features column for such categorical fields:

categorical_features = []

for col in categorical_columns:

 vocabulary = data[col].unique()

 cate = tf.feature_column.

        categorical_column_with_vocabulary_list

            (col, vocabulary)

 categorical_features.append

        (tf.feature_column.indicator_column(cate))

Note we first obtain the vocabulary by calling the categorical_column_with_vocabulary_list method and then append the indicator columns to the list.

This list is passed as a parameter to the estimator constructor, as seen in Figure 6-4. If your model requires both numerical and categorical features, you will need to append both to your destination Features column.

The left block in Figure 6-4 shows data which is actually built by an Input function. This data will be input to the estimator’s train/evaluate/predict method calls.

Next, I will describe how to write the Input function.

Input Function

The purpose of the Input function is to return the following two objects for the use of our estimator model object:

• A dictionary of feature names (keys) and Tensors or Sparse Tensors (values) containing the corresponding feature data

• A Tensor containing one or more labels

The basic skeleton looks like this:

def input_fn (dataset):

    # create dictionary with feature names

    # and Tensors with corresponding data

    # create Tensor for label data

    return dictionary, label

You may write separate functions based on this prototype for training/evaluation/inferencing.

I will now present you the practical implementation of the Input function taken from the example that is later discussed in this chapter. This would further clarify the concepts behind the Input function in your minds.

The input function is a Python function with the following prototype:

def input_fn(features, labels, training =

              True, batch_size = 32):

Here the features and labels arguments represent tensors containing the features and the labels data. Within the function, the data is converted into tensor by calling the function from_tensor_slices of the tf.data.Dataset module.

#converts inputs to a dataset

Dataset = tf.data.Dataset.from_tensor_slices

            ((dict(features),labels))

The input parameter to the function is a Python dictionary of features and the corresponding labels. Finally, the function returns this data to the caller in batches by using the batch method of tf.data.Dataset.

return dataset.batch(batch_size)

The function structure is visualized in Figure 6-5 for further understanding.

The whole matter may look quite complicated; a practical example will help in clarifying the entire implementation, and that is what I am going to do next.

Loading Data

You will see the output shown in Figure 6-6.

Each image is of size 8 by 8 pixels.

Preparing Data

If you check the shape of data, you would notice that there are 1797 images, each consisting of totally 64 pixel values.

At this point, we are now ready to define our input function for the estimator.

Estimator Input Function

The first parameter defines the features data, the second defines the target values, and the third parameter specifies whether the data is to be used for training or evaluation. The data is processed in batches – the batch size is decided by the last parameter.

The data is converted into tensors by calling the from_tensor_slices method. The method takes an input consisting of the dictionary of features and the corresponding labels.

The shuffle method shuffles the dataset. We specify the buffer size of 1000 for shuffling. To handle datasets that are too large to fit in memory, the shuffling is done in batches of data. If the buffer size is greater than the number of data points in the dataset, you get a uniform shuffle. If it is 1, you get no shuffling at all.

Now comes the time to create an estimator instance.

Creating Estimator Instance

The constructor takes five parameters. The first parameter defines the network architecture. Here, we have defined three hidden layers in our architecture; the first layer contains 256 neurons, the second layer contains 128, and the third layer contains 64. The second parameter specifies the list of features that the data will have. Note that we have earlier created the feature_columns vector, so this is set as a default parameter. The third parameter specifies the optimizer to be used, set to Adagrad by default here. Adagrad is an algorithm for gradient-based optimization that does just this: It adapts the learning rate to the parameters, performing smaller updates. The n_classes parameter defines the number of output classes. In our case, it is 10, which is the number of digits 0 through 9. The last parameter model_dir specifies the directory name where the logs will be maintained.

Next comes the important part of model training.

Model Training

The method takes our input_fn as the parameter. The input_fn itself takes the features and labels data as the first two parameters. The training parameter is set to True so that the data will be shuffled. The number of steps is defined to be 2000. Let me explain what this means. In machine learning, an epoch means one pass over the entire training set. A step corresponds to one forward pass and back again. If you do not create batches in your dataset, a step would correspond to exactly one epoch. However, if you have split the dataset into batches, an epoch will contain many steps – note that a step is an iteration over a single batch of data. You can compute the total number of epochs executed during the full training cycle with the formula:

Number_of_epochs = (batch_size * number_of_steps) / (number_of_training_samples)

In our current example, the batch size is 32, the number of steps is 2000, and the number of training samples is 1707. Thus, it will take (32 * 2000) / 1707, that is, 38 epochs, to complete the training.

Model Evaluation

Note that the training parameter is set to false. You can print the evaluation result by simply printing the value of eval_result. The output is shown in the screenshot given in Figure 6-7.

A sample loss plot from the logs is shown in Figure 6-8.

Next, you will learn how to predict an unseen data using our trained estimator.

Predicting Unseen Data

The input function uses the test data that we create for testing the two unseen data items.

Experimenting Different ANN Architectures

The execution produced the evaluation results shown in Figure 6-9.

The evaluation results for this configuration are shown in Figure 6-10.

Thus, you can easily experiment with several model architectures. You may also experiment with different datasets – for example, by changing the number of features included in the Features column list. Once satisfied with the model’s performance, you can save it to a file and then take the saved file straightaway to the production server for everybody’s use.

Project Source

Now, as you have seen how to use a dense neural network classification estimator, I will show you how to use a premade classifier for regression problems.

Project Description

The regression problem that we are trying to solve in this project is to determine the estimated selling price of a house in Boston. We will use the Airbnb dataset for Boston (www.kaggle.com/airbnb/boston) for this purpose. The dataset is multicolumnar and the various columns must be examined carefully for their suitableness as a feature in model development. Thus, for the regression model development, you would need to have a stringent preprocessing of data so that we sufficiently reduce the number of features and yet achieve a great amount of accuracy in predicting the house’s price.

Creating Project

Loading Data

The data is read into a pandas dataframe. You may check the data size by calling its shape method. You will know that there are 3585 rows and whopping 95 columns. Finding out a regression relationship between these 95 columns is not an easy task even for a highly skilled data analyst. That’s where once again you will realize that neural networks can come to your aid.

Out of the 95 columns that we have in the database, obviously not all will be of use to us in training our model. So, we need some data cleansing to remove the unwanted columns and standardize the remaining. We now do the data preprocessing as described further.

Features Selection

The first thing that we do is to list out all columns, which we do by calling the columns method on the data. When you do this, you will see the output in Figure 6-11.

If you prefer, you may use data.info() to get a better understanding of the data structure. You can easily notice that there are many fields that may be irrelevant to us. To get a better idea on what columns can be eliminated, get the data description by calling the describe method. The screen output is shown in Figure 6-12.

After selecting our features, we need to ensure that the data it contains is clean.

Data Cleansing

We first check if the data contains any null values. You do this by using the isnull function and taking the sum on its output. The screenshot in Figure 6-13 shows the columns containing null values.

We can examine the data by printing a few records. The command and its output are shown in Figure 6-14.

data.head()

In the loop, we also replace the null values with the field’s mean value.

Next, we look into the categorical column – property_type. As this is a categorical column, we cannot simply replace null values with a mean. We need to find an appropriate value for the replacement of the null fields. For this, let us check what the unique values are in this column. The unique values can be seen in Figure 6-15.

We can check the frequency of each value by calling the value_counts method. This is shown in Figure 6-16.

Creating Datasets

The histogram is shown in Figure 6-17.

After the transformation, the distribution looks like the one shown in Figure 6-18.

At this point, your data processing is complete. We now create the training and testing datasets.

We reserve 20% of our data for testing. The last thing that we need to do before building our estimator is to create Feature columns, which we do next.

Building Feature Columns

We include all columns with type np.number from our feature list and create a new list called numeric_columns. When you print this list, you would see the output shown in Figure 6-19.

If we print this list, you would see the output shown in Figure 6-20.

When you print this list, you will see the output shown in Figure 6-21.

We first extracted the unique values in the categorical column, then built the vocabulary list, and finally added it to the categorical_features list. If you print this list, you would see the output shown in Figure 6-22.

Defining Input Function

As seen in the previous project, the function takes features and labels as the first two parameters. The training parameter determines whether the data is to be used for training; if so, the data is shuffled. The from_tensor_slices function call creates a tensor for the data pipeline to our model. The function returns the data in batches.

Creating Estimator Instance

Note we use the LinearRegressor class as a premade estimator for regression. The first parameter is the Features columns, which specify the combined numerical and categorical columns list. The second parameter is the name of the directory where the logs would be maintained.

Model Training

The input function takes the features data in the xtrain dataset and the target values in the ytrain dataset. The training parameter is set to true that enables data shuffling. The number of steps would decide the number of epochs in the training phase.

Model Evaluation

A loss curve displayed on the TensorBoard is shown in Figure 6-23.

Project Source

This project has demonstrated how to use a premade LinearRegressor for solving the linear regression problems. Next, I will describe how to build a custom estimator of your own.

Creating Project

Loading Data

As you are already familiar with the data, I am not going to describe it again. I will not also include any data preprocessing here. I will straightaway move on to features selection.

Creating Datasets

Defining Model

The output consists of a single neuron which emits a float value for the wine quality. Note that we are treating this problem as a regression problem like the wine quality problem discussed in the previous chapter.

We consider this as a regression problem and thus use mse for our loss function. We will use this compiled model in the estimator instantiation. Before that, we define the input function.

Defining Input Function

The function definition is identical to the earlier example and does not need any further explanation.

Now, it is time for us to convert our model to the estimator.

Model to Estimator

The function takes two arguments; the first argument specifies the previously existing compiled Keras model, and the second argument specifies the folder name where the logs would be maintained during the training. The function call returns an estimator instance which you would use like in the earlier example to train, evaluate, and predict.

Model Training

For training, the input function takes the features data in xtrain and the labels in ytrain. The number of steps is set to 2000, which decides the number of training epochs. The mode set by the training argument takes the default value of True.

Evaluation

Note that we pick up the metrics from the earlier specified logs folder.

Project Source

Creating Project

Importing VGG16

Building Your Model

We build a sequential model by taking the VGG16 as the base layer. On top of this, we add a pooling layer, followed by two Dense layers. The last layer is a binary output layer.

You may print the summaries of the two models to see the changes you have made.

The output is shown in Figure 6-24.

As you can see in Figure 6-24, the VGG16 models have several hidden layers, and the total number of trainable parameters equals more than 14 million. You can imagine the time and resources it would have taken to train this model. Obviously, in your application, when you use this model, you will never think of retraining this model.

The output is shown in Figure 6-25.

In your model, you have only about 100,000 trainable parameters.

Compiling Model

Creating Estimator

Before you train the estimator, you need to process your images.

Processing Data

Training/Evaluation

Note that I have used the same training dataset with a different step size for model evaluation, as no separate testing dataset is available for the purpose.

Project Source

This trivial example has demonstrated how you can use well-trained models into your own models.