In order to understand how TensorBoard helps, let's initialize a model structure as follows, one that is bound not to work:

Note that, in this code snippet, the validation accuracy is only around 19%.

The reason for such a low validation accuracy is that the input dataset is not scaled and we are performing ReLU activation on top of an unscaled dataset.

Note that, in the preceding code, we are storing the logs of the model run in the directory logs/tensor_new6 (the sub-directory could be named anything).

Once the logs are stored in this location, we start TensorBoard as follows:

The preceding code starts TensorBoard, which looks like this:

Note that, by default, the output gives a measure of the scalars, that is, the accuracy and loss values of both the train and test datasets.

The outputs can be visualized adjacent to each other using the regular expression .* in Filter tags, as follows:

Note that the first two graphs in this screenshot represent the accuracy and loss of train datasets, while the next two graphs represent the accuracy and loss of validation datasets.

When we look at the histogram of weights and bias across various layers, we learn that the weights and biases do not change across epochs:

This is an indication that no learning is happening in the network architecture.

The same can be noted when we look at the distribution of weights and biases across epochs in a different tab:

From this screenshot, we can conclude why the accuracy of the model is so low; it's because the model is not able to update the weights.

Now, by clicking on the GRAPHS tab, let us explore whether the model was initialized correctly:

You should notice that the training block is connected to every other block in the graph. This is because, in order to compute the gradients, one needs to connect to every variable in the graph (as every variable contains weights that need to be adjusted).

Let us, for now, remove the training block from the graph. This is done by right-clicking on the training block, as follows:

The resultant graph after removing the training block is as follows:

Note that the input layer is connected to hidden layer, which in turn is connected to output layer, from which the metrics and loss are calculated. Let us explore the connections by double-clicking on the individual blocks, as follows:

A zoom-in of these connections helps us understand the shapes at various blocks:

The input layer is (784) in dimension, as there could be any number of input samples but each of them are 784-dimensional. Similarly, the kernel (weight matrix) is 784 x 784 in dimensions and the bias would have 784 initialized values, and so on.

Note that, in the preceding diagram, we take the values in input layer and perform matrix multiplication with the kernel that is initialized using random_normal initialization. Also note that random_normal initialization is not connected to the training block, while the kernel block is connected to the training block.

Let us also find out whether the output layer is connected to all the relevant blocks per expectations. Given that the graph looks very complicated, we can use another functionality provided in TensorBoard: Trace inputs. Trace inputs help in highlighting only those blocks that are connected to any block of interest. It is activated by selecting the block of interest and toggling the switch in the left-hand pane, as follows:

Now all the connections look fine, but the gradients are still not getting updated; let us change the activation function to sigmoid and then check the weight histograms:

We build a neural network with sigmoid activation as follows:

Once the neural network structure is defined and compiled, let us fit the model as follows:

In order to open TensorBoard, we will execute the following code:

from google.datalab.ml import TensorBoard as tb
tb.start('./logs/tensor_neww3')

We will then receive the following output:

At the same time, we should notice that the accuracy and loss metrics have improved considerably:

One would also be able to visualize the histogram of gradients by specifying write_grads=True in the TensorBoard function. The output would then be as follows: