© Umberto Michelucci 2018
Umberto MichelucciApplied Deep Learninghttps://doi.org/10.1007/978-1-4842-3790-8_1

1. Computational Graphs and TensorFlow

Umberto Michelucci1 
(1)
toelt.ai, Dübendorf, Switzerland
 

Before we delve into the extended examples later in this book, you will require a Python environment and a working knowledge of TensorFlow. Therefore, this chapter will show you how to install a Python environment ready to run the code in this book. Once that’s in place, I’ll cover the basics of the TensorFlow machine-learning library.

How to Set Up Your Python Environment

All the code in this book was developed with the Python distribution Anaconda and Jupyter notebooks. To set up Anaconda, first download and install it for your operating system. (I used Windows 10, but the code is not dependent on this system. Feel free to use a version for Mac, if you prefer.) You may retrieve Anaconda by accessing https://anaconda.org/ .

On the right side of the web page, you will find a Download Anaconda link, as shown in Figure 1-1 (top right).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig1_HTML.jpg
Figure 1-1

On the Anaconda web site, at the top right of the page, you will find a link to download the required software

Simply follow the instructions to install it. When you start it after the installation, you will be presented with the screen shown in Figure 1-2. In case you don’t see this screen, simply click the Home link on the navigation pane at left.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig2_HTML.jpg
Figure 1-2

The screen you see when you start Anaconda

Python packages (like NumPy) are updated regularly and very often. It can happen that a new version of a package makes your code stop working. Functions are deprecated and removed, and new ones are added. To solve this problem, in Anaconda, you can create what is called an environment. This is basically a container that holds a specific Python version and specific versions of the packages you decide to install. With this, you can have a container for Python 2.7 and NumPy 1.10 and another with Python 3.6 and NumPy 1.13, for example. You may have to work with existing code that has been developed with Python 2.7, and, therefore, you must have a container with the right Python version. But, at the same time, it may be that you require Python 3.6 for your projects. With containers, you can ensure all this at the same time. Sometimes different packages conflict with each other, so you must be careful and avoid installing all packages you find interesting in your environment, especially if you are developing packages under a deadline. Nothing is worse than discovering that your code is no longer working, and you don’t know why.

Note

When you define an environment, try to install only the packages you really need, and pay attention when you update them, to make sure that any upgrade does not break your code. (Remember: Functions are deprecated, removed, added, or frequently changed.) Check the documentation of updates before upgrading, and do so only if you really need the updated features.

You can create an environment from the command line, using the conda command , but to get an environment up and running for our code, everything can be done from the graphical interface. This is the method I will explain here, because it is the easiest. I suggest that you read the following page on the Anaconda documentation, to understand in detail how to work within its environment: https://conda.io/docs/user-guide/tasks/manage-environments.html .

Creating an Environment

Let’s get started. First, click the Environments link (the one that has a small icon representing a box) from the left navigation panel (Figure 1-3).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig3_HTML.jpg
Figure 1-3

To create a new environment, you first must go into the Environments section of the application, by clicking the related link from the left navigation pane (indicated in the figure by a black rectangle)

Then click the Create button in the middle navigation pane (as indicated in Figure 1-4).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig4_HTML.jpg
Figure 1-4

To create a new environment, you must click the Create button (indicated with a plus sign) from the middle navigation pane. In the figure, a red arrow indicates the location of the button.

When you click the Create button, a small window will pop up (see in Figure 1-5).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig5_HTML.jpg
Figure 1-5

The window you will see when you click the Create button indicated in Figure 1-4

You can choose any name. In this book, I used the name tensorflow. As soon as you type a name, the Create button becomes active (and green). Click it and wait a few minutes until all the necessary packages are installed. Sometimes, you may get a pop-up window telling you that a new version of Anaconda is available and asking if you want to upgrade. Feel free to click yes. Follow the on-screen instructions until Anaconda navigator starts again, in the event you received this message and clicked yes.

We are not done yet. Click again the Environments link on the left navigation pane (as shown in Figure 1-3), then click the name of the newly created environment. If you’ve followed the instructions until now, you should see an environment named “tensorflow.” After a few seconds, you will see on the right panel a list of all installed Python packages that you will have at your disposal in the environment. Now we must install some additional packages: NumPy, matplotlib, TensorFlow, and Jupyter. To do this, first select Not installed from the drop-down menu , as illustrated in Figure 1-6.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig6_HTML.jpg
Figure 1-6

Selecting the Not installed value from the drop-down menu

Next, in the Search Packages field, type the package name you want to install (Figure 1-7 shows that numpy has been selected).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig7_HTML.jpg
Figure 1-7

Type “numpy” in the search field, to include it in the package repository

Anaconda navigator will automatically show you all packages that have the word numpy in the title or in the description. Click the small square to the left of the name of the package that has the name numpy. It will become a small downward-pointing arrow (indicating that it is marked for installation). Then you can click the green Apply button at the lower right corner of the interface (see Figure 1-8).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig8_HTML.jpg
Figure 1-8

After you have selected the numpy package for installation, click the green Apply button. The button is at the lower right of the interface.

Anaconda navigator is smart enough to determine if other packages are needed by numpy. You may get an additional window asking if it is OK to install the additional packages . Just click Apply. Figure 1-9 shows what this window looks like.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig9_HTML.jpg
Figure 1-9

When installing a package, the Anaconda navigator will check if what you want to install depends on other packages that are not installed. In such a case, it will suggest that you install the missing (but necessary) packages from an additional window. In our case, 52 additional packages were required by the NumPy library on a newly installed system. Simply click Apply to install all of them.

You must install the following packages to be able to run the code in this book. (I added within parentheses the versions I used for testing the code in this book; subsequent versions are fine.)

  • numpy (1.13.3): For doing numerical calculations

  • matplotlib (2.1.1): To produce nice plots, as the ones you will see in this book

  • scikit-learn (0.19.1): This package contains all the libraries related to machine learning and that we use, for example, to load datasets.

  • jupyter (1.0.0): To be able to use the Jupyter notebooks

Installing TensorFlow

Installing TensorFlow is slightly more complex. The best way to do this is to follow the instructions given by the TensorFlow team, available at the following address: www.tensorflow.org/install/ .

On this page, click your operating system, and you will receive all the information you need. I will provide here instructions for Windows, but the same can be done using a macOS or Ubuntu (Linux) system. The installation with Anaconda is not officially supported but works perfectly (it is community supported) and is the easiest way to get up and running and check the code in this book. For more advanced applications, you may want to consider other installation options. (For that, you will have to check the TensorFlow web site.) To begin, go to the Start menu in Windows and type “anaconda.” Under Apps, you should see the Anaconda Prompt item, as in Figure 1-10.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig10_HTML.jpg
Figure 1-10

If you type “anaconda” in the Start menu search field in Windows 10, you should see at least two entries: Anaconda Navigator, where you created the TensorFlow environment, and Anaconda Prompt

Start the Anaconda Prompt. A command line interface should appear (see Figure 1-11). The difference between this and the simple cmd.exe command prompt is that here, all the Anaconda commands are recognized, without setting up any Windows environment variable.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig11_HTML.jpg
Figure 1-11

This is what you should see when you select Anaconda Prompt. Note that the user name will be different. You will see not umber (my username) but your username.

At the command prompt , you first must activate your new “tensorflow” environment. This is necessary to let the Python installation know in which environment you want to install TensorFlow. To do this, simply type the following command: activate tensorflow. Your prompt should change to this: (tensorflow) C:Usersumber>.

Remember: Your username will be different (you will see not umber in your prompt but your username). I will assume here that you will install the standard TensorFlow version that uses only the CPU (and not the GPU) version. Just type the following command: pip install --ignore-installed --upgrade tensorflow.

Now let the system install all the necessary packages. This may take a few minutes (depending on several factors, such as your computer speed or your Internet connection). You should not receive any error message. Congratulations! Now you have an environment in which you can run code using TensorFlow.

Jupyter Notebooks

The last step to be able to type code and let it run is to start a Jupyter notebook. The Jupyter notebook can be described (according to the official web site) as follows:

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.

It is widely used in the machine learning community, and it is a good idea to learn how to use it. Check the Jupyter project web site at http://jupyter.org/ .

This site is very instructive and includes many examples of what is possible. All the code you find in this book has been developed and tested using Jupyter notebooks. I assume that you already have some experience with this web-based development environment. In case you need a refresher, I suggest you check the documentation you can find on the Jupyter project web site at the following address: http://jupyter.org/documentation.html .

To start a notebook in your new environment, you must go back to Anaconda navigator in the Environments section (see Figure 1-3). Click the triangle to the right of your “tensorflow” environment (in case you have used a different name, you will have to click the triangle to the right of your new environment), as shown in Figure 1-12. Then click Open with Jupyter Notebook.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig12_HTML.jpg
Figure 1-12

To start a Jupyter notebook in your new environment, click the triangle to the right of the “tensorflow” environment name and click Open with Jupyter Notebook

Your browser will start with a list of all the folders you have in your user folder. (If you are using Windows, this is usually located under c:Users<YOUR USER NAME>, within which you must substitute <YOUR USER NAME> with your username.) From there, you should navigate to a folder where you want to save your notebook files and from which you can create a new notebook by clicking the New button, as illustrated in Figure 1-13.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig13_HTML.jpg
Figure 1-13

To create a new notebook, click the New button located at the top-right corner of the page and select Python 3

A new page will open that should look like the one in Figure 1-14.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig14_HTML.jpg
Figure 1-14

When you create an empty notebook, an empty page will open that should look like this one

For example, you can type the following code in the first “cell” (the rectangular box in which you can type).
a=1
b=2
print(a+b)
To evaluate the code , simply press Shift+Enter, and you should see the result (3) immediately (Figure 1-15).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig15_HTML.jpg
Figure 1-15

After typing some code in the cell, pressing Shift+Enter will evaluate the code in the cell

The preceding code gives the result of a+b, that is, 3. A new empty cell for you to type in is automatically created after the result is given. For more information on how to add comments, equations, inline plots, and much more, I suggest that you visit the Jupyter web site and check the documentation provided.

Note

In case you forget the folder your notebook is in, you can check the URL of the page. For example, in my case, I have http://localhost:8888/notebooks/Documents/Data%20Science/Projects/Applied%20advanced%20deep%20learning%20(book)/chapter%201/AADL%20-%20Chapter%201%20-%20Introduction.ipynb. You will notice that the URL is simply a concatenation of the folders in which the notebook is located, separated by a forward slash. A %20 character simply means a space. In this case, my notebook is in the folder: Documents/Data Science/Projects/… and so forth. I often work with several notebooks at the same time, and it is useful to know where each notebook is located in case you forget (I sometimes do).

Basic Introduction to TensorFlow

Before starting to use TensorFlow, you must understand the philosophy behind it. The library is heavily based on the concept of computational graphs, and unless you understand how those work, you cannot understand how to use the library. I will give you a quick introduction to computational graphs and show you how to implement simple calculations with TensorFlow. At the end of the next section, you should understand how the library works and how we will use it in this book.

Computational Graphs

To understand how TensorFlow works, you must understand what a computational graph is. A computational graph is a graph in which each node corresponds to an operation or a variable. Variables can feed their values into operations, and operations can feed their results into other operations. Usually, nodes are plotted as a circle (or ellipsis), with variable names or operations inside, and when one node’s value is the input to another node, an arrow goes from one to another. The simplest graph that can exist is simply one with a single node that is simply a variable. (Remember: A node can be a variable or an operation.) The graph in Figure 1-16 simply computes the value of the variable x.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig16_HTML.png
Figure 1-16

The simplest graph that we can build, showing a simple variable

Not very interesting! Now let’s consider something slightly more complex, such as the sum of two variables x and y: z = x + y. It can be done as in the following graph (Figure 1-17):
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig17_HTML.png
Figure 1-17

A basic computational graph for the sum of two variables

The nodes at the left of Figure 1-17 (the ones with x and y inside) are variables, while the bigger node indicates the sum of the two variables. The arrows show that the two variables, x and y, are inputs to the third node. The graph should be read (and computed) in topological order, which means that you should follow the arrows that will indicate in which order you have to compute the different nodes. The arrow will also tell you the dependencies between the nodes. To evaluate z, you first must evaluate x and y. We can also say that the node that performs the sum is dependent on the input nodes.

An important aspect to understand is that such a graph only defines the operations (in this case, the sum) to perform on two input values (in this case, x and y) to obtain a result (in this case, z). It basically defines the “how.” You must assign values to the input x and y and then perform the sum to obtain z. The graph will give you a result only when you evaluate all the nodes.

Note

In this book, I will refer to the “construction” phase of a graph, when defining what each node is doing, and the “evaluation” phase, when we will actually evaluate the related operations.

This is a very important aspect to understand. Note that the input variables do not need to be real numbers. They can be matrices, vectors, and so on. (We will mostly use matrices in this book.) A slightly more complex example can be found in Figure 1-18 and uses a graph to calculate the quantity A(x + y), given three input quantities: x,  y, and A.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig18_HTML.png
Figure 1-18

A computational graph to calculate the quantity A(x + y), given three input quantities: x, y, and A

We can evaluate this graph by assigning values to the input nodes (in this case, x, y, and A) and evaluate the nodes through the graph. For example, if you consider the graph in Figure 1-18 and assign the values x = 1, y = 3, and A = 5, we will get the result b = 20 (as plotted in Figure 1-19).
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig19_HTML.png
Figure 1-19

To evaluate the graph in Figure 1-18, we must assign values to the input nodes x, y, and A and then evaluate the nodes through the graph

A neural network is basically a very complicated computational graph, in which each neuron is composed by several nodes in the graph that feed its output to a certain number of other neurons, until a certain output is reached. In the next section, we will build the simplest neural network of all: one with a single neuron. Even with such a simple network, we will be able to do some pretty fun stuff.

TensorFlow allows you to build very complicated computational graphs very easily. And by construction, it separates their evaluation from the construction. (Remember that to compute a result, you must assign values and evaluate all the nodes.) In the next sections, I will show you how this works: how to build computational graphs and how to evaluate them.

Note

Remember that tensorflow first builds a computational graph (in the so-called construction phase) but does not automatically evaluate it. The library keeps the two steps separate, so that you can compute your graph several times with different inputs, for example.

Tensors

The basic unit of data handled by tensorflow is—try to guess from its name—a tensor. A tensor is simply a collection of primitive types (such as, for example, floating numbers) shaped as an n-dimensional array. Here are some examples (with relative Python definitions) of tensors:

  • 1 → a scalar

  • [1,2,3] → a vector

  • [[1,2,3], [4,5,6]] → a matrix or a two-dimensional array

A tensor has a static type and dynamic dimensions. You cannot change its type while evaluating it, but the dimensions can be changed dynamically before evaluating it. (Basically, you declare the tensors without specifying some of the dimensions, and tensorflow will infer the dimensions from the input values.) Usually, one talks about the rank of a tensor, which is simply the number of dimensions of the tensor (whereas a scalar is intended to have a rank of 0). Table 1-1 may help in understanding what the different ranks of tensors are.
Table 1-1

Examples of Tensors with Ranks 0, 1, 2, and 3

Rank

Mathematical Entity

Python Example

0

Scalar (for example, length or weight)

L=30

1

A vector (for example, the speed of an object in a two-dimensional plane)

S=[10.2,12.6]

2

A matrix

M=[[23.2, 44.2], [12.2, 55.6]]

3

A 3D matrix (with three dimensions)

C = [[[1],[2]],[[3],[4]], [[5], [6]]]

Supposing you import tensorflow with the statement import tensorflow as tf, the basic object, a tensor, is the class tf.tensor. A tf.tensor has two properties:

  • data type (for example, float32)

  • shape (for example, [2,3], meaning a tensor with two rows and three columns)

An important aspect is that each element of a tensor always has the same data type, while the shape need not be defined at declaration time. (This will be clearer in the practical examples in the next chapters.) The main types of tensors (there are more) that we will see in this book are

  • tf.Variable

  • tf.constant

  • tf.placeholder

The tf.constant and the tf.placeholder values are, during a single-session run (more on that later), immutable. Once they have a value, they will not change. For example, a tf.placeholder could contain the dataset you want to use for training your neural network. Once assigned, it will not change during the evaluation phase. A tf.Variable could contain the weights of your neural networks. They will change during training, to find their optimal values for your specific problem. Finally, a tf.constant will never change. I will show you in the next section how to use the three different types of tensors and what aspect you should consider when developing your models.

Creating and Running a Computational Graph

Let’s start using tensorflow to create a computational graph.

Note

Remember: We always keep the construction phase (when we define what a graph should do) and its evaluation (when we perform the calculations) separate. tensorflow follows the same philosophy: first you construct a graph, and then you evaluate it.

Let’s consider something very easy: the sum of two tensors
$$ {x}_1+{x}_2 $$
that can be calculated with the computational graph depicted in Figure 1-20.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig20_HTML.png
Figure 1-20

The computational graph for the sum of two tensors

Computational Graph with tf.constant

As discussed, first we must create this computational graph with tensorflow. (Remember: We begin with the construction phase.) Let’s start using the tf.constant tensor type. We need three nodes: two for the input variables and one for the sum. This can be achieved with the following code:
x1 = tf.constant(1)
x2 = tf.constant(2)
z = tf.add(x1, x2)
The preceding code creates the computational graph in Figure 1-20 and, at the same time, tells tensorflow that x1 should have the value 1 (the value in the parentheses in the declaration), and x2 should have the value 2. Now, to evaluate the code, we must create what tensorflow calls a session (wherein an actual evaluation can take place), and then we can ask the session class to run our graph with the following code:
sess = tf.Session()
print(sess.run(z))

This will simply give you the evaluated result of z that is, as expected, 3. This version of the code is rather simple and does not require much, but it is not very flexible. x1 and x2 are fixed and cannot be changed during the evaluation, for example.

Note

In TensorFlow, you first must create a computational graph, then create a session, and finally run your graph. These three steps must always be followed to evaluate your graph.

Remember: You can also ask tensorflow to evaluate only an intermediate step. For example, you might want to evaluate x1 (not very interesting, but there are many cases in which it will be useful, for example, when you want to evaluate your graph and, at the same time, the accuracy and the cost function of a model), as follows: sess.run(x1).

You will get the result 1, as expected (you expected that, right?). At the end, remember to close the session with sess.close() to free up used resources.

Computational Graph with tf.Variable

The same computational graph (the one in Figure 1-20) can be created with variables, but that requires a bit more work. Let’s re-create our computational graph.
x1 = tf.Variable(1)
x2 = tf.Variable(2)
z = tf.add(x1,x2)
We want to initialize the variables with the values 1 and 2, as before.1 The problem is that when you run the graph, as we did before, with the code
sess = tf.Session()
print(sess.run(z))
you will receive an error message. It is a very long error message, but near the end, you will find the following message:
FailedPreconditionError (see above for traceback): Attempting to use uninitialized value Variable
This occurs because tensorflow does not automatically initialize the variables. To do this, you could use this approach:
sess = tf.Session()
sess.run(x1.initializer)
sess.run(x2.initializer)
print(sess.run(z))
This works now without errors. Line sess.run(x1.initializer) will initialize the variable x1 with the value 1, and the line sess.run(x2.initializer) will initialize the variable x2 with the value 2. But this is rather cumbersome. (You don’t want to write a line for each variable you need to initialize.) A much better approach is to add a node to your computational graph that has the goal of initializing all the variables you define in your graph with the code
init = tf.global_variables_initializer()
and then again create and run your session , running this node (init) before evaluating z.
sess = tf.Session()
sess.run(init)
print(sess.run(z))
sess.close()

This will work and give you the result 3, as you would expect.

Note

When working with variables, remember always to add a global initializer (tf.global_variables_initializer()) and run the node in your session at the beginning, before any other evaluation. We will see in many examples during the book how this works.

Computational Graph with tf.placeholder

Let’s declare x1 and x2 as placeholders.
x1 = tf.placeholder(tf.float32, 1)
x2 = tf.placeholder(tf.float32, 1)
Note that I have not provided any value in the declaration.2 We will have to assign a value to x1 and x2 at evaluation time. That is the main difference between placeholders and the other two tensor types. The sum, then, is given again by
z = tf.add(x1,x2)
Note that if you try to see what is in z using, for example, print(z), you will get
Tensor("Add:0", shape=(1,), dtype=float32)
Why this strange result? First, we have not given tensorflow the values for x1 and x2 and, second, TensorFlow has not yet run any calculation. Remember: Graph construction and evaluation are separate steps. Now let’s create a session in TensorFlow, as before.
sess = tf.Session()
Now we can run the actual calculation, but to do that, we must first have a way of assigning values to the two inputs x1 and x2. This can be achieved by using a Python dictionary that contains all the placeholders’ names as keys and assign to them values. In this example, we assign to x1 the value 1 and to x2 the value 2.
feed_dict={ x1: [1], x2: [2]}
Feeding this code to the TensorFlow session can be done with the following command:
print(sess.run(z, feed_dict))
You finally get the result you expected: 3. Note that tensorflow is rather smart and can handle more complicated inputs. Let’s redefine our placeholder to be able to use arrays with two elements. (Here, we report the entire code, to make it easier to follow the example.)
x1 = tf.placeholder(tf.float32, [2])
x2 = tf.placeholder(tf.float32, [2])
z = tf.add(x1,x2)
feed_dict={ x1: [1,5], x2: [1,1]}
sess = tf.Session()
sess.run(z, feed_dict)
This time, you will get an array with two elements as output.
array([ 2., 6.], dtype=float32)

Remember that x1=[1,5] and x2=[1,1] meaning that z=x1+x2=[1,5]+[1,1]=[2,6], because the sum is done element by element.

To summarize, here are some guidelines on when to use which tensor type:

  • Use tf.placeholder for entities that will not change at each evaluation phase. Usually, those are input values or parameters that you want to keep fixed during the evaluation but may change with each run. (You will see several examples later in the book.) Examples include input dataset, learning rate, etc.

  • Use tf.Variable for entities that will change during the calculation, for example, the weights of our neural networks, as you will see later in the book.

  • Use tf.constant for entities that will never change, for example, fix values in your model that you don’t want to change anymore.

Figure 1-21 depicts a slightly more complex example: the computational graph for the calculation x1w1 + x2w2.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig21_HTML.png
Figure 1-21

The computational graph for the calculation x1w1 + x2w2

In this case, I defined x1, x2, w1, and w2 as placeholders (they will be our inputs) containing scalars (remember: when defining placeholders, you must always pass the dimensions as second input parameters, in this case, the 1).
x1 = tf.placeholder(tf.float32, 1)
w1 = tf.placeholder(tf.float32, 1)
x2 = tf.placeholder(tf.float32, 1)
w2 = tf.placeholder(tf.float32, 1)
z1 = tf.multiply(x1,w1)
z2 = tf.multiply(x2,w2)
z3 = tf.add(z1,z2)
Running the calculations means simply (as before) defining the dictionary containing the input values, creating a session, and then running it.
feed_dict={ x1: [1], w1:[2], x2:[3], w2:[4]}
sess = tf.Session()
sess.run(z3, feed_dict)
As expected, you will get the following result:
array([ 14.], dtype=float32)

This is simply 1 × 2 + 3 × 4 = 2 + 12 = 14 (remember that we have fed the values 1, 2, 3, and 4 in feed_dict, in the previous step). In the Chapter 2, we will draw the computational graph for a single neuron and apply what we have learned in this chapter to a very practical case. Using that graph, we will be able to do linear and logistic regression on a real dataset. As always, remember to close the session with sess.close() when you are done.

Note

In TensorFlow, it can happen that the same piece of code runs several times, and you can end up with a computational graph with multiple copies of the same node. A very common way of avoiding such a problem is to run the code tf.reset_default_graph() before the code that constructs the graph. Note that if you separate your construction code from your evaluation code appropriately, you should be able to avoid such problems. We will see later in the book in many examples how this is working.

Differences Between run and eval

If you look at blogs and books, you may find two ways of evaluating a computational graph with tensorflow. The one we have used up to now is sess.run(), in which the function wants as argument the name of the node you want to evaluate. We have chosen this method because it has a nice advantage. To understand it, consider the following code (the same you have seen previously)
x1 = tf.constant(1)
x2 = tf.constant(2)
z = tf.add(x1, x2)
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
sess.run(z)
This will only give you the evaluated node z, but you can also evaluate several nodes at the same time, using the following code:
sess.run([x1,x2,z])
This will give you
[1, 2, 3]

And that is very useful, as will become clear in the next section about the life cycle of a node. Additionally, evaluating many nodes at the same time will make your code shorter and more readable.

The second method of evaluating a node in a graph is to use the eval() call. This code
z.eval(session=sess)

will evaluate z. But this time, you must explicitly tell TensorFlow which session you want to use (you may have several defined). This is not very practical, and I prefer to use the run() method to get several results at the same time (for example, the cost function, accuracy, and F1 score). There is also a performance reason to prefer the first method, as explained in the next section.

Dependencies Between Nodes

As I mentioned before, TensorFlow evaluates a graph in topological order, which means that when you ask it to evaluate a node, it automatically determines all the nodes that are required to evaluate what you are asking and evaluate them first. The problem is that TensorFlow might evaluate some nodes multiple times. Consider the following code, for example:
c = tf.constant(5)
x = c + 1
y = x + 1
z = x + 2
sess = tf.Session()
print(sess.run(y))
print(sess.run(z))
sess.close()
This code will build and evaluate the computational graph in Figure 1-22.
../images/463356_1_En_1_Chapter/463356_1_En_1_Fig22_HTML.png
Figure 1-22

The computational graph that the code cited at the beginning of the section builds

As you can see, z and y both depend from x. The problem with the code as we have written, is that TensorFlow will not reuse the result of the previous evaluation of c and x. This means that it will evaluate the node for x one time when evaluating z and again when evaluating y. In this case, for example, using the code yy, zz = sess.run([y,z]) will evaluate y and z in one run, and x only once.

Tips on How to Create and Close a Session

I showed you how to create a session with the template
sess = tf.Session()
# Code that does something
At the end, you should always close a session, to free up used resources. The syntax is quite easy:
sess.close()
Keep in mind that the moment you close the session, you cannot evaluate anything else. You must create a new session and perform your evaluation again. In a Jupyter environment, this method has the advantage of allowing you to split your evaluation code in several cells and then close the session at the very end. But it is useful to know that there is a slightly more compact way of opening and using a session , using the following template:
With tf.Session() as sess:
# code that does something
For example, the code
sess = tf.Session()
print(sess.run(y))
print(sess.run(z))
sess.close()
from the previous section could have been written as
with tf.Session() as sess:
    print(sess.run(y))
    print(sess.run(z))
In this case, the session would automatically be closed at the end of the with clause. Using this method makes using eval() easier. For example, the code
sess = tf.Session()
print(z.eval(session=sess))
sess.close()
will look like this with the with clause
with tf.Session() as sess:
    print(z.eval())

There are some cases in which the explicit declaration of the session is preferred. For example, it is rather common to write a function that performs the actual graph evaluation and that returns the session, so that additional evaluation (for example, of the accuracy or similar metrics) can be done after the main training has finished. In this case, you cannot use the second version, because it would close the session immediately after finishing the evaluation, therefore making additional evaluations with the session results impossible.

Note

If you are working in an interactive environment such as Jupyter notebooks and you want to split your evaluation code on multiple notebook cells, it is easier to declare the session as sess = tf.Session(), perform the calculations needed, and then, at the end, close it. In this way, you can intercalate evaluations, graphs, and text. In case you are writing code that will not be interactive, it is sometimes preferable (and less error-prone) to use the second version, to make sure that the session is closed at the end. Additionally, with the second method, you don’t have to specify the session when using the eval() method.

The material covered in this chapter should give you all you need to build your neural networks with tensorflow. What I explained here is by no means complete or exhaustive. You should really take some time and go on the official TensorFlow web site and study the tutorials and other materials there.

Note

In this book I use a lazy programming approach. That means that I explain only what I want you to understand, nothing more. The reason is that I want you to focus on the learning goals for each chapter, and I don’t want you to be distracted by the complexity that lies behind the methods or the programming functions. Once you understand what I am trying to explain, you should invest some time and dive deeper into the methods and libraries, using the official documentation.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.128.197.221