Simple CPU Example

Here’s a simple example that creates a tensor, performs a tensor operation, and uses a built-in method on the tensor itself. By default, the tensor data type will be derived from the input data type and the tensor will be allocated to the CPU device. First, we import the PyTorch library, then we create two tensors, x and y, from two-dimensional lists. Next we add the two tensors and store the result in z. Notice we can just use the “+” operator since the torch.Tensor class supports operator overloading. Finally, we print the new tensor z, which we can see is the matrix sum of x and y, and we print the size of z. Notice that z is a tensor object itself and the size() method is used to return its matrix dimensions, namely two by three.

import torch

x = torch.tensor([[1,2,3],[4,5,6]])
y = torch.tensor([[7,8,9],[10,11,12]])
z = x + y
print(z)
> tensor([[ 8, 10, 12],
 [14, 16, 18]])
print(z.size())
> torch.Size([2, 3])

Simple GPU Example

Since accelerating tensor operations on a GPU is a major advantage of tensors over Numpy arrays, we’ll show you an easy example to do so. Here’s the same example but instead we move the tensors to the GPU device if one is available. Notice that the output tensor is also allocated to the GPU. You can also use the device attribute (e.g. z.device) to double check where the tensor resides. In the first line, the torch.cuda.is_available() function will return True is your machine has GPU support. This is a convenient way to write more robust code that can be accelerated when a GPU exists but also runs on a CPU when a GPU is not present. Also, notice that the device=cuda:0 indicates that the first GPU is being used. If your machine contains multiple GPUs, you can also control which GPU is being used.

device = "cuda" if torch.cuda.is_available() else "cpu"
x = torch.tensor([[1,2,3],[4,5,6]], device=device)
y = torch.tensor([[7,8,9],[10,11,12]], device=device)
z = x + y
print(z)
> tensor([[ 8, 10, 12],
    [14, 16, 18]], device='cuda:0')
print(z.size())
> torch.Size([2, 3])
print(z.device)
> cuda:0

Moving Tensors between CPU & GPU

The previous code uses torch.tensor() to create a tensor on a specific device; however, you may want to move an existing tensor between devices. You can do so by using the torch.to() method. When new tensors are created as a result of tensor operations, PyTorch will create the new tensor on the same device. In the following code, z resides on the GPU because x and y reside on the GPU. The tensor z is moved back to the CPU using torch.to(“cpu”) for further processing. Also note that all the tensors within the operation must be on the same device. If x was on the GPU and y was on the CPU, we would get an error.

device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)
x.to(device)
y.to(device)
z = x + y
z.to(“cpu”)

Tensor Attributes

One PyTorch feature that has contributed to its popularity is the fact that it’s very Pythonic and object oriented in nature. Since a tensor is it’s own data type, you can read attributes of the tensor object itself. Now that you can create tensors, it’s useful to quickly find information about them by accessing their attributes. Assuming x is a tensor, you can access several attributes of x as follows:

• x.dtype indicates the tensor’s data type (see Table 2-2 for list of PyTorch data types)

• x.device indicates the tensor’s device location (e.g. CPU or GPU memory)

• x.shape shows the tensor’s dimensions

• x.ndim identifies the number of tensor’s dimensions or rank

• x.requires_grad is a boolean indicating whether Tensor keeps track of graph computations (see “Automatic Differentiation (Autograd)” )

• x.grad contains the actual gradients if requires_grad is True

• x.grad_fn stores the graph computation function used if requires_grad is True

• x.is_cuda, x.is_sparse, x.is_quantized, x.is_leaf, x.is_mkldnn are booleans indicating whether the tensor meets certain conditions

• x.layout indicates how a tensor is laid out in memory

Remember when accessing object attributes, do not include parentheses () like you would with a class method (e.g. use x.shape not x.shape() ).

Data Types

During deep learning development, it’s important to be aware of the data type used by your data and its calculations. So when you create tensors, you should control what data types are being used. As we mentioned before, all tensors elements have the same data type. You can specify the data type when creating to tensor by using the dtype parameter or you can cast a tensor to a new dtype using the casting method or to() method as shown in the code below.

# Specify data type at creation using dtype
w = torch.tensor([1,2,3], dtype=torch.float32)

# Use casting method to cast to a new data type
w.int()    # w remains a float32 after cast
w = w.int()  # w changes to int32 after cast

# Use to() method to cast to a new type
w = w.to(torch.float64) 
w = w.to(dtype=torch.float64) 

# Python automatically converts data types during operations
x = torch.tensor([1,2,3], dtype=torch.int32)
y = torch.tensor([1,2,3], dtype=torch.float32)
z = x + y 

Pass in data type

Define data type directly with dtype

Python automatically converts x to float32 and returns z as float32

Note that the casting and to() methods do not change the tensor’s data type unless you reassign the tensor. Also, when performing operations on mixed data types, PyTorch will automatically cast tensors to the appropriate type.

Most of the tensor creation functions allow you to specify the data type upon creation using the dtype parameter. When you set the dtype or cast tensors, remember to use the torch namespace such as torch.int64 not just int64. Table 2-2 lists all the available data types in PyTorch. Each data type results in a different tensor class depending on the tensor’s device. The corresponding tensor class is shown in the two rightmost columns for CPU and GPU respectively.

Creating Tensors from Random Samples

The need to create random data comes up often during deep learning development. Sometimes, you will need to initialize weights to random values or create random inputs with specified distributions. PyTorch supports a very robust set of functions that you can use to create tensors from random data. As with other creation functions, you can specify the dtype and device when creating the tensor. Table 2-3 lists some examples of random sampling functions.

You can also create tensors of values sampled from other distributions using torch.empty() and in-place functions including cauchy, exponential, geometric, and log_normal. Remember in-place methods use the underscore post-fix. For example, x = torch.empty([10,5].cauchy_() creates a tensor of random numbers drawn from the Cauchy distribution.

Creating Tensors Like Other Tensors

Sometimes you may want to create and initialize a tensor that has similar properties to another tensor including the dtype, device, and layout properties to facilitate calculations. Many of the tensor creation operations have a similarity function that allows you to easily do this. The similarity functions will have the post-fix _like. For example, torch.empty_like(tensor_a) will create an empty tensor with the dtype, device and layout properties of tensor_a. Some examples of similarity functions include empty_like(), zeros_like(), ones_like(), full_like(), rand_like(), randn_like(), and rand_int_like().

Indexing, Slicing, Combining & Splitting Tensors

Once you have created tensors, you may want to access portions of the data and combine or split tensors to form new tensors. The following code demonstrates how to perform these types of operations. You can slice and index tensors in the same way you would slice and index NumPy arrays as shown in the first few lines of code. Note that indexing and slicing will return tensors even if it’s only a single element. You will need to use the item() function to convert a single element tensor to a Python value when passing to other functions like print(). Next, we see that slicing uses the same [start:end:step] format that is used for slicing Python lists and NumPy arrays, and boolean indexing allows you to extract portions of the data that meet certain criteria. PyTorch supports transposing and reshaping arrays as shown in the next few lines of code. Finally, you can combine or split tensors by using functions like torch.stack() and torch.unbind() respectively as shown.

x = torch.tensor([[1,2],[3,4],[5,6],[7,8]])
print(x)
> tensor([[1, 2],
    [3, 4],
    [5, 6],
    [7, 8]])

# Indexing, returns a tensor
print(x[1,1])
> tensor(4)

# Indexing, returns value as Python number
print(x[1,1].item())
> 4

# Slicing
print(x[:2,1])
> tensor([2, 4])

# Boolean indexing
print(x[x<5])
> tensor([1, 2, 3, 4])

# Transpose array, x.t() or x.T can be used
print(x.t())
> tensor([[1, 3, 5, 7],
    [2, 4, 6, 8]])

# Changing shape, Usually view() is preferred over reshape()
print(x.view((2,4)))
> tensor([[1, 3, 5, 7],
    [2, 4, 6, 8]])

# Combining tensors
y = torch.stack((x, x))
print(y)
tensor([[[1, 2],
     [3, 4],
     [5, 6],
     [7, 8]],

    [[1, 2],
     [3, 4],
     [5, 6],
     [7, 8]]])

# Splitting tensors
a,b = x.unbind(dim=1)
print(a,b)
> tensor([1, 3, 5, 7]) tensor([2, 4, 6, 8])

PyTorch provides a robust set of built-in functions that can be used to access, split, and combine tensors in different ways. Table 2-4 lists some commonly used functions to manipulate tensor elements.

Some of these functions may seem redundant. However, the following key distinctions and best practices are important to keep in mind:

• torch.item() is an important and commonly used function to return the Python number from a tensor.

• Use view() instead of reshape() for reshaping tensors in most cases. Using reshape() may cause the tensor to be copied depending on it’s layout in memory. view() ensures that it will not be copied.

• Using x.T or x.t() is a simple way to transpose 1D or 2D tensors. Use transpose() when dealing with multidimensional tensors.

• The torch.squeeze() function is used often in deep learning to remove an unused dimension. For example batch of images with a single image can be reduce from 4D to 3D using squeeze().

Tensor Operations for Mathematics

Deep Learning development is strongly based on mathematical computations so PyTorch supports a very robust set of built-in math functions. Whether you are creating new data transforms, customizing loss functions or building your own optimization algorithms, you can speed up your research and development with the math functions provided by PyTorch. The purpose of this section is to provide a quick overview of many of the mathematical functions available so that you can quickly build your awareness of what currently exists and find the appropriate functions when needed.

PyTorch supports many different types of math functions including pointwise operations, reduction functions, comparison calculations, linear algebra, spectral and other math computations. The first type of useful math operations are considered to be pointwise operations. Pointwise operations perform an operation on each point in the tensor individually and return a new tensor. They are useful for rounding and truncation as well as trigonometrical and logical operations. By default, the functions will create a new tensor or use one passed in by the out parameter. If you want to perform an in-place operation, remember to append the underscore to the function name. Table 2-5 lists some commonly used pointwise operations.

Use Python hints or refer to the PyTorch documentation for details on function usage. Note that true_divide() converts tensor data to floats first, and should be used when dividing integers to obtain true division results.

A second type of math functions are called reduction operations. Reduction operations reduce a bunch of numbers down to a single number or smaller set of numbers. That is, they reduce the dimensionality or rank of the tensor. Reduction operations include functions like maximum or minimum values as well as many statistical calculations like mean or standard deviation. These operations are frequently used in deep learning. For example, deep learning classification often uses the argmax() function to reduce softmax outputs to a dominant class.

Table 2-6 shows a list of commonly used reduction operations. Note that many of these functions accept dim parameter which specifies the dimension of reduction for multi-dimensional tensors. This is similar to the axis parameter in NumPy. By default, when dim is not specified, the reduction occurs across all dimensions. Specifying dim = 1, will compute the operation across each row. For example, torch.mean(x,1) will compute the mean for each row in tensor x.

A third type of mathematical operations are called comparison functions. Comparison functions usually compare all the values within a tensor or compare one tensor’s values to another’s. They can return a tensor full of booleans based on each element’s value such as torch.eq() or torch.is_boolean(). They may also find the maximum or minimum value, sort tensor values, or return the top subset of tensors elements. Table 2-7 lists some commonly used comparison functions for your reference.

Comparison functions seem pretty straight forward, however, there are a few key points to keep in mind. Some common pitfalls are described as follows:

• The torch.eq() function or == returns a tensor of the same size with a Boolean result for each element. The torch.equal() function tests if the tensors are the same size and that all elements within tensor are equal then returns a single Boolean value.

• The function torch.allclose() also returns a single boolean value if all elements are close to a specified value.

The next type of mathematical functions are considered linear algebra functions. Linear algebra functions facilitate matrix operations and are important for deep learning computations. Many computational including gradient descent and optimization algorithms use linear algebra to implement their calculations. PyTorch supports a robust set of built-in linear Algebra operations, many of which are based on the BLAS and LAPACK standardized libraries for linear algebra calculations.

Table 2-8 lists some commonly used linear algebra operations. They range from matrix multiplication to batch calculations and solvers. It’s important to point out that matrix multiplication is not the same as pointwise multiplication with torch.mul() or the * operator. A complete study of Linear algebra is beyond the scope of this book, but you may find it useful to access some of the linear algebra functions when performing feature reduction or developing custom deep learning algorithms. See PyTorch Linear Algebra Documentation for a complete list of available functions and more details on how to use them.

The final type of mathematical operations are considered spectral and other math operations. Depending on the domain of interest, these function may be useful for data transforms or analysis. For example, spectral operations like the Fast Fourier Transform can play an important role in computer vision or digital signal processing applications. Table 2-9 lists some built-in operations for spectrum analysis and other mathematical operations.

Automatic Differentiation (Autograd)

One function is worth calling out in it’s own subsection because it’s what makes PyTorch so powerful for deep learning development. The backward() function uses PyTorch’s automatic differentiation package, torch.autograd, to differentiate and compute gradients of tensors based on the chain rule. Here’s a simple example of auto differentiation. We define a function, f = sum(x^2). If we want to find dx/df, we need to set the requires_grad = True flag for x. f.backward() performs the differentiation with respect to f and stores dx/df in the x.grad attribute.

x = torch.tensor([[1,2,3],[4,5,6]], dtype=torch.float, requires_grad=True)
f = x.pow(2).sum()
print(f)
> tensor(91., grad_fn=<SumBackward0>)
f.backward()
print(x.grad) # dx/df
> tensor([[ 2., 4., 6.],
    [ 8., 10., 12.]])

Only Tensors of floating point dtype can require gradients.

Training neural networks requires us to compute the weight gradients on the backward pass during training. As our neural networks get deeper and more complex, this feature automates the complex computations.

This chapter provided a quick reference to create tensors and perform operations. Now that you have a good foundation on tensors and how to use them, we will focus on how to use tensors and PyTorch to perform deep learning research. In the next chapter, we will review and provide a quick reference to the deep learning development process before jumping into writing code.