We will now use our accumulated knowledge of GPU programming to implement our very own deep neural network (DNN) with PyCUDA. DNNs have attracted a lot of interest in the last decade, as they provide a robust and elegant model for machine learning (ML). DNNs was also one of the first applications (outside of rendering graphics) that were able to show the true power of GPUs by leveraging their massive parallel throughput, which ultimately helped NVIDIA rise to become a major player in the field of artificial intelligence.

In the course of this book, we have mostly been covering individual topics in a bubble on a chapter-by-chapter basis—here, we will build on many of the subjects we have learned about thus far for our very own implementation of a DNN. While there are several open source frameworks for GPU-based DNNs currently available to the general public—for example, Google's TensorFlow and Keras, Microsoft's CNTK, Facebook's Caffe2, and PyTorch—it is very instructive to go through an implementation of one from scratch, which will give us a greater insight and appreciation of the underlying technologies required for DNNs. We have a lot of material to cover here, so we'll cut right to the chase after a brief introduction to some of the basic concepts.

In this chapter, we will be looking at the following:

• Understanding what an artificial neuron (AN) is
• Understanding how many ANs can be combined together in a deep neural network (DNN)
• Implementing a DNN from scratch in CUDA and Python
• Understanding how cross-entropy loss can be used to evaluate the output of a neural network
• Implementing gradient descent to train an NN
• Learning how to train and test an NN on a small dataset