Now, let's take a step back and start with a simple, working definition of DL. As we work through this book, our understanding of this term will evolve, but, for now, let's consider a simple example. We have an image of a person. How can we show this image to a computer? How can we teach the computer to associate this image with the word person?
First, we figure out a representation of this image, say the RGB values for every pixel in the image. We then feed that array of values (together with several trainable parameters) into a series of operations we're quite familiar with (multiplication and addition). This produces a new representation that we can use to compare against a representation we know maps to the label, person. We automate this process of comparison and update the values of our parameters as we go.
This description covers a simple, shallow ML system. We'll get into more detail in a later chapter devoted to neural networks but, for now, to make this system deep, we increase the number of operations on a greater number of parameters. This allows us to capture more information regarding the thing we're representing (the person's image). The biological model that influences the design of this system is the human nervous system, including neurons (the things we fill with our representations) and synapses (the trainable parameters).
The following diagram shows the ML system in progress:
So, DL is just an evolutionary twist on the 1957's perceptron, the simplest and the original binary classifier. This twist, together with dramatic increases in computing power, is the difference between a system that doesn't work and a system that allows a car to drive autonomously.
Beyond self-driving cars, there are numerous applications for DL and related approaches in farming, crop management, and satellite image analysis. Advanced computer vision powers machines that remove weeds and reduce pesticide use. We have near-real-time voice search that is fast and accurate. This is the fundamental stuff of society, from food production to communication. Additionally, we are also on the cusp of compelling, real-time video and audio generation, which will make today's privacy debates or drama about what is fake news look minor.
Long before we get to AGI, we can improve the world around us using the discoveries we make along the way. DL is one of these discoveries. It will drive an increase in automation, which, as long as the political change that accompanies it is supportive, can offer improvements across any number of industries, meaning goods and services will get cheaper, faster, and more widely available. Ideally, this means people will be set increasingly free from the routines of their ancestors.
The darker side of progress is not to be forgotten either. Machine vision that can identify victims can also identify targets. Indeed, the Future of Life Institute's open letter on autonomous weapons (Autonomous Weapons: an Open Letter from AI & Robotics Researchers), endorsed by science and tech luminaries such as Stephen Hawking and Elon Musk, is an example of the interplay and tensions between academic departments, industry labs, and governments about what the right kind of progress is. In our world, the nation-state has traditionally controlled the guns and the money. Advanced AI can be weaponized, and this is a race where perhaps one group wins and the rest of us lose.
More concretely, the field of ML is progressing incredibly fast. How might we measure this? The premier ML conference Neural Information Processing Systems (NIPS) has over seven times the registrations in the year 2017 that it did in 2010.
Registrations for 2018 happened more in the manner of a rock concert than a dry technical conference, reflected in the following statistic tweeted out by the organizers themselves:
The de facto central repository of ML preprints, arXiv, has a hockey-stick growth chart of such extremes, where tools have emerged to help researchers to track all of the new work. An example of this is the director of AI at Tesla, Andrej Karpathy's site, arxiv-sanity (http://www.arxiv-sanity.com/). This site allows us to sort/group papers and organize an interface by which we can pull research we're interested in from the stream of publications with relative ease.
We cannot predict what will happen to the rate of progress over the next five years. The professional guesses of venture capitalists and pundits range from exponential to the next AI winter is nigh. But we have techniques and libraries and compute power now, and knowing how to use them at their limits for a natural language processing or computer vision task can help to solve real-world problems.
This is what our book aims to show you how to do.