We will now briefly cover the history of DL and the historical context from which it emerged, including the following:
- The idea of AI
- The beginnings of computer science/information theory
- Current academic work about the state/future of DL systems
While we are specifically interested in DL, the field didn't emerge out of nothing. It is a group of models/algorithms within ML itself, a branch of computer science. It forms one approach to AI. The other, so-called symbolic AI, revolves around hand-crafted (rather than learned) features and rules written in code, rather than a weighted model that contains patterns extracted from data algorithmically.
The idea of thinking machines, before becoming a science, was very much a fiction that began in antiquity. The Greek god of arms manufacturing, Hephaestus, built automatons out of gold and silver. They served his whims and are an early example of human imagination naturally considering what it might take to replicate an embodied form of itself.
Bringing the history forward a few thousand years, there are several key figures in 20th-century information theory and computer science that built the platform that allowed the development of AI as a distinct field, including the recent work in DL we will be covering.
The first major figure, Claude Shannon, offered us a general theory of communication. Specifically, he described, in his landmark paper, A Mathematical Theory of Computation, how to ensure against information loss when transmitting over an imperfect medium (like, say, using vacuum tubes to perform computation). This notion, particularly his noisy-channel coding theorem, proved crucial for handling arbitrarily large quantities of data and algorithms reliably, without the errors of the medium itself being introduced into the communications channel.
Alan Turing described his Turing machine in 1936, offering us a universal model of computation. With the fundamental building blocks he described, he defined the limits of what a machine might compute. He was influenced by John Von Neumann's idea of the stored-program. The key insight from Turing's work is that digital computers can simulate any process of formal reasoning (the Church-Turing hypothesis). The following diagram shows the Turing machine process:
So, you mean to tell us, Mr. Turing, that computers might be made to reason…like us?!
John Von Neumann was himself influenced by Turing's 1936 paper. Before the development of the transistor, when vacuum tubes were the only means of computation available (in systems such as ENIAC and its derivatives), John Von Neumann published his final work. It remained incomplete at his death and is entitled The Computer and the Brain. Despite remaining incomplete, it gave early consideration to how models of computation may operate in the brain as they do in machines, including observations from early neuroscience around the connections between neurons and synapses.
Since AI was first conceived as a discrete field of research in 1956, with ML coined in 1959, the field has gone through a much-discussed ebb and flow—periods where hype and funding were plentiful, and periods where private sector money was non-existent and research conferences wouldn't even accept papers that emphasized neural network approaches to building AI systems.
Within AI itself, these competing approaches cannibalized research dollars and talent. Symbolic AI met its limitations in the sheer impossibility of handcrafting rules for advanced tasks such as image classification, speech recognition, and machine translation. ML sought to radically reconfigure this process. Instead of applying a bunch of human-written rules to data and hoping to get answers, human labor was, instead, to be spent on building a machine that could infer rules from data when the answers were known. This is an example of supervised learning, where the machine learns an essential cat-ness after processing thousands of example images with an associated cat label.
Quite simply, the idea was to have a system that could generalize. After all, the goal is AGI. Take a picture of your family's newest furry feline and the computer, using its understanding of cat-ness, correctly identifies a cat! An active area of research within ML, one thought essential for building a general AI, is transfer learning, where we might take the machine that understands cat-ness and plug it into a machine that, in turn, acts when cat-ness is identified. This is the approach many AI labs around the world are taking: building systems out of systems, augmenting algorithmic weakness in one area with statistical near certainty in another, and, hopefully, building a system that better serves human (or business) needs.
The notion of serving human needs brings us to an important point regarding the ethics of AI (and the DL approaches we will be looking at). There has been much discussion in the media and academic or industry circles around the ethical implications of these systems. What does it mean for our society if we have easy, automated, widespread surveillance thanks to advances in computer vision? What about automated weapons systems or manufacturing? It is no longer a stretch to imagine vast warehouses staffed by nothing with a pulse. What then for the people who used to do those jobs?
Of course, full consideration of these important issues lies outside the scope of this book, but this is the context in which our work takes place. You will be one of the privileged few able to build these systems and move the field forward. The work of the Future of Humanity Institute at Oxford University, run by Nick Bostrom, and the Future of Life Institute, run by MIT physicist, Max Tegmark, are two examples of where the kind of academic debate around AI ethics issues is taking place. This debate is not limited to academic or non-profit circles, however; DeepMind, an Alphabet company whose goal is to be an Apollo program for AGI, launched DeepMind Ethics & Society in October 2017.
This may seem far removed from the world of code and CUDA and neural networks to recognize cat pictures, but, as progress is made and these systems become more advanced and have more wide-ranging applications, our societies will face real consequences. As researchers and developers, it behooves us to have some answers, or at least ideas of how we might deal with these challenges when we have to face them.