Humans act on what they perceive

We don’t know exactly how it works, but humans make decisions based on loops of perception and action. We perceive, then we act. Very rarely do we stop and make explicit calculations like machines when we control complex processes. For example, do you stop and calculate angles during your golf swing or explicitly search through options while driving?

Humans build complex correlations into their intuition with practice

Many times during brain design workshops, people have told me that humans aren’t good at managing many variables. I disagree. We don’t think visually in more than three dimensions and for most of us it is hard to calculate in multi-dimensional space, but we don’t use either of these methods to make complex decisions. We use our intuition. So, while it’s true that we don’t calculate in many dimensions well, we do learn to manage many variables while making decisions. It just takes time (a lot of time), feedback, repetition and exploration of the task landscape to build the correlations of these many variables into our intuition. I have a friend who knits. She tells me that it is important to select the right kind of yarn to match the kind of garment you are making. She doesn’t use a lookup table to select the right type of yarn for each knitting project, she uses her intuition from previous knitting projects to make the decision.

Humans abstract to strategy for complex tasks

The AI researchers on my team had a strong hunch that the best way to train AI was through imitation. Reasonable hypothesis. There are even special algorithms that learn by imitating the actions taken by experts. Imitation is indeed powerful, but every time I interviewed experts, they told me about strategy.

At the steel mill, experts filled whiteboards with different strategies they use to coat steel evenly to the right thickness (this process is called galvanization and prevents rust). Some strategies work best when the steel is thick and narrow and the coating is thick. Other strategies work best when the steel is wide and thin and the coating is thin. Why was I hearing so often about strategies when talking to industrial experts about how they manage their processes? It’s because humans use strategies to make complex decisions and to teach each other.

Here are some examples of strategies from well-known games and industrial processes that I discuss in this book. Notice how the strategies (and the goals; see Chapter 4, “Setting Goals for Autonomous AI,” for more details) come in pairs. This is because most real and natural systems have a fuzzy trade-off: you know, yin and yang, both sides of the coin, playing the devil’s advocate. See how many phrases we have to describe this phenomenon?

So, the experts were telling me that they use strategy to communicate and teach what to do, but the AI researchers and data scientists were telling me that it’s better to search for options. Enter the Dreyfus Model of Skill Acquisition!

Hubert (1929-2017) and Stuart Dreyfus developed a model to describe how humans acquire skills. These expert systems and computer science pioneers developed their model of skill acquisition for the United States Air Force in 1980 to help fighter pilots improve their emergency response. There are many models that attempt to describe how humans learn, and this model has its critics, but I’ve not seen a better model for describing how to acquire skills in a way that’s useful for designing Autonomous AI.

Figure 2-2. The Dreyfus Model of Skill Acquisition adapted for designing AI brains

Beginner

The first-stage learner, the beginner, learns the rules and goals of the game and begins to practice simple expert rules. The beginning chess player must first learn the goal of the game: you win when you capture (land on a square occupied by) your opponent’s king. Then, the beginner learns the rules of the game: which movements each piece can make turn by turn (bishops can move any distance along diagonal unoccupied squares, rooks can move any distance along lateral unoccupied squares). Next, the beginner is usually given one or more strategies to practice. One such common strategy is a point system that assigns a point value to each captured piece (<<#Figure_2-2>>). The beginner then practices many games, tallying the points accumulated with the pieces she captures. This system emphasises a few mental shortcuts: Queens are extremely important, so protect them; Rooks are more important than Knights and Bishops. Why Does this matter for designing AI? Autonomous AI learns by practicing just like human beginners, so you can teach it by giving it rules to practice.

Figure 2-3. Point system for chess.

There’s a new kind of AI in town

Each of the Autonomous AI applications that I cite in this book leverage an exciting new form of AI called Deep Reinforcement Learning (DRL). DRL allows optimization algorithms to exhibit useful human-like decision-making in some controlled situations. In order to design highly capable brains, it is important to acknowledge the capability of DRL but also to recognize that the principles of brain design transcend any particular technology, including Deep Reinforcement Learning. There are two key components to DRL: Reinforcement Learning algorithms and Deep Neural Networks.

Reinforcement Learning

In reinforcement learning, an AI brain acquires skill for a specific task through trial and error and by receiving feedback. The brain practices completing the task, gets feedback from a digital simulation (or from the real thing), and learns to take actions that lead to the best rewards.

Reinforcement Learning (RL) algorithms approximate learning by changing their behavior based on feedback. Reinforcement Learning algorithms try actions and receive feedback in the form of a “reward”: they translate the resulting actions into an index of success toward the goal for the task. These learners are like young children: infinitely curious (often well beyond their years) and persistent (to the point of being obstinate).

Algorithms do not apply reasoning to their exploration: they learn based on significant trial and error. They efficiently explore large spaces and succeed at tasks by learning complex behaviors that achieve rewards under a huge variety of scenarios. Along the way, they often learn the fundamental dynamics of the task, which can let them succeed in scenarios for which they were not trained. For example, the brain I referenced above that Microsoft built for Siemens practiced so much (it made more than 100 million decisions in the simulation) that it learned to successfully calibrate real-life machines, even on operations that it had never experienced in training.

For details on how Reinforcement Learning Algorithms work, I recommend Reinforcement Learning by Phil Winder Ph.D.

Figure 2-4. Reinforcement learning algorithms “learn” by trying actions, getting feedback and pursuing maximal future reward.

Autonomous AI perceives, then acts

One unique quality that sets Autonomous AI apart from automated systems is its ability to learn from “what it sees and hears” and make supervisory decisions. For example, when human operators control kilns that cook limestone as the first step in the cement-making process, they use their senses. They supervise and control the process based on how the kiln flame looks: its shape, its color, and the haziness of the air around it. When an AI can do the same, this kind of “sensory perception” moves it toward higher-level executive functioning (but only in the context of machine and process control).

While automated systems cannot make supervisory decisions based on visual and auditory perception, autonomous AI can. Deep Reinforcement Learning, which uses reinforcement learning algorithms to train neural networks, is the only control system technology that can input visual, sound, and categorical perception and act on this perception to control the system.

Autonomous AI learns and adapts when things change

I once met with a room full of Model Predictive Control (MPC) experts who quizzed me (it was perfectly friendly, I promise) about the benefits of DRL compared to MPC. After over 60 minutes of conversation, one of the researchers gave me this example that helped me explain. Have you ever borrowed your friend’s car and screeched the tires (accelerated too quickly) because the accelerator on their car was touchier than yours? I drive an old Honda Pilot that has a really mushy gas pedal. You have to press the pedal far down to get it to accelerate. MPC control systems (and most other control systems) have a single behavior when it comes to the variable that they control. The distance that you have to press the pedal to get additional acceleration is called a gain. Touchy gas pedals have a high gain and mushy gas pedals have a low gain. Either way, a gain-based control system will never change its behavior. It will either be touchy or mushy.

Autonomous AI can learn to change its behavior based on the circumstances it is in. It can be a mushy gas pedal when it needs to be and a touchy gas pedal when it needs to be.

Autonomous AI can spot patterns

Humans often use pattern recognition from previous experience to make decisions. Supply chain and logistics professionals described to me many times how they make decisions about how much of each product type and how to deliver each product type by matching patterns. One of these companies makes water sports equipment like kayaks and canoes and built an AI to help them make better plans and schedules. They produce and ship the boats to large box stores like Walmart, Target and Costco who sell them. Here’s the challenge. Each of those stores provides a forecast for how many boats of each type and color they want. But these forecasts are never completely accurate and they never will be. So, the best way to deal with this is for planners to match patterns based on past experience. The patterns sound like these made up examples:

• Costco tends to ask for more canoes than they actually need during the off-season.

• Target tends to underestimate the amount of blue boats they need during peak season.

• Walmart overestimates how many kayaks they need in urban areas but underestimates how many they need in rural areas.

Much like humans, AI brains can match patterns like the ones above even complex patterns that involve many variables.

Autonomous AI infers from experience

One of the key motivations for developing reinforcement learning algorithms was to handle situations where some system changes cannot be measured. Of course, key states need to be measured in the environment but Autonomous AI can infer and respond to some changes in the environment by building correlations with other state variables. Pepsi built an AI that does this. This AI (which performs as an expert operator making CheetosTM snack foods) responds to the moisture level of the incoming corn and controls the manufacturing equipment differently based on the way it responds to changes in moisture, even though this moisture level is not measured.

Autonomous AI infers from experience

Deep Reinforcement Learning (DRL) is the only technology of any kind that has demonstrated the ability to learn strategy. Early chess-playing machines utilized programmed strategies , but brains learn strategy as they gain experience. The AlphaChess AI program learned and regularly used the 12 most common opening move sequences in chess on its own without being taught.

If brains can learn chess strategies, they can also learn strategies to control high-value equipment and processes. For example, a piece of equipment called a gyratory crusher is commonly used to crush rocks as the first step in many mining processes. The goal is to crush as much rock as possible (measured in tons per hour) to the particle size that will fit through the holes in a large shaking sieve. A gyrating steel arm rotates inside the cone and crushes the rocks against the steel cone. The rocks then fall through the bottom of the funnel. If you stuff the crusher chock-full, the resulting compression forces crush even the largest, hardest rocks, but it takes more time for the rocks to move through the crusher. If you fill the crusher ⅔ to ¾ full, you can move more material through the crusher per hour, but the crusher doesn’t generate as much compressive force. The first control strategy efficiently crushes large, hard rocks. This second control strategy is perfect for increasing throughput when the rocks are softer and smaller. An Autonomous AI can learn to move between these two strategies to maximize throughput.

Figure 2-5. Choke the gyratory crusher when the rocks are large and hard. This ensures that they get crushed efficiently. Regulate the crusher when the rocks are softer and smaller. This maximizes the throughput of crushed rocks.

Autonomous AI can plan for the long-term future

Complex situations where the optimal move in the short term is shortsighted, not optimal in the long term. Let me give you an example: squirrels store up nuts and save them for later. Bears store energy as fat, not because they need it now but because they will need it later when they hibernate during the winter.

Researchers at Microsoft Bonsai experienced this forward thinking in an amusing way while teaching the Jaco robotic arm to grasp and stack one block on another. This seven-jointed robot arm was folding in on itself while performing the task, so the researcher set a reward for maximizing the distance between the block and the shoulder of the robotic arm and a stiff penalty for bringing the block closer to the base and shoulder.

After more practice, it began to wind up, bringing the block closer to its shoulder—at great short-term penalty—to strike or throw the block. This machine agent learned that it could achieve much greater rewards in the long term by winding up and throwing the block, enough to cancel out the penalty for winding up. Of course, what the researcher really intended was to reward the agent for keeping the robot wrist far from the shoulder. Once they changed the parameters of the reward to be more specific, this disincentivized the throwing behavior and prompted the robot to learn the intended behavior of extending its arm.

You might just be thinking, why not just reward the AI for the distance to incentivize this behavior from the beginning? That’s a great question. Why did the brain learn to throw when it was penalized for winding up and what might it do if it was not penalized for winding up? We got lucky. Either way, if we tried this again, the AI would probably explore a lot of movements that don’t include a wind up. Best to teach winding up explicitly as a skill.

So far in this chapter, we’ve demonstrated that the human-like decision-making capabilities of Autonomous AI can outperform automated machine intelligence. The rest of this book will describe how to design Autonomous AI, but let’s finish off this chapter with a discussion of when to use Autonomous AI.

When the superpowers matter most

You now know how to compare the decision-making capabilities of brains with automated systems and humans. Now you need to decide whether or not this difference in performance matters. Almost every brain I’ve ever designed is for a system or process where a 1% improvement in the key performance indicators leads to over $1 million USD in savings or increased revenue for one facility. That matters.

When humans need to take over the decision making process

When humans have to take over, it’s a sure fire sign that you need more human-like decision making. Here’s a few final tips for when to use Autonomous AI:

• Look for high-value decisions

• Look for decisions that require human intervention

• Look for complex decisions that require complex perception.