by Roman V. Yampolskiy

When you’re ready to incorporate artificial intelligence technologies in your business, the analysis you should perform is this: What can possibly go wrong? What is our product or service expected to do? What happens if it fails to do so? Do we have a damage mitigation plan? Consider the embarrassing situation that Microsoft found itself in with its Tay chatbot fiasco, where internet trolls exploited vulnerabilities in the bot’s code, feeding it racist, homophobic, and sexist content that millions read on social media.

Accidents, including deadly ones, caused by software or industrial robots can be traced to the early days of such technology, but they are not necessarily caused by the systems themselves. AI failures, on the other hand, are directly related to the mistakes produced by the intelligence such systems are designed to exhibit. We can broadly classify such failures into “mistakes made during the learning phase” and “mistakes made during performance phase.” A system can fail to learn what its designers want it to learn and might instead learn a different, but correlated function.

A frequently cited example is a computer vision system that the U.S. Army had hoped to use to automatically detect camouflaged enemy tanks. The system was supposed to classify pictures of tanks, but instead learned to distinguish the backgrounds of such images. Other examples include problems caused by poorly designed functions that would reward AIs for only partially desirable behaviors, such as pausing a game to avoid losing, or repeatedly touching a soccer ball to get credit for possession.

It can help to look at some recent examples of AI failure to better understand what problems are likely to arise and what you can do to prevent them—or at least to clean up quickly after a failure. Consider these examples of AI failures from the past few years:¹

• An automated email-reply generator created inappropriate responses, such as writing “I love you” to a business colleague.
• A robot for grabbing auto parts grabbed and killed a man.
• Image-tagging software classified black people as gorillas.
• Medical AI classified patients with asthma as having a lower risk of dying of pneumonia.
• Adult-content-filtering software failed to remove inappropriate content, exposing children to violent and sexual content.
• AI designed to predict recidivism acted racist.
• An AI agent exploited a reward signal to win a game without actually completing the game.
• Video game NPCs (nonplayer characters, or any character that is not controlled by a human player) designed unauthorized superweapons.
• AI judged a beauty contest and rated dark-skinned contestants lower.
• A mall security robot collided with and injured a child.
• A self-driving car had a deadly accident.

And every day, consumers experience more common shortcomings of AI: Spam filters block important emails, GPS provides faulty directions, machine translations corrupt the meaning of phrases, autocorrect replaces a desired word with a wrong one, biometric systems misrecognize people, transcription software fails to capture what is being said; overall, it is harder to find examples of AIs that don’t fail.

Analyzing the list of AI failures, we can arrive at a simple generalization: An AI designed to do X will eventually fail to do X. While it may seem trivial, it is a powerful generalization tool, which can be used to predict future failures of AI. For example, looking at cutting-edge current and future AI, we can predict that:

• AI doctors will misdiagnose some patients in a way a real doctor would not.
• Video-description software will misunderstand movie plots.
• Software for generating jokes will occasionally fail to make them funny.
• Sarcasm-detection software will confuse sarcastic and sincere statements.
• Employee-screening software will be systematically biased and thus hire low performers.
• The Mars robot explorer will misjudge its environment and fall into a crater.
• Tax-preparation software will miss important deductions or make inappropriate ones.

What should you learn from these examples and analysis? Failures will happen! It’s inevitable. But we can still put best practices in place, such as:

• Controlling user input to the system and limiting learning to verified data inputs
• Checking for racial, gender, age, and other common biases in your algorithms
• Explicitly analyzing how your software can fail, and then providing a safety mechanism for each possible failure
• Having a less “smart” backup product or service available
• Having a communications plan in place to address the media in case of an embarrassing failure (Hint: Start with an apology.)

I predict that both the frequency and seriousness of AI failures will steadily increase as AIs become more capable. The failures of today’s narrow domain AIs are just the tip of the iceberg; once we develop general artificial intelligence capable of cross-domain performance, embarrassment will be the least of our concerns.

TAKEAWAYS

There are countless examples of an AI designed for a task eventually failing to do that task. As AI performs more consequential functions, damage caused by single failures will become more severe.

• When your business is ready to incorporate AI technologies, perform this risk analysis: What can possibly go wrong with our AI? What is our product or service expected to do? What happens if it fails to do so? Do we have a damage mitigation plan?
• To mitigate damage from AI failures, companies should follow these best practices:

• Control user input into system and only use verified data.
• Check for racial, gender, age, and other common biases in algorithms.
• Analyze how the software can fail, and then provide a safety mechanism for each possible failure.
• Have a less “smart” backup product or service available.
• Have a communications plan in place to address the media in case of an embarrassing failure.