CHAPTER 12
Can Machine Learning Solve Your Business Problem?
by Anastassia Fedyk
As you consider ways to analyze large swaths of data, you may ask yourself how the latest technological tools and automation can help. AI, big data, and machine learning are all trending buzzwords, but how can you know which problems in your business are amenable to machine learning?
To decide, you need to think about the problem to be solved and the available data, and ask questions about feasibility, intuition, and expectations.
Assess Whether Your Problem Requires Learning
Machine learning can help automate your processes, but not all automation problems require learning.
Automation without learning is appropriate when the problem is relatively straightforward—the kinds of tasks where you have a clear, predefined sequence of steps that is currently being executed by a human, but could conceivably be transitioned to a machine. This sort of automation has been happening in businesses for decades. Screening incoming data from an outside data provider for well-defined potential errors is an example of a problem ready for automation. (For example, hedge funds automatically filter out bad data in the form of a negative value for trading volume, which can’t be negative.) On the other hand, encoding human language into a structured data set is something that is just a tad too ambitious for a straightforward set of rules.
For the second type of problem, standard automation is not enough. Such complex problems require learning from data—and now we venture into the arena of machine learning. Machine learning, at its core, is a set of statistical methods meant to find patterns of predictability in data sets. These methods are great at determining how certain features of the data are related to the outcomes you are interested in. What these methods cannot do is access any knowledge outside of the data you provide. For example, researchers at the University of Pittsburg in the late 1990s evaluated machine-learning algorithms for predicting mortality rates from pneumonia.1 The algorithms recommended that hospitals send home pneumonia patients who were also asthma sufferers, estimating their risk of death from pneumonia to be lower. It turned out that the data set fed into the algorithms did not account for the fact that asthma sufferers had been immediately sent to intensive care, and had fared better only because of the additional attention.2
So what are good business problems for machine learning methods? Essentially, any problems that meet the following two criteria:
- They require prediction rather than causal inference.
- They are sufficiently self-contained or relatively insulated from outside influences.
The first means that you are interested in understanding how, on average, certain aspects of the data relate to each other, and not in the causal channels of their relationship. (Keep in mind that the statistical methods do not bring to the table the intuition, theory, or domain knowledge of human analysts.) The second means that you are relatively certain that the data you feed to your learning algorithm includes more or less all there is to the problem. If, in the future, the thing you’re trying to predict changes unexpectedly and no longer matches prior patterns in the data, the algorithm will not know what to make of it.
Examples of good machine learning problems include predicting the likelihood that a certain type of user will click on a certain kind of ad, or evaluating the extent to which a piece of text is similar to previous texts you have seen. (To see an example of how an artificial intelligence algorithm learned from existing customer data and test marketing campaigns to find new sales leads, see the sidebar “Artificial Intelligence at Harley-Davidson.”)
Bad examples include predicting profits from the introduction of a completely new and revolutionary product line, or extrapolating next year’s sales from past data when an important new competitor just entered the market.
ARTIFICIAL INTELLIGENCE AT HARLEY-DAVIDSON
by Brad Power
It was winter in New York City, and Asaf Jacobi’s Harley-Davidson dealership was selling one or two motor cycles a week. It wasn’t enough.
Jacobi went for a long walk in Riverside Park and happened to bump into Or Shani, CEO of an AI firm, Adgorithms. After discussing Jacobi’s sales woes, Shani suggested he try out Albert, Adgorithm’s AI-driven marketing platform. It works across digital channels, like Facebook and Google, to measure and then autonomously optimize the outcomes of marketing campaigns. Jacobi decided he’d give Albert a one-weekend audition.
That weekend, Jacobi sold 15 motorcycles—almost twice his all-time summer weekend sales record of eight.
Naturally, Jacobi kept using Albert. His dealership went from getting one qualified lead per day to 40. In the first month, 15% of those new leads were lookalikes, meaning that the people calling the dealership to set up a visit resembled previous high-value customers and therefore were more likely to make a purchase. By the third month, the dealership’s leads had increased 2,930%, 50% of them lookalikes, leaving Jacobi scrambling to set up a new call center with six new employees to handle all the new business.
While Jacobi had estimated that only 2% of New York City’s population were potential buyers, Albert revealed that his target market was larger—much larger—and began finding customers Jacobi didn’t even know existed.
How did it do that?
Albert drove in-store traffic by generating leads, defined as customers who express interest in speaking to a salesperson by filling out a form on the dealership’s website. Armed with creative content (headlines and visuals) provided by Harley-Davidson and key performance targets, Albert began by analyzing existing customer data from Jacobi’s customer relationship management system to isolate defining characteristics and behaviors of high-value past customers: those who either had completed a purchase, added an item to an online cart, viewed website content, or were among the top 25% in terms of time spent on the website.
Using this information, Albert identified lookalikes who resembled these past customers and created micro segments—small sample groups with whom it could run test campaigns before extending its efforts more widely. Albert used the data gathered through these tests to predict which possible headlines and visual combinations, and thousands of other campaign variables, would most likely convert different audience segments through various digital channels (social media, search, display, and email or SMS).
Once it determined what was working and what wasn’t, Albert scaled the campaigns, autonomously allocating resources from channel to channel, making content recommendations, and so on.
For example, when it discovered that ads with the word call—such as, “Don’t miss out on a pre-owned Harley with a great price! Call now!”—performed 447% better than ads containing the word buy, such as, “Buy a pre-owned Harley from our store now!” Albert immediately changed buy to call in all ads across all relevant channels. The results spoke for themselves.
For Harley-Davidson, AI evaluated what was working across digital channels and what wasn’t, and used what it learned to create more opportunities for conversion. In other words, the system allocated resources only to what had been proven to work, thereby increasing digital marketing ROI. Using AI, Harley-Davidson was able to eliminate guesswork, gather and analyze enormous volumes of data, and optimally lever age the resulting insights.
Adapted from “How Harley-Davidson Used Artificial Intelligence to Increase New York Sales Leads by 2,930%” on hbr.org, May 30, 2017 (product #H03NFD).
Brad Power is a consultant who helps organizations that must make faster changes to their products, services, and systems to compete with startups and leading software companies.
Find the Appropriate Data
Once you verify that your problem is suitable for machine learning, the next step is to evaluate whether you have the right data to solve it. The data might come from you or from an external provider. In the latter case, ask enough questions to get a good feel for the data’s scope and whether it is likely to be a good fit for your problem.
Ask Questions and Look for Mistakes
Once you’ve determined that your problem is a classic machine learning problem and you have the data to fit it, check your intuition. Machine learning methods, however proprietary and seemingly magical, are statistics. And statistics can be explained in intuitive terms. Instead of trusting that the brilliant proposed method will seamlessly work, ask lots of questions.
Get yourself comfortable with how the method works. Does the intuition of the method roughly make sense? Does it fit conceptually into the framework of the particular setting or problem you are dealing with? What makes this method especially well-suited to your problem? If you are encoding a set of steps, perhaps sequential models or decision trees are a good choice. If you need to separate two classes of outcome, perhaps a binary support vector machine would be best aligned with your needs.
With understanding come more realistic expectations. Once you ask enough questions and receive enough answers to have an intuitive understanding of how the methodology works, you will see that it is far from magical. Every human makes mistakes, and every algorithm is error prone too. For all but the simplest of problems, there will be times when things go wrong. The machine learning prediction engine will get things right on average but will reliably make mistakes. And these errors will happen most often in ways that you cannot anticipate.
Decide How to Move Forward
The last step is to evaluate the extent to which you can allow for exceptions or statistical errors in your process. Is your problem the kind where getting things right 80% of the time is enough? Can you deal with a 10% error rate? 5%? 1%? Are there certain kinds of errors that should never be allowed? Be clear and upfront about your needs and expectations, both with yourself and with your solution provider. And once both of you are comfortably on the same page, go ahead. Armed with knowledge, understanding, and reasonable expectations, you are set to reap the benefits of machine learning. Just please be patient.
__________
Anastassia Fedyk is a PhD candidate in business economics at Harvard Business School. Her research focuses on finance and behavioral economics.
NOTES
1. G. F. Cooper et al., “An Evaluation of Machine-Learning Methods for Predicting Pneumonia Mortality,” Artificial Intelligence in Medicine 9 (1997): 107–138.
2. A. M. Bornstein, “Is Artificial Intelligence Permanently Inscrutable?” Nautilus, September 1, 2016, http://nautil.us/issue/40/learning/is-artificial-intelligence-permanently-inscrutable.
Adapted from “How to Tell If Machine Learning Can Solve Your Business Problem” on hbr.org, November 25, 2016 (product #H03A8R).