Chapter 1. Introduction

This chapter covers

• The state of the AI project landscape today
• Distinguishing between critical and nice-to-have elements of a successful AI project
• Understanding business actions you can take based on AI project results
• A high-level overview of the process that a successful AI project should use

Today, the topic of AI comes up quite often, not only in the technical and business communities but also in the news intended for nontechnical audiences. Discussions of AI are even entering the domain of public policy. It’s likely that your own organization is considering the impact of AI and big data on its business, and that will lead to projects that use AI. I’ve written this book to help organizational leaders succeed with those AI projects.

As a consultant and trainer, I’ve been privileged to work with a large number of clients since topics like big data, AI, and data science have been taking off. Those clients have ranged from startups to Fortune 100 companies. Between projects, I’ve witnessed an emerging picture of the state of the industry. That picture includes many positive elements, with many millions of dollars made on successful projects. It also includes less talked-about projects. Those projects were managed in a way that doomed them from the start. But before they met their doom, those projects sent millions of dollars circling down the drain. The goal of this book is to help your project avoid becoming one of those doomed projects.

You might have heard of an AI platform called IBM Watson. The University of Texas MD Anderson Cancer Center partnered with IBM to create an advisory tool for oncologists. It was reported that Watson was canceled after $62 million was spent on it [5]! This example shows that even a high-profile project supported by a famous company with highly visible technology isn’t a guaranteed success.

In this book, you’ll see that the successful use of AI requires significant human involvement and insight. AI on its own isn’t a substitute for business knowledge, nor can it improve your business by solely looking at the data and making recommendations.

Warning

The fastest way to fail with AI is for the executive and business leaders to think, “AI can solve our problems; we don’t need to do anything except hire the right tech geeks and unleash them on our data,” or for the whole data science team to think, “Businesspeople take care of the business; we focus on technology.” Business and technology must work together for success.

Initially, this situation may seem disappointing, but on closer examination, it’s good news for a project leader. If AI could figure out your business, it would quickly put you out of a job. It can’t, so your job is safe for the foreseeable future. But to intelligently apply AI, you need special skills and knowledge to enable you to combine your business domain with it.

Warning

Technical knowledge regarding AI algorithms isn’t sufficient to get business results using AI.

This book teaches you what you need to know to run and get good results from an AI-based project. It’s assumed that you can run a general technical project.

The methods and techniques I teach are process neutral; you can use them in organizations of all sizes. To help you successfully run AI projects in such a wide range of organizations, this book focuses on the principles and skills that you must apply to your project, as opposed to providing rigid checklists and sequences of steps that you must perform in only one way. By learning these principles and skills, you can apply the techniques that are critical for the success of your project to any environment and process. But before we talk about how to get results with AI, let’s first review the skills you need to have to get the most out of this book.

1.1. Whom is this book for?

If you feel that insufficient information has been published on how to deliver business results with AI, this book is for you. You’ll find many books on the technical side of AI, data science, and big data, and some universities recently have started to add data science and AI programs. The result is that a lot of data scientists (and academics) know a lot about AI technology; however, they know far less about the business applications of AI.

I wrote this book for the business leader who is tasked with delivering results with AI and views technology as a vehicle to deliver those results. I also wrote it for the leadership team that’s working with and advising such a business leader. This section provides an overview of the skills these leaders need to follow the book.

To get the most value out of this book, you’ll need the following qualifications:

• You’ve been part of the leadership team of a successful software project.
• It doesn’t matter whether the project used Agile or some other software development methodology. It doesn’t matter whether the project used Java, Python, or some other programming language. What matters is that this isn’t your first software project and that you’re confident you can deliver a successful software project with the technologies you’ve used before.
• Whatever software development methodology you’re using (Agile or not), you must understand how your organization manages software development. This includes managing the requirements, deliverables, resources, and reporting mechanisms used to track progress in a timely fashion.
• You understand the basics of the business your organization is in, on a level commensurate with your position in the organization.
• This means that you understand your organization’s day-to-day business and what it involves, what business actions are possible for your organization, the main sources of income for your organization, and the basics of its budgeting process.
• If you’re a leader with profit and loss (P&L) responsibility, it’s also assumed that you understand how your business generates profit, as well as how to succeed in your business.
• You have experience with using business metrics to score the success of a business initiative.
• You know why metrics are important, how to measure the value of metrics, and how to recognize a metric that’s inappropriate for your business. Data science and AI are quantitative fields, and the data sizes used make it difficult to get an intuitive feel for how well a project is progressing based on a few examples.
• Although an engineering background or previous deep knowledge about AI isn’t required, an open mind and a willingness to facilitate conversations between people with technical and business backgrounds is.

With regards to the prior technical knowledge you’ll want to have, the due diligence you already did before you decided to join the AI project should suffice. You need to have a basic understanding of what terms such as AI and big data mean. You need to know that you’ll need data scientists on your AI team. And you need to know that before you can use an AI algorithm, you must collect the necessary data and train that AI algorithm. As long as you meet these qualifications, your official title doesn’t matter. Your position in an organization might range from executive or senior vice president to product manager. You may also be a software architect or a business-focused data scientist in one of the teams working as part of a much bigger project. Your team may be working on a big data infrastructure or may be analyzing data using various techniques from the fields of data science, machine learning, artificial intelligence, and statistics.

I also expect that the organization you’re working in already has a team with some basic technical knowledge of data science or can hire some specialists in the area, or at the very least has enthusiastic engineers willing to learn. You already know that you can’t hope to manage successful web projects unless you have people on your team who know something about web development. AI is no different. As long as your organization meets this basic requirement, you can use techniques in this book whether you’re in a large or small organization.

Warning

A team that pays attention to how to link technology with the business problem will beat a team that consists of (slightly) better specialists in a limited area of technology or business. If you aren’t taking a genuine interest in what the other side does, this book isn’t for you.

If your interest in business is limited, and your main interest is in AI methods and algorithms (or if your interest is mainly academic), this is not the book for you. My focus in covering any AI methods isn’t on the technology but on what those methods mean in the context of the overall business.

Note

If you’re a data scientist, this book will be of value to you, as long as you’re interested in how to achieve business results with AI. This book expects the reader who has a business background to be willing to learn the basics of how AI works. Similarly, if you’re a leadership-focused data scientist, you’re expected to be willing to learn about the business side of the equation.

If you’re interested solely in the technical side of the AI, you’ll be better off with books that focus on the technical side of AI. If you’re just getting familiar with AI, some good places to start for some current trends are Practical Data Science with R [6], as well as Deep Learning with Python [7] or Deep Learning with R [8] (depending on your programming language preference).

1.2. AI and the Age of Implementation

Organizations new to AI have a lot of confusion regarding how to best organize projects around it. The root cause of that confusion is that the ways AI is best used in academia and industry are different. This section highlights the differences and advises you on what aspects of AI are most applicable to the typical project in business and industry.

When you hear terms like data science or things like “we should use the methods described in this scientific paper,” it’s easy to imagine academic disciplines and diligent scientists trying to discover new scientific principles. That’s certainly happening, as plenty of recent and widely publicized research has been going on in the AI field.

Warning

Successfully applying AI to business problems is not research in the traditional academic sense. Unless you’re a researcher at a university (or in an industry-supported basic R&D effort) who’s getting paid for conducting research itself, additional research results are difficult to monetize. You need the best practices for implementing AI projects, and those practices are the focus of this book.

If you’re working in a typical corporation, you shouldn’t organize your AI efforts in the mold of traditional scientific research. There are debates as to whether business success in the AI field as a whole is even contingent on the need for new AI discoveries. Influential voices [9] in the AI community believe that, as far as AI is concerned, we are entering the Age of Implementation of our existing knowledge.

Kai-Fu Lee, in AI Superpowers [9], uses the terms Age of Implementation and Age of Discovery to describe the state of AI today. He states that AI could be considered to be in the Age of Implementation in the sense that, in such an age, a scientific discovery (for example, the steam engine or electricity) that’s already known needs to be widely disseminated throughout different areas of business and society. This is contrasted to the Age of Discovery, in which progress is driven by regular new discoveries of new scientific principles (for example, the second law of thermodynamics or the existence of X-rays). His argument that we are in the Age of Implementation is founded on the fact that many of the basic technical principles behind the current AI explosion have been known to the research community for a long time.^[1] What we’re doing today is applying known principles of AI to concrete business problems. For example, we know that AI excels in recognizing what’s in an image, and we’re now using that ability to recognize what is in that picture in a business context.

¹

For that matter, the argument could be made that a lot of the recently published papers are more focused on how to apply deep learning on the new problem domain rather than on investigating new fundamental principles of AI.

Tip

If we’re in the Age of Implementation, what we need most are practices for the best application of AI to new areas of the business. This book describes such practices.

Kai-Fu Lee is not alone. Andrew Ng, another prominent AI pioneer, believes that we need wider applications of already-known AI techniques to problems in business and industry, as opposed to more academic publications describing research results [10].

At the time of this writing, it isn’t clear whether AI’s long-term progress is more representative of the Age of Implementation or the Age of Discovery. While this may not be clear on the level of AI as a field of endeavor and its influence on broader society, in the context of an AI project you start in business and industry today, that question is settled. For the duration of such a project, you’re working in the Age of Implementation mindset. This book, therefore, focuses on applying what’s already known, so let’s start talking about how to succeed with AI in the context of your business.

Warning

In almost all industry projects, it’s dangerous for both your project and your career to depend on the need to make new scientific discoveries to be able to deliver the project.

1.3. How do you make money with AI?

There’s often a perception that using AI makes it possible for you to make a lot of money (or, in the case of nonprofits, help some cause). This section highlights the relationship between AI and making money.

What you may have heard is

Data + AI = $

This is partially true. Actually, in at least one situation, it’s completely true: if $ stands for cost, you can rest assured that, most certainly, those systems (and people operating them) would come with a significant cost. But if you want $ to stand for profit, the equation looks somewhat different:

Data + AI + CLUE = Profit

This book is about the CLUE part of the equation. I define CLUE in more detail in section 1.10; briefly, what I mean here is that you need to think (have a clue) about how to make a profit.

Often, people hope that data and technology will pave the way forward. That hope may tempt you to proceed with a significant investment without being sure of how to monetize the results of your data science project.

Tip

The main insight to remember is that using AI to analyze data doesn’t make money; properly reacting to the results of the correct analysis does.

If you don’t have any idea how you’d react to the results of a data project before those results are delivered, chances are you won’t have a clue about what to do with the results once you receive them. People who make money correctly tie technology to the business problem. The right time to figure out how to tie technology and business is at the start of the project, not once the analysis is already complete. Both now and in the foreseeable future, that tie-in would have to be conceived, engineered, and executed by people, not AI.

1.4. What matters for your project to succeed?

Before we talk about what matters most in an AI project, let’s talk about best practices of time and project management. Not every technology you’d use on your project is equally important for success. Nor is every part of your project equally important. Your time should be focused on those parts of your project that matter the most for success.

An AI project’s leadership has to make many choices, and it’s easy to get lost in them. Those choices are in many different technical areas—areas like what infrastructure to use, how to handle big data, whether to use cloud or on-premise solutions, and which AI algorithms to use. Within each one of those categories are many additional choices you’d have to make.

Here are some concrete examples of some of those further choices:

• If you’re using the cloud, is it AWS [11], Google Cloud Platform [12], Microsoft Azure [13], or something else?
• Do you need a big data solution? If you do, do you use Apache Spark [14], Apache Hadoop [15], or Apache Flink [16]?
• What monitoring and security infrastructure do you need?

It’s understandable that with so many choices to make, the temptation is to put the early focus of management’s attention on making the choices I’ve listed. If you see that happening, you should stop the discussion. You’re getting pulled into the details and should refocus the conversation back on the business problems you’re trying to solve.

Tip

Think about the following analogy. There is a similarity between going on a sailing vacation and running an AI project. If you’re sailing for the first time and have an experienced captain with you, your main expectations are that the sailboat is a seaworthy vessel that will sail and that it won’t sink. It’s of little relevance whether the sailboat is capable of a speed of 9 knots or is limited to 8 knots. Just as the first question for your sailing vacation should be where you intend to sail, the first question for your AI project is how to properly react to results of the analysis.

Although you must choose your infrastructure correctly (on a do not sink level), the primary determinant of the AI project’s success lies in how well you connect the answer to your research question with a specific business action. Once you’re confident in that connection, then you can focus your attention on infrastructure and making good infrastructural choices.

Tip

When embarking on an AI project, the highest priority for management attention must be on linking a research question to specific business action.

Speaking of infrastructure, it’s easy to assume that you need to start by building infrastructure that’s capable of supporting any conceivable AI project. You have plenty of choices in the space of big data and AI frameworks. But always remember that those frameworks are just enablers. As a project leader, you must be careful that you don’t intermix the enabler with the value you’re building. Your primary focus should be on value. You should never allow yourself to be in the situation in which you’re discussing infrastructure but have no concrete idea of what specific business questions you intend to use AI to answer.

Tip

Vendors make the fair point that if you have many AI projects using different and incompatible infrastructural stacks, you’re risking a significant amount of chaos in the infrastructure space. I also agree that infrastructure is an important consideration, as there are real and substantial differences between the technology stacks of different vendors. I encourage you to pay attention to infrastructure, but not at the expense of losing focus on the business value you’re creating.

Starting with technology stack considerations as your main focus can lead you into the trap of overconcentrating on the infrastructure, and you can lose focus on the sequence of data science use cases that you should implement. Remember, your attention is a finite resource too.

Tip

Let’s embrace the analogy that data is the new oil. If you need a new oilfield, you’d put your greatest focus into finding oil and understanding what an oilfield looks like. You most certainly wouldn’t start by buying the best oil drilling equipment in the world, with the drilling location as an afterthought. The same thing applies to data—concentrate on finding oil, not on the drilling equipment.

There is one exception to the “Don’t start with the infrastructure” rule. If your enterprise is large and has already made the decision to adopt data science across the board, you may need an infrastructure that supports as wide a range as possible of technologies used in AI today. Only the largest companies with the biggest data science departments are in this position, and, for them, my advice is that they build infrastructure in parallel with the definition and development of data science use cases that need to be supported.

Finally, because this field is rapidly evolving, in this book I don’t discuss any of the individual frameworks available today. Some are better than others, but I want this book to outlast the current generation of frameworks.

1.5. Machine learning from 10,000 feet

For the successful application of AI to business problems, close human involvement, with the application of engineering and business skills, is essential. This section explains the roles that humans will need to fill for a project using AI to succeed.

AI and machine learning (ML) are closely related. ML allows computers to learn an underlying pattern in data on their own, without being programmed with the prior knowledge of those data [17]. Today, in business and industry, most AI projects use algorithms that are (strictly speaking) part of ML.^[2] There’s a lot of confusion among the wider public about what ML can do, so if you aren’t an expert in ML, it’s easy to imagine that ML is some magical box that’s given data and somehow produces an impressive result.

²

Exceptions are in the field of robotics, which can use algorithms that are traditionally not part of ML but are part of AI as a broader field. The broader field of AI includes not only ML, but also fields of robotics and machine perception [18]. To address the broadest category of projects used in the industry today, the early chapters in this book primarily concentrate on projects using ML.

Let’s clarify just how that somehow happens. ML uses various algorithms—a sequence of predefined steps—that produce an answer. An ML algorithm guarantees that if you provide it data in a particular format with some metric related to its output, it will generate some mathematical transformation that reveals the optimal value of the metric that you provided. ML isn’t a genius that discovers what you missed. It’s an idiot savant that performs complex math on the numbers you feed it, but it has no comprehension of the meaning of those numbers. The only goal an ML algorithm has is to find an optimal value of a given evaluation metric, all the while not knowing or caring about why it’s optimizing that particular metric. Figure 1.1 shows what an ML algorithm is actually doing.

Figure 1.1. A machine learning algorithm produces a result that minimizes the evaluation metric. At its core, machine learning is data transformation from some predefined input to an answer in a predefined output format. It’s a human’s job to align the evaluation metric with the business goals.

Tip

Informally, ML is the application of some mathematical algorithm that requires data input in a certain format, produces an answer in a predefined output format, and doesn’t provide any other guarantee except that it will minimize some number that we call the evaluation metric.

When you look at the definition provided in the TIP, ML certainly doesn’t sound magical, nor is there some genius lurking around in the algorithm. The trick of using ML isn’t so much in what ML algorithms do, but in how you use them.

Let’s expand on how data scientists use ML. Who defines the format of the data, the answer, and the metrics? To understand how to apply ML algorithms, it helps if you think about ML in the form given in the paper “A Few Useful Things to Know About Machine Learning” [19]:

Machine Learning = Formulation + Optimization + Evaluation

Those terms are defined as follows:

• Formulation— Finding a clever way to describe a business problem so that it relates to the input and output data that your ML algorithm knows how to process. This means you package data related to your business problem in the format an ML algorithm expects as its input and find a way to interpret the answer provided by that ML algorithm in your business context.
• Optimization— What the ML algorithm does internally. It’s the process of applying math to get an optimal solution. How is that optimal solution defined? It’s defined according to an evaluation metric.
• Evaluation— The application of the aforementioned evaluation metric to measure the success of the optimization.

It’s up to you as the user of the algorithm to understand which evaluation metric is best and what the best format is for your data. Figure 1.2 shows the roles that humans and computers play in ML.

Figure 1.2. Machine learning is a combination of formulation, optimization, and evaluation. The only part that the computer can do (mostly) on its own is optimization.

Wait—what do all these concepts in figure 1.2 even mean? What do they have to do with your business? As far as ML or AI is concerned, it doesn’t care at all about your business. That’s your job.

Tip

If you’re an executive, one way to think about ML is as a black box that operates on numbers. Who then is responsible for bringing intelligence to that ignorant black box? Humans, through the proper formulation (that’s the job of data scientists) and application of the appropriate evaluation metric that measures something relevant to your business (that’s your job as an executive, in cooperation with the data scientist).

If you’re a data scientist, you’re aware of what ML is. But here’s a detail that’s easy to miss: because the only guarantee provided by the algorithm is that it will optimize the evaluation metrics, those metrics must have a business meaning. Those metrics can’t just be something you’ve seen in an academic paper, and certainly not something you don’t know how to relate to anything concrete in your business.

When a data science team is choosing the evaluation metrics, they must do so in close cooperation with the executive team. Here’s the scary thing—a lot of ML projects use evaluation metrics that aren’t clearly tied to business results. Evaluation metrics have their roots in ML theory and history, as opposed to in business. At best, most of the technical evaluation metrics are merely correlated with but not identical to the business result you’re trying to get. At worst, those evaluation metrics that your data scientist can’t even explain in business terms are in the will not help your business category.

Tip

If you’re an executive, a good habit for you to form is to speak up when you see an acronym in a business presentation that you don’t know (for example, RMSE).^[3] Similarly, with ML, demand an explanation in the form of, “Don’t tell me the mathematical definition of that metric; tell me how to relate the value of that metric to something in my business.”

³

RMSE stands for root mean square error metric, defined with a mathematical formula whose terms aren’t inherently related to any business concept. Chapter 4 shows you how you can express the RMSE using metrics related to business.

If your evaluation metric isn’t tied to your business result, you’re using a black box (the ML algorithm) to generate a random number (a value of the evaluation metric) and then trying to figure out how to run your business based on that random number. Good luck!

1.6. Start by understanding the possible business actions

Focused human involvement and attention makes or breaks an AI project. This section discusses the first questions you should ask in the context of every AI project, as well as the most important ideas to keep in mind during any AI project.

You don’t make money simply by knowing the answer to a business question—you make money when you take action. And that action is constrained by what you can and can’t do in the physical world (for example, the number of possible management decisions). A business action can require approval from your boss(es), the creation of external partnerships, and getting the whole team to buy in on that decision—that’s the real world we’re talking about.

Tip

The number of good, effective actions you can take to affect the physical world is relatively small. Many analyses you can perform will yield results that are not actionable.

The limits to business actions you can take can come in many forms. They may be limits of knowledge and know-how in your organization, or the limits of your budget. They may be imposed by the organization’s internal politics and what you’re able to rally people around, or they can be the result of which battles you’re willing to fight. Whatever has summoned these limits into being, you’re stuck with them.

You can spend a lot of time on analysis, get some results, and then say, “Well, there’s nothing I can do to change this.” The time and money spent on such analysis is wasted, and it is a preventable waste.

Tip

Don’t ask a question if you can’t imagine what you’d do with the answer. You should start an AI project by asking, “What actions can I take, and what analysis do I need to do to inform those actions?”

Once you know the business actions you can take, you should use these actions to drive the analysis, not vice versa. Figure 1.3 illustrates the relationship between the business actions you can take and the possible analyses you can perform.

Figure 1.3. You can perform many more analyses than there are business actions you can take. Don’t spend a ton of time and money on an analysis just to figure out it was never an actionable one. (Earth image is from Wikipedia [20].)

There’s always something in the data, and it’s possible to conduct hours of analysis on only a few hundred numbers.^[4] Often, you’ll find some interesting properties in even a small dataset. The problem is that the large majority of those analyses fall into the category of that’s interesting, but not really relevant to the business. If you’re focused primarily on analysis, you can spend a lot of money and time on interesting analysis and then not have any idea of how to execute on the results.

⁴

Improving business with data is not new. There’s a long history of projects that were done in the factory and business process improvement space prior to the rise of big data. Those projects were done on small datasets but often used complex statistical analyses that required high technical proficiency with statistics and significant time to perform. Many such projects would fall under the umbrella of Six Sigma—see [21] and [22] for details.

In every AI project, these are the two most important ideas to remember:

1. Action is where you make profit; analysis without action is just cost. You make money when you perform an appropriate business action, not when some analysis is completed. Analysis can be an enabler of making profits, but from an accounting perspective, analysis is a cost. Analysis stops being a cost and becomes an investment only when it can help you take good business actions.
2. To succeed, focus on the whole system, not on its individual parts. Your customers will never see the individual parts of the system. For instance, they don’t care about the ML algorithms that you’re using. They care only about the result, and that result depends on how well the system works as a whole.

1.7. Don’t fish for “something in the data”

Teams that fail to start a project by focusing on which business actions they can take will routinely encounter two failure modes. They typically start by copying something that “worked for another organization,” or use AI to look for “something in the data.” This section explains why neither of those approaches works well in practice.

Copying parts of a system that worked for someone else doesn’t work, because that copying then confines you to the decisions someone else made for their context, whereas your project should deliver a whole system customized to solve your business problem. When you copy someone else’s solution, you have no way of knowing if that system will be useful in the context of your problem and your organization. You may then learn too late that it’s too difficult to modify the copied system to solve your business problem.

Nor does it work to collect a lot of data and then throw various AI methods at it hoping that one of them will reveal an action you’ve never considered. Suppose it does. What’s the chance that you’d actually be able to execute on some random action that the analysis just divined? How much money do you want to spend on the off chance of getting an idea unrelated to your day-to-day business operations?^[5]

⁵

One extreme case of this question is “How much would you pay for just an idea of a totally new business that you can undertake, without any associated execution of that idea?” All venture capitalists have encountered this situation, and the usual answer from that community is “I wouldn’t pay anything.”

Sometimes you can execute on an unexpected insight. Your batting average will be much higher if the studies you perform are related to areas of your business where it’s obvious that you can take action based on the possible results of your study. For example, one of those famous anecdotes you might have heard is that analysis has shown that in supermarkets, beer and diapers are often bought together [23].^[6] What happened next? Supermarkets supposedly put diapers next to beer and saw an increase in beer sales. The ability to react on the analytical result was already there, which is why the association study was done in the first place.

⁶

This popular story has many details wrong or changed in the retelling, but, for the sake of argument, let’s go with it in the form it’s most often told.

1.8. AI finds correlations, not causes!

One word of warning, which I’ll cover in more detail in chapter 8, regards a common misunderstanding that I want you to be aware of early on. AI as practiced in business and industry today finds correlations, not causes. This is important to recognize because newcomers to AI, after seeing AI answer a complicated question, sometimes anthropomorphize AI and attribute to it the ability to know the reason behind the answer when it’s often just guessing.

Humans are causal model driven. Even if they don’t know how a given system works, they usually have a theory, a model, of how that system works. The models built make it possible for humans to reason about cause and effect. Because that’s how we work, we have the tendency to assume that AI works that way too. That’s not correct.

When a human sees A and B going together, the human would easily give you some theory (very often, not a correct theory) of why A and B are causally related. AI wouldn’t be able to give you any causal theory (much less ensure that the theory is correct), it would just say, “I think that A and B go together.” You as a human can reason about causes, but AI as practiced today can’t establish causal relationships on its own.

Note

If your neighbors look distant every time they see you, you might assume that they don’t like you or that something is troubling them. You’ve just assumed a model of why people behave in certain ways. AI might predict that your neighbors will appear distant the next time you see them, but few AI algorithms would impute any cause: they don’t say why it is, just that it is.

When you’re thinking about what you need AI to do for your business, keep in mind the differences between the human use of causal models and how AIs work:

• The expectation that AI can automatically find causes (as opposed to patterns) isn’t realistic in most cases.
• If you’re required to explain why an AI algorithm made some decision, you must carefully consider how to translate the algorithm’s behavior in a way that you and your audience can understand.
• Most AI algorithms used today, especially in the big data domain, have a limited ability to guide change. That ability is limited to situations in which you have data for both where you are now and where you want to go. Because AI recognizes patterns, if you have no data (or lack the ability to collect new data) on how the world will look after a change occurs, AI can’t do much to guide you.

Always tread carefully when you think about using AI to impute causes or to guide change.

1.9. Business results must be measurable!

For an AI project to succeed, you need to be able to measure the results of the data science project in the context of its business impact. That measurement needs to be quantifiable. AI and ML algorithms can’t use gut feeling metrics as feedback on how well the project is doing, so someone needs to define a quantitative metric. This section discusses important considerations and pitfalls of the business metric used in the context of AI projects.

Before you can measure how your data scientists would impact your business, you need to be able to measure your business as it is today. As a business leader, you should have a way to measure how well your business is doing based on some numerical metrics directly tied to the business. Some examples of such metrics are revenue, number of customer purchases, and internal rate of return (IRR). Once you have those business metrics, you should have a way to tie those metrics to the technical evaluation metric that the ML algorithm uses. Chapter 4 describes in detail how to make that connection.

Depending on the methodology you’re using to run your business, business metrics could be readily available, or you might have to develop them yourself. Various methodologies are available for running a business and measuring its results. If you’re looking for a starting point, the Lean Startup methodology described in Ries’s book [28] is popular in startup and IT settings. I also recommend reviewing the Business Performance Excellence (BPE) model described in Luftig and Ouellette’s book [1].

Note

You may not have a direct business metric at your department level to which you can direct your data science project. If that’s the case and you don’t already have a metric to use, define one yourself.

Although I could provide examples of business metrics you could use, defining appropriate business metrics isn’t a simple operation; there are many considerations to take into account to do it right. If you define the wrong metrics for the business problem you’re trying to solve, your employees may work toward maximizing metrics as opposed to solving the business problem. At a minimum, the methodology you’re using should ensure that the metric correctly measures whether the business goal is achieved and that the metrics on the department level help you align individual departments toward a common goal. A number of resources are available to you [1,2,29–33], and you may want to review some of them for a few guidelines.

The complexity associated with recognizing and defining good business metrics is exactly why I recommend that you develop a thorough understanding of the operations of your business. You need to recognize the appropriate metric to run your business with, versus which metrics might sound reasonable but will actually not measure what’s really important for your business success. For example, it’s your job to understand if it’s better to direct your business in the next six months based on preferring to increase revenue growth or to increase total number of users.^[7]

⁷

Quite often, early stage startups are looking for high growth. Such startups prefer to increase users in the short term, before they reach profitability.

Note

A good business metric yields a number that’s directly related to some business result. Such a metric is actionable. In the case of a recommendation engine, a good business metric could be expected increase in profit. Any technical metric that has no clear meaning in the context of business is obviously not a good business metric. In the absence of quantitative business metrics, you’re running a project based on gut feeling.

What if you can’t measure business results in a numerical way and can only use gut feelings? Then you’re at a severe competitive disadvantage, at least as far as AI projects are concerned. Businesses that can’t systematically measure business results aren’t well suited for running AI projects. If the project’s progress is measured based on the gut feeling of a business leader, then measuring that progress takes considerable time and involvement of that leader. Even if a business leader’s gut feeling is 100% accurate, due to the limited amount of times the data science team can consult the business leader, they will only get one or two data points on how well their AI project is doing. It’s difficult (and often impossible) for data scientists to optimize their approach based on so few data points.

Without a business metric to measure progress, you’re asking your data science team to use mathematics to produce something you already told them can’t be quantified. To me, if a business result can’t be quantified (at least approximately) with a metric, there’s no reason to believe that math-based tools can help. And make no mistake—AI algorithms are a math-based tool.

1.10. What is CLUE?

Many data science projects today are taking a haphazard approach, which doesn’t amount to much more than “Let’s look at the data and test the hypothesis that comes to our team members’ minds, hoping we’ll find the right one.” The elephant in the room is that few things you find in data are actionable. The end result is likely to be a long, unpredictable research project that leaves you uncertain about its ability to ever turn a profit.

In the absence of a systematic process for generating and evaluating hypotheses, and then cost effectively managing that, you’re missing a clue. Figure 1.4 shows you elements of the CLUE.

Figure 1.4. Elements of CLUE. AI projects that don’t have good answers for all elements of CLUE experience difficulties.

CLUE is an acronym for

• Consider (available business actions)— What business actions are possible for you to take? What business decisions can you make? What are some possible options for those decisions? How can you best choose among those options? Once you know what business actions you can take, the obvious question is, “Why haven’t you taken that action?” Clearly, you have some doubt, some question that needs to be answered before you can take that action. That’s good news! You now have a business question that should be answered with AI.
• Link (research question and business problem)— The information you need to make a choice among available business options leads you to research questions that your data science should answer. You need to ask that research question in the form that a typical data science project can answer. You’ll need to link the business problem, the business action to be taken, and the research question with each other.
• Understand (the answer)— How will you know what the answer means when your chosen data science method produces it? How can you translate some technical metrics produced by the data science method (for example, a metric like RMSE) to your business domain? For that matter, how many people in your average business meetings know what RMSE means? If they don’t know it, then why is a result presented using that metric?
• Economize (resources)— Your resources are scarce. Run your data science project in the way that provides you the highest expected payoff for resources invested.

Although subsequent chapters elaborate on each one of these elements, let’s just quickly expand on why these steps are important.

You need to start contemplating your options from the business side. You can find a lot of the non-actionable things in the data. Conversely, there are much fewer actions you can execute on. You should start from the few things you can execute on.

Note

If you’ve never encountered RMSE and are wondering what it is and what it has to do with your business, you’re not alone. Just throwing a technical metric at business users is confusing. What units is 2.83 in—dollars, meters, points? Depending on the business problem, that RMSE can be all of the previous units—or none of them. If you are a data scientist, always use business metrics in presentations to the business user. You’re better positioned to translate technical metrics to business terms, and you should do so before a meeting. If for any reason you can’t do it, how can you possibly expect a much less mathematically inclined business user to do it, in real time, during the presentation and using only mental math?! Chapters 3 and 4 discuss the process of linking technical and business metrics, and appendix A has an explanation of RMSE.

Once you know what business actions are available, you can start an AI project that will help you choose between those actions. Every project has finite resources, so you should be a good steward of those resources. The foundations of the project you establish (for example, the architecture of the software or the business relationships you need to form to get data and execute on the project) need to be right the first time, because it will cost you a fortune to change them. The E in CLUE (economize) is about how you combine project management and data science best practices so that you can assign resources optimally in your project.

An overarching theme here is that a properly organized AI project can be managed with the management tools that successful executives are already familiar with. CLUE is the glue between the existing management practices you know and those specific to data science projects.

1.11. Overview of how to select and run AI projects

Throughout the rest of this book, I construct a process for getting concrete business results with AI. Let’s briefly review the process here (figure 1.5).

Figure 1.5. High-level overview of the process that a successful AI project should use. This workflow shows you how to make sure that all elements of CLUE are part of your project.

This process should be understood as a group of considerations that must be addressed in a successful AI project, not as a sequence of steps that always must be executed in the same order. If some part of your existing process already addresses the given considerations, you don’t necessarily have to modify it.

For your process for running an AI project to be successful, you must do the following:

• Distinguish between what’s most important to get right for success (having a CLUE) versus what’s nice to get right—that which is mostly supporting your project. In the sailing example I gave earlier, the most important question was, “Where do you want to sail?” not “What is the max speed of the boat you’re on?” Similarly, in the AI project, having a CLUE matters more than the infrastructure you’re using, the collection of volumes of data, or the knowledge of the exotic, recently published AI algorithms.
• Start your analysis with the business actions you can take. If you instead start by saying, “Let me try some AI algorithms on data I already have,” chances are that even if something is found in the data, you won’t know what business action to take based on that discovery.
• Once you understand the business actions you can take, consider why you haven’t taken them yet. You typically have a question or uncertainty there, and that’s how you find a research question that your project should answer. On the business level, it’s a question that, when answered, will cause you to take some business action. It’s important to understand that a single business question can require from one to many research questions to answer. For example, you can break an overarching question such as “What inventory mix should I hold?” into several research questions like: “What are the best-selling items I have in inventory? What’s the predicted supplier’s availability of the items I have in inventory? What are the predicted future sales of these items?” You might be able to think of others as well.
• Not all of the projects are equally valuable or equally easy to do. As you already know, you need to triage them so that the easiest to implement, high-value projects are scheduled first.
• Formulate the business metrics you’ll use to measure the results of your business actions. One of the common errors facing AI projects is that they make technical decisions without carefully considering the business impact of those decisions. The business metric is there not only to help your executives run the business, but also as a quantitative measure whose importance both the AI and business teams agree on. This business metric could be specific to each research question you ask, or a single business metric can apply to multiple research questions.
• Initiate individual efforts that will answer your research questions. It isn’t necessary to initiate one project per research question, and the exact number of research questions that a project should answer depends on the research questions, team expertise, and the difficulty of answering the questions (or obtaining data for them).

At the end of this process, you’ll have a set of AI projects that are in advantageous positions to succeed and make a difference. You’ll know how difficult the implementation of those projects will be. You’ll know that there’s an implementable business action that you can take once AI provides you with an answer, and you have a way to measure how beneficial that action will be for your business.

1.12. Exercises

I strongly encourage you to complete these exercises to get a better understanding of the material covered in this chapter. This book’s exercises highlight and reinforce the best practices in, and common pitfalls of, AI projects in business. If you elect to skip the exercises, I still recommend reading the answers to the questions.

These exercises may introduce some new concepts that aren’t discussed in the chapters, but they should be already familiar to you or, if not, well within your ability to grasp. This is intentional and will help you practice the application of the skills and concepts you learned in this chapter to new business situations.

Solutions to the exercises are in appendix B at the end of this book.

1.12.1. True/False questions

Answer the following questions with True or False.

Question 1:

You always need a lot of data to make significant money with AI.

Question 2:

The first step when starting an AI project is to select the right technology tools to use.

Question 3:

Sometimes, simple AI algorithms can produce large business results.

Question 4:

Some tools can significantly automate AI projects. Just by using those tools, you can ensure a significant and lasting advantage over your competitors.

Question 5:

Making money with AI requires a PhD in math, physics, or computer science.

Question 6:

Every AI PhD is guaranteed to know how to make money with AI.

Question 7:

All AI tools are created equal.

Question 8:

You’re a project executive, and you leave the definition of the evaluation metrics to your data science team. Unfortunately, your data science team doesn’t have strong business domain knowledge, and they provide you with a metric that you don’t understand—let’s call it the Gini coefficient. If they do well on that metric, the project will help your business.

1.12.2. Longer exercises: Identify the problem

A short narrative description of a hypothetical project or actions taken during the individual projects follows. What’s your opinion of the situation described?

Question 1:

A friend who works in the IT department of an organization somewhat similar to yours uses tool X and approach Y with great success. Should you use that tool and approach because your friend was successful with them?

Question 2:

X, a Fortune 100 company, begins their AI efforts by creating an infrastructure holding petabytes of data and buying an array of tools capable of solving a broad spectrum of AI problems. They’ve also created a department responsible for using and maintaining all those tools. Should you buy the same set of tools?

Question 3:

You want to start your AI efforts with the use cases that other people successfully employed. Can you ask consultants with AI experience for an example of AI use cases often seen in your industry?

Question 4:

What’s wrong with the following approach? You’re seeing that AI is getting better in video recognition. You plan to start an AI project that would apply AI to recognizing and scoring Olympic skating. By using such an AI, you can show the viewers what the predicted scores would be as soon as the skaters are done, without needing to wait for the judges. Your AI solution must be ready before the next Olympics.

Question 5:

Is the following a good idea? You’re in a heavily regulated industry that delivers products to end consumers. You have to run all your changes by a regulator, and changes are evaluated (almost exclusively) based on legal compliance, with a typical change taking five years to be approved. You plan to use AI to understand online customer feedback and your customers’ satisfaction. The technical term for this process is sentiment analysis.

Question 6:

What are some problems with the following proposal? We’ll use this AI and feed it patterns of our customer behavior, and it will reveal to us the causes of our customers’ decisions.

Question 7:

You’re working in a domain in which it isn’t easy to define business metrics that you can use to measure the business result. Someone has proposed to use AI and make business decisions based only on technical metrics. Is this a good idea?

Summary

• Machine learning (ML) is a combination of problem formulation, optimization, and result evaluation. The only part of ML that artificial intelligence (AI) can (mostly) do on its own is optimization—a human has to do everything else.
• Profit isn’t made when AI provides an answer to a research question; it’s made when you take a business action. Always start by asking, “What business actions can I take?”
• AI is mostly about finding correlative patterns: A is often present in vicinity of B. AI has significant limitations in its ability to impute cause or operate in the changing world.
• Succeeding with AI requires a CLUE, the acronym for Consider the business actions available, Link a research question to the available business action, Understand the answer to the research question in the business context, and Economize scarce resources. Total absence of any of these elements is usually enough to doom the AI project.
• You should use a standardized workflow in every AI project. That workflow consists of triaging the possible business actions, defining research questions, defining business metrics to track, and estimating project difficulty and expense.