Chapter 3. Choosing your first AI project

This chapter covers

• Selecting AI projects that are matched to your organization’s AI capabilities
• Prioritizing your AI projects and choosing which AI project to run first
• Formulating a research question that’s related to a business problem
• Pitfalls to avoid when selecting AI projects, and best practices of such projects

To develop a sustainable analytical organization, you shouldn’t start with an AI project that involves complex technical challenges. Instead, you should choose your initial project so it provides clear and actionable results quickly. Your whole process should be organized to optimize time to success.

This chapter shows you how to select your first AI project. It also teaches you how to check if the research question that your AI project uses correctly reflects the business concerns it’s supposed to address. Finally, it presents a list of common pitfalls that young AI teams might fall into.

3.1. Choosing the right projects for a young AI team

I assume that your long-range goal is to build an AI team that will help the success of your parent organization by delivering a series of successful AI-related projects. To achieve that, you need to understand the journey that a successful AI team will take. This section explains that journey.

Tip

If you’re after a one-off AI project, you might be better off buying an off-the-shelf solution or contracting with an outside partner to do it for you.

One of the most crucial decisions that you as a leader need to make is how you want to prioritize the order of the initial AI projects that your team must undertake. Before you’re ready to make that decision, you need to understand its impact. To understand how AI teams succeed or fail, you first need to understand what success and failure look like.

3.1.1. The look of success

Leo Tolstoy wrote in Anna Karenina [70]:

• All happy families are alike; each unhappy family is unhappy in its own way.

Likewise, all successful AI teams are alike: your AI team is growing in expertise (and possibly headcount) and is solving more and more complicated problems. Unsuccessful AI projects result from an assortment of errors (many of which are described in section 3.4), and they may take your whole AI team down with them. This section explains why you should start with projects that are quick to deliver but still provide significant value for your business.

If you’re initiating AI efforts in your organization (or even if you’re part of an established analytical organization), you’re subject to three forces:

1. Plentiful opportunities— You’re operating with technologies (AI and big data) that weren’t present in business and industry historically, and you’re the first to apply them to the many datasets your organization has.
2. Limited time and resources— You have limited resources to devote to analysis. Chances are, you don’t have enough qualified people to run AI projects you’re thinking about.
3. Success makes you stronger— If you make money for your business, your analytical resources will increase over time. Management invests in teams with a good track record of providing value. Additional data scientists will want to join projects with a history of success. Solve some easy problems first, and you’ll get the resources you need to address larger problems.

How do you succeed in an environment like this? Does it make sense to first concentrate on large wins regardless of how difficult they are (for example, projects that provide significant monetary value)? Clearly not. What’s difficult today will be easier tomorrow, so start with a project that has significant monetary value and can be delivered quickly.

Tip

The key is a fast turnaround on initial projects so that you can learn quickly. You’re the first one in your company that has applied AI to your data and business problems, and there are plenty of opportunities to succeed with AI. Frankly, if you can’t find an easy win in such a setting, then AI simply can’t help your business.

Let’s use another analogy here. The position your team is in (with respect to the opportunities) would be similar to a hunter that finds a rich hunting ground. If opportunities to make money were animals, and you were a prehistoric hunter, you’d be operating in a target-rich environment (see figure 3.1).

Figure 3.1. You’re in a rich hunting ground—plenty of rabbits and a big mammoth are in sight. Which animal should you try to catch first?

Building a successful analytics organization is similar to surviving and prospering as a hunter. Now, ask yourself, “If I were in that position, would I aim for the biggest animal in the field first?”

The answer is probably no, if for no other reason than that you’re at the end of a long and distinguished line of ancestors that have succeeded in passing their genes to the next generation. They had the common sense and the skills to survive. You don’t initially have the hunting skills that bring you success if you go after the mammoth first! If you’re like me, the author, chances are that by the time that mammoth turned around, you’d deep-six it, run far away, and convert to vegetarianism for the rest of your life. But I’d like to believe that even I can be successful snaring a rabbit. All successful hunters should be able to catch rabbits (see figure 3.2), right?

Figure 3.2. Start with easy projects. Success with those projects enhances your skills and reputation within the rest of the company, allowing you to attempt more difficult hunts later.

You don’t want to start with a technically difficult project that requires a long time to deliver, even if that project is perceived to have a high business value. If your team is the hunter, the easier projects are the rabbits of the project world.

Tip

Once you’re known as a good hunter, the rest of the tribe is going to be more willing to help you with hunting mammoths. As your AI team learns and builds a solid reputation with the executive team, you’ll get more resources. That’s the time to take on difficult AI projects.

3.1.2. The look of failure

What must you avoid to prevent a massive failure? Yes, there are many ways in which individual small AI projects can fail, but there’s only one general way in which a whole AI team fails. Total team failure occurs when the AI team places all their bets on a single AI project that subsequently fails.

Let’s extend our previous analogy of the hunter. How does the hunter fail? At the end of the day, a successful hunter has something to eat, and an unsuccessful one doesn’t. Why do unsuccessful hunters starve in a target-rich environment?

In a target-rich environment, you don’t starve because your hunt for the biggest animal failed. You starve because you spent too much time chasing the biggest animal, overlooking smaller ones that would have been an easy dinner. The time to hunt big animals is when you’ve honed your skills on the smaller ones and have a full belly.

Note

When you’re just starting your AI efforts, you’ll have a hard time assembling a large enough team that’s capable of tackling your biggest, toughest opportunity. The dangerous approach, “Let’s go for the big opportunities first,” too often becomes a risky bet that can destroy your team.

Choosing to start with the technically challenging projects, even if they have a higher perceived value, is dangerous. If your project is complicated, your analytics team might not have enough resources to run any other significant projects. All of your eggs are in one basket.

Figure 3.3 shows what can happen if you take on a project that’s too hard for the initial skills of your team.

Figure 3.3. You’ve cornered the mammoth on your first hunt. What are you going to do now?

Sometimes, if you start with a complicated project, you might get lucky. Perhaps you’d be able to keep management’s trust for long enough to be able to deliver a successful project, and success can help when tackling larger efforts later. But is betting on that kind of support wise, or is it a huge risk in which your most precious resource (your team’s time) is spread thin on one large attempt? Furthermore, remember that this type of success will also set high expectations for future accomplishments. Even if your first project succeeds, now you have to find another large and risky project. How long will it be before your luck runs out?

When running a data science team, the real dangers lie in taking on a complex project, persisting too long on the wrong track, monopolizing your scarce resources, and having nothing to show for your efforts. All the while, you’re incurring significant costs. Also, you’re putting management in the position that they have to continue supporting an expensive project that doesn’t deliver any result quickly. What if they decide that pulling the plug is the rational thing to do?

Warning

When you’re building an AI project that will be used as a decision support system, your business organization needs time to learn how to implement the results of your AI-powered analytics. If your management team gets nervous before the technical part of the project is completed, your project is dead on arrival.

3.2. Prioritizing AI projects

How do you choose the right first AI project? Simple: it must be, from a business standpoint, viable, valuable, and simple. That means each project must be able to deliver a result that’s actionable for your business. It must have a significant business value, and you must be able to estimate how difficult it is to deliver. This section shows you how to create a list of projects that meet that criteria. Figure 3.4 describes the process that you should use to create a list of projects.

Figure 3.4. The C part of the CLUE allows you to create a list of viable AI projects and estimate their complexity.

Figure 3.4 shows these elements:

1. Start by looking at all the business areas your team is responsible for.
2. In which of those areas is it possible to apply AI and make sure you cover all elements in the Sense/Analyze/React loop? (A good reference for this step is figure 2.4 in chapter 2.)
   • Start with React, by finding the available actions that can be taken. (See section 3.2.1 for details.)
   • Then make sure that you can cover the Sense and Analyze side of the loop. (See section 3.2.2 for details.)
3. Determine which business metric you’ll use to measure how much your AI project is helping you to achieve the business goal. (See section 3.2.3 for details.)
4. Estimate the business value of the given AI project.
5. Estimate the difficulty of implementing this business case and how long it will take to implement (section 3.2.4).

I’ll assume you’re able to estimate the business value (step 4) of the AI project on your own, as I don’t know your organization or your business as well as you do. The rest of this section shows you how to implement the other steps in this workflow.

3.2.1. React: Finding business questions for AI to answer

After reading chapter 2, you’re already aware of AI taxonomy based on the role AI plays in business. Using that taxonomy is a good way to elicit business actions that can be used in the React loop. This section shows you how to apply that taxonomy.

You find the appropriate React part of the Sense/Analyze/React loop by using a process of elimination. First, look at all the areas of your business. Then determine which of those areas will benefit from AI by applying an AI taxonomy at a high level (as described in figure 2.5 in chapter 2). Figure 3.5 shows you how to use that taxonomy to help facilitate discussions to discover the domain actions that cover the React part of the loop.

Figure 3.5. The React part of the Sense/Analyze/React loop: finding business problems that AI can react to. Once you’ve identified the role AI plays in your business, ask the questions provided here.

Figure 3.5 simply applies the existing business-related taxonomy of AI and asks a question designed to find a business problem that needs to be answered by AI. I’ll show you an example of how you can use it.

Imagine you’re working with a retailer that’s nominally independent but part of a bigger franchise. Any process changes require the franchise owner’s approval, and store management isn’t willing to ask for that until you’ve shown them that AI can improve the bottom line. Consequently, the retailer’s management isn’t willing to change or automate their business processes yet, but it can change the product mix (how many products of which type are in the store).

Tip

Management’s skepticism toward an AI-based solution is an issue that often needs to be addressed in the real world. Technologists often take AI capability for granted, but businesspeople can be skeptical. You need to earn their trust before larger and more effective AI projects can be adopted. Put yourself in the shoes of the retailer’s management. How would you feel about going to the franchise owner and opening up with, “Can we change the processes that impact all the stores in the franchise? I want to try something called AI in my store, but I don’t know how it will work out.”

In this example, the only use of AI you can make is helping the store management itself. This is clearly an example of creating a decision support system. Look at figure 3.5 in the branch AI for Decision Support System. Questions applicable to this branch are which decisions can management make and why haven’t they made them yet? That’s where you learn that all your management team can do right now is change the product mix—you can put different products on different shelves. This is the React part of the Sense/Analyze/React loop.

The business question you’re answering is, “What is the most profitable product mix in my store, based on historical sales?” Now you can move to the Sense/Analyze part of the loop.

3.2.2. Sense/Analyze: AI methods and data

Once you’ve established the React part of the loop, you must decide which AI algorithms you’ll use, and make sure that you have sufficient data to use them. This section talks about the relationships between AI methods and data.

This is one area where, if you aren’t proficient in AI yourself, you need an expert with a strong command of a broad range of AI and ML algorithms to help you. The field of AI is simply too large and changing too fast for any single book to teach you enough to replace that expert. Still, the taxonomies presented in figure 2.6 in chapter 2 can be useful to frame the discussion and remind you of high-level AI capabilities that you can apply to the business questions you want to address.

Figure 3.6. Data science methods and data are interconnected and influence each other. Never discuss a method without asking where you can get the data needed to train it.

Once you have an idea of the type of AI methods you’ll use, you need to be aware of the relationship between those methods and data, as shown in figure 3.6.

You must always consider the data and AI methods that you’re planning to use together. You can divide data into two groups: data that your team has and data that your team can collect.

Tip

Data you can collect isn’t only data that you have somewhere in your organization but that isn’t immediately available to your team. It can also be data you can acquire from sources external to your organization or data you can purchase from business partners. Getting access to such data often requires negotiation and signed contracts.

In the retail example given in the previous section, your data scientist assures you that you can use one ML method to predict sales trends and another to optimize product mix. The names of these algorithms and methods don’t mean much to you; some terms like ARIMA, LSTM, and operational research might pop up. Those methods require data about past sales. Such data is available internally in your organization, and your project can access it immediately. You’ve now identified a use case for which all elements of the Sense/Analyze/React loop are covered—this is a viable use case for an AI project.

What if some of your business questions could have benefited from some AI technique that isn’t known to the best data scientist you were able to find? If the best AI experts you can find aren’t even remotely familiar with that particular AI technique, drop it from consideration, as it’s unlikely that you can assemble a team that’s strong enough to deliver using it.

3.2.3. Measuring AI project success with business metrics

By making sure that all parts of the Sense/Analyze/React loop are covered for a specific use case, you’ve verified that this case is technically possible and actionable. But how can you know how it affects the bottom line? This section shows you why you should use business domain metrics to measure the outcome of your AI project.

AI is metric driven, but you must use AI to satisfy a business goal. That goal should be represented by a business metric. That business metric should, in turn, indicate how valuable an answer to your question is when used to improve your business. The measured metric doesn’t have to be a single, exact number like “Profit improved 10%.” It can also be estimated, such as “Profit improved between 8% and 12%.”

Warning

The business metric must be defined for every single AI project. AI methods are, by their nature, quantitative methods, in the sense that they operate only with hard data. Don’t attempt to use AI if you’re unable to define a business metric first that you can use to measure the project’s result.

Being able to choose an appropriate business metric is a business skill that’s anything but trivial. But the good news is that if your organization is using metrics correctly, you should already have a business metric defined for you—the same one that already measures business results in the area to which you’re trying to apply AI. That metric should also measure how much AI improves your business.

This book can’t provide all the best practices for building good organizational metrics; that would be a book in itself (Luftig and Ouellette’s book [1] and Ries’s book [28] discuss the topic of business metrics). But I’d like to point out that a good business metric should be specific to your organization, quantifiable, measurable, relevant to the desired result, and free of unintended consequences.

Tip

Sometimes your organization is using a business metric, but you suspect that what it’s measuring is wrong and isn’t helpful for running your business. If you’re in such a situation, fix the metric before you start an AI project, and use that fixed metric to measure the end results of your AI project. Otherwise, you’ll optimize AI to produce the wrong business results.

Once you’ve selected a business metric that you can use to measure the business contribution of your AI project, you can define the threshold. This threshold represents the minimum value that AI directed action must accomplish for your project to be worthwhile. As an example, if the business metric you choose to use is “profit increase,” your threshold might be that your profit must increase by at least $2M/year for the AI project to be worthwhile.

Now that you have the metrics that you intend to use with your project, you need to confirm that it’s possible to measure the results of your AI project using those metrics. Present the business metrics to your AI expert, and request that they confirm that their team will be able to report the result of the AI project using those metrics. Your AI expert needs to establish a link between that business metric and one of the technical evaluation metrics (as in RMSE, for example) that they intend to use in the AI project. Chapter 4 shows you how to tie together business and technical metrics.

Tip

A business metric is appropriate for an AI project when it correctly measures the business result you want to achieve, and when technical experts know how to report their technical progress using that business metric.

3.2.4. Estimating AI project difficulty

Now you’ve confirmed that the AI project you’re considering is technically possible, and you have a way to measure its business impact and its business value. To determine if the AI project is viable, you need to know its cost, the difficulty of its implementation, and how long it will take. This section details considerations in estimating those quantities.

To estimate difficulty, you need to sketch an outline of the technical solution that will be used in the AI project. You need to have representatives from your data science and data engineering teams, plus your software architect, work together on this outline. Your goal is to provide a high-level outline of the solution, to compare a selection of possible AI projects.

Once you have this outline, use it to estimate the difficulty, cost, and length of time to deliver the project. These are, again, rough estimates intended for comparing different AI project options.

Tip

Remember that people aren’t particularly good in estimations [75], are worse if the estimate is based on just a sketch of the solution, and are worse still if they must estimate in technical areas they know little about. Your estimate will be very rough by necessity. It’s only intended to compare different AI project options, and you should make no strong commitments to management based on this estimate.

At this point, you have all the information you need to create a list of viable and actionable AI projects. You know how to determine whether the proposed project is actionable and technically possible. You know how to measure the business value of the AI project, and you can make a rough estimate of the cost, difficulty, and duration. Now it’s time to select the first AI project to run. The next section guides you through the selection and preparation for running your first AI project.

3.3. Your first project and first research question

As discussed in section 3.1, if your goal is to build an AI team that’s a long-term asset to your business organization, then initial projects should be simple and fast to deliver. The criteria for selecting your first AI project to run is therefore simple: choose the project that’s fast to deliver and has significant business value. When you’ve chosen that project, you need to do the following:

• Define the research question that the project would answer (section 3.3.1).
• Organize the project so that if it fails, it fails fast (section 3.3.2).

The rest of this section shows you how to define the research question and explains what fail fast means.

3.3.1. Define the research question

You’ve chosen your first AI project. That project has a clear business question that needs to be answered, and that question is written in a form that a business decision maker will understand. Now that question needs to be translated into a format that AI can understand—the “research question.” This section shows you how to ensure that your research question matches your business question.

Suppose you’re a manufacturer, and your research question is, “Should I go with supplier A or supplier B based on the quality of their product?” You need AI to answer this question, but there’s a problem: AI has no idea what the concept of supplier means.

AI doesn’t understand business concepts

People unfamiliar with AI capabilities are often under the impression that AI can find some novel business reaction that’s escaping human capacity. With the current level of AI, that’s rarely possible.

There’s no AI algorithm that could look at a retailer and figure out how to improve profits. The reason is that AI doesn’t know what the words retailer, product, and profit mean. AI has no idea what a supplier even is, much less what makes one supplier better than another. Those are business concepts. Nor does AI understand that there could be ethical, public relations, and legal considerations regarding the use of AI in analyzing surveillance video in the store.

Business concepts may be understandable to humans, but the data regarding those concepts must be packaged in a format that AI/ML algorithms expect. That’s your data science team’s job. To do that, they must first formulate a research question. You can view the research question as a translation of the business question into a form that AI can understand.

The contractual language of the technical domain

AI methods operate in a technical domain. Language used in that domain is contractual in nature and is of the form, “If you present me an input in format X, I guarantee that I’ll provide answer Y.” Those contracts are often convoluted and require an expert in the field to really understand their precise meaning, as well as all the implications of what is said.

Let’s demonstrate this contractual language in some concrete examples:

• AI that’s based on classical statistical methods will use the language of hypothesis testing: “Is there a statistically significant difference between part samples from supplier A and supplier B at p = 0.05?”
• AI that’s based on image recognition will express in the language of ML: “If this is a picture of your part, I can tell you with 95% confidence that it’s much more similar to the class of parts you labeled as defective, compared to other classes.” AI can also tell you that, overall, it reaches 98% accuracy in correctly classifying parts between classes.
• AI used in the publishing industry, to predict how many physical books should be printed in a second batch if the first batch sold in three months, might be based on a time series model. Depending on the model used, one research question can be, “Predict with a 95% confidence interval the book sales in the next three months, based on the sales in the previous three months.”

If you aren’t a data scientist, your head is probably spinning right now. What do those sentences even mean? Sounds geekish? That’s because it is! AI methods are defined in an abstract format that’s intended to be understood by computers and data scientists, not by business users. To solve business problems, you need to translate them into AI language. That’s what I mean by formulation of the problem when I say that ML is a combination of formulation, optimization, and evaluation. Figure 3.7 shows this process.

Figure 3.7. The translation of a business question into a research question. AI doesn’t understand business concepts. If you aren’t familiar with statistics, a research question formulation might be difficult to understand.

The job of your AI experts is to select the appropriate research question and translate it into a format the AI methods can answer without compromising its relationship to your business question.

Warning

Projects often go awry when the business and data science teams don’t communicate closely. Business questions can be incorrectly translated to poor research questions, causing you to get an answer to a question you didn’t ask. The problem is compounded if you then use that answer to take the wrong business action.

It’s important to understand that this translation is a highly complex activity requiring that your team share an understanding of both the business and AI domains. This translation isn’t straightforward. It’s almost impossible to devise a translation that evaluates all possible business actions. It’s the job of the business leader to guard against misalignment between the business question and the research question.

Best practices for the business leader of an AI project

The “Misaligned business and research questions” sidebar shows an example of a project that could have gone horribly wrong, as the business leader and data scientist are talking about totally different questions. But until that conversation happens, you won’t notice that you’re not talking about the same thing. The following provides best practices for business leaders who are starting an AI project:

• Never sign a document that proposes a highly technical research question you don’t understand. Instead, call a meeting with the data scientists. Ideally, someone who has a strong background in both business and AI should facilitate this meeting. If there’s no such person in your organization, you should enlist a consultant to help you.
• Ask your data scientist to provide you with a couple of the possible answers that the proposed research question can produce.
• Play out a scenario analysis (as in the previous example). Take the answer to a research question and describe what you think it says. Then, state the exact business decisions you’d make, explaining all your assumptions.
• Always repeat how you interpret the answer that the data scientist gives you. Use simple, nontechnical terminology.
• Don’t worry about looking stupid if you misunderstand something. Finding misunderstandings at this stage saves a ton of money later. You’d really look stupid if you waited until the wrong AI project was run.
• State clearly the business actions that you intend to take, based on their answers. Ask if those business actions are reasonable.
• If the data scientist responds to your planned business action with anything other than a simple yes or no, investigate in which circumstances it would be wrong for you to take such a business action, based solely on the results of the analysis.
• Don’t be afraid to further explore your research question, together with your AI experts. Get educated about the details of what the research question means; by the same token, educate your data scientist on the details of your business.
• Understand that the mapping between a business question and a research question is never perfect. Research questions rarely correctly capture all aspects of the business problem. The question is, “Are those aspects important to you?”
• Beware of discussing business problems using highly technical terminology. Terminology that isn’t shared between all meeting participants is an excellent vehicle to hide misalignment between technology and business.
• If a business doesn’t have properly thorough discussions to ensure that research questions and business problems are aligned, they’re in fact leaving it up to the technology team to fill in critical business details.
• Always make your business experts available to the technical team for consultation about clarification of important details. It isn’t realistic to expect that the technical team will make optimal business decisions when the specification of the business problem is vague.

Warning

Always perform a scenario analysis of a research question before the AI team starts its work. The chance of a project succeeding if you ask the wrong research question is about zero.

A final point to remember when defining a research question is that correspondence between business questions is not “one research question per one business question.” You might need multiple research questions to cover a single business question. It’s also possible (although rare in practice) to have a single research question that can answer multiple business questions.

3.3.2. If you fail, fail fast

You’re starting this first AI project with the assumption that it will be an easy project to deliver. However, your estimates were coarse; it’s quite possible that the project, once you start it, will be more difficult to deliver than you initially calculated. This section explains how you should manage that initial AI project so that if the project is unexpectedly difficult, that becomes immediately obvious.

Tip

You should optimize your project delivery process for speed to success. When hunting in rich hunting grounds, the more you hunt, the more you catch. In the early days of your AI efforts, you shouldn’t persist on projects that looked easy early on, but on closer examination were difficult. Instead, stop them early and use the remaining project’s time to start an easier project instead.

Your project should start with a proof of concept that builds a quick prototype. That prototype serves four purposes:

1. It demonstrates to you that the engineering team has the technical expertise needed to deliver the project.
2. It gives you a concrete AI implementation, which is the Analyze in the Sense/Analyze/React loop you identified. Now you can test the React part of the loop (for example, you can test how difficult it would be to implement the required business action).
3. The prototype can be analyzed to determine how your proposed system solution behaves when exposed to either more data or different ML algorithms. Chapters 6 and 7 of this book show you how this analysis is performed.
4. The prototype shows you how difficult it will be to implement your AI project. If the level of difficulty is much greater than you expected, that should be quickly obvious.

Until you have an experienced team that has passed through such processes many times, don’t get stuck on projects that take a lot of time to implement. As a hunter, you won’t starve because you failed to catch a single prey; you’ll starve because you were trying to catch a single prey for way too long.

Tip

If your project is more difficult than expected, pause it and choose an easier one instead. If you stumble upon the Gates of Hell, turn around and run!

3.4. Pitfalls to avoid

When running an AI project, there are some common pitfalls that you should avoid. Some of the most important ones are as follows:

• Not communicating with the organizational actors that own the React part of the Sense/Analyze/React loop, or, even worse, not working with them at all until your AI project is well on its way
• Transplanting use cases (and metrics) from other projects or organizations
• Running fashionable AI projects that are likely to grab headlines
• Believing that you can buy a tool, any tool, that will give you a sustainable advantage
• Hoping that throwing random analysis at your data will produce results
• Selecting which project to run based on a “gut feeling” instead of the results of analysis

This section discusses each one of these pitfalls in more detail.

3.4.1. Failing to build a relationship with the business team

When using AI as a decision support system, it’s never enough to just deliver a good analysis; you need to execute well on the specific business actions recommended by an AI-powered analysis. That means that executive attention must hone in on the link between the analytical result and the business action. This section highlights why the AI team must build good relationships with the department of your organization that will take business actions, based on your AI analysis.

Tip

Analytics is just like a speed gauge in a car. If the speedometer is telling you that you’re going too fast, the driver must reduce the car’s speed. Who’s the equivalent of the driver in your project/organization?

A nonspecialist can misinterpret analytical results. A classic problem is that nonspecialists won’t understand the limits of the analysis and when the assumptions basic to the analysis are violated. An example of that problem was shown in section 3.3.1, when the research question and business question were misaligned.

I’ve personally witnessed several organizations hand off an analysis report to separate business teams. These business teams proceeded to take business actions without the input of data scientists. This is always a mistake.

Tip

Analytical expertise should always be represented in any group that’s discussing how to react to analytical results. You need to ensure that the business team understands your analytical results and prescriptions fully and correctly. The analysis result must be valid in the light of the intended business actions.

If you’re the leader of an analytical project, your job isn’t done when you deliver the results. Your job is done when the analyst’s prescription is successfully implemented. You must build good working relationships with departments that will implement the analysis. If the analysis has prescribed a particular business action, don’t underestimate the need for you to help and follow through with its execution.

3.4.2. Using transplants

People are often tempted to copy what worked for the people and organizations that surround them. As a result, you see what I call transplant projects. Here, an enterprise decides to form an AI team and embarks on some AI project they’ve heard other organizations similar to theirs performed. This section explains why transplants are a bad idea.

Examples of transplant projects abound. Some examples are projects like “let’s have our own recommendation engine” or “let’s do sentiment analysis of customer feedback.” Sometimes these projects make sense in the context of the business, but all too often they’re just vanilla use cases that you heard about from someone else and didn’t analyze in the context of your own business.

Note

For some reason, people have more common sense when they’re thinking about real transplants as opposed to business transplants. You’d never get a kidney transplant just because it worked well for your neighbor. Why should you behave differently in your business?

Instead of just blindly adapting a project that worked well for someone else, consider it to be just one of many possible AI projects. Use the analytical approach presented in this chapter to determine which AI project you should start first.

3.4.3. Trying moonshots without the rockets

Many of the world’s largest technology companies have made fortunes based on the use of data. In the core, companies such as Google, Microsoft, and Baidu are heavily dependent on AI for their success. They have significant research capabilities and have a vested interest in ensuring that they won’t miss the train of important AI advancements. This section explains why your organization shouldn’t blindly follow those companies.

While the logic from the sidebar “Imagine that you’re a CEO” applies to businesses such as Google, Baidu, or Microsoft, there’s an unfortunate tendency for many enterprises to emulate these companies without understanding the rationale behind their actions. Yes, the biggest players make significant money with their AI efforts. They also invest a lot in AI research. Before you start emulating their AI research efforts, ask yourself, “Am I in the same business?”

If your company were to invent something important for strong AI/AGI [76], would you know how to monetize it? Suppose you’re a large brick-and-mortar retailer. Could you take full advantage of that discovery? Probably not—the retailer’s business is different from Google’s.

Almost certainly, your company would benefit more from AI technology if you used it to solve your own concrete business problems. This means that instead of teams populated by the smartest researchers and processes oriented toward the acquisition of new AI knowledge, your organization needs people who know how to make money in your business domain with existing AI technologies.

Don’t emulate organizations richer than yours without first understanding how you’d exploit success. For most organizations, the road to success isn’t found in advancing the frontiers of AI knowledge, but in knowing how to tie AI results into their business. You need a data science team focused on applications, not research. That doesn’t mean that you shouldn’t hire bright PhDs, but that the leadership of your AI teams must primarily be experts in applying AI to the task of making money.

3.4.4. It’s about using advanced tools to look at the sea of data

Another common pitfall is the belief that you can buy an AI or big data tool that will make it trivial to look at your data, find insights, and then monetize the insights found. Some organizations adopting AI might even take the attitude that the main focus of early AI efforts should be on finding the right tools. This section explains why this is a pitfall to avoid.

Tip

If monetization is trivial, so is explaining how it happens. Ask vendors detailed questions until you really understand the finest points of how to apply an out-of-the-box tool all the way from the point of purchase to the endpoint where your organization makes profit as a result of use of that tool.

In most business verticals, it isn’t trivial at all to monetize AI. And while many tools can help you get there, it’s unlikely that these tools can solve monetization problems for you. Even if there are tools that let you monetize by just installing and running them, what you’re dealing with is a commoditized use case. Heck, someone already has a product that does it!

Tip

The early focus for your business should be on finding AI projects that provide a concrete business value. Tools are enablers of those projects.

A salesman might advise you to “Build a large data lake and unleash your data scientists on it; there has to be something in all that data.” You might even have been given an example of the unexpected insights that only analytics on a big dataset can provide. However, those situations are rare and unpredictable. Don’t count on the tooth fairy. Don’t start with the Analyze part of the Sense/Analyze/React loop. In our framework, always start with the React part.

Warning

It’s always possible that there might be something special lurking deep within your data. With the proper analysis, it might give you some unexpected business idea that you can implement and with which you can make a ton of money. While possible, this is certainly not guaranteed, isn’t predictable, and there’s a big question of is it likely enough to justify it as a main strategy you should adapt? Worse, the lucrative bluefin tuna you hope to catch might turn out instead to be a slimy monster of the deep. Instead of going on a fishing expedition, organize your early AI projects for predictable success.

3.4.5. Using your gut feeling instead of CLUE

Often a decision about running an AI project is made in a haphazard way, as little more than a technical idea that excites the team. Running an AI project primarily because you want experience with the underlying technology is the tech equivalent of buying a sports car. This section explains why following your gut may result in poor business results.

Be extremely skeptical about counting on intuition to select which AI project to run first. Section 3.2 showed you the steps necessary to correctly determine which is the best project to run. When selecting a first project, there are simply too many moving parts to consider for gut feeling to provide the right answer. You need to verify that the project is actionable, technically possible, and business valuable. You need to know its cost, as well as the outline and difficulty of the proposed technical solution. It’s highly unlikely that you can assess all those attributes of a proposed AI project by just thinking about the problem for a minute or two.

Tip

Above all, be on the lookout for any situation in which, during a well-attended and important company meeting, everyone immediately exclaims, “This looks like a great idea!” Such a social situation isn’t exactly conducive to encouraging people to perform the careful analysis needed to disprove the group consensus. In short, beware of groupthink.

The biggest cause of failure of AI projects today might be technical. But even among technically successful projects, there are far too many that aren’t even used by the businesses that paid for them. Those AI projects shouldn’t have been started at all and were usually started because a gut feeling about their value was wrong.

3.5. Exercises

The following questions each give a concrete business scenario and then ask follow-up questions about that scenario. Please answer the following questions:

Question 1:

Suppose you’re working in the publishing industry, and you’re wondering if it’s better to release printed, electronic, and audiobooks at the same time or one after another. Also, if delivery is staged so that printed books are released first, how long should you wait before releasing the other formats? Within this setting, answer the following question: “What business metrics should you use?”

Question 2:

If you’re a business leader, define a business question and an appropriate metric to measure it. Think about some hypothetical scenarios not directly applicable to your organization (for example, some scenarios related to philanthropy). Think about actions that you can take while running a nonprofit. Use the techniques introduced in chapter 3 to select your first hypothetical business question, as well as the metrics you’d use to measure success.

Question 3:

Once you’ve identified your business question from the previous exercise, take your senior AI expert to lunch and talk about the business problem. Ask them how they’d formulate a research question. Use the process described in chapter 3 to check whether or not the answer supports the business action you intend to take. And, while you’re having that lunch, talk about how you’d find a dataset to answer such a research question. Do you think you can acquire that dataset?

Summary

• AI, when introduced to new businesses, falls on rich hunting grounds. Don’t start by chasing difficult projects that tie up all your resources and destroy you if they fail. Start instead with simple projects that have big business value and are quick to complete.
• Use CLUE to select and organize AI projects. The C part of CLUE (figure 3.4) allows you to create a list of actionable AI projects that you can implement and helps you estimate their size and value.
• The business question that AI needs to answer must be translated into a technical format by defining a research question. If that translation is incorrect, it can wreck the business outcomes. Before starting a project, check your research question using scenario-based testing.
• Use business metrics to measure the progress of your AI project. Business metrics should be customized for your project and organization. Don’t start an AI project if you don’t have the business metric for measuring its success.
• Organize your AI project so that if it fails, it fails fast.
• Start with the proof of concept. If the project happens to be more difficult than you thought, stop it and work on an easier project instead. The goal is to optimize time to the next success.
• There are common pitfalls to avoid when running AI projects. They include failure to build a relationship with relevant business leaders, transplanting use cases, adopting a “moonshot” project but missing the rockets, placing too much hope in random tools (or random analysis), and substituting a gut feeling for CLUE.