Chapter 4. Linking business and technology

This chapter covers

• Linking business and technology metrics
• Measuring technical progress in business terms
• Applying the L and U of the CLUE process
• Overcoming organizational obstacles to measuring technical progress

In chapter 3, you selected your first AI project to run. Now you have the research questions that project needs to answer. This chapter shows you how to properly organize that project by using metrics to link a business problem with the technical solution you’re building. It also shows you how to understand what your technical progress means when translated into a business result. Finally, it shows you how to avoid typical organizational obstacles you might meet in your quest to link business and technology. But before we get to all that, I’ll first explain what causes the average organization to fall into the trap of making AI project decisions based on intuition as opposed to data.

4.1. A project can’t be stopped midair

Running any project is more like piloting a plane than driving a car. When you’re piloting a plane, if something happens, you don’t have the option of pulling onto the side of the road and sorting out the problems. The flight has to continue, and you have to sort things out while you’re still flying. This section shows why such a situation could incentivize an organization to make many decisions based on instinct rather than data.

Once a project starts, you have people working on it and money and time being spent on it. That puts pressure on the team to do something because all team members want the project to succeed. And once the project starts, team members on all levels of the project devote themselves to working on what they’re directed to work on. However, in the absence of directions, they’d most likely work on what they perceived to be the most important activity for the project’s success. As every manager knows, team members working in isolation and without knowing what a project’s priorities are often working on the wrong thing because they aren’t aware of what should really happen first.

Managers are people too. In the absence of knowing what the best thing to do is, they still need to make management decisions, because there comes a point in the project when it’s in flight, and decisions must be made. In the absence of guiding principles, managers would try to help a project the only way they know how. That’s why gut feeling projects are so common in the industry.

How do you know what or how well your AI project is doing? The simple answer is that when you’re getting good business results, it’s doing well. But how do you know you’re getting good business results? Let’s look at one typical situation in which it’s difficult to have an intuitive feel for what good results are for your AI project.

4.1.1. What constitutes a good recommendation engine?

You’re in charge of the recommendation engine of a large retailer. The retailer is selling 200 K products, and it has a total of 80 million customers and close to 2 million products viewed every day. Your recommendation engine suggests to every customer additional products they might be interested in buying. You’ve just made an update to your recommendation engine. How do you know that the latest update is moving the system in the right direction?

You can look at a few products overall, but looking at a few products doesn’t actually tell you if your latest change is doing well across all of your customers. Instead, you need to develop some metric that would summarize the behavior of the system as a whole. Such a metric should tell you how you’re doing overall, across all products and customers.

In this situation, you have the following alternatives, all of which I’ve seen used in the context of Fortune 500 companies (even when they should have known better):

• Manually look at the recommendations given by the engine and see if they appear to make sense to a human. Clearly, one problem is what “makes sense” and how to ensure that two different testers have the same criteria. It’s difficult to ensure reliable, repeatable, and correct scoring of the recommendations with this approach.
• Application of various metrics seen in the scientific papers (such as “Evaluating Recommendation Systems” [77]) on recommendation engines. An example of such a metric is novelty [78], which measures how many new items that the user didn’t know about were recommended. What is often not clear is if such technical metrics positively impact any aspect of the business—I might not have known that the retailer stocks garden hoses, but do I care?
• Measure the sales increase from improved recommendations. This approach has clear business relevance and is unambiguous. You don’t have to deploy your recommendation engine fully in production to test the sales increase. You can also perform a test on a limited subset of your customers to measure the progress of the recommendation engine.

The very nature of the AI project is that it’s difficult to get a feel for the results by simply inspecting a few samples. By definition, if you’re using AI, you’re using it because the answer to the question of what is the best thing to do isn’t self-evident, and/or the size of the data is too large for any single human to manually inspect.

Note

This characteristic of an AI project makes managing such projects different from managing other software projects. It’s much easier to say what good progress is for a web application than for a recommendation engine. Similar problems emerge when you’re wondering what task the team should work on next.

Once a project is in flight, decisions need to be made. You’re better off if those decisions are made in a systematic way. Figure 4.1 uses an analogy from another domain to contrast systematic decisions with making decisions based on feel.

Figure 4.1. Once a project starts, decisions have to be made. In which plane would you rather be?

4.1.2. What is gut feeling?

If you can’t explain the reasons for the business decisions you make, you’re making those decisions based on gut feeling. You can’t be sure that you’ve explained something well when people present at the meeting nod their heads and say, “That makes sense!” People often nod their heads just to fit in with a group. You’ve explained something well when someone not present at the meeting looks at the same data and makes the same decision you did.

Making decisions based on gut feeling doesn’t mean that you’re making wrong decisions, but good gut-feeling decisions in the field of AI aren’t easy. This means that even if it was the right call, it’s improbable that anyone else would make the same decision, that anyone (including you) would always make the same decision in similar circumstances, or that an organization would learn from the decision. Such lack of consistency is a red flag for any project, even if a few decisions happen to work out.

Making your decisions based on business metrics ensures that they’re explainable, repeatable, and predictable. It also makes it easy to train new team members on how to make good decisions and learn as an organization from your past decisions.

4.2. Linking business problems and research questions

So far in this book, I’ve shown you how to select actionable business problems to solve with AI and how to use business metrics to measure business results. You’ve also selected your first AI project to run. This section elaborates on how to run that AI project. Your first step is to link the research questions with the business questions.

To link a business problem with the research questions for your AI project, you must make sure that you have the right research questions and that you’re using the correct business metrics. You do that by following the CLUE process, and it’s time for the L part of CLUE.

4.2.1. Introducing the L part of CLUE

In section 1.10, I defined CLUE in detail. Now we’ll talk about the L part of CLUE: Link the research question and the business problem. This is done by scrutinizing and further refining both the research questions and the right business metrics. Figure 4.2 (and the rest of this section) guides you through how that linking is done and why it’s necessary.

Figure 4.2. Linking business and technology. You must make sure you have the right relationship between business questions and business metrics.

To correctly link a business problem and a research question (or questions), you must ensure that you have the right research question and business metric. Yes, you already thought about the research questions and business metrics in chapter 3 (when doing the C part of CLUE), but there you thought about them at the granularity necessary for a triage between the projects. You needed only a level of precision necessary for ranking the projects and choosing the first AI project to run. Now you’re running that project and need better estimates.

Additionally, it’s possible that the team running the project is a different team than the one that participated in the initial triage. Your new team may have different skillsets and maybe even a different idea about how to approach the problem. Even if the team is the same (as is usually the case in smaller organizations and early efforts), it’s now more focused on the specifics of your AI project and able to refine initial estimates.

Note

This iterative refinement is part of the fail-fast philosophy that I advocated in chapter 3. You’d start an AI project by addressing early the questions that, if you were unable to answer them, would sink a project. Only once you knew that the answers to those early questions justified running the project would you make additional investments—such as detailed refinement of the research questions.

4.2.2. Do you have the right research question?

As already discussed in section 3.3.1, the answer to the right research question needs to support business actions that you would take based on that answer. You determine this through a conversation between the data scientists and business management team, during which you test the research question by describing scenarios of what answers are possible and what business actions are planned based on those answers.

The reason why you’re repeating this exercise at the start of the AI project isn’t just due diligence, it’s also to make sure that the team performing the AI project is familiar with the business problem you’re trying to solve.

Note

If you’re a startup or part of a small team piloting AI in your organization, describing possible business actions to the AI team at the start of the project is less important. In a small company, the team members may have already participated in previous conversations about the business objectives. However, in large organizations, it’s typical that the team that performed the C part of CLUE is different from the AI project team.

When you know that you have the correct research question you’re trying to answer, have your AI team discuss how they’d go about solving the problem. You only need to know the high-level outline of the ML algorithms that they’re considering so that you can identify which evaluation metrics you would most likely use to evaluate the results of those algorithms. Now we can take a closer look at the metrics you’d typically use (and the ones that you should use) to evaluate an AI project.

4.2.3. What questions should a metric be able to answer?

How do you know that you have the right metrics to run your AI project? You know because good metrics answer the business questions that you would encounter in the project. This section gives examples of some of those business questions.

Warning

It’s generally not a productive activity to ask the average AI software team in the industry today, “Do you have a good business case, and are you aware of how well your product is doing in business terms?” What do you expect the team that spent a lot of time and money on a project to say when put on the spot? I expect them to say, “Yes!”

The most important question you should be able to answer when running a project is: “For this project, am I going to be able to get the answers to the business questions that I’m likely to ask?” Here are some typical questions that you might encounter on the project:

• If I release the software today in the state it’s currently in, how much money will we make or lose?
• If we can’t release a product today, how much better would our results need to be before we could release it?
• Is it worth investing $100 K extra in getting 5% better results than we have today?

A good project metric should be able to answer such questions. But how good is the typical metric used in an AI project today at doing that?

4.2.4. Can you make business decisions based on a technical metric?

If this isn’t your first AI project, chances are you’ve sat in on a business meeting in which some technical metric was represented as a measure of progress. If you never were part of an AI project, it’s just a matter of time before you sit in on such a meeting. Let’s briefly review what this experience typically looks like.

Suppose you’re a decision maker on a project, and that project uses as a metric RMSE^[1], which is one of the more common technical metrics. The team has presented that the current value of RMSE is 5.143. Quickly, answer the following questions:

¹

A definition for RMSE is included in appendix A—Glossary of terms.

• Is it worth investing 5% of the whole project budget into further improving RMSE?
• How much should you improve RMSE?
• Is an improvement of 3.1415927 enough?

If you’re in the majority that’s not able to answer these questions (or are wondering what in the world is RMSE), then I have several easy questions for you:

• Have you ever made (or have you seen other people make) business decisions impacting an AI project after being at a presentation similar to the one I described?
• How did you establish that the audience at the meeting knew what RMSE was? Note that they need to know it so well that they can immediately apply it to making business decisions.
• What mechanisms were used in such a meeting to ensure that the few people who knew what RMSE is, and knew it well, were the most influential voices in the meeting?

When you’re done with the easy questions, let’s move on to the more difficult ones:

• Why was RMSE even presented to the audience of that meeting?
• A year from now, how would you find out what the team thought about what an RMSE improvement of 3.1415927 would even mean in business terms?

Making business decisions based on the value of a technical metric isn’t trivial. Most businesspeople won’t know how to do it. It’s not just a problem of having someone at the meeting who’s able to translate the value of the business metric into a technical metric. The voice of that person needs to be heard and needs to be influential in making decisions. It’s also not clear what business result would be expected from an RMSE improvement of 3.1415927.

Tip

A business meeting is not the time to learn new technical metrics. Attempting to do so may even result in a meeting dynamic that looks like an unruly college class headed by a difficult-to-follow professor. Decisions become tentative, people are afraid to speak because they aren’t confident they know the class material, and decisions are made based on the personalities of the people present rather than comprehension of the situation.

The preceding example is by no means hypothetical: every single AI project would, at some point, have to decide if further expenditure for improving results would be worthwhile. Similarly, management would have to decide if the project would be worth pursuing further, or if it would be better to reassign the AI team to other projects.

Clearly, the inability to use metrics to answer common and unavoidable business questions means that a metric, while useful for the technology team, is less useful to management. If you find yourself in this situation, you’ve suffered a case of a technical metric that has escaped into the wild, as figure 4.3 shows.

Figure 4.3. Technical metrics are known to escape into the wild. This is not a reason to invite them to business meetings.

One more problem with presenting technical metrics at a business meeting is the negative impact they have on training new engineering managers of AI projects. Put yourself in the position of a new engineering manager. A new manager can see that most of the other experienced managers are appearing to use the technical metric to make business decisions. No one is questioning, “What does that technical metric have to do with business?” The new manager may be excused if they conclude that the connection between the technical metric and business is evident to all their peers, who instinctively know how to relate RMSE to business decisions—as opposed to being just as puzzled with RMSE as the new manager is.

Many new managers in that position end up trying to learn the technical details of what RMSE is and then try to (somehow) make business decisions based on that. Sometimes, an unfortunate new manager only remembers the conclusion—that some magic happens when RMSE is improved to a value of 3.1415927, which also happens to approximate the value of the constant of π.

Note

Every data scientist knows that you should never use the value of π when looking to improve RMSE. The right value to use instead should have been e (approximately 2.7182818). All joking aside, in a business context, there’s no good value for RMSE; there’s only a good value for the business metric—like making a profit.

The previously described situation unfortunately happens a lot today. In some rare circumstances, such as when your company can afford to hire a management team that’s so knowledgeable that they know all the technical metrics cold, it might even be reasonable to present the technical metrics directly. Good for you! You’re working in a company that’s so affluent that you can afford tech-savvy managers.

The rest of this chapter shows you how to translate a technical metric to a business metric. When you see how that’s done, you’ll see why I’m skeptical that even brilliant people are particularly good at doing this right immediately, in real time, in their head, and in the same way.

4.2.5. A metric you don’t understand is a poor business metric

One of the main problems in a typical AI project is what I call the technology smokescreen, and it works in the following manner:

1. Some technical concept (like RMSE, or some other technical metric) is presented to you.
2. You start by trying to understand what RMSE is.
3. It becomes a game of who’ll give up first—the data scientist trying to explain to you what the concept means, or you trying to understand what it means.
4. After a while, everyone is tired, some decision needs to be made, and that decision is made based on the understanding of the concept that has emerged thus far.

Most of the time, that situation emerges without anyone intending it. The team is honestly trying to educate management, and if management proceeds down this path, they’re contributing to the situation.

Trying to understand a technical metric in such a situation is always a part of the trap. Instead, in this chapter, I’ll concentrate on what helps you succeed. It’s not a question of what the concepts are that you don’t understand. It’s a question of are those concepts something that’s relevant to making business decisions?

The answer to that question for any technical metric is always going to be “No.” Any technical metric is either irrelevant to a business decision or is relatable to some other business metric that makes the business decision straightforward. In the former case, the metric shouldn’t have been presented in the first place. In the latter case, what should have been presented is a corresponding business metric. The question isn’t “What is RMSE?” It’s “What is an exchange rate between dollars and RMSE?”

People who know what RMSE is may object, “It’s often easy to translate RMSE to dollars!” But if it’s easy to translate RMSE to a business metric, then it’s also easy to use a business metric instead of RMSE, so you should use the business metric.

Warning

Sometimes you’ll encounter technical metrics you don’t understand. Never try to learn that metric during the meeting and translate it to a business metric in real-time. Insist that business metrics be presented instead.

Moving past RMSE, you’ll encounter many technical metrics in the AI field. Stay focused on business! Running a successful AI project requires knowing some AI concepts but shouldn’t require the equivalent of a PhD in AI and ML. The number of managers who’d be able to acquire such a PhD is always going to be limited. Consequently, any argument that a PhD is needed to manage an AI project is actually an argument that a successful AI project is a rarity.

4.2.6. You need the right business metric

As explained in chapter 3, it’s important to use the right business metrics to measure the progress of your AI project. Surprisingly, often the right business metrics aren’t developed early in the project. This section reminds you of the pitfalls of having the wrong business metrics.

One common anti-pattern is to allow business metrics to emerge from the nomination of various team members with no additional analysis. Often, those metrics are transplants that someone has seen used on previous projects.

Warning

It’s not uncommon for teams to think that because previous projects used a business metric successfully, they should use the same metric again. A history of past use of the metric is irrelevant—what matters is applicability of the business metric to measure the desired business goal on the current project.

Common errors when defining the business metric are as follows:

• Having a business metric that’s not related to the business goal at all
• Having a business metric that’s too vague to measure the underlying business goal
• Trying to substitute a business metric that’s more difficult to measure with a business metric that happens to be easy to measure

The first situation typically emerges when you take or transplant a metric from a previous project. An example (exaggerated for effect) is using a metric based on cost plus in an early startup. Cost plus pricing is common in government and defense contracts, where you might be guaranteed a profit of (for example) 15% above your costs. Clearly, the meaning of high cost in such environments isn’t necessarily negative.

Second, metrics that are too vague to measure business goals are much more common than they should be. They might be vague because they’re poorly formulated and aren’t correctly measuring a concrete business goal. However, they could also be vague because they’re trying to measure ambiguous business goals. An example would be a business goal of our products having a fanatical following, which happens to be better formulated as “We’ll have industry-leading retention and referral rates among our customers.”

Note

A good way to determine if a business goal is too vague is to ask people to state the business goal and then ask “So, how is it (the goal) going?” [3]

The third error of metric substitution happens in practice because you know what an ideal metric would be, but you don’t know how to precisely measure it.

4.3. Measuring progress on AI projects

You should measure the progress that your analytical teams are making on the research questions by referring to the business metric that measures the success of the business problem you’re trying to solve. This section will show you how.

Guiding the construction of a business metric that’s capable of measuring business impact should be the overall responsibility of the business leader. Of course, work on it could be delegated, but ownership and responsibility should stay with the executive sponsor of the project. That business metric could be considered a contract among the executive sponsor, the business team, and the data science team.

Defining business metrics is a team sport. Let’s look at an example that you can also present to your team and determine if you’re lucky enough to have a team that’s perfectly capable of choosing the right metrics on their own.

Suppose you’re at a university that wants to help students bike between buildings on the campus. A few years ago, as a trial, the university created three bike rental stations at various locations on the campus. The university also collects bike rental data that includes weather information and rental times for each bike. You’re probably wondering if there’s a way to improve the use of the bikes. How many bikes do you need at each station? To know the answer to that question, you clearly would have to predict what the demand for bikes would be at each station.

For the time being, let’s assume that you choose an appropriate ML algorithm to use.^[2] Ask your team (and yourself), how would we evaluate how well that algorithm is doing?

²

If you’re a data scientist, assume that you’ve chosen multiple regression as your baseline model for this example.

If you have a data science team handy, ask them to help you select an evaluation metric they’d use to measure the algorithm’s performance. If you want to save time, you might even skip constructing the algorithm and metric and just ask them if they feel that the RMSE could be a reasonable metric to measure the progress of such a project.

Also ask the team if they’d use additional metrics and what they’d report to you when showing progress. And, of course, if you’re a data scientist, you might answer those questions yourself.

If you don’t have a data science team handy, you can just assume that they advised you to use RMSE. If your team chooses a different evaluation metric, it doesn’t matter—regardless of the metric you choose, you can follow the rest of this section with it.

Note

Be prepared. Your team might not choose the correct business metric. Guide a discussion by asking business questions and how the metric could be modified to better answer those questions. This is a learning exercise, not an exercise in saying “Gotcha!” to the team.

When you’ve chosen a metric and confirmed that the team will use it to report on the progress of the project, you’ll need to make some simple management decisions. Let’s try to answer a few questions based on the RMSE. For the sake of argument, assume that the value of the evaluation metric came back as 0.83. Answer these three questions:

1. I’m a student who just finished a class on a Monday, at 5 p.m., in July. What’s the chance that there wouldn’t be a bike available for me to use?
2. I’m a facilities administrator in charge of the project. How many bikes should I have on the lot to ensure that 95% of the people who’d like to rent a bike can do so?
3. I’m the college administrator running the project. What do you think is an important business question to ask for the project? Answer it, under the assumption that your evaluation metric has a value of 0.83.

What happens when you try to answer the three questions? If you know the data science yourself, what answers do you come up with? If not, ask your data science team. How do they do with answering such questions? These are the type of questions that a business is likely to ask on such a project, and if you aren’t able to use RMSE to answer them, then RMSE is clearly not the right metric to use for making business decisions on this project.

Now, what would be the right business metric? That depends on the business goal, but I’ll provide a business metric for one of the scenarios. The first question to ask is, “What are we trying to achieve with this system?” Are you trying to maximize profit of the bike rental operation? Or are you trying to encourage the maximum number of students who can bike?

Note

A business metric that’s appropriate to use depends on the problem you’re trying to solve. You can’t define a good metric for a problem you poorly understand, so don’t hesitate to ask questions about business goals.

If you’re trying to maximize profit, chances are you’re charging people to rent bikes, and profit is maximized when you don’t have many bikes lying around unused. A good metric will be expected profit (in dollars) after accounting for all costs of bike rentals.

If you’re just trying to encourage people to ride bikes, they might be free to rent, and having extra bikes lying around unused is much better than not having enough bikes. The best business metric could be the percentage of the time the rental station is out of bikes.

There’s also a hybrid case. Suppose the campus wants to encourage biking, but it also wants to minimize the cost of the program. Then your business metric should be based on making the maximum profit with the constraint that you want to make sure there are always enough bikes per lot during peak usage times.

Note

Remember this bike example and, in particular, the hybrid business metric. We’ll use that metric later in this chapter when constructing a profit curve.

Once you’ve defined the business metric, it’s time to link technical progress to the business metric.

4.4. Linking technical progress with a business metric

Linking technical progress and business metrics is done using an artifact known as the profit curve. It lets you translate the value of the technology metric into the value of the business metric.

Once you’ve defined a business metric, if it’s the right business metric, you should be able to make a business decision based on it. Suppose your business metric is profit increase. If I told you that an ML algorithm would likely increase profits 10%, you’d have an easy time answering the question: “Is it worth investing $100 K in developing such an ML algorithm?”

But ML algorithms don’t operate with business metrics. They operate with the technical evaluation metrics. You’d need to link the business metric you already have with the technical metric that your team has just selected to measure their ML algorithms.

4.4.1. Why do we need technical metrics?

Why do we need technical metrics in the first place? Why don’t we just use business metrics directly in the ML algorithms? The reasons for that are both technical and historical and come from the fact that business and technology metrics are intended for different purposes. This section introduces you to technical metrics.

Let’s quickly revisit chapter 1. ML is a combination of formulation, optimization, and evaluation. An evaluation metric is a technical metric that’s optimized by the ML algorithm. Your data science team chooses evaluation metrics based on what’s technically appropriate for your project.

Note

Technical metrics are intended to work well when used with particular ML and AI algorithms. It’s not uncommon for the algorithm itself to dictate which technical metrics can be used with it.

Technical metrics have properties that make it easy for the ML algorithms to optimize the value of such metrics. Those properties are mathematical in nature, highly technical, and typically unrelated to business. For example, one of the common properties of technical metrics is that they are differentiable, and many optimization algorithms used in the context of AI and ML would require that the metric be differentiable. Unfortunately, business metrics aren’t necessarily differentiable. As a result, you can’t use a business metric directly in many of the ML and AI algorithms.

Note

If you’re wondering what differentiable means, Google it and read the results until you’re satisfied that whoever invented the concept didn’t worry much about it having any obvious relationship to a typical business problem you’re likely to face.

In addition to the technical reasons, there are also historical reasons why many of the AI and ML algorithms require metrics that don’t look like a business metric. When many of the AI and ML algorithms were invented, the use of AI and ML in business wasn’t as common as it is today. There were competitions between researchers, such as the KDD cup [80], and originally, technology metrics were perfectly fine for measuring who won such a competition.

4.4.2. What is the profit curve?

A profit curve establishes the relationship between the business and technology metrics. It allows you to use a technical metric for your ML algorithms. It also lets you translate the threshold of business metrics (the minimum value the business metric project must achieve to be viable) into the corresponding value of a technical metric. This section shows you how to construct a profit curve.

The profit curve, in the context of data science projects, was originally proposed in the book Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking [81], although the general concept of establishing mathematical relationships between metrics predates it and was known before [1,3]. Figure 4.4 shows the process of constructing a profit curve.

Figure 4.4. A profit curve specifies the relationship between a technical metric and a business metric. It allows you to understand what the technical answer (in the form of a technical metric) means for the business terms.

When defining a profit curve, you’d combine business and technical metrics through a mathematical relationship. That joins technology and business by linking the research question with the business problem you’re trying to solve. You can think about it as a form of the exchange rate when answering the question of “How many US dollars is one unit of RMSE worth?” (if your business metric is measured in dollars, and your technology metric is RMSE).

The value threshold is the minimum value of the business metric that must be achieved for your project to be viable. Suppose the business metric is profit, and the cost of starting the project is $30 K. Clearly, there’s no point in starting the business unless it’s expected that it would bring in more than $30 K in profit. In this example, $30 K is the value threshold.

4.4.3. Constructing a profit curve for bike rentals

Now that you understand what a profit curve is, we’ll construct one for the bike rental example from section 4.3. The construction of the profit curve requires the cooperation of business and engineering. If you don’t have an engineering background, here’s one area where you’ll be learning a little about the engineering side—see the sidebar for more information.

Back to the bike rental example from section 4.3. You need to predict how many rental bikes will be needed on rental lots. As a reminder, you’re trying to minimize the number of bikes bought, subject to the constraint that you want to make sure there are bikes available during peak usage times. For simplicity, let’s also assume that the rental fleet offers only a single bike model.

The main question for constructing a profit curve is, “How much would an error in prediction cost?” For the technical metric to measure the ML algorithms we chose, let’s use RMSE. (In this particular case, take my word for the correct interpretation of what an RMSE measure is in the domain of bike rentals). For our question we’ll use: “How many bikes can my prediction be off, on average?”

Note

If you’re a data scientist, you know that RMSE would have a tendency to penalize large errors more, and that, as such, the technical interpretation I’m using might have some caveats and corner cases. In the construction of an initial profit curve, you can disregard the corner cases of technical metrics, as you’re just trying to decide how profitable a project is likely to be.

The RMSE given in the bike example was 0.83. This means that we’re approximately 0.83 bikes per rental lot off in our demand prediction. To ensure that enough bikes are available, we’d need 0.83 extra bikes per lot. Of course, the number of bikes must be a round number, so 0.83 is rounded to one extra bike per lot, right?

Wrong! If you’re looking to ensure that there’s always a bike available, you’re not interested in the average error in prediction of available bikes. If you’re five bikes off in your prediction of demand for bikes at 3 a.m., but the lot is full of unrented bikes, who cares? There are enough bikes for everyone when the lot is full.

What you’re interested in is the error of prediction during peak usage hours. If there are enough bikes to rent at peak usage hours (let’s say 2 p.m.), chances are, there would be enough to rent at 3 a.m. too! Your profit curve in this case isn’t

Cost of Error Prediction = Price of Bike * RMSE

Instead, it’s

Cost of Error Prediction = Price of Bike * Max prediction error during peak usage hours

Let’s give the quantity “Max prediction error during peak usage hours” a simpler name: Peak usage hour’s RMSE.

Figure 4.5 shows you how the profit curve we constructed for the bike rental project looks.

Figure 4.5. Profit curve for bike rentals. Note that in the case of a business metric being a cost, the goal is to minimize the business metric.

The construction of that curve is simple but comes with two caveats. The first is that the cost is the same for every peak usage hour’s RMSE between 0 and 1. The reason is that you can’t purchase half of a bike, so any error smaller than 1 has the same business implication. The second caveat is that any time you’re working with business metrics such as cost, less is better. When operating with this business metric, the goal of your AI team would be to minimize the business metric. If your business metric was something like profit, you’d want to maximize the profit curve.

Note

Some organizations prefer this rule: we always build a profit curve in such way that more is better. This is for uniformity and for making understanding and training easier. If that’s your preference, use signed numbers; for example, a cost of $10 would be written as –$10.

What about the value threshold you should choose? If you happen to make an RMSE prototype that on the first try has an RMSE of 0, good for you. But if it happens that you have a peak hour RMSE of 1.2, for example, should you try to improve that? It depends on how the cost of your engineering time compares with the cost of a bike. If the cost associated with such a peak hour’s RMSE is smaller than the cost of your engineering team’s time to look at the problem, then don’t worry about improving it. Consequently, the value threshold for this profit curve would be based on the estimate of the minimum cost of the engineering team’s time to look at the problem further.

Once you’re able to express your data science project’s progress in terms of a business metric, understanding the value threshold you should select is typically simple. If you’re looking to make a profit, the value threshold would be based on that profit actually existing when you take all costs (including costs of capital) into account.

4.4.4. Why is this not taught in college?

If you recently had college courses in AI and ML and never heard of a concept such as a profit curve, you might wonder why that’s the case. First, some courses (like the ones that use as the textbook Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking, by Provost and Fawcett [81]) do teach it. As for the courses that don’t teach the profit curve, the reason may be that the profit curve is much more important and applicable to business environments than academic ones. This section explains the differences between these environments.

As shown in figure 4.6, the shape of the typical profit curve in academia is very simple. Once your proposed AI methods get better results than was achieved before, your work becomes publishable. So, a profit curve in academia is typically that your work has no value until you achieve better results than previous researchers.

Figure 4.6. A typical profit curve when you’re working in an academic environment. Such a profit curve allows you to focus only on the technical metric.

The shape of the typical profit curve in academia makes it easy to collapse the whole profit curve into a single question. That single question is, “Is my technical metric better than what anyone that published before me has achieved?” Replacing the profit curve with that question, and the fact that most of the engineering courses don’t talk much about a business environment in the first place, might explain why you didn’t encounter this concept in college.

4.4.5. Can’t businesses define the profit curve themselves?

It’s not uncommon for data scientists to assume it’s possible for business teams to translate a technical metric into a business metric on their own. This section explains why it’s usually better if the data science team helps the business team here, rather than asking the business team to do it on their own.

The whole purpose of the data science project is to allow management to make the right business decisions based on some quantitative criteria. Don’t be like that security device manufacturer that asks their users to manually do the work that they should have automated (see the “What would you do?” sidebar). Even a person who knows how to construct a profit curve might object to the task of manually calibrating a device. It doesn’t get better if business users are less current on math and programming than your data science team.

Never present technical metrics to the business audience and ask them to translate the metrics themselves. At best, you’re saying, “It’s so easy to translate this metric, but I didn’t feel like doing it for you.” At worst, you’re saying, “It’s not possible to present in business terms what the project is doing.”

The team that’s creating a profit curve should consist of data scientists and business analysts. Once you construct the profit curve, use it to present technical results in the form of business metrics to the business audience.

4.4.6. Understanding technical results in business terms

Once you have a relationship between the technical and business metric, that relationship is bidirectional. Just as you can use a profit curve to translate the value of the business threshold to the value of a technical metric, you can also translate the value of the technical metric to the business metric. This section shows you how.

One of the outputs of running a typical ML algorithm would be a technical metric. Once you understand the relationship between a technical result and a business result, you can use a profit curve to understand the technical results in business terms. This is the U part of CLUE, as shown in figure 4.7.

Figure 4.7. A profit curve is a bidirectional relationship. You can use it to answer a business question based on the technical metric.

When you have a profit curve, suddenly it’s possible to measure technical progress with business terms. You do that by translating the value of a technical metric to a business result.

Again, suppose your technical metric is RMSE, and your business metric is profit (in US dollars). Let’s say you have an RMSE of 0.83. Looking at the profit curve, you see how an RMSE of 0.83 translates into the business metric. And let’s say that, in this example, an RMSE of 0.83 is a profit of $1.325 million per year. Now, use the profit of $1.325 million per year to answer your business question.

4.5. Organizational considerations

While construction of the profit curve is technically simple, you must be cognizant of the organizational aspects of constructing a profit curve. When constructing a profit curve, you must not only consider what’s the best mathematical formula to describe the profit curve, but also how best to interact with your organization to get the information you need to construct that formula. This section describes some of the important organizational concerns you must address when constructing a profit curve.

4.5.1. Profit curve precision depends on the business problem

A profit curve shows you what your organization currently knows about the relationship between the technical and business metrics you’re using on the project. As in any other project management decision, there’s going to be a dichotomy between the cost of obtaining precise information and the value of that information.

Note

You can spend a ton of time trying to construct the perfect profit curve with a precision of up to five decimal places. If you’re on a project that’s bringing in $1 billion per year to your employer, you definitely should spend that time. If you’re on a project that’s bringing in $20 K per year in profit, you don’t need that much precision.

When constructing a profit curve, the first question should be: “At this stage of the project, what level of precision do we need to make sound project management decisions?” A general rule you should follow is that in the early phases of projects, a profit curve should be approximate. That’s the approach that I followed in section 4.4.3.

Tip

When you’re just starting a project, there are many unknowns. You don’t know how much exactly it would cost, yet you’re comfortable with estimating cost. You also are comfortable with initially providing a rough cost estimate and revisiting it over time. Use those skills you already have to construct a profit curve.

4.5.2. A profit curve improves over time

A project that’s just building proof of concept (POC) and you don’t know if it’s technically possible is quite different from a project that has been in production for years for a business operation that has $1 billion of revenue per year. If you have the latter type of project, an improvement of 0.1% is typically worthwhile.

Note

It isn’t unheard of for an AI project to be applied to the critical business areas of large companies with revenue of (or exceeding) $1 billion. There are hundreds of companies in this group.

As an AI project progresses through its life cycle, the profit curve is significantly refined. It would be a ridiculous idea for an early startup to hire a full-time economist to construct the best profit curve. But by the time you’re making $1 billion per year with some product, putting more effort in getting better profit curves (or hiring full-time economists) is a reasonable thing to do.

As a result, refinements in the profit curve are expected, and they are not a sign that your initial profit curve was in error or that the decisions you made based on that version of the profit curve were wrong. They were the best decisions you could have made based on what you knew then.

As your project grows, the economic value of information changes. Large, successful projects need (and can afford) a more refined version of the profit curve.

Tip

After refining your profit curve, don’t forget to revise old decisions you made based on previous versions. Those decisions were based on a business metric value that changes every time a profit model changes. Perhaps they should be updated based on what you know now.

4.5.3. It’s about learning, not about being right

The nature of the profit curve is that it should be approximate, and that it codifies the incomplete knowledge that your organization has. The first attempt at the profit curve will likely be the best guess that captures most of the relationship between the technical and business metrics, although it may disregard a few rarely encountered corner cases. It’s your job to create an environment in which people feel safe providing their best guess.

Explain to your team (and management) that a profit curve is expected to evolve over time. A good analogy to use is estimating the cost of a project. Such estimates are also expected to change and evolve over the lifetime of the project.

Build an organization that learns over time more about its business domain and codifies that learning with improvements in the profit curve. See Elastic Leadership [82] for some of the techniques you can use to build a learning organization.

4.5.4. Dealing with information hoarding

Information hoarding is a fact of life for any AI project. Often, some organizational members are going to be afraid that providing access to data and information can result in the loss of their influence and power. This section contains some strategies for dealing with this problem.

If your organization has hoarding problems, it’s a safe bet that some of these issues will come up during the construction of the profit curve too. In general, there’s no magic wand to resolve information hoarding, as it’s a sign of the general organizational culture (and health). Having said that, the rules for getting good information for profit curve construction are the same as the rules for getting any other data in an organization:

• If some parts of the organization are comfortable with working among themselves but less comfortable with working closely with the rest of the organization, you have the problem of organizational silos. You’ll need the support of higher-level management to break through this problem.
• Identify which organization is in charge of the data you need and which specific people know where that data is.
• People in charge of the necessary data (let’s call them data owners) should be part of the team responsible for constructing the profit curve. You don’t necessarily need access to all their data; you just need their help in constructing the profit curve.
• If need be, always make the case to the owner of the information that they’re better off sharing the data voluntarily. Otherwise, the next best thing is to change incentives—can you affect business metrics on which those owners of information are measured so that sharing the information improves the metric?

In general, resistance to breaking organizational silos always comes down to the fact that you’re not important enough to the silo owner, which is almost always a sign that you don’t have enough buy-in from higher-level management for the AI project. Consider what you can do to remedy that situation.

4.5.5. But we can’t measure that!

One complaint that you often hear when constructing a profit curve is that this AI project is special, and that it’s not possible to construct a metric that would measure progress on it. This section presents some of the common arguments you’re likely to hear.

The most common objection you might encounter is that the business metric can’t be precisely measured. This is typically mentioned in relationship to intangibles such as, for example, customer satisfaction. The solution is given in Hubbard’s book [75], which points out that everything can be measured. The impression that some things can’t be measured is based on the misunderstanding of what measurement is: people believe that the result of measurement is always a number.

Note

The result of any measurement is actually an interval. How tall are you? You say six feet? Is that closer to 5 feet, 9.5 inches, or 6 feet, 0.5 inch? You say 6 feet, 0.5 inch? Is that closer to 6 feet, 0.25 inches, or 6 feet, 0.75 inches? Does it include your hair style? You can’t provide a single number even for such a trivial and clearly measurable thing as your height.

Similarly, for intangibles, you measure them using an interval. The interval is just a lot larger. You can attempt to make the case that you can’t quantify the value of satisfaction for your early customers to measure the success of your startup. What would you invest in customer satisfaction improvement? Would you invest $0.25 in it? How about $1 billion? You’ve just quantified that the value of customer satisfaction is between $0.25 and $1 billion. Keep trying; I’m sure you can do better. You can refer to Hubbard’s book [75] for many examples of how to measure intangibles.

The next common objection is that the data you need to construct the profit curve simply doesn’t exist. For example, it might be that you’re trying to measure the profit of some activity, but in your P&L statement, most of the costs your enterprise is incurring are grouped together under the name Overhead. Consequently, you have revenue for some activities but no associated per activity cost, so you can’t calculate profit per activity. Or, in the case of a small team, you have a cost for the team but no breakdown of which tasks the team spent its time doing, so you don’t know how much the per activity cost was.

When you encounter a situation like that, the right long-term solution is to collect the data you’re missing: what were the per activity costs, and which team members spent time on those activities? A short-term solution would have to be based on estimating how overhead cost breaks down among different activities.

4.6. Exercises

Question 1:

If your organization has run AI projects before, look at some progress reports and the metrics used in those reports. Answer the following:

• If we release software today, in its current state, how much money will we make/lose?
• If we can’t release today, how much better do our results need to be before we can release?
• Is it worth investing $100 K extra in getting 5% better results than we have today?

Question 2:

Based on the answers to the previous questions, do you feel that your organization is making decisions in its AI projects based on the data, or is it possible that in some cases you had to make important decisions based on intuition?

Question 3:

Suppose the cost to start a project is $100 K, and the policy of your organization is that no project that can’t create a 10% return on investment is worth doing. If your business metric is profit, what would be your value threshold for the project?

Question 4:

Go back to the bike rental example from this chapter. Suppose the estimated cost to assign a data scientist to the project is $10 K, and each extra bike costs $1 K. How much should you expect to improve the peak hour’s RMSE to make it worthwhile to assign a data scientist to the project?

Summary

• Most of the current projects run in businesses today use some form of technical metrics to measure progress and success. But it’s difficult for most organizations to use such metrics to make quantitative and repeatable business decisions.
• Link business and technology metrics, which is the L part of CLUE, so that you can measure technical progress in business terms. That’s done by constructing a profit curve.
• You should measure and report the progress of your AI project using business metrics. Such an approach helps you understand the AI project’s progress in business terms. That’s the U part of CLUE.
• Constructing a good profit curve isn’t just an exercise in abstract math. If anything, it’s more of a detective job in which you’re tracking which part of the organization has the information you need and thinking how to get that information.