Appendix B. Exercise solutions
This appendix provides solutions for the exercises in all of the chapters. To help you reference the questions, I repeat those questions for each chapter and then provide the answers beneath the questions.
I strongly encourage you to complete the exercises to get a better understanding of material covered in this book. This book’s exercises highlight and reinforce the best practices in, and common pitfalls of, AI projects in business. If you elect to skip the exercises, I still recommend reading the answers to the questions.
The exercises may introduce some new concepts that are not discussed in the chapters but that should be already familiar to you or, if not, well within your ability to grasp. This is intentional and will help you practice the application of skills and concepts you learned in each chapter to new business situations.
B.1. Answers to chapter 1 exercises
This section contains answers to the questions asked in chapter 1. For ease of reference, each question is repeated with the answer below it.
B.1.1. True/False questions
Answer the following questions with True or False.
You always need a lot of data to make significant money with AI.
False. Often, just a simple analysis on a small dataset can be all the analysis you’ll need to make money, if you’re the first person who figured out how to link that analysis with the business actions you can take. In the early days of what eventually became one of the biggest hedge funds in the world, Ray Dalio used a single computer much less powerful than anything you can buy today [29] for all his analyses. Such a computer would struggle to run many of the complicated models we use in AI and ML today.
The first step when starting an AI project is to select the right technology tools to use.
False. No tools are best in class among all possible sizes of data and business applications. The first step in any AI project should always be to think about the business problem you’re trying to solve.
Sometimes, simple AI algorithms can produce large business results.
True. Making money is a function of the size of the business opportunity to which you apply AI, not of the technical excellence of an algorithm. A simple algorithm can make a lot of money in plenty of situations, and in plenty of situations even the best AI algorithms can’t solve a problem in a way that would make for a viable business product.
Some tools can significantly automate AI projects. Just by using those tools, you can ensure a significant and lasting advantage over your competitors.
False. If a tool is something anyone can buy, then that tool is a commodity and can’t form the basis of a sustainable competitive advantage.
Making money with AI requires a PhD in math, physics, or computer science.
False. Plenty of AI practitioners don’t have PhDs.
Every AI PhD is guaranteed to know how to make money with AI.
True . . . if you hire them, but at that point, they made money for themselves, not for you. Such money is called salary. As far as making money for you goes, the reality is that it isn’t unheard of to find people who’ve earned PhDs in AI programs who have limited business understanding.
All AI tools are created equal.
False. Some tools are definitely better than others in their specific areas, but that’s not what’s most important. I’m not claiming that tools don’t matter at all, or that there aren’t good business reasons to choose one tool over another. I’m just saying that a tool itself isn’t what matters the most for success of an AI project.
You’re a project executive, and you leave the definition of the evaluation metrics to your data science team. Unfortunately, your data science team doesn’t have strong business domain knowledge, and they provide you with a metric that you don’t understand—let’s call it the Gini coefficient. If they do well on that metric, the project will help your business.
Good luck! This is an example of using a technical metric that has an unclear relation to your business. Consequently, it’s not clear what value of the Gini coefficient is “good” for your business. Now you have a technical team measuring themselves based on improving “Gini,” and a business team making decisions based on some other criteria. Those two teams are disconnected, and a good dose of luck is needed for the project to succeed.
B.1.2. Longer exercises: Identify the problem
A short narrative description of a hypothetical project or actions taken during the individual projects follows. What’s your opinion of the situation described?
A friend who works in the IT department of an organization somewhat similar to yours uses tool X and approach Y with great success. Should you use that tool and approach because your friend was successful with them?
The most important question to ask yourself here is how similar are the business problem and its data to your issue and its data? If the answer is very, then using a similar tool is valid. The problem with mimicking your friend’s approach is that you might be copying the solution to a problem that’s different from yours.
X, a Fortune 100 company, begins their AI efforts by creating an infrastructure holding petabytes of data and buying an array of tools capable of solving a broad spectrum of AI problems. They’ve also created a department responsible for using and maintaining all those tools. Should you buy the same set of tools?
If you’re in a Fortune 100 company with the same budget and business needs, then, yes, this applies to you too. If you don’t work in a Fortune 100 company, remember that, in the last few years, we’ve faced a market that was excited about AI and was putting less pressure on you to get demonstrable business results quickly. Is that still the case at the time you’re reading this book? Overall, the problem here is that you may be copying someone who can afford what you can’t.
You want to start your AI efforts with the use cases that other people successfully employed. Can you ask consultants with AI experience for an example of AI use cases often seen in your industry?
You can, but, again, you’re copying what someone else did. If they’re in your industry and are making money with their approach, you’re at best a follower. If they aren’t in the same industry, make sure that you can monetize such an approach. The problem here is too much focus on how to do it and not enough focus on talking about whether you should do it.
What’s wrong with the following approach? You’re seeing that AI is getting better in video recognition. You plan to start an AI project that would apply AI to recognizing and scoring Olympic skating. By using such an AI, you can show the viewers what the predicted scores would be as soon as the skaters are done, without needing to wait for the judges. Your AI solution must be ready before the next Olympics.
You’ve just set an AI goal that’s limited in time and, to the best of my knowledge, beyond the scope of current AI ability. You need a new scientific discovery to have a viable business, and you have limited time. Good luck.
Is the following a good idea? You’re in a heavily regulated industry that delivers products to end consumers. You have to run all your changes by a regulator, and changes are evaluated (almost exclusively) based on legal compliance, with a typical change taking five years to be approved. You plan to use AI to understand online customer feedback and your customers’ satisfaction. The technical term for this process is sentiment analysis.
How are you going to change anything based on that analysis? And, if you do, it would likely take you five years to get it approved. Would your customers still remember their feedback by then, even if you were to eventually act on it? Would they care by that time? Although the argument could be made that overwhelming customer feedback can influence eventual change in regulation or a regulator’s enforcement policy, chances are that you’ve just chosen to use AI in a context in which the result of the analysis isn’t actionable.
What are some problems with the following proposal? We’ll use this AI and feed it patterns of our customer behavior, and it will reveal to us the causes of our customers’ decisions.
Although it’s a common misconception that AI can find the cause of behavior, today it can’t. AI can only find a correlation. Causes are something for which humans (and methods different than ones typically used in AI today [24–27,151,164]) are needed.
You’re working in a domain in which it isn’t easy to define business metrics that you can use to measure the business result. Someone has proposed to use AI and make business decisions based only on technical metrics. Is this a good idea?
AI methods are quantitative methods; informally, that means that they’re methods intended to operate and optimize numbers. If you really can’t define a reasonable business metric, you most likely shouldn’t be using quantitative methods in the first place. Fortunately, it’s usually possible to define business metrics quantitatively, if not with a single number, then with some range of values representing the business value of such an outcome (for example, “This outcome would be worth between $1 million and $2 million to us”).
Chapters 3 and 4 talk more about business metrics. Here, the important thing to understand is that the unwillingness to quantify business smells of a data science/AI project that’s not on the right track.
B.2. Answers to chapter 2 exercises
This section contains answers to the questions asked in chapter 2. For ease of reference, the question is repeated with the answer below it.
B.2.1. Short answer questions
Please provide brief answers to the following questions:
Think about a failed project in your enterprise. Would that project have failed in the exact same way if it also had a component based on AI?
Most likely, the answer is yes. I’m talking about a large cross-section of possible projects, but the answer given here should apply to most cases. The point is that AI projects can fail in any of the ways that other normal projects can fail, as well as in a few ways that are specific to AI projects. (I talk about this later throughout the book, for example in section 5.1.2.) Again, systems matter more than any individual component that provides AI functionality. There is no reason to hope that introducing AI to your product will somehow prevent (or resolve) problems that would cause a failure of the typical software project.
Do you personally have enough knowledge of data science and data engineering to understand the gap between the technical skills that your team has and the skills that they need for this project?
For this one, you’re the best person to answer the question as yes or no. A more important question is what will you do if the answer happens to be no? Can you ask someone else in your organization? Can you hire a consultant to help you?
Do you have a good enough relationship with your team members that they’re comfortable admitting the limitations of their skillset to you?
If the answer happens to be no, what are you going to do about it? How can you make your team members feel safe enough to disclose their perceived skill gaps? Longer term, the question is this: what should you do to improve that trust?
B.2.2. Answers to the scenario-based questions
Answer the following questions based on the scenario described.
One of the important skills in applying a Sense/Analyze/React loop is to identify who will execute on the React part of the pattern. For the following scenarios, answer this question: Who or what will carry out the action and fulfill the React part of the Sense/Analyze/React loop?
- Scenario 1: You’re making an automated car, and the AI that you’re using will allow fully autonomous driving under all conditions (so-called Level 5 autonomy [38], in which there are no available controls for the driver).
- Scenario 2: You’re writing a recommendation engine in which products are suggested to the customer.
- Scenario 3: You’re writing an AI program to regulate a smart thermostat that controls the temperature in your home.
- Scenario 1: The car itself is fulfilling the React part of the loop because AI is controlling all standard functions of the car. Some of the possible actions that the car can perform include driving within the given speed, stopping, turning, signaling, and using the horn.
- Scenario 2: The customer, when they make a purchase based on your recommendations.
- Scenario 3: In this case, it’s the HVAC system.
Use AI to create a new job. Find an example of an AI capability that would let you offer a new service that your organization doesn’t yet provide. (For the job to count as a solution to this exercise, it must be a job that’s so unrelated to the software development team that’s building the AI that the person hired for the job is unlikely to ever meet that team.)
This is a free-form exercise, so no single answer can be provided. The “Creating new jobs with AI” sidebar in section 2.5.3 gives one such example when using AI to monitor pets while owners are at work.
Suppose you’re using an AI algorithm in the context of a medical facility—let’s say a radiology department of a large hospital. You’re lucky to have on the team the best AI expert in the field of image classification, who has you covered on the AI side. While you’re confident that expert will be able to develop an AI algorithm to classify medical images as either normal or abnormal, that expert has never worked in a healthcare setting before. What other considerations do you need to address to develop a working AI product applicable to healthcare?
A nice thing about this question is that there are so many considerations in a project of this size that you’re almost certainly able to think of at least some that are applicable. The following is an (incomplete) list of considerations on projects like this:
- What’s the exact action that you should take to solve a problem? What does “problem” even mean for a medical image? Medical images of a child and an elderly person are different, and so are medical images of an athlete in top shape and someone who is living a more sedentary lifestyle. Do you provide medical analysis even for the medical images of healthy people, or is the problem you’re addressing limited to finding abnormal medical conditions? Do you diagnose medical condition precisely? Do you notify a person? Which person?
- Where would you get data to train the algorithm? Is that available from the hospital? Does the data need to be labeled? Does HIPAA [72,73] apply?
- How would you get an image from a patient into your AI system? How would you interface with the hospital’s system? What’s the workflow?
- Is the system reliable enough for use in a clinical setting? What types of errors are permissible? Can it misclassify 10% of normal images as a problem? Can it misclassify a problem image as being normal?
- Which regulations apply to you? Do you need regulatory approvals?
- And, all other considerations that apply to any other AI project: what infrastructure do you need, where would the data be stored, what are your organizational process and standards for developing software, and so on?
Apply the previous example from a hospital setting to a classification problem in your industry. What are the new considerations that exist in your industry as compared to the healthcare industry?
This is a free-form exercise, so no single answer can be provided.
Provide an example of an AI that has replaced a human role but doesn’t provide as good of an experience as a human would.
For me, automated voice prompts (those that are commonly encountered when you call customer support these days) are one example.
You’re a manufacturer of security cameras, and you’ve developed an AI algorithm that can detect a person in a picture. Regarding the taxonomy of its role in your business, how would you classify this use of AI?
This is an example of AI as a part of the larger product. Depending on how much confidence you have that you can correctly recognize a human as an intruder and the follow-up action you can take (for example, calling the police), this might also become a fully autonomous system.
You’re an insurance company, and you’ve developed an AI program that, based on static images from an accident site, could recognize which parts of the car are damaged in a wreck. Can this replace an insurance adjuster?
It’s unlikely, if all you have to go on is images from an accident site. Insurance adjusters need to check the car itself to be able to assess the damage. As such, this isn’t an example of a simple AI that can replace humans with a large cost savings, and, if used as such, it would likely fail. There’s a reason why even qualified mechanics need to open the hood to see what’s damaged underneath. But applied across all insurance adjusters and wrecks, this could be useful for fraud detection. For example, AI could flag for further investigation if individual adjusters often write off parts that look undamaged. As such, this type of AI can help the employees working in the fraud department (or even create new jobs in that department).
B.3. Answers to chapter 3 exercises
This section contains answers to the questions asked in chapter 3. For ease of reference, the question is repeated with the answer below it.
Suppose you’re working in the publishing industry, and you’re wondering if it’s better to release printed, electronic, and audiobooks at the same time or one after another. Also, if delivery is staged so that printed books are released first, how long should you wait before releasing the other formats? Within this setting, answer the following question: “What business metrics should you use?”
The appropriate business metrics to use depend on how the business of your specific organization is structured. Metrics are always specific to your organization. You should be suspicious of any statement of the form “Always use metric X” made before the person making such a statement takes a closer look at your organization.
The only correct answer to this question is, “It depends—what do you hope the book would achieve?” You should never just transplant a metric you’ve seen someone else use without analyzing if and why such a metric applies to you.
Maybe the total profit for the lifetime of a book would be the best metric. For some publishing organizations, and if the book’s only purpose was to make a profit, it would be. However, if you’re releasing a free book for a philanthropic organization, the best metric could just as well be, “The total number of new volunteers that you’ve recruited as a result of them reading the book.”
If you’re a business leader, define a business question and an appropriate metric to measure it. Think about some hypothetical scenarios not directly applicable to your organization (for example, some scenarios related to philanthropy). Think about actions that you can take while running a nonprofit. Use the techniques introduced in chapter 3 to select your first hypothetical business question, as well as the metrics you’d use to measure success.
This is a free-form exercise.
Once you’ve identified your business question from the previous exercise, take your senior AI expert to lunch and talk about the business problem. Ask them how they’d formulate a research question. Use the process described in chapter 3 to check whether or not the answer supports the business action you intend to take. And, while you’re having that lunch, talk about how you’d find a dataset to answer such a research question. Do you think you can acquire that dataset?
- For a research question, this is a free-form exercise.
- Finding a dataset for your research question clearly depends on the problem you’re trying to address, but it’s not uncommon that the answer to the question “Is it possible to acquire a dataset?” is no. Often, obtaining labeled data is the real obstacle to an application of AI.
- Also, if during this hypothetical conversation, neither of you thought about which data science/AI/ML methods you could use on the dataset, chances are, you might have missed some of the needed data. Remember that needed data and its quantity depend on the AI methods you use (and vice versa).
B.4. Answers to chapter 4 exercises
This section contains answers to the questions asked in chapter 4. For ease of reference, the question is repeated with the answer below it.
If your organization has run AI projects before, look at some progress reports and the metrics used in those reports. Answer the following:
- If we release software today, in its current state, how much money will we make/lose?
- If we can’t release today, how much better do our results need to be before we can release?
- Is it worth investing $100 K extra in getting 5% better results than we have today?
The answers for this exercise, of course, depend on the project. But, after completing this exercise, you already have the answers to whether your historical projects used metrics that business decisions could be based on.
Based on the answers to the previous questions, do you feel that your organization is making decisions in its AI projects based on the data, or is it possible that in some cases you had to make important decisions based on intuition?
Most organizations today have some ways to go before making the most of management decisions on AI projects based strictly on data. Don’t feel bad if your organization is in the same position. While transforming organizational mindsets is always a job that must be customized, I hope that the material in chapter 4 gives you some starting points. You might also want to review Osherove’s book [82] and Kotter International’s website [190] for a larger discussion on techniques for organizational transformation.
Suppose the cost to start a project is $100 K, and the policy of your organization is that no project that can’t create a 10% return on investment is worth doing. If your business metric is profit, what would be your value threshold for the project?
A minimum value threshold would be $100 K + 10%, or $110 K. However, in practice, you’re unlikely to be certain that the cost of the project is really going to be $100 K, so add whatever safety factor you think you should to $110 K. This is a simplified example in which the only consideration is ROI. In many organizations, you would also need to account for the cost of capital/how long you would need to earn those $10 K.
Go back to the bike rental example from this chapter. Suppose the estimated cost to assign a data scientist to the project is $10 K, and each extra bike costs $1 K. How much should you expect to improve the peak hour’s RMSE to make it worthwhile to assign a data scientist to the project?
If an extra bike costs $1 K, then you’d need to believe that you can improve the peak hour’s RMSE for at least $10 K/$1 K = 10 for it to be worthwhile to assign a data scientist to the project.
B.5. Answers to chapter 5 exercises
This section contains answers to the questions asked in chapter 5. For ease of reference, the question is repeated with the answer below it.
Construct an ML pipeline for this AI project: the project takes feedback from your customers and analyzes it. If a customer appears unhappy, an alert is issued so that you can contact the customer and try to appease them before they decide to leave. (That part of AI which determines whether a customer is happy or not is technically called sentiment analysis.) You already have an AI software library that performs sentiment analysis. The data is in your customer support system, which is a web application.
Figure B.1. ML pipeline for sentiment analysis of the customer feedback
- More than one result is an acceptable answer to this question; after all, there’s no universal ML pipeline that works the best in all cases!
- Figure B.1 shows one ML pipeline I’d start with.
Suppose you implement the ML pipeline from the previous example in your organization. Which departments would be responsible for the implementation of which parts of the pipeline?
The answer depends on your organization. The goal of this question is to get you thinking about your organization and visualizing people who would be involved.
What business metric would you use to measure the success of the ML pipeline from question 1?
- It depends on what you’re trying to achieve. As mentioned in chapter 3, you shouldn’t just blindly transplant metrics from other organizations to your project, even the metrics that happened to be suggested by me in the answers to the exercises in this book.
- Suppose that what you’re trying to achieve is the reduction of total customer turnover (churn). Clearly, reduction in churn is one such business metric.
- What if you’re trying to maximize profit from future business with customers? There are some questions as to how you’d exactly measure it, but let’s suppose that, for the sake of argument, you’d assume that past recurring business is a good predictor of future recurring business. Then the metric would be profit per customer saved, which in itself is different from revenue per customer saved. Moreover, as not all of the customers are of the same value (which is a common situation in many businesses), the result you’d get from this metric is very different from the result you’d get from churn reduction.
What is the history of the coordination between departments from question 2 in past projects that they’ve participated in? Were projects on which those teams worked successful?
Note
The next questions (5 and 6) are targeted toward data scientists. You can skip them if you don’t have data science expertise.
- The history, of course, would depend on your organization. What matters for the hypothetical project (and any new project) is to ask, “Which of the historical patterns you’ve seen are likely to repeat on the new project?”
- What has worked well? What didn’t work well? Can you fix issues that historically didn’t work well before they create a problem on the new project?
- How would you assess the risk that your organizational structure and way of working pose to the hypothetical project?
As a part of the installation of an AI security product, you’re offering a 30-day, money-back guarantee. Your customers have taken a survey about their satisfaction with the product, which they completed as soon as the product was installed. You’re interested in predicting if your customers would return the product. During discussions, the team has mentioned that this problem could be solved using either an SVM, a decision tree, logistic regression, or a deep learning-based classification. Should you use deep learning? After all, it’s an exceedingly popular technology, has a substantial mindshare, and could solve the problem. Or should you use one of the other suggested options?
- You can use a deep learning-based classifier, but I typically wouldn’t try it as my first (or even second) choice. Unless your survey is a monster with a thousand questions, it’s unclear that you’ll be able to train a large deep learning network at all.
- I’m not persuaded that for the typical survey of only a few questions, more complicated methods are going to produce better results. It is possible to analyze whether you need to apply a more complicated AI method to the problem (such as variance-bias tradeoff analysis [183]). However, in practice, I would start with a simple method such as logistic regression, a decision tree, or an SVM classifier. I would also try gradient boosting machines (GBMs) prior to trying deep learning; in most cases, they work better on tabular data anyway.
You answered question 5 using an algorithm of your choice. Suppose the algorithm you chose didn’t provide a good enough prediction of a customer returning the product. Should you use a better ML algorithm? Is it now time to use the latest and greatest from the field of deep learning?
- Your data may not be related to the problem you’re trying to solve (a customer returning your security product)! Remember, the survey result is completed as a mandatory step immediately after installation of the AI security system. Does the customer know enough at that time to know if they like the system as it is when they’re using it?
- Can you collect data about what the system was doing between installation and the moment a customer returned it? Is that data a better predictor of whether the customer will return the system? For that matter, can you survey customers when they’re returning the system?
- And a larger topic you should remember for practical AI projects is that, unlike an academic or Kaggle competition [191], you control which data you can collect! Don’t just take data you have as a given and assume you must use better ML algorithms.
B.6. Answers to chapter 6 exercises
This section contains answers to the questions asked in chapter 6. For ease of reference, the question is repeated with the answer below it. To help you look at the questions and solutions all in one place, in addition to the text of the questions, I’m also repeating the figure that the exercises refer to.
Figure B.2. An example ML pipeline. This figure is a repeat of figure 6.10.
You’ll also need to refer to table 6.1 (which is repeated here for your convenience as table B.1).
Table B.1. Summary of the possible results of MinMax analysis
|
Min result/Max result |
Max passed |
Max failed |
|---|---|---|
| The ML pipeline is business-viable. | This combination can’t happen. | |
| The ML pipeline needs improvement to be business-viable. | The current ML pipeline is not suitable for solving the business problem. |
Remember that because analyzing the ML pipeline is a technical and business skill, it is hoped that you were able to form a team consisting of a business specialist and an engineer and that your team does some of these exercises together.
Note that in table B.1, you don’t have any guidance for the situation in which the Min part of the MinMax has passed, but the Max part of the MinMax failed. Explain why this is the case.
The Min part of the MinMax analysis returns what the ML pipeline you have today can achieve. The Max part shows you the best that can be achieved. By definition, “the best you can do” can’t be worse than what you’ve already done.
For the ML pipeline in figure B.2, assume that the value threshold at which the project becomes business-viable is $1 million. Determine whether the pipeline is worth pursuing if the results of the MinMax analysis are as follows:
- Scenario 1: The Min part is $2.3 million, and the Max part is $23 million.
- Scenario 2: The Min part is $500 K, and the Max part is $1 million.
- Scenario 3: The Min part is $500 K, and the Max part is $2 million.
- Scenario 4: The Min part is $1.1 million, and the Max part is $900 K.
- Scenario 5: The Min part is $500 K, and the Max part is $900 K.
- Scenario 1: The Min part is above the value threshold; you already have a business-viable ML pipeline.
- Scenario 2: I would say that you need a new pipeline. Your Max analysis barely reaches the value threshold. I’m typically skeptical of the possibility that business and industry teams will be able to reach or exceed the best currently published result. I’m also skeptical that any value threshold’s estimate is entirely on target. There should be some safety factor here, and I’d assume that this isn’t a viable business pipeline.
- Scenario 3: In this case, I’d assume that the Min pipeline isn’t good enough, but (unless I have a reason for a safety factor above 2) I’m working with an ML pipeline that could be made business-viable. I’d perform a sensitivity analysis and see what happens.
- Scenario 4: Excuse me, how did this situation happen? Find a way to (politely) ask your engineering team to repeat the complete analysis. See the answer to this chapter’s question 1 for details.
- Scenario 5: Your ML pipeline isn’t business-viable. I wouldn’t attempt to pursue it unless I had one of the world’s leading teams in that particular area of AI, and I’d also be conscious of the possibility that the project might fail. Even if I had such an all-star team, I’d first check if I’d be able to construct a better pipeline.
If you’re a data scientist or technical manager, take a technical problem of your choice and construct an ML pipeline for it. Perform the Max part of the MinMax analysis for it.
This is a free-form exercise, and the goal is to begin to get you comfortable with thinking about which tools you’d use for the MinMax analysis:
- Are you familiar with the academic literature in the area and comfortable following it? Would you instead ask an expert? Who are the experts who could help you?
- What would be a good proxy problem for your ML pipeline?
- How much of a safety factor should there be? For that matter, do you feel that in your organization the safety factor should be based primarily on a technical opinion or primarily on a risk management (how much am I afraid of being wrong) decision?
If you’re a data scientist or technical manager, look at the examples given in section 6.4.1 and perform a MinMax analysis as described in that section. Determine where the dollar amount given in that section comes from. Hint: a profit curve was constructed from the confusion matrix of the classifier.
- Because the city is hedging you 50 parking overstays a year, your worst-case scenario is 51 illegal parking overstays per year. At that point, you still need to make a profit.
- I used the following formula to calculate the result:
(Accuracy * Profit_when_right – (1 – Accuracy) * Loss_when_wrong) * 51
How would you classify the use of AI in the context of saving litigation costs during the e-discovery process described in section 6.5.5? Use the taxonomy of AI uses introduced in section 2.5. It’s shown in figure 2.5, duplicated here as figure B.3, which summarizes the taxonomies discussed in that section.
Figure B.3. An AI taxonomy based on the high-level role it plays in business. You could use this taxonomy to guide you in eliciting available business actions you can help with AI. This figure is a repeat of figure 2.5.
- In the problem formulation presented in section 6.5.5, the role of AI is limited to just rejecting the document, as in, “For sure not related to litigation.” You are automating a step in the business process, and therefore such a use of AI would best fit under the category of automation of the business process.
- Note that if we were to extend the use of AI to help the attorney with all aspects of the e-discovery, then it would be better classified as a decision support system.
B.7. Answers to chapter 7 exercises
This section contains answers to the questions asked in chapter 7. For ease of reference, the question is repeated with the answer below it. To help you look at the questions and solutions all in one place, in addition to the text of the questions, I’m also repeating the figure that the exercises refer to.
Figure B.4. An example ML pipeline. We use this pipeline as a motivating example for sensitivity analysis. This is a repeat of figure 6.10 for the reader’s convenience.
This question gives you the results of the sensitivity analysis for the pipeline in figure B.4. Assume that the business metric is profit and the value threshold is $2 million/year. The results of your MinMax analysis are the Min part being $1.9 million/year and the Max part being $3 million/year. You decide to perform a sensitivity analysis. Why is it necessary to perform the sensitivity analysis? You’ve worked on all the stages for a while, and you’ve reached a point where it’s more and more challenging to improve any of the stages. Determine in which stage of the pipeline you should invest if the results of the sensitivity analysis are as follows:
- Stage A would require 6 months to improve by 1%. When you improve stage A, the overall improvement in the ML pipeline will be $10 K/%.
- Stage B would require 2 months to improve by 1%. When you improve stage B, the overall improvement in the ML pipeline will be $200 K/%.
- Stage C would require 1 year to improve by 1%. When you improve stage C, the overall improvement in the ML pipeline will be $800 K/%.
- The ML pipeline doesn’t show any appreciable improvement in results when stages D and E are improved. When does such a situation occur in practice?
- You needed to perform sensitivity analysis because your Min analysis failed and your Max analysis passed. Your ML pipeline isn’t business-viable as-is but hopefully can be made so.
- I’d personally try to improve stage B first or, if I had the resources, work on improving stages B and C in parallel. You’re reasonably close to business viability, and improvement in stage B is much faster to develop than in stage C. A lot can happen in a year, so try to make your AI pipeline business-viable soon.
- It’s not atypical for some of the stages in the ML pipeline (such as stages D and E in this pipeline) to contribute little to the quality of its overall result. The data those stages are producing may be unimportant for the ML pipeline as a whole. As an example, holiday sales data may have only minimal value for predicting customers’ ultimate satisfaction with the product—the person making a holiday purchase may not be the same person who would be using the product in the end. In that case, improving the quality of that data might not do much for the ML pipeline when predicting future customer satisfaction.
This question gives you the results of the sensitivity analysis for the pipeline in figure B.4. Assume that the business metric is profit and the value threshold is $2 million/year. The results of your MinMax analysis are the Min part being $1.9 million/year and the Max part being $3 million/year. You decide to perform a sensitivity analysis. You haven’t constructed any prototype or tried to clean the data. Determine in which stage of the pipeline you should invest if the results of the sensitivity analysis are as follows:
- Stage A would require 3 months to improve by 2%. When you improve stage A, the overall improvement in the ML pipeline will be $200 K/%.
- Stage B would require 2 months to improve by 1%. When you improve stage B, the overall improvement in the ML pipeline will be $100 K/%.
- Stage C would require 1 year to improve by 1%. When you improve stage C, the overall improvement in the ML pipeline will be $800 K/%.
- The ML pipeline doesn’t show any appreciable improvement in results when stages D and E are improved.
- If you haven’t constructed any implementation of even a minimal ML pipeline, you have no reason to believe that only small, incremental improvements are possible in your system. After all, it’s not like you tried hard to improve and are facing diminishing returns in every stage.
- Consequently, I don’t believe that assuming a linear response in the ML pipeline to the analysis is reasonable. I would recommend performing a full range sensitivity analysis of this pipeline.
- If I was facing a rare case in which linearization of this pipeline would be a reasonable assumption (for example, I didn’t try to improve it, but many other people tried and failed), I’d probably choose to upgrade stage A or stage B, based on how critical it was to deliver this ML pipeline quickly.
- Without a prototype, how was your Min analysis completed—is it only an estimate, or did you use commercial-off-the-shelf (COTS) products? If you used COTS, why can’t you use that COTS to build a Proof of Concept (POC) of your pipeline? Is that COTS appropriate for performing Min analysis (as opposed to Max analysis)? Similarly, how did you perform a sensitivity analysis? There are situations in which you will have a Min analysis completed without any POC. However, in the absence of POC, the questions above should be asked, as they would influence safety factors and the confidence you have in the results.
Your AI project is investigating if, by installing an IoT sensor to monitor a vehicle’s sound, you’d be able to determine what kinds of changes in tone would indicate a mechanical problem in the vehicle. You’ve deployed a sensor in 150 vehicles and waited for a month. Only a single vehicle had a mechanical problem. After the monthlong investigation, your data scientists tell you that from the data collected, they can’t predict breakage of the vehicles, and that a single broken vehicle is an insufficiently small dataset. Does this mean you can’t make an AI that can predict vehicle breakage?
- No, it doesn’t mean that you can’t make an AI that can predict vehicle breakage! There’s only a single breakage, and your team already hinted that they don’t think they got enough data to train AI algorithms.
- With more data, they might be able to succeed. However, with the current rate of breakage, it might take you many years to get enough data.
- This project should be classified as “We need more data to know the answer,” not as “AI prediction of breakages was tried and is impossible.” However, you should pause the project and try a different and easier one.
Suppose you have two ML pipelines. Your business metric is revenue. The value threshold is constant at $10 million/year. You have two parallel teams that could work on both ML pipelines. Pipeline 1 would deliver $20 million/year, and pipeline 2 would provide $30 million/year. The cost of the team to develop the pipeline is small compared to the lifetime profit expected from the AI project. Your organization can implement pipeline 1 in 4 months and pipeline 2 in 1 year. Determine which of the two pipelines you should release, and when. Also, draw a timing diagram showing these two pipelines.
- You should release both pipelines, as they both exceed the value threshold, and the cost of the team is small compared to the lifetime value of the pipeline.
- Figure B.5 shows the timing diagram.
Figure B.5. A timing diagram that answers question 4 of chapter 7’s exercises
B.8. Answers to chapter 8 exercises
This section contains answers to the questions asked in chapter 8. For ease of reference, the question is repeated with the answer below it.
Explain whether AI as practiced today is applicable to the following projects:
- Scenario 1: Predicting short-term retail and economic activity based on using satellite images to track the movement and number of ocean-going container ships
- Scenario 2: Predicting new prices of a portfolio in the case of an unexpected event that completely disrupts the foundations of the global economic system
- Scenario 3: Predicting the effect of genetic research on healthcare spending costs 20 years from now
- Scenario 1: AI is definitely applicable to this problem. You can recognize a container ship on a satellite picture. The picture can also recognize the approximate size of the ship and direction of travel. That allows you to know the amount of goods that are being transported. There’s a clear relationship between transportation and sale of goods, and in the globalized economy, transportation of goods comes before retail activity.
- Scenario 2: AI is likely hopeless on this problem because it doesn’t account for a causal relationship. Remember, correlation-based AI is valid only in the world that looks like the one in which it’s trained. This question doesn’t describe such a world.
- Scenario 3: Not likely. We certainly don’t have data about such a world. It’s not clear that we can predict what genetic research will come up with or how that would be applicable to healthcare. It also isn’t clear that we even understand what causal relationships exist between genetic data and future healthcare costs. Any model trying to extrapolate that far out is probably extrapolating well beyond the breaking point.
Reflect on the last three times your organization adapted a new and popular technology, after which the consensus in hindsight was that the project didn’t succeed. Now, find the reasons why the project was not successful. By the way, saying that the wrong people were on the team or that you weren’t experienced enough with technology aren’t acceptable reasons.
Avoiding discussions of people and personalities makes the dialogue more palatable in a corporate setting. More importantly, after these exclusions, what remains are the inherent process weaknesses that your organization has in evaluating new technologies. The goal is to recognize those same weaknesses the next time a new technology comes in vogue. This time, the technology in vogue is AI.
- The answers for each organization are going to be specific. The goal of this question is for you to concentrate on the system your organization uses, not on the personalities or one-time events. Chances are, you’d see problems that are produced by the weaknesses of the system even after employees changed.
- For extra credit, you can apply counterfactual analysis on the characteristics of your organization that you’ve identified. Has your organization made any recent changes that are likely to result in a different outcome than the ones that have happened historically?
For each of the trends introduced in this chapter (as exemplified by the headings of sections 8.1 to 8.6), answer the following questions. As a reminder, those trends are listed in table 8.1 (which is repeated here for your convenience as table B.2).
Table B.2. Trends introduced in this chapter. How do those trends affect you?
Section 8.1 The meaning of the term AI changes through time. Section 8.2 AI use in physical systems must account for safety. Section 8.3 AI systems today don’t account for causal relationships. Section 8.4 AI algorithms don’t typically account for the variable veracity in data. Section 8.5 AI systems’ mistakes are different from human errors. Section 8.6 AutoML is approaching.
- Does this trend affect your current project?
- Is the trend likely to affect your organization in general?
- Is the trend likely to affect your personal career?
- How do you intend to follow the trend?
- How will you know if the trend is materializing or not?
Every organization will have specific answers to these questions.
Are there any trends of specific applications of AI in your industry that affect you but weren’t enumerated in this chapter?
Almost certainly there are some, because AI use is still in its infancy and is not widely adopted across many different industries today. Chances are, if you performed a serious and complete analysis and still have trouble identifying an AI trend that affects your industry, you’ve just discovered a business opportunity. You might be among the first people who seriously thought about the best ways to use AI in your industry!