The commercialization of machine learning (ML) models and artificial intelligence (AI) is bringing about a dramatic transformation in the way businesses view the value of their data. Although there is no doubt that there is incredible potential, this era represents a challenging transition time for businesses and technology leaders. Amid all the potential benefits of AI and ML, there are serious risks that often are ignored. Organizations must be able to deal with an abundance of both business and technical risks. Some risks are obvious, such as the ethical risks of collecting personal data when individuals have not given their permission. However, business leaders need to understand the vast array of risks and issues they need to prepare for before they can leverage the power of AI and ML.

In this chapter, we discuss risks that businesses need to consider when applying machine learning to business tasks that have the potential to augment how work gets accomplished in everyday business life. We are separating risks from the ethical issues of gathering data. The issue of ethics and governance will be discussed in Chapter 7. In Chapter 7, we discuss the potential for litigation stemming from ethical failures that impact customers’ lives and finances.

Before we get into the details of the risks a business needs to be prepared to address, let’s take a step back and understand what is actually happening inside the machine learning components as they apply to business tasks. Let’s assume that a business plans to replace straightforward repetitive process tasks with machine learning models. Although there are wild promises that machines will be smart enough to replace everything a human does, it is not quite that simple—even when the tasks seem relatively routine. For example, the machine learning model would have to learn the purpose and context is for every task being automated. What is the goal of executing a process? What happens if an action related to one task is done incorrectly because of a problem with the data that has been ingested? What if the data is old and hasn’t been updated? What if two different processes are using different data sources and different conflicting rules? Unlike a human team member, an ML component cannot alert other team members about the job it performs, or work around activities if something seems wrong.

Furthermore, we can ask: Do the results of the process really serve the business objectives of the organization? Remember that ML algorithms and models are code created by developers. Successful models require enough knowledge and nuance from the team of experts to ensure that the results are correct and useful for the business task. If the developer does not understand the business and the processes behind the way the business should operate, the models are likely to fail to achieve their goals and add a significant level of risk to the organization.

Once a machine learning component is available to perform a business task without a team member, the way the business process is executed will have to change. At a fundamental level, there are simply different expectations for how automation handles a problem compared to how a person will approach the same task. In the traditional business, staff members have a set process for conducting business and rarely alter their routine unless forced to by changing business conditions. Needless to say, people are creatures of habit. In contrast, a machine learning model is not bound by “the way things have always been done.” One of the powerful characteristics of a machine learning model is that it is capable of creating an innovative approach to achieving a business objective. The objective of the model requires that business processes must evolve so that they are more efficient, more adaptable, and more innovative. But there is a danger in simply trusting a machine to understand the constraints required for a responsible business. So, the question remains, how do you innovate and transform without putting the business at risk?

Building an innovative model is not automatic. The data scientist must work side by side with subject matter experts to understand the purpose of the business process and how it can be changed and adapted in a consistent and predictable manner. If the team builds a model in isolation, it is possible to create unanticipated consequences that risk real harm to the business. For example, consider a business process that requires the business to guarantee that if a product is not in stock, the company will offer the customer a third-party alternative if they agree. What if the ML model that predicts product availability fails to include the permission agreement? In this case, a tried-and-true business process has violated the trust between vendor and customer. Failure to support this adaptation and provide it with sufficient resources assures that the augmentation of the team will fail or perform poorly.

It is tempting to assume that once a business builds a machine learning component, the staff’s job is done. The process may have been directly automated or even dramatically changed. However, this simplistic view does not consider some key considerations. First, let’s assume that the process has been completely redesigned so that there is no historical data available for testing. This new process has to be integrated with the rest of the overall business process that operates a key part of how business is conducted. The machine only executes the model based on the data being ingested. The model often must be viewed as part of a larger unit of work that defines a combination of new innovations and basic best practices for an industry. Although digital disruption is a rallying cry for businesses that need to change to compete, there are certain business principles of how commerce is conducted that will never change. For example, the customer must be provided with the right level of information to ensure the accountability of the vendor. The product and service must be delivered at the right time and the right price. Accounts payable (AP) processes have to work as required by the company’s business practices.

The basic truth is that the human team still understands the context of the organization’s overall business process. It is critical that team members be able to quickly manage important changes to business processes as the market changes. It may not be obvious how the ML component will impact a business process. All team members, not just the data scientist and subject experts, need to know how the ML model is intended to work and where it fits into the overall process. Then they can assess if the model is contributing to the business process changes and whether the changes are having the intended goals. If the model is not contributing well, then the team will need to step, document the failures, and make changes in the model. Therefore, the model has to be designed to anticipate errors and the fact that customers do not always act rationally.

How do you know how well a machine learning model is performing? Is it the responsibility of senior managers? Does the data scientist have the ability to understand what is happening within that model? In reality, many risks can impact the validity and consequences of deploying a model. What if the data scientist has selected an algorithm that is not appropriate for the task at hand? What if the algorithm is not given enough data? There may be a situation when the wrong features (measurable property) are selected. Selecting a wrong feature can mean that the wrong characteristic is being measured and, therefore, will not be helpful to the business. The risk of wrong feature selection can be significant because management will pay attention to the wrong processes and issues. Even if the features are correct, the model’s performance may be hindered by the new data being ingested into the model. When data is not correct, the business process simply will fail to deliver the required business results.

A new ML component is a bit like a new employee who seems to have all the right credentials for the job but is unable to fulfill the needs of the business. The employee might have potential but may not understand the business well enough to make the right decisions. Likewise, as the business adopts new business processes based on changed machine learning models, there is a risk that the model will not perform as intended and cause problems. An even more dangerous risk is that the process in which the model does its work will be com promised. Therefore, both the data scientists and the business team members have to understand the components of the model well enough to be able to assess the performance of the model with the ingested data and how it is impacting the overall business processes in which it operates. The team must regularly check that the model is doing what it is expected to do and have a means of performing digital audits, that is, simply said, running tests to ensure that the model is working well.

One technique that has emerged to help is to have experts conduct digital auditing to assess how well an algorithmic model is working. That assessment is called algorithmic accountability. Algorithmic accountability is not just good “business hygiene.” It is essential for knowing that models coming from machine learning algorithms work properly, and for conforming with new federal laws being developed by the US Congress to address machine learning and its role in business.

Although outside consulting firms are now offering special services for digital auditing, this process can be managed in house if senior management engages a business to undertake this process. Digital auditing involves assessment of what goals the algorithm has been asked to address, what decisions it is making, and what data has been used to create the model decisions. The whole process of developing the algorithmic model, including the training and testing of the model, must be scrutinized to assure that the model has not introduced bias in its decision making.

Algorithm accountability will be even more useful if the tests that make up a digital audit use both past as well as new business data (even if the team has to “make up” the data because there is none at hand). Because the needs of customers evolve over time, those changes will be reflected in the data the business inputs to the model. If new data is markedly different from the original data used to build the model, then the algorithm could make different decisions given the new data. Assessing the model’s performance on old and new data will give some indication of the overall behavior of the model.

Furthermore, some knowledge of the differences in the use of new data will allow the team to continually assess whether the model is changing or instead failing, as more and more new data come online. The value of new data is its use in seeing new trends before human teams become aware of them. However, if there is not enough new data to retrain the model, the old model will hold sway and make fewer good predictions. If there is enough new data and the new results are dramatically different, the model might seem unreliable to the team. It is imperative to keep the team informed about how well an algorithm is performing in order to mitigate risks in using machine learning–based models.

Other factors can be critical in digital auditing. Not only can a machine learning algorithm be faulty, but efforts to correct predictions can be suppressed. For example, a model’s predictions might be ignored if it surfaces problems in product safety that the team does not want to acknowledge. Correct predictions could also be manipulated to add results that favor a direction desired by some part of the management team. Although overall digital auditing goes beyond the concerns of AI augmentation, its practice will be valuable to many businesses in mitigating the risks in machine learning models.

Many companies are beginning to use powerful machine learning techniques as a way to transform their business processes. Leveraging data in order to drive business decisions through artificial intelligence and machine learning models can be surprisingly powerful. However, teams may rush to begin using machine learning models without understanding the full impact on the business. One of the biggest issues that organizations face is that they simply do not have enough data to ensure the accuracy of results from the use of machine learning algorithms. Gathering an ample amount of data often requires procuring information from partners or other third-party data sources. Sharing this data can result in exposing private customer data. The typical process of anonymizing data is intended to sanitize private information. Anonymizing is done either through encryption or by removing personally identifiable information from data. Masking company data for others to use is challenging and risky. First, the business has to be very careful that critical private or mission critical data isn’t accidently disclosed. Although experienced security experts are often able to protect data, there are no guarantees that mistakes will not occur. Avoiding risks of data security breaches requires a team effort to ensure that everyone understands the rules: what data must be kept private and what data can be freely shared. It is not surprising that a number of businesses simply refuse to share data that they consider too private and confidential to expose the information outside their data center. The risks may not even with obvious. For example, there have been situations in which business leaders did not even realize that data rules have been violated. For example, there was a situation at a state governmental agency where data scientists were analyzing citizen data. All social security numbers and names were blacked out. However, there was enough other identifiable information that it was easy to determine the identities of individuals.

Although it is impossible to anticipate all the risks that can result from the way data is used within machine learning models, there are some fundamental risks that are a good starting point. In this next section we will detail seven important risks that should help you get started. Keep in mind that as you work with your models in context with business change, new risks may emerge.

If you are not prepared to deal with risks, you will be putting your business in danger. One of the ironic factors leading to these risks is that you are innovating by beginning to break down silos between your sources of data.

Machine learning models for business decision making are incredibly powerful in helping subject matter experts gain better insights into the steps needed to execute decisions. Being able to harness all of the data collected over years to find patterns and anomalies is powerful but not without risk. If an organization approaches machine learning with a focus on avoiding pitfalls, the models can be a game changer.

This chapter has presented seven key risks for businesses using machine learning models to enhance business decision making: overfitting or under-fitting the data used in the model, changing business processes, failure to understand the ethical implications in biases in data and models, poor business decisions based on bias, overreliance on the machine learning model, lack of explainability of the model, revealing confidential information in sharing data for models, and constructing a team with little regard for their skill and knowledge in using the model. Each of these risks can be significantly mitigated by awareness of these risks by a team of business and machine learning experts chosen to work on business processes, including the machine learning models, along with regular review of the data used in the model as well as the model’s performance. In addition, the business team should take the time to brainstorm new ways to deal with changes in the model as the business strategy transitions.