Staffing

Staffing AI projects entirely through new hires may be prohibitively expensive, since these required people (data scientists, for example) tend to be in high demand. That approach also limits people within the company from moving into AI projects and bringing along their domain expertise about the business. Good strategies for staffing AI projects will blend new hires with programs for upskilling existing staff, as well as promoting lateral moves in-house through mentoring programs.

One important role, often referred to as a “unicorn,” is the person who has both product management experience and familiarity with the AI technologies involved. These people become crucial for translating between available technology and the business use cases. Over time, the product management programs will incorporate more AI-specific skills, and more people coming from the technology side will advance into product roles. Meanwhile, this represents a staffing bottleneck.

Collaboration

As your AI projects move into production, multiple teams will need to coordinate closely, sharing responsibilities for oversight. How can those respective teams collaborate without having to learn one another’s disciplines in depth? For example, a given project might require a data science team, an operations team, and a risk team to review one another’s work. It would be prohibitive for each of those teams to learn one another’s respective disciplines; some cross-training can help build mutual trust and interdisciplinary collaboration, but there are limits on requiring people to wear too many hats at the same time. Finding a proper balance between abstracting another team’s contributions and obstructing the other teams is key.

Instead of finding out about possible limits or obstructing factors at the last minute, we can be proactive by engaging early with people across the organization who will be working on AI projects. They need to have a complete understanding about the data on which those projects will depend:

• What is the scope of the data required, and does that data even exist within the organization?

• Which team collects and manages that data?

• Do we have access and permission to use it?

• Which other teams use this data, and is there existing code or analysis that we could reuse?

• Are there any known data quality issues, and can we trust this data source?

Resolving cross-departmental collaboration issues often requires business stakeholders to step in and set priorities overall. Such leadership may also clarify who to talk with about what, across the organization. Establishing this leadership from stakeholders often becomes the most difficult step.

Executive Support

Moving up an organizational level, far too often the executive ranks do not support the initiatives or even the kind of thinking required for successful AI projects. Executive staff and board members may think in terms of once-popular management ideologies such as Six Sigma and the 1990s era of business intelligence, which do not fit well with contemporary needs for AI. For example, one core principle in Six Sigma is to reduce process variation: to get rid of sources of uncertainty. The language of Six Sigma and related business philosophies tends to emphasize terms such as predictable and consistency and control, which are at odds with AI technologies that depend on probabilistic techniques and embrace “variation” in the data. Aversion to uncertainty is prevalent among executives. Meanwhile, data from that period of Six Sigma was generally organized in data warehouses and leveraged through business intelligence tools. Data that didn’t fit into SQL queries effectively didn’t matter, but that’s become legacy thinking.

Now, the world leverages machine learning. While SQL queries are still important for data management, reporting, and even getting training data ready for building models, we most definitely build machine learning and deep learning models using completely different techniques! The more valuable algorithms at their very core are stochastic in nature: the algorithms are based on probabilistic approaches—that is, parts are randomized—which cannot be predicted precisely. Nor do we necessarily want to reduce variance within the training data; in fact, that can become a way of introducing harmful bias.

As we noted, AI approaches tend to embrace uncertainty—for example, taking advantage of uncertainty in data to explore/exploit regions of opportunity. To those who still think in terms of earlier times, the priorities for AI will seem inverted: up is down, left is right. Unfortunately, all too often that confusion translates into missteps for organizational priorities.

An important point to keep in mind is that the enterprise transformations required to overcome these challenges—such as getting robust data infrastructure into place, with widespread adoption within an organization—are generally long-term projects that may take years to complete. Even if the top executive ranks give their full buy-in about priorities, the results won’t be immediately visible. Without long-term vision and commitment from the executive ranks, the foundational components become nonstarters for AI adoption. Meanwhile, the competition is probably leveraging AI and moving fast, gaining ground.

People: Training and Mentoring

From the perspective of the technical stakeholders, one of the most immediate concerns should be to get their people trained and prepared for the new roles required for AI work. To reinforce the upskilling and training, it’s also important to support mentoring programs internally so that people help one another learn and grow in their roles. In the next few sections, we’ll cover the steps needed to achieve an AI Center of Excellence, which addresses this need for training and mentoring.

Process: Unique Platforms and Processes

AI work often requires unique platforms and processes. Software engineers who work within integrated development environment (IDE) tools may feel bewildered by a flurry of new terms and techniques that now become potentially more crucial than stepping through code with a debugger: sampling, cross-validation, imbalanced data, dropout, learning rate, entropy, vanishing gradients, etc.

The typical processes used in AI development will lead to priorities, concerns, and means of evaluation that differ substantially from software engineering. This is where so much of the current MLOps dialogue in industry is coming from. Moreover, the best-of-breed tools for AI are evolving and changing rapidly. For example, among the deep learning frameworks, TensorFlow had been quite a buzzword in recent years, although PyTorch gets used more widely; even so, JAX usage is rising, and practitioners should probably be learning their way through HuggingFace libraries, too. For a technical stakeholder, embracing AI represents yet another set of steep learning curves to navigate, in order to make appropriate decisions at the organizational level.

Platform: Collaborative Data Infrastructure

Another hurdle faced by data science teams in many organizations is the matter of gaining access to the data needed for AI projects. Far too often, access to data gets hampered by outdated data management policies or data frameworks that are locked in silos. For example, one department may not be able to read another department’s datasets without authorization. Organizations must provide data platforms as infrastructure that allow for cross-departmental collaboration and data sharing. Often, that requires paying down infrastructure technical debt as a first step.

Other Challenges and Balancing Time-to-Market

Even when the necessary tools and practices are in place to begin developing AI solutions, there are still other areas of concern where the technical stakeholders must lead: trust, data privacy, fairness and bias, security, risks due to model drift, and so on. Given infinite time to research a problem and refine a solution for it, we could do wonders with AI, although we’ll probably never get to enjoy that situation. Instead, we have budgets, schedules, reviews, audits, and so on. The metric that puts all of these points into context is time-to-market. Technical stakeholders, therefore, need to focus on managing the time-to-market metrics in order to guide their teams toward success.

Bias and Fairness

When developing AI solutions, we need to ensure that privileged groups are not at a systematic advantage compared to nonprivileged groups. The definition of a privileged group depends on industry and use case. In general, features like age, gender, and ethnicity are particularly sensitive elements when defining privileged groups. Take the example of a credit risk use case. The artificial intelligence/machine learning (AI/ML) model classifies whether a certain credit application is risky or not. We need to ensure that the model outcomes (risky or not risky) are not biased toward a particular age, gender, or ethnicity.

Eliminating bias and ensuring fairness in our AI/ML models is not as simple as avoiding sensitive features for model training and scoring.

Implicit bias often becomes an issue when other features can act as proxies for sensitive features. Zip code, for example, is a particularly common proxy, but there can be other nonobvious proxies, especially when new features have been created through automated feature-engineering steps. Regardless of whether it is a direct or implicit bias scenario, we need to have the proper guardrails to ensure fair outcomes.

Drift

Most runtime environments will be able to monitor standard performance metrics used in an AI/ML model (for example, metrics like accuracy, AUC, precision, recall, and F1 score in a classification use case). The performance of a model can drift over time for various reasons. Model drift could be caused by changes in the relative weights or contributions of different features toward the outcome. Drift can also relate to the data that is being fed into the model. Data drift occurs when payload data (input data that is sent to an AI/ML model plus the model’s output) is considerably different from training data. For example, training data may not have included data ranges or combinations seen in real life. It is important to monitor for both model performance drift and data drift over time and take corrective action when thresholds are breached.

Robustness

We need our AI/ML model to be consistent in its behavior. This is especially true when executing models in environments different from original development, training, and testing environments. Models also need to be defended against adversarial attacks. For example, we need a mechanism to check if training data has been poisoned with malicious samples. Here again, we see the importance of monitoring the model over time.

Explainability

Traditional statistical models are relatively simple to interpret and explain. Interpretability and explainability become more difficult when working with models that use techniques like neural networks and gradient boosting. Explainability takes on different forms. It could be a global explanation of why a model behaves a certain way. We are seeing an increasing requirement for local explainability. This is where we need to provide an explanation for an individual transaction, either to the end user or to a governing or regulatory body. The organization should be able to play back the transaction, explain why the model produced a certain outcome, and then trace it all the way back through model version, engineered features, the datasets that features came from, and so on. There is also increasing demand for contrastive explanations, which support “what-if” analyses—for example, “At what point would an outcome have been different?”

All of these challenges need to be addressed as you operationalize AI, end-to-end and across the life cycle stages (which we’ll cover shortly). Emerging regulations around AI ethics are suggesting the need for a structured approach to detecting, monitoring, and correcting these issues to deliver trusted AI.

Regulations and Roles

Various roles in an organization need to be involved to ensure trusted AI. Traditionally, highly regulated industries like finance and health care have had model risk management (MRM) teams. For example, an MRM team in a US-based financial sector organization (like a retail bank) would look into ways to manage and mitigate the reputational or financial risk of using financial models, in compliance with the Federal Reserve’s SR 11-7 guidance. AI/ML models introduce additional challenges, as we previously discussed, and the MRM/validation teams need to work alongside the data science teams to address these challenges at the speed and scale at which model versions are produced.

Operationalizing AI in a trustworthy manner is not just the responsibility of a risk management or compliance team, but rather the responsibility of all the different roles and personas involved across the AI life cycle.