Lack of Structured Deployment Processes

Analytical projects are often conducted in silos within most organizations—a reflection of how data is organized according to each internal department. This has led to the formation of fragmented analytics initiatives and capabilities. It also means that the governance and structuring of analytical initiatives is left to the responsibility of each department in turn, leading to longer model deployment cycles, without having a standardized enterprise-wide approach for analytical projects.

Another hurdle is the separation between development and deployment environments. In some instances, organizations have adopted an experimental approach to analytics, which means that the model development is done in an ad-hoc way, rather than following a standardized process that is designed for a simplified deployment in production. Often, models are prevented from successful deployment due to the incorrect specification of the model logic, the input data, or limitations in operating systems (e.g., some production systems cannot handle nonlinear transformations).

Many organizations are actively seeking technological capabilities, which help centralize model development and deployment processes. The SAS platform introduces this capability by allowing organizations to define and track custom workflows for model lifecycle management. Figure 6.1 is an example of how the end-to-end process can be structured in a series of tasks for developing, validating, and approving a model.

The Need to Manually Recode Complex Models

As mentioned above, as models are typically developed in siloes, it requires translation of the model logic from one application system to another. Taking a hypothetical example, the back office may use a custom software application for statistical modeling to develop a new loss forecasting model. This model will need to be applied during a regular monthly impairment calculation process, as well as in a front office system for the pricing of new commercial loans. This will require the translation of the model logic into the system used for monthly impairment calculations, as well as into the front office system used by credit analysts as they structure new deals. This is a challenge as it increases the operational model risk (e.g., potential for coding error in translation resulting in incorrect impairment values), delays in the model delivering value to the organization (as it takes time to recode the new model), and a lack of flexibility to make changes to the model in the event of a required model change.

With consistently packaged algorithmic code and data that follows a “build once, deploy anywhere” approach, inefficient, repetitive, and manual deployment tasks can be avoided (Figure 6.2). In addition, it improves the scalability and agility of model deployments.

Managing Multiple Analytical Tools and Programming Languages

In recent years, with the rapid development of open-source technologies, risk modelers have more flexibility and access to a range of modeling techniques, combined with support from open community forums. For the IT (information technology) departments, responsible for managing the deployment of new risk models, it requires knowledge of a range of different programming languages. A single deployment approach, regardless of model development language, that follows the same standardized process (Figure 6.3) for a range of risk models provides flexibility to model developers in their preferences of programming language.

Signoff and Approvals

To productionize new risk models, proper signoffs, documentation, and approvals from internal and sometimes external stakeholders are needed. To include a new model in a “live” production system may require additional technical approvals and documentation. The model deployment and the ongoing maintenance of a model in a production system are subject to the governance framework, including having the necessary model contingency plans in place.

Adoption of Agile Practices for ModelOps

Success and proficiency in the use of agile practices in developing and deploying software has led to the increased adoption of these practices for developing and deploying analytical models as well. Crucial to enabling agile practices is the use of the appropriate technology for an organization's specific needs. The right technology enables efficient and reliable deployment of software, fast feedback for improvement, and enables greater experimentation to test new ideas—all of which are key goals of an agile method.

Similarly, organizations are looking to achieve the same goals for analytical models. However, analytical models are more than just code—they are data dependent, have precise monitoring requirements, and are used to drive key business decisions. It is because of these distinctions that there is a challenge in taking only an IT-led agile approach, where tools designed to be effective for software development are ineffective for analytical models. Therefore, for an organization to adopt agile practices for models, also known as ModelOps, it must be able to design a process that includes appropriate analytical tools that can integrate with its existing successful agile software development practices.

Deploying Models in Batch Processes

Batch processing (Figure 6.2) is a common way to deploy many risk models. Large volumes of data that are collated over a period can be processed in batch when convenient. This method is a highly efficient way to score models for use cases that are not time sensitive. Typically, the infrastructures required for batch deployments are simpler than those required for real-time deployments. For example, to perform monthly scoring for limit increase approvals, a process will extract data from source systems and apply the data transformation, model scoring, and other post-processing logic to calculate the model output, including the pre-approved limits. One of the main benefits of batch processing is that compute resources can be planned for. It can be made available for regular processing or scheduled to run off-peak:

• Generate model output for business decisions that are not time sensitive.
• Data inputs that are accumulated over a period before models are scored.

Deploying Models in Real Time

In today's world, society has grown used to services and systems that respond instantaneously. In this scenario, model outputs are time sensitive, often with the expectation to be available on demand.

Financial institutions are increasingly competing in supplying services and risk decisions instantly. In such a scenario, risk models must be deployed in real time. For example, for new loan approvals, the risk assessment of a customer based on the input data gets generated immediately on request to decide to approve or reject the application.

Another example is in fraud detection and prevention. Financial institutions often have in place fraud detection models that were developed on historical cases of suspicious transactions. These systems score each new transaction based on a combination of customer and transaction information to find suspicious activity. As the goal is to stop a fraudulent transaction from being processed, these models need to supply answers in real time:

• Model outputs for business decisions are time sensitive.
• Model outputs are generated at the transactional level.
• Model output rates must be as close as possible to data input rates.

Modern infrastructures are evolving where we see variations to the deployment of models to batch or real-time applications.

In some instances, mostly for batch deployments, the model logic can be deployed to execute within a database, reducing the need for data movement, and increasing the speed and efficiency of the scoring process.

Financial institutions are also increasingly adopting a cloud-based, microservices architecture where models are deployed as analytic services in containers. In this way, models are deployed in the cloud as lightweight as possible. The container software manages the operating layer and is accessed easily through an API endpoint. This option supports both batch and real-time deployment scenarios.

We next explain three deployment options for risk models: in-database deployment, to lightweight containers, or directly in business decision workflows.

Deployment of Models in Database Management Systems

With the explosion of data, big datasets are typically stored in database management systems. To reduce the need for data movement, modern technologies enable the deployment of data preparation, model logic, and decision logic to where the data is residing, rather than moving the data to a compute environment for processing. The benefits are reduced input/output to transfer the data between systems, better performance, and a smaller memory footprint needed for analytical processing.

Deployment of Models to Lightweight Containers

Models can also be deployed as lightweight containers. In this case and in general, the container is exposed by an API to allow it to be accessed from anywhere within a wider ecosystem. With the progressive adoption of software applications being deployed as containers, this approach has been adopted for models as well—it being particularly effective for open-source models, as running in a container allows for easier management of the packages and dependencies of the models.

Putting models into containers makes it ideal for an organization to run efficient analytical workloads in the cloud. Models deployed in containers allow an organization to flexibly scale the infrastructure for running models, as per their business needs. Model container deployment can suit both batch and real-time paradigms, as containers can be scaled to meet computational requirements—for example, every month (batch) or as more users are requesting model output from a live web application (real time).

Additionally, within a cloud-based architecture, containerizing models can greatly ease the integration of models with other applications. Models as containers are managed by the same container orchestration tools and CI/CD infrastructure as other software applications, essentially managing them as analytical microservices.

Deployments in Business Decision Workflows

The output of models can take on the form of predicted probabilities or predicted numerical values. While informative, these results often need to be integrated with business rules to deliver business-specific outcomes (Figure 6.4).

In the case for fraud detection, in isolation, the fraud model would produce a predicted probability of a transaction being fraudulent. However, there needs to be an integration with post-processing decision logic. An example of that could be a combination of the transaction amount and the fraud probability to compute an expected fraud amount. The expected fraud amount can help in prioritizing flagged transactions for investigation.

It is through the integration with decisioning workflows and business logic that the true business value can be extracted from analytical models. Organizations with the technological capability to automate the integration will realize the return on investment in AI and machine learning more efficiently.

Begin with the End in Mind

In reviewing the risk model lifecycle as depicted in Chapter 3 and shown in Figure 6.5, each step affects the deployment of the model. For example, the first step in the process is the initiation of the model development project. First, it makes sense to ask, where will the model be deployed? And how will the model outputs be consumed? How often and how soon do users need the model outputs?

Second, the data preparation and transformations of the model will form part of the scoring logic. It makes sense to ask, what data sources will be needed to prepare the data and score the model?

Third, keep in mind, in general, that AI and machine learning algorithms require heavy compute power for development, but may be surprisingly light when it comes to scoring (e.g., neural networks for function approximation).

Furthermore, the post-modeling processes, including decision logic and the feedback for regular monitoring of the model, are all design decisions that will affect the ease of deployment. We next describe key considerations of regular, continuous model monitoring.

Continuous Model Monitoring

Traditionally, financial institutions have in place standard monitoring processes for risk models. Risk models are typically subject to regulatory compliance that requires regular performance tracking.

As the number of risk models are ever increasing, the sophistication of modeling methods deployed by financial institutions is also increasing. This is particularly clear with the adoption of AI and machine learning, where enhanced transparency is often needed. As new data becomes available, AI and machine learning are often dynamically updated, which requires that the performance of the model is monitored dynamically. Many metrics are tracked to ensure that the model is performing as expected, including data quality, model drift, explainability, model robustness, and bias.

By automatically and dynamically tracking the performance, including champion and challenger models, risk departments will be better equipped to proactively find issues that may lead to model failure (Figure 6.6).