Role in the Machine Learning Model Lifecycle

Subject matter experts usually come to the table — or at least, they should come to the table — with clearly defined goals, business questions, and/or key performance indicators (KPIs) that they want to achieve or address. In some cases, they might be extremely well defined (e.g., “In order to hit our numbers for the quarter, we need to reduce customer churn by 10%” or “We’re losing $N per quarter due to unscheduled maintenance, how can we better predict downtime?”) In other cases, less so (e.g., “Our service staff needs to better understand our customers to upsell them” or “How can we get people to buy more widgets?”). 

In organizations with healthy processes, starting the machine learning model lifecycle with a more well-defined business question isn’t necessarily always an imperative, or even an ideal scenario. Working with a less-defined business goal can be a good opportunity for subject matter experts to work directly with data scientists upfront to better frame the problem and brainstorm possible solutions before even beginning any data exploration or model experimentation. 

Without this critical starting point from subject matter experts, other data professionals (particularly data scientists) risk starting the machine learning lifecycle process trying to solve problems or provide solutions that don’t serve the larger business. Ultimately, this is detrimental not only to the subject matter experts who need to partner with data scientists and other data experts to build solutions, but to data scientists themselves who might struggle to provide larger value (data teams might in turn see trust in — or budgets for — data initiatives fall). Business decision modeling methodologies can be applied to formalize the business problems to be solved and frame the role of machine learning in the solution. 

Subject matter experts have a role to play not only at the beginning of the machine learning model lifecycle, but the end (post-production) as well. Oftentimes, to understand if a machine learning model is performing well or as expected, data scientists need subject matter experts to close the feedback loop — traditional metrics (accuracy, precision, recall, etc.) are not enough. 

For example, data scientists could build a simple churn prediction model that has very high accuracy in a production environment; however, marketing does not manage to prevent anyone from churning. From a business perspective, that means the model didn’t work, and that’s important information that needs to make its way back to those building the machine learning model so that they can find another possible solution — e.g., introducing uplift modeling that helps marketing better target potential churners who might be receptive to marketing messaging.

Role In and Needs From MLOps

Given their role in the machine learning model lifecycle, it’s critical when building MLOps processes for subject matter experts to have an easy way to understand deployed model performance in business terms. That is, not just model accuracy, precision, and recall, but its results or impact on the business process identified upfront. In addition, when there are unexpected shifts in performance, subject matter experts need a scalable way through MLOps processes for flagging model results that don’t align with business expectations.

On top of these explicit feedback mechanisms, more generally, MLOps should be built in a way that increases transparency for subject matter experts. That is, they should be able to use MLOps processes as a jumping-off point for exploring the data pipelines behind the models, understanding what data is being used, how it’s being transformed and enhanced, and what kind of machine learning techniques are being applied.

For subject matter experts who are also concerned with compliance of machine learning models with regulations (either internal or external), MLOps serves as an additional way to bring transparency and understanding to these processes. This includes being able to dig into individual decisions made by a model to understand why the model came to that decision — this should be complementary to statistical and aggregated feedback.

Ultimately, MLOps is most relevant for subject matter experts as a feedback mechanism and a platform for communication with data scientists about the models they are building. However, there are other MLOps needs as well — specifically around transparency, which ties up into Responsible AI — that are relevant for subject matter experts and make them an important part of the MLOps picture.

Role in the Machine Learning Model Lifecycle

Though most see data scientists’ role in the machine learning model lifecycle as strictly the model building portion, it is actually — or at least, it should be — much wider. From the very beginning, data scientists need to be involved with subject matter experts, understanding and helping to frame business problems in such a way that they can build a viable machine learning solution.

The reality is that this very first, critical step in the machine learning model lifecycle is often the hardest. It’s challenging particularly for data scientists first and foremost because it’s not where their training lies; that is, both formal and informal data science programs in universities and online heavily emphasize technical skill and not necessarily skills for communicating effectively with subject matter experts from the business side of the house who usually are not intimately familiar with machine learning techniques. Once again, business decision modeling techniques can help here.

It’s also a challenge because it can take time — for data scientists who want to dive in and get their hands dirty, spending weeks framing and outlining the problem before getting started on solving it can be torture. To top it all off, data scientists are often siloed (physically, culturally, or both) from the core of the business and from subject matter experts, so they simply don’t have the organizational infrastructure that facilitates easy collaboration between these profiles. Robust MLOps systems can help address some of these challenges.

After overcoming the first hurdle, depending on the organization, the project might get handed off to either data engineers or analysts to do some of the initial data gathering, preparation, and exploration. In some cases, data scientists themselves manage these parts of the machine learning model lifecycle. But in any case, data scientists step back in when it comes time to build, test, robustify, and then deploy the model. 

Following deployment, data scientists’ roles include constantly assessing model quality to ensure the way it’s working in production answers initial business questions or needs. The underlying question in many organizations is often whether data scientists monitor only the models they have had a hand in building, or if there is one person who handles all monitoring. In the former scenario, what happens when there is staff turnover? In the latter scenario, building good MLOps practices is critical, as the person monitoring also needs to quickly be able to jump in and take action should the model drift and start negatively affecting the business. If they weren’t the ones who built it, how can MLOps make this process seamless?

Role In and Needs From MLOps

All of the questions in the previous section lead directly here: data scientists’ needs when it comes to MLOps. Starting from the end of the process and working backwards, MLOps must provide data scientists with visibility into the performance of all deployed models as well as any models being A/B tested. But taking that one step further, it’s not just about monitoring — it’s also about action. Top-notch MLOps should also allow data scientists the flexibility to select winning models from tests and easily deploy them.

Transparency is an overarching theme in MLOps, so it’s no surprise that it’s also a key need for data scientists. The ability to drill down into data pipelines and to make quick assessments and adjustments (regardless of who originally built the model) is critical. Automated model packaging and delivery for quick and easy (yet safe) deployment to production is another important point for transparency, and it’s a crucial component of MLOps, especially to bring data scientists together to a place of trust with software engineers and DevOps teams.

In addition to transparency, perhaps another theme for mastering MLOps — especially when it comes to meeting the needs of data scientists — is pure efficiency. In an enterprise setting, agility and speed matter. It’s true for DevOps, and the story for MLOps is no different. Of course, data scientists can deploy, test, and monitor models in an ad-hoc fashion. But they will lose enormous amounts of time re-inventing the wheel with every single machine learning model, and that will never add up to scalable machine learning processes for the organization.

Role in the Machine Learning Model Lifecycle

In large organizations, managing the flow of data itself outside of the application of machine learning models is a full-time job. Depending on the technical stack and organizational structure of the enterprise, data engineers might, therefore, be more focused on databases themselves than on pipelines (especially if the company is leveraging data science and machine learning platforms that facilitate the visual building of pipelines by other data practitioners, like business analysts).

Ultimately, despite these slight variations in the role by an organization, the role of data engineers in the lifecycle is to optimize the retrieval and use of data to eventually power machine learning models. Generally, this means working closely with business teams, particularly subject matter experts, to identify the right data for the project at hand and possibly also prepare it for use. On the other end, they work closely with data scientists as well to resolve any data plumbing issues that might cause a model to behave undesirably in production.

Role In and Needs From MLOps

Given data engineers’ central role in the machine learning model lifecycle, underpinning both the building and monitoring portions, MLOps can bring significant efficiency gains. Data engineers will namely require not only visibility into the performance of all models deployed in production, but the ability to take it one step further and directly drill down into individual data pipelines to address any underlying issues. 

Ideally, for maximum efficiency for the data engineer profile (and for others as well - including data scientists), MLOps must not consist of simple monitoring, but be a bridge to underlying systems for investigating and tweaking machine learning models.

Role in the Machine Learning Model Lifecycle

Software engineers usually aren’t building machine learning models, but on the other hand, most organizations are not only producing machine learning models, but classic software and applications as well. It’s important that software engineers and data scientists work together to ensure the functioning of the larger system. After all, machine learning models aren’t just stand-alone experiments; the machine learning code, training, testing, and deployment has to fit into the CI/CD pipelines that the rest of the software is using.

For example, consider a retail company that has built a machine learning-based recommendation engine for their website. The machine learning model was built by the data scientist, but to integrate it into the larger functioning of the site, software engineers will necessarily need to be involved. Similarly, software engineers are responsible for the maintenance of the website as a whole, and a large part of that includes the functioning of the machine learning models in production.

Role In and Needs From MLOps

Given this interplay, software engineers need MLOps to provide them with model performance details as a larger picture of software application performance for the enterprise. MLOps is a way for data scientists and software engineers to speak the same language and have the same baseline understanding of how different models deployed across the silos of the enterprise are working together in production.

Other important features for software engineers include versioning, in order to be sure of what they are currently dealing with; automatic tests, in order to be as sure as possible that what they are currently dealing with is working; and the ability to work in parallel on the same application (thanks to a system that allows branches and merges like Git).

Role in the Machine Learning Model Lifecycle

DevOps teams have two primary roles in the machine learning model lifecycle: first, they are the people conducting and building operational systems as well as tests to ensure security, performance, and availability of machine learning models. Secondly, they are responsible for CI/CD pipeline management. Both of these roles require tight collaboration with data scientists as well as data engineers and data architects. Tight collaboration is, of course, easier said than done, but that is where MLOps can add value.

Role In and Needs From MLOps

For DevOps teams, MLOps needs to be integrated into the larger DevOps strategy of the enterprise, bridging the gap between traditional CI/CD and modern machine learning. That means systems that are fundamentally complementary and that allow DevOps teams to automate tests for machine learning just as they can automate tests for traditional software.

Role in the Machine Learning Model Lifecycle

When it comes to the machine learning model lifecycle, model risk managers play the critical role of analyzing not just model outcomes, but the initial goal and business questions machine learning models seek to resolve to minimize overall risk to the company. They should be involved along with subject matter experts at the very beginning of the lifecycle to ensure that an automated, machine learning-based approach in and of itself doesn’t present risk. 

And, of course, they have a role to play in monitoring — their more traditional place in the model lifecycle — to ensure that risk stays at bay once models are in production. In between conception and monitoring, MRM also is a factor post-model development and pre-production, ensuring initial compliance with internal and external requirements.

Role In and Needs From MLOps

MRM professionals and teams have a lot to gain from MLOps, as their work is often painstakingly manual, and as teams often use different tools, standardization can offer a huge leg-up in the speed at which auditing and risk management can occur. 

When it comes to specific MLOps needs, robust reporting tools on all models — whether they are currently in production or have been in production in the past — is the primary one. This reporting should include not just performance details, but the ability to see data lineage. Automated reporting adds an extra layer of efficiency for MRM and audit teams in MLOps systems and processes.

Role in the Machine Learning Model Lifecycle

Machine learning architects play a critical role in the machine learning model lifecycle, ensuring a scalable and flexible environment for model pipelines. In addition, data teams need their expertise to introduce new technologies (when appropriate) that improve machine learning model performance in production. It is for this reason that the data architect title isn’t enough; they need to have an intimate understanding of machine learning — not just enterprise architecture — to play this key role in the machine learning model lifecycle.

This role requires collaboration across the enterprise, from data scientists and engineers to DevOps and software engineers. Without a complete understanding of the needs of each of these people and teams, machine learning architects cannot properly allocate resources to ensure optimal performance of machine learning models in production.

Role In and Needs From MLOps

When it comes to MLOps, machine learning engineers’ role is about having a centralized view of resource allocation. As they have a strategic, tactical role, they need an overview of the situation to identify bottlenecks and use that information to find long-term improvements. Their role is one of pinpointing possible new technology or infrastructure for investment, not necessarily operational quick fixes that don’t address the heart of the scalability of the system.