The Continuous Integration Pipeline

Another great practice for maintaining stability is to promote code through a series of quality assurance environments. Together, these environments make a continuous integration pipeline. A pipeline can consist of any number of environments, but most include the following three:

• The development environment, which enables internal developers and product managers to access the latest version of the code
• The stage environment, also sometimes called the test environment, which is configured similarly to the production environment and runs new code and models that will eventually become production if they pass all quality assurance checks
• The production environment, which is the live environment that end users are actively using

The first environment in this continuous integration pipeline is the development environment. It serves as the latest and greatest version of the system. The development environment can be redeployed after every successful code/model integration, or once a day (typically in the evening). In the latter case, the development environment is said be to running the nightly build. Developers and product managers use the development environment to check that closed tickets have been implemented correctly and, separately, that no feature misunderstandings have occurred. This distinction is so important in software engineering that there are defined terms for each. Verification is defined as ensuring that the code performs a function without causing errors, and validation is defined as ensuring that each function is useful and correctly fulfills the software's intended requirements. Although automated tests can perform verification, it is rare that they are also set up to perform validation since the same developer typically writes the tests for their code. In most cases, other developers and product managers perform validation manually.

Since the development environment is primarily used to ensure feature correctness, it tends to be a smaller version of the fully deployed production environment. This enables the development environment to be updated and redeployed more quickly, but it also means that representative performance tests and other scalability tests cannot be performed until the next stage of the pipeline.

When a sufficient number of features and bug fixes have been pushed to development, it is time to promote the development environment to the stage environment (see Figure 6.1). Promoting code to the next level of the pipeline involves deploying the same version of code and models as in the previous environment. Since the stage environment is the same size as the full production environment, it is a good place to test performance and scalability. New code additions, especially if they are the first implementation of a feature, may be fine when run in isolation but cause issues when run hundreds of times concurrently. For instance, new code might call an external system that is not set up to handle scale itself. In this case, reducing calls to the external system by caching results may be necessary to ensure that your users do not experience performance issues from excess load.

Because the stage environment is always the last stage in the continuous integration pipeline before production, quality assurance becomes critical. The stage environment is the last line of defense before bugs are introduced to the production environment, which will affect real users. The stage environment's acceptance process should include both code and models.

Once the stage environment has been thoroughly tested, it is time to promote its code and models to production. In this way, features that were implemented and originally deployed into the development environment have now made their way to providing value to real users. The improved models that are now deployed will provide more accurate results or have increased capabilities. For instance, an object detection model may have been improved through this process to recognize some new objects that have been frequently requested by its users.

For machine learning models in particular, this continuous integration pipeline can promote new models without their even requiring additional programming (see Figure 6.2). Simply including additional data can be sufficient for a model to be retrained and enhanced. This additional data can come from an external source or, more interestingly, from the production users of the system. In the latter case, this feedback loop is perpetually self-improving, an incredibly powerful concept.

True Continuous Integration

Up until now, we have primarily discussed feedback loops in the context of Agile and as a way to ensure that the system is meeting stakeholder requirements. Now, we see that feedback loops can also be applicable, and very useful, in building self-improving machine learning systems. This simple yet revolutionary concept of using the results from a previous activity to partially determine the next activity is a fundamental principle that drives most human learning and behavior. Once you are aware of it, you will start to see this pattern in most intelligent processes. Even a human adjusting how hard to press on a car's accelerator while looking at the speedometer comes from a type of feedback loop, with the desired speed being the goal. We will dive deeper into this concept in the next chapter.

In a world where organizations are global, systems have to stay live 24 hours a day. Any downtime can hurt users' immediate productivity and start to erode their trust in the brand. Therefore, upgrades to production must forgo incurring any downtime whenever possible. In large systems, multiple instances of code may be deployed for intentional redundancy. It is common to replace each redundant instance with the new deployment, one at a time, upon updating. Similarly, models might be deployed multiple times to handle scalability, and they too can be replaced one at a time to ensure that there is no modal downtime.

An alternative upgrade option involves swapping the stage environment completely with the current product environment with a simple networking change. In this scenario, you would have two production environments; call them A and B. Let's say users are currently using production environment A. You would then use production environment B as a staging environment. Once environment B has passed quality assurance, the network is changed so that my-system.my-company.com points to environment B instead of environment A. There is some complexity with this approach (e.g., handling currently-in-flight transactions), but this process will ensure that the system experiences no downtime.

The benefit of continuous integration is that code pushes are immediately tested and validated so that stage could, if stakeholders trust the process enough, be promoted to production as soon as all the tests are complete. Gone are the days where bug fixes have to wait weeks for the next update to be released. This approach works well for a centralized AI system such as one implemented using a software as a service (SaaS)-like architecture. If your system is physically deployed in different locations, this aspect of continuous integration is less practical. There are a number of continuous integration tools, such as Travis CI and Jenkins, that provide detailed usage instructions on their websites.

Continuous integration principles are applicable not only to software, but also to any machine learning models that are being built. For instance, updating an object detection model to recognize one additional object may break its ability to identify a previously recognized object. For scenarios such as this, it is important to have a test suite of images to which precision and recall metrics (discussed in more detail in the next chapter) can be calculated for each object type each time the model is updated or re-created. This way, problems with new models can be identified before they make it through the pipeline and are integrated with the production system.

Ideally, this continuous integration system would have been established during the prototype phase to ensure quality, but as is typical with most prototypes, testing functionality is not the priority. The logic behind this is that if the prototype does not pay off, no further investment has been made in building test frameworks or other supporting infrastructure. Alternatively, if the prototype is successful and quality assurance was postponed until after the prototype was successful, it can be a large undertaking to set up the infrastructure at this point. Unit tests and other automated testing would need to be created for the existing code. Additionally, any incurred technical debt (coding shortcuts, which can also introduce bugs into the system) will need to be resolved at this point in time. If you decide to start your production code from a clean slate or use the aforementioned hybrid approach, most of these issues can be mitigated while either writing fresh code or reviewing it as you copy over pieces of the prototype.

Test Types

Automated tests are typically broken into three types:

• Unit tests, which validate self-contained pieces of code.
• Integration tests, which validate code that controls the interactions between systems—for instance, code that talks to a database or a third-party API.
• Acceptance tests, which validate the system as a whole, testing functionality in ways that an end user would use the system. These can be direct methods like user interface inputs or programmatic API calls if your solution is an API.

Each of these test types helps validate the code base at different levels and are ordered by increasing scope. Ideally, most bugs will be caught with targeted unit tests where bugs are fairly easy to isolate. As scope increases and multiple systems become involved, problems become more complex and harder to identify. Integration tests are responsible for all bugs that make it past unit testing and will catch them if an API you are using had an interface change (e.g., renamed a parameter). Acceptance tests then become your last line of defense against remaining bugs as they peruse and test large swaths of user features. A good sample acceptance test might be logging in as a user and updating your account information. These sets of tests flow together as depicted in Figure 6.3.

AI Model Testing Example

Given that your AI system will contain AI models that you train, it is important that these models are tested as well. With appropriate tests, you can be certain that new models are as good, if not better, before they replace an older one in production. Let's dive into an example of creating unit tests that evaluate our AI model's ability to determine the correct user intent for a particular user input. Let's use a chatbot in the banking domain for this example. Assume our chatbot contains an AI model that is able to recognize and handle the following three user intents and a “catch-all” intent for if the user's input does not match an intent with enough confidence:

• HOURS_OPEN: Provide the user with the hours that the bank is open.
• OPEN_BANK_ACCOUNT: Provide the user with instructions for opening a bank account.
• CLOSE_BANK_ACCOUNT: Provide the user with instructions for closing a bank account.
• CATCH_ALL: A generic response pointing the user to the existing FAQs webpage or that provides the ability to fall back to a human operator.

Since we want to ensure that our system correctly classifies inputs into these intents, we should have unit tests for the AI model that prove it is correctly classifying intents. The tests could be as simple as the following code:

# Test HOURS_OPEN
assert classifyUserIntent("What time do you open?") ==
HOURS_OPEN
assert classifyUserIntent("Are you open tomorrow at 9am?") ==
HOURS_OPEN
 
# Test OPEN_BANK_ACCOUNT
assert classifyUserIntent("I want to open a new
account with you guys?") == OPEN_BANK_ACCOUNT
assert classifyUserIntent("I am interested in a
checking account?") == OPEN_BANK_ACCOUNT
 
# Test CLOSE_BANK_ACCOUNT
assert classifyUserIntent("I want to close my
account?") == CLOSE_BANK_ACCOUNT
assert classifyUserIntent("I am moving to a new bank,
how do I do that?") == CLOSE_BANK_ACCOUNT
 
# Test CATCH_ALL
assert classifyUserIntent("How do I add my spouse to
my account?") == CATCH_ALL
assert classifyUserIntent("Does your bank provide IRA
accounts?") == CATCH_ALL

In this example there are only two unit tests per intent, which may not seem like enough but they will provide a sanity check that your system is still generally working as expected. Remember that these unit tests will be run after any model is created and before a new model is promoted into production. This should increase the confidence that problems will not sneak into production and negatively affect real users.

When working with AI models specifically, there are methods of splitting your model's ground truth into a training set (data used to create the model) and a test set (data used to evaluate the accuracy of the model). This process should be used to generate and select a model, which will then be further checked by the unit tests as defined earlier.

As you increase the number of recognized intents and therefore your chatbot's capabilities, you will also want to make sure that you add their respective unit tests. Classifiers that have to distinguish between a large number of classes (e.g., intents) have to learn more nuanced complexity, causing the overall model to suffer. This shows just how important tests are to feeling confident that your system has not regressed in any way.

What if You Find a Bug?

What happens, however, when a defect is found in the production code? Well, this means that there was no appropriate test case to catch the bug. Therefore, the first step is to write a new test that catches this issue. In doing so, other bugs in the code might also become apparent, and tests for those defects should be written as well. With the test written, we need to validate that the test does indeed catch the offending issue(s). Once we are sure it does, we can then fix the code and validate that the issue is resolved by running the new test cases again. Not only has this particular issue been fixed, but if that same error is ever reintroduced into the code (e.g., code rollbacks), the new test coverage will immediately identify the issue and bring it to your attention before it is rolled out to production with real end users.

Infrastructure Testing

The automated tests we have been discussing are important for ensuring code and model quality. However, a system is only as stable as its deployment configuration and the hardware it is deployed on. If you deploy your solution on a single server and that server goes down, your users will be affected. If you deploy your solution on three servers in a redundant fashion and one server goes down, your users might not even notice a difference. Therefore, it is important to ensure a robust, multiserver deployment.

But how do you know that your deployment is robust enough to handle issues and recover gracefully? This is the problem Netflix faced in 2011. The solution they came up with is the result of some out-of-the-box thinking. They decided to intentionally cause random errors in their deployment and ensure that users were not affected. Since they were causing the errors, they could immediately stop the issue if the error impacted users. Using this technique, they could identify errors in the deployment configuration, which could then be updated to fill any gaps. That way, the next time any of the errors occurred for real-world reasons, their deployment would stay resilient. Netflix named their successful project Chaos Monkey, after an imagined monkey running around their data center causing havoc. Netflix has since open-sourced the Chaos Monkey code and made it available on GitHub.¹

Because the Chaos Monkey approach operates at the deployment level, there is no need to build a separate Chaos Monkey for your machine learning models. Since the models will be deployed on infrastructure for execution, Chaos Monkeys can easily be pointed to test that part of your deployment as well. In this way, code and models alike will be tested for robustness via the Chaos Monkey approach.

Scalability and the Cloud

Some might argue that modern cloud platforms make load testing irrelevant. They assume that the cloud can automatically scale to any load, as needed. Unfortunately, this is not always the case. At the very least, you will need to perform load tests to validate that the cloud scales as expected. You might also find that the cloud scaling is not instantaneous and there is up to three minutes of inaccessibility as it scales to the correct capacity. This happens especially often with workloads that spike or that have a sudden increase and decrease in usage (see Figure 6.5).

The next validation point is to ensure that your system is implemented in such a way that it takes advantage of the cloud scalability. It may be true that the cloud is scalable, but if user information is stuck in a single database instance, then a scaled application layer, though no longer a bottleneck, will still provide slow responses. In this case, the addition of database mirrors or a server-side cache to maintain session data is required. This is also something that may not become apparent until a load test is performed.

Even for machine learning models, there are dedicated clouds that can spawn multiple model instances, as needed, to limit latency. These clouds are relatively new, such as Google's Cloud Machine Learning Engine. For scalability, it is important to determine which parts of your system will likely be a bottleneck using load testing and ensure that those components are designed and deployed accordingly. We will discuss cloud deployments further in the next section.

Pitfall 1: End Users Resist Adopting the Technology

This pitfall is common with all new technology but especially with AI solutions. Automation technology can be unsettling for end users, since it replaces some of the work they are used to doing themselves. Opinions range from “This technology will just be a hindrance to how I work,” to “It's only a matter of time until my skills are obsolete, the robots take over, and I'm out of a job.” Change is hard, no matter what form it takes.

Another issue with AI solutions in particular is that most AI systems require input from a subject matter expert (SME) to create the ground truth used to train the underlying machine learning models. These SMEs are also typically the ones directly affected by the integration of a new AI solution. For many reasons, it is important that the AI solution be an augment to the SMEs' knowledge and capabilities, instead of a direct replacement of their role. Remember that a machine learning model is only as good as the ground truth used to train it.

To avoid this pitfall, early end user engagement is critical. End users need to be part of the planning process to ensure that they fully understand the solution and feel they have contributed to the end product. This might even mean inviting a few influential end users during the Ideation/Use Case phases to build excitement and a voice for your user base. While early input is in no way a guarantee (end users might think they want one thing at first, only to realize they need something else once they start using the solution), it will help mitigate the fear associated with adopting new technology.

Pitfall 2: Micromanaging the Development Team

Under Agile, the development team is given full responsibility for the successful technical implementation of the project. The team works on the combined values of transparency and mutual trust. In such an environment, it would not be prudent to control every aspect of the development team. Nor would it be a good practice to set the targets for each sprint for the developers. This will lead to a lack of motivation, weakening Agile. The development team should be able to focus on the project by themselves with minimal intervention from the product owner. Some teams pick the Scrum manager from among the dev team as well. This is to ensure that the benefits of Agile are preserved.

Pitfall 3: Not Having the Correct Skills Available

Since building a machine learning system requires a number of specialized skills, it is critical to have these skills available and ready to go before your project starts. Whether this means hiring full-time employees or establishing relationships with contracting firms, it is worth the up-front effort to avoid delays. Skills we have mentioned thus far that will be required include AI, data science, software engineering, and DevOps. The hiring issue is twofold in that you must find the individuals with the proper skillsets for your project and, of course, have the appropriate budget in place to fund them. With these items addressed, there should not be any skill roadblocks in the way of getting your system deployed.