Version Control and Audit

Version control has always been a crucial part of software engineering teams. A version control program manages the different versions or states of a project. Initially, they were designed to track the changes in the source code of a project. The purpose of a version control system is to track the changes in a file, a directory, or a system. In software engineering practices, they are applied to a project directory, the source file, and the other files within it. Version controls are also used to track changes users make. Your organization wants to see transparency across projects. Version control systems not only provide tracking and transparency, but they also allow you to roll back any unnecessary changes. Most version control systems ship with the features that are specific to a department or field. Hosted solutions, especially integrated modules, provide such offerings. BitBucket, for example, does not have issue boards and can be used by teams that only work on source code and CI/CD. As a team administrator, you are responsible adding Jira for ticketing and issue management, Trello for Kanban boards, and so on. You can purchase Jira only and manage the projects that do not require source code version control and automation of builds and deployments.

As a software engineer, you might have experienced all these types of version control systems. But if you are new to version control, then this section is important, as most DevSecOps stages depend on version control. Among the types of version controls, I will not talk about the local version control systems, as they will be covered automatically.

It is important to understand the difference between a centralized and a distributed version control system before diving into version control protocols and hosting platforms.

Centralized Version Control Systems

A centralized version control system is a system in which your repositories are saved to a central server. It is just the network topology that is used in the system. Think of it as a web server that stores the original version of the repository. Organizations use centralized version control systems to provide a central repository where engineers can collaborate. This provides better control over the repository. A software engineer must request access to a resource (a file, a configuration, etc.) before they can modify it. This lets other team members know who is modifying the resource.

A centralized repository provides access to users on the network and outside the network, all the same, as shown in Figure 4-1. Resource access and identity authorization policies are enforced by the centralized server. Your users can come from anywhere around the globe and perform their actions. The benefits of this mode of version controlling are as follows:

• Centralized control over the repository. The owning organization can always enforce their own policies and do not have to wait for all the parties to accept them.

• The “original” code copy is maintained on a single server, so it is easy to manage.

• User account policies can be easily controlled on the server. A user can be granted access to a repository and then revoked from the server. There is no need to keep track of the machines.

• An organization’s user accounts database (active directory) can be used for authentication. This enables IT Ops to control the user accounts management.

• It enables the owning party to provide infrastructure costs for build servers. A centralized datacenter can be used to build and release the software.

• It prevents the tampering of the test suites and other policies on code standards. If a code standard fails on the server, it means the user must verify and apply the necessary changes before the code is accepted.

Each time a resource has been modified, it is checked back into the repository. This notifies¹ other team members that a resource is available for access. You can see the workflow in Figure 4-2.

Multiple users exist on the repository database, but only a single user can make any changes. This requires every other user to wait for the first user to release their lock on the file. When a user checks in the changes, the next users can check the file out to make their own changes. This design prevents conflicts in the source code.

The most prominent and well-known service that uses this method of version control is Team Foundation Server (TFS) by Microsoft. TFS is powerful and available on-premises as a source version control system.

Note

Do not confuse TFS with the online hosted source control systems provided by Microsoft. The hosted platforms are known as Visual Studio Team Services (formerly known as Visual Studio Online). TFS is a protocol as well as the software that provides support for the TFS protocol.

TFS was the default protocol used by Microsoft for their internal projects as well as projects created and worked on in Visual Studio. More recently, Microsoft has started recommending (and providing support for) Git version control system in Visual Studio as well as their hosted products. Git is a distributed version control system, which we explore in the next section.

Distributed Version Control Systems

A distributed version control system , as its name suggests, is a version control system that is distributed across machines, networks, and regions. Unlike centralized VCS, distributed VCSs enable anyone to work on the repository without being provisioned by a central server.

Note

In this book, I use the distributed version control system called Git. You can explore other options such as TFS as well, as most of the concepts are similar. To a single user, the differences are not obvious. You will only feel the differences in these two version control systems when you work in a team environment.

With this design, the users maintain a copy of the repository on their own machine. There is no original repository, just remote repositories. This concept is illustrated in Figure 4-3.

Each user can have their own copy of the repository. There is no direct link linking to the “original” repository. Each user is free to make their own changes and start the project from there. This is a concept called forking. In forking, we create a clone of the repository and work on the clone separately. Note that the original and the clone are two different repositories, but since they have a similar history tree, we can merge them later if we approve the contributions. Every repository can be a source of truth, and every repository can be a temporary repository for a new feature addition or for testing.

The common traits of distributed VCSs are as follows:

• The code base and its copies are distributed across multiple machines and networks. There is no central repository to share or track the changes with.

• A distributed version control system also supports user accounts and authentication policies and rules.

• Policies and control are managed per repository.

• Since each repository can be a source of truth, every repository needs to manage its own set of policies.

Distributed VCSs specialize in the management of source code in a distributed environment in so many ways. Let’s look at an example of Git² version control. You can get a copy of Git version control for your platform from https://git-scm.com. Install it and then you can proceed with the next steps in this chapter. Git version control supports user accounts. If you are new to the Git environment, this is the first thing you need to perform in order to finish setting up Git version control.

$ git config --global user.name "John Doe"

$ git config --global user.email "[email protected]"

This step configures the Git version control to share the author details with each commit. Git also takes security measures and provides support for SSH. SSH is secure shell access that uses public/private certificates to authenticate and secure the traffic. This protects the data that you are transferring to and from the remote³ repositories.

In a distributed environment , you are not required to wait for other users to finish their work. You can work on your projects on your own machine, independently. This applies to everyone on the network as well as off the network. Git keeps a history of the changes that are made on each repository and on the clone of the repository. Each user (along with their user details) then commits their changes to the remote repositories to publish them.

Note that there is no concept of a central project, but a well-known remote repository is used to keep the track of the “master” state of the repository. Several well-known projects use Git version control deployed either on-premises or using one of the hosted code storage facilities offered by third-party vendors.

The Git protocol is used by GitHub and GitLab (it’s a part of their name and the service that they provide) to manage project resources, track changes, and version the state of the project.

A few features in Git include:

• Branches

• Hooks

• Staging areas

• Workflows

Some of these features (like workflows) are supported in other version control systems too. These features make it easier for the software engineering teams to manage the lifecycle of a project. Branches, for example, can support feature addition and feature try-outs. You can create a new branch, add the code that must be tested, and run a DevOps pipeline on it. A branch is an abstraction layer that can be used to create two copies of your code. Similarly, you can create hooks that process the incoming commits, and also manage the workflows for your contributions.

Note

You can check out another book published by Apress, Pro Git, which explores the Git version control in more detail. For more version control options and features, please consult that book, as I will not discuss how Git and its commands work internally.

GitOps

The Git protocol does one job, and does it so well, that they named the complete design and flow of the development to deployment lifecycle after it: GitOps. The term is a bit interesting in that it uses Git-based operations to perform a complete lifecycle. The lifecycle does not have to be of software development. Many website developers, blog authors, and magazine and book publishers use GitOps for their regular operations as well.

Git as software helps all these stages. GitHub, GitLab, and several other online hosting solutions provide support for tasks management and code building using Git. Let’s start by reviewing some of the features of Git, to understand how it can be used to manage the lifecycle of our projects.

If you are using an IDE, it will show the current branch that you are on. Visual Studio Code shows the branch information, as shown in Figure 4-4.

Now the IDE will show the master branch to be the active branch (see Figure 4-5).

Visual Studio Code also provides GUI tools to work with the repository. For example, you can click the branch name in the bottom-left corner to preview all the branches⁶ that are created in the repository, as shown in Figure 4-6.

Likewise, you can create the branches from here. This gives you good control over the environment that you are working with.

Coming back to the source code, the active branch will also revert to its original state. Remember that we did not make any changes to the master branch after creating the test branch. This helps us keep the master branch free from all changes that might be made. If we are not happy with our new features, we can simply change the branch and continue working from where we left off. Git tracks the changes to each file and resource. Thus, at every step we know what changes are made, and who made those changes to our repository. GitOps also enables owners to apply different policies to external contributors to prevent unauthorized access and actions to be taken on the resources. This helps owners prevent external contributors from triggering a production release on a change that is not yet verified, or code that is not yet reviewed by peers. GitOps is something that we can come up with once we have introduced security and performance automation to our DevOps; the end goal of this guide.

Other benefits of GitOps include the capability to verify the infrastructure design. I will talk more about that in the later section, Infrastructure as Code. GitOps is a pattern that uses Git software to automate project development and delivery, while also supporting transparency and audit of the contributions.

Hosted Code Storage

One of the best solutions made available to software engineers is the suite of code hosting solutions. GitHub, GitLab, Azure DevOps, you name it. The benefit of hosted code storage is that we get to use a cloud-hosted space to store our projects and their source code. This lets our teams connect to, work on, and modify the projects at their own pace. Code hosting is free. As a customer, we only pay for the resources that we consume. It also depends on the service provider, as some providers such as GitLab or GitHub charge per user in the subscription. But the jobs that are run on the code—including build time, test minutes, and release environments—are charged separately.

Several other methods and means of code hosting are also available. You can create your own code servers—TFS, Azure DevOps on-premises, Jenkins, and so on, are all means of creating your own repositories.

Hosted code storage options also come with user account management and policy management settings. You can also grant public access to your repositories or prevent access to your repositories altogether. Enterprise version managers also provide support for custom user accounts databases, such as Active Directory.

Infrastructure as Code (IaC)

For decades, software engineers have seen the power of plain-text information and have learned how to control different aspects of it. One of these aspects is version control. Now our software programs can be made rock solid against unwanted (or poor quality) changes. We have written tests that verify the quality of our solutions and our engineers can quickly verify the code quality. We have static code analysis tools that can tell us when our code is missing a policy enforcement so our engineer can make the changes. But what about our infrastructure/ What about the virtual machines and web servers that run the programs? Even if our programs are ~100% tested to work perfectly, if our infrastructure or the hosting environment is not working well, we cannot ensure a good quality service.

By the time that you answer these questions (25-100 seconds), your customers would be long gone to your competitors. Therefore, you need to think proactively and not reactively. Infrastructure as Code is a way of doing that. The name is somewhat confusing, but it simply means that you write down the structure and architecture of your infrastructure as source code. Just like a blueprint of source code for your software, your infrastructure is written down as a blueprint. There are special software tools that are used to verify the validity of your infrastructure drafts, its state, quality, and responsiveness to your performance load. Every organization defines their own set of principles and steps that are followed to create the resources on the cloud. But if a company is using hybrid cloud solutions, then it becomes difficult to manage the infrastructure. Despite supporting all features on the cloud, some vendors provide a different selection of compute power for their customers. For example, every virtual machine profile on Microsoft Azure might not be available on AWS.

A resource template can help with this. You can create a resource template for a cloud platform and then use it to deploy the resources on the cloud. This has a direct connection to how version-controlled software is deployed. If you have a software package, you know it will compile to the same binary and provide the same output. Regardless of the number of times we build the package, the result is always the same. Similarly, once your infrastructure is designed as a version-able resource, you can deploy it any number of times and the result will always be similar. This will be easier to understand with an example, so let’s look at an example of the Azure Resource Manager as an IaC toolkit for .NET Core applications.

Azure Resource Manager as an IaC Toolkit

Infrastructure as Code is an important aspect of Microsoft Azure as a cloud platform. For .NET Core applications, this means that we can test and verify the infrastructure before we deploy the applications. This will help you improve the SLAs of your solution by meeting the objectives. When you deploy a solution on Microsoft Azure, you create a new resource group to hold all the resources for that solution. A resource group is a grouping of resources that work together to provide a service. A resource group profile has the following information:

• Name of the resource group

• Location where the resources are to be deployed

• Subscription to use for billing purposes

That is it. You can include other information such as the tag information with a resource group, but that is not necessary. A resource group defines where your resources will be created. Resource groups also help manage the quota for your subscription and region.

Note

I assume that you have a Microsoft Azure subscription from this point onward. If you do not have an active Azure subscription, you can always create a free account with Microsoft Azure. You’ll get free credits to try out the services and follow along with this book.

We start by creating a new resource group on Microsoft Azure. To create one, search for “Resource Group” on the Azure marketplace. Figure 4-7 shows the Azure Portal web page with the list of products available.

You will create the resource group resource in your subscription, as shown in Figure 4-8. This resource group will contain our hosting platform as well as other configurations to deploy our web applications.

The form asks for the subscription to use to create this resource group. It also asks for a name (I am naming it Chapter4-RG to keep things simple; RG stands for resource group) and lastly it asks for the region for this resource group. Note that the region is used to deploy your resources in a specific datacenter. You need to determine which location to select. Here is a quick rule of thumb—if you are testing the environment, then this location does not matter much. If you are deploying a production release on the server, then you need to make sure that the region is as close to your customers as possible. This region is physical datacenter location where your web applications will exist. If the distance from this region to your customers is far, then your customers will face latency and other performance issues. These issues cannot be fixed through software improvements, rather they require a redeployment of the web servers to a closer region.

Note

If you cannot redeploy your web servers to provide better performance to your customers, then you can use CDNs (content delivery networks). A content delivery network takes a static resource—such as an image, a stylesheet, or an HTML web page—and replicates a copy of this page across the globe. This way, the data is replicated and made available to your customers closer to them. You only pay for the traffic and not the online resources, such as the hard disks and compute power. CDNs are very easy to configure and almost every cloud provider supports a CDN profile. You can create a new profile in seconds and have your static content replicated in minutes. One downside however is that CDNs only work with static data. If your web application generates dynamic content based on user profiles and sessions, CDNs cannot provide support to your business model. Instead, you need to create distributed web service deployments.

Our resource group is now ready to accept our resources. Right now, we are not interested in the deployment. We want to see how it controls and manages the resources in the group. We can get the resource template to preview the automation scripts that are used on the cloud by Microsoft Azure. Microsoft Azure provides a feature called Export Template, which enables you to generate an automation script for the resource group, with all the resources that are created in it and all configurations applied. The option can be found in the menu side bar for the resource group, as shown in Figure 4-9.

Once you click the option (from either selection), it will generate the automation script on runtime and provide you with the scripts that you can run to create the resources using PowerShell or other automation tools. See Figure 4-10.

The template in Figure 4-10 is empty because we have not created any resources. We can fill in the details for the template by creating new resources within the resource group. Let’s quickly create a new web application staging resource (an Azure App Service) and then preview the changes in the template. See Figure 4-11.

This web application will serve as a resource while we explore how Azure manages the infrastructure as a template. Now, if you go back to the template generation section, Figure 4-12 shows what Azure will provide.

Figure 4-12 shows the Azure Resource Manager template. You can download the template and add it to your version control. If you explore the entire script you will see that Microsoft has enforced privacy and security policies. The passwords fields, for example, are made private and no password is stored directly in the template. You will have to input these later, or you can connect with other services such as Azure Key Vault to fill in the details. Microsoft supports downloading the templates as well as storing the templates in your subscription for later use (see Figure 4-13). I will save the template to my library (my subscription) and recreate the resources.

The core concept of Infrastructure as Code was to redeploy the resources a number of times. Now that we have a template, let’s see how it works when we redeploy the resources. See Figure 4-14.

As shown in Figure 4-14, the exact details of our infrastructure were fed into the portal by the template. This shows how a template can be reused to deploy resources. My resource was a simple web server. If I had created a virtual machine, it would have used the IP address, network address, hostname, and other details as the template parameters and reused them during the resource deployment phase. You can use this script in PowerShell with Azure CLI to automate the deployments of your projects.

Note

It is important to note that resources can be deployed for testing and QA purposes as well. You can create a profile known to be used for production and spin up resources on the cloud with one template. This would help your testers quickly create test environments and run test suites on your application.

If you have configured Azure to use your own custom Docker image (I use my own afzaalahmadzeeshan/express-nodejs Docker image as a starter, as it lets me verify everything is wired up correctly), you will see that the Azure Resource Manager template also stored this information for future reference, as shown in Figure 4-15.

The first highlighted box shows the property name (…dockerimage_name…) and the second one shows the value for this property. There is a lot more to an Azure Resource Group than what we have covered, such as the virtual network setups and IP address mapping to your virtual machines, but they are off-topic for this book.

Automating Code Building and Deployment

A build pipeline will also run test suites against a software project. There are multiple ways in which you can add the security verification phase to a pipeline. For a .NET Core application, the easiest way is to use NuGet packages and add the build steps. In previous chapters, we added the NuGet packages to the projects and ran static code analysis for the code. We can do the same thing here, and a build pipeline will automatically run the tests against our code.

Regardless of the service⁸ you use, you can always create a build pipeline and create the artifacts that your hosting platforms will receive. I will use the example of Azure DevOps, because it integrates smoothly with Microsoft Azure. The guides for this are available on the Azure DevOps documentation website.⁹ I will try to explain how the automation aspects of Azure DevOps work, and how you can control the pipelines. I assume that you have a .NET Core project created and a Git repository initialized with a Azure DevOps repository.

An Azure DevOps project gives you complete control over the project. The dashboard gives you a clear visual report for all the tasks in your project. You can preview the number of on-going builds as well as the percentage of success and failure in the build pipelines. Figure 4-16 shows the dashboard for my project.

This page shows the details of your project. I have not added any Wiki pages or Markdown README.md files to explain the concept. You can add those files to make it easier for newcomers to catch up on the project and understand how things work in this repository. The screenshot also shows the Project Stats that clearly state the number of activities in the Repos, as well as those on the Pipelines. The dashboard can be configured to show more stats about the project. Similar dashboards are created on other platforms as well. Just for the same explanation, Figure 4-17 shows the sample from my GitLab project.

GitLab offers DevOps tools on the project, as a feature of Auto DevOps (see Figure 4-18).

As a developer, you will use the Boards to store the issue list and feature requests. This provides an automated way to create the issues and bug listings in the project and assign them to developers. You can also clear the list when the issues are resolved, or when they are to be closed. See Figure 4-19.

You can create new issues as they are needed. Stakeholders are provided with different controls that allow them to read the state of the projects. A team can collaborate on these issues to work toward a solution. Scrum runs can help keep the team focused on the issues with a deadline. Issue boards help the teams organize what is important and what features to work on.

The benefit of having DevOps tools is that they provide a complete set of solutions to help develop the project. You can drag-and-drop the features from one list to another. You can create repositories against this too. Most issues and bugs can be assigned to the developers that are directly responsible for working on them. Azure Active Directory can help automate this process of user identity management.

Creating Build Pipelines

Now the meat of the topic, the build pipelines. A build pipeline in Azure DevOps is a YAML file that contains the configuration for the tasks that are to be run. The default profile for ASP.NET Core (Azure DevOps will know the project type by the source code file and structuring) will contain the following script:

# ASP.NET Core

# Build and test ASP.NET Core projects targeting .NET Core.

# Add steps that run tests, create a NuGet package, deploy, and more:

# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:

- master

pool:

  vmImage: 'ubuntu-latest'

variables:

  buildConfiguration: 'Release'

steps:

- script: dotnet build --configuration $(buildConfiguration)

  displayName: 'dotnet build $(buildConfiguration)'

This script will build our ASP.NET Core application automatically each time we commit changes to Azure DevOps. If you commit changes to the repository, you will see that the build triggers and executes automatically; that’s continuous integration for you folks. You can create extra tasks to perform other tasks, such as integration and deployments. Each step will be executed in sequence and you can pass the artifacts from the previous stage to the next. Here, in this example, I am publishing the artifacts of the publish process for my ASP.NET Core application:

trigger:

- main

pool:

  vmImage: 'ubuntu-latest'

variables:

  buildConfiguration: 'Release'

steps:

    - task: UseDotNet@2

      displayName: "Building ASP.NET Core app"

      inputs:

        version: '3.1.x'

        packageType: sdk

    - script: dotnet build --configuration $(buildConfiguration)

      displayName: 'dotnet build $(buildConfiguration)'

    - task: DotNetCoreCLI@2

      displayName: "Testing"

      inputs:

        command: test

        projects: '**/*tests/*.csproj'

        arguments: '--configuration $(buildConfiguration)'

    - task: DotNetCoreCLI@2

      displayName: "Publishing App"

      inputs:

        command: 'publish'

        publishWebProjects: true

        arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'

        zipAfterPublish: true

    - task: PublishBuildArtifacts@1

      displayName: "Upload Artifacts"

      inputs:

        pathtoPublish: '$(Build.ArtifactStagingDirectory)'

        artifactName: 'app'

This is the complete pipeline for my continuous integration. It specifies the .NET Core version to use.

Note

As mentioned previously, you do not need to copy anything from this book. The reason is that Microsoft has been changing the specification for .NET Core as well as the CLI. Other products that they offer are also in the same category. Within a span of five years, users have seen Visual Studio Online change to Visual Studio Team Services to Azure DevOps to a new brand of Visual Studio Online. Therefore, I am recommending that you learn the best practices from this book and use the tools and scripts available at the time of reading (not this writing) to execute the tasks. I will try my best to keep the GitHub repository fresh with the latest scripts and project samples.

Rest assured, the samples are tested for validity with multiple user accounts and sample projects.

This job will execute on the hosted VMs and provide with the results. If you open the Job status (from Pipelines ➤ Select Your Pipeline ➤ Pipeline Execution Instance) you will see the dashboard shown in Figure 4-20.

The job dashboard shows the VM used as well as information about the job state. You can click these items to preview more information about them.

So far, we have only created the integration part. We have not yet deployed the artifacts on the cloud. That is where the Release phase comes in.

Note

A typical DevOps pipeline contains three phases:

Continuous integration

Continuous delivery

Continuous deployment

Continuous integration is the start of what is known as the DevOps pipeline. When you commit the project’s changes, your code is automatically built to verify the status of the code changes. Some developers include the testing phase of a software package in the CI phase. I believe the distinction can be used to simplify DevOps. For me, the testing phase is part of the continuous delivery phase. Your disagreements are welcome. Continuous delivery is the phase that ensures that you are “delivering” the package that has good quality UX and policies.

The last of these is continuous deployment. Once your package has been tested against bugs and vulnerabilities—note that a simple build cannot do that, so integration alone cannot support this requirement of DevOps—you will need to provide the software to customers. You can do that manually by uploading the artifacts to the web servers, which is fine in DevOps. If you automate this step, then you have continuous deployment. The goal of DevOps is to simplify the process for the teams, take what you want from the buffet of best practices, and leave what you cannot afford.

This job will also upload our artifacts for the next step. After this, we will have our Release stage automatically deploy it. This process can be done automatically from the pipeline or it can be done with a web hook¹⁰. In my Azure DevOps project, I handled this within the pipeline because my project was a .NET Core based deployment and not a Docker image.

As seen in Figure 4-21, the release stage will list all the steps that took place. Figure 4-22 shows the homepage view after this deployment.

You see that the DevOps pipeline in this case is also a part of the project repository and resources. You can version control that too. Git will allow you to check who made changes to the IaC files along with other resources in the version control. This is where Git blame comes in to the play. The Git blame feature allows you to check who committed the last change in the file, and on which line. Open a Git repository of your own (or if you downloaded this repository) and run Git blame against a file that is version controlled in your repository:

$ git blame filename.type

If I run this command against my azure-pipelines.yml file, Figure 4-23 shows what I will see.

$ git blame azure-pipelines.yml

You can easily review who made a change and whether that change was approved. Integrated solutions in the DevOps tools can help control these changes further by introducing issues and bug boards. Good team collaboration can improve the velocity of software deployment because your teams will quickly have their changes accepted and deployed.

In a nutshell, you see how an automated pipeline can help your team achieve more, in less time. It also shows how you can get insight into the contributions of external users and control the access of internal members. Every DevOps tool provides access to testing suite execution, application quality analysis, bugs and issue boards, and platform- or license-specific features.

Utilizing a Bug Database

A bug database is a database that contains the known bugs, code smells, and vulnerabilities of a framework, runtime, language, or deployment platform. One can be created manually by the organization based on their experience with the platform, or it can be purchased as a subscription from a security organization.

Do not confuse a bug database with an issue tracker system. By a bug database, I mean the database that contains references to known and incoming vulnerability reports. This also includes any software that is written using the same framework and runtime. In the previous chapter, we explored two such options that provide an expert¹¹ opinion about a code base to solve the challenges in the project. If you add the package analyzers that we used previously (Microsoft.CodeAnalysis.FxCopAnalyzers), then you will see that our build pipelines show the warning messages in the build output (see Figure 4-24).

You can configure the build systems to terminate the process if there are any warning messages. This is entirely dependent on the company policies for the code. We explore the usefulness as well as false positives of the code analysis plugins. Based on that, it is good for your teams to discuss how to apply these changes to your tools.

Risk and Bugs Analysis

DevOps requires that your teams enforce policies to prevent risk and bugs. A risk can come in different ways, but most notably it comes with a new feature addition. Most common bugs and code problems arise when a new feature has been rolled out. A new feature can bring a new set of problems, including code-related issues, but also problems that relate to the infrastructure and the online hosting resources. Imagine that you develop a new feature that your customers have been requesting. You know that this feature is important, and that many active users will try to access it. If you do not run an infrastructure load or stress test on the feature, it could lead to failure since the resource might not be ready to accept the traffic load. Infrastructure as Code comes in handy in this regard, because you can run the tests on the infrastructure before they go live.

There are other ways to solve this problem, some include that you roll out the software feature in percentages. Some teams also try to develop features and them release them to individual regions, like the United States first, then to Europe, then Asia, and so on. But how do you manage this in your software? One common practice is to write the software in a branch (branches that we discussed earlier in this chapter) and then merge the branches with a production branch to deploy the changes on the server. Do you see the problem here?

The problem is the lack of control over the environment, as well as the software state. You cannot predict the time it will take for the build environment to finish building your project. Similarly, it does not guarantee that your build will succeed at the 11th hour. What’s real is that it might fail when the time comes. This sort of approach to deploying an application also leads to a poor UX since your users might not get the best software they were expecting. Software you have tested for code quality does not ensure a stable execution on the Internet. Your testers and engineers only tested the code based on their knowledge and experience.

One way to perform these code checks is using feature flags. A feature flag is a conditional block in your application, written to abstract the underlying code.

Feature Flags

A feature flag is provided by the DevOps tools, pipelines, and cloud providers, and it can be developed manually by a team as well. The story of a feature flag is as simple as a basic if...else block in the code that controls whether to grant access to a feature or not. Feature flags can be hardcoded features or soft conditional blocks that prevent access to a specific feature for a time.

Imagine a scenario where you must broadcast an update to your customers for a New Year or Black Friday deal. You cannot expect the customers to wait until the update has been pushed out to the online Android or iOS app stores to download and then use the discounts. Your users expect the benefits to be unlocked once the clock ticks 12. Another major problem is that the clock does not tick 12 across the globe at the same time. There are different time zones that the software needs to consider, and many other similar problems. With feature flags, we can easily control the values for the discounts from an external server. This server acts as the hosting platform for the configuration values. After all, a feature flag is just a configuration value that is hosted elsewhere.

I have used several platforms that support feature flags. I personally like Alibaba Cloud Application Configuration Manager¹² for its simplicity, features, and pricing model.

Note that if you do not wish to use Alibaba Cloud Application Configuration Manager, you can use an online resource storage service.¹⁴ Google Drive, OneDrive, and other online storage options are also valid. You can create a simple JSON file that holds the data that your application connects to. This data can contain the configuration information for that instance in time.

Summary

Automation is the core of DevOps and it helps teams reach their goals of publishing features quickly. This chapter was about the ways in which teams can achieve scalable automation. There are no best solutions, only useful solutions in the DevOps market. In this chapter, I used Azure DevOps to demonstrate a few steps in DevOps, but you can use any other DevOps solution, such as GitLab or Jenkins.

The introduction of version control systems was necessary to understand how a DevOps pipeline gets triggered on changes. The most important aspect of DevOps is IaC, which is a way to control the infrastructure deployments. We looked at Azure Resource Manager, but there are several other options, like Alibaba Cloud ROS.

We also discussed the automation pipelines and the stages of DevOps and CI/CD. We created a new pipeline in Azure DevOps. Then we connected our repository to DevOps and triggered a new build pipeline. We also created a new Release pipeline to deploy the solution to Microsoft Azure. DevOps build pipelines can also be used to analyze code for bad code standards or for code that is ignoring the code policies of an organization.

Feature flags were also introduced. They can help development teams deploy features at their own pace and help operations teams control access to the features themselves. One thing you might notice is that I did not talk much about security in this chapter. Security is not the cherry on the top, instead it should be the DNA of the software, build servers, and production environments. That is what I focus on in the next chapter. I talk about security for build systems, and how you can verify that the deployment environments are safe from external access. I use several concepts introduced earlier and some new interesting topics as well.