Although we have learned that DevOps is more than just tools, there are a number of tools that are commonly used in the enterprise, such as the following:
- A source code repository: Computer source code has become a very valuable asset. It is usually stored in a repository with advanced version management features. The repository manages the many versions of code that are checked in, so developers can collaborate on each other's projects. This concept is not new and has been around for 30 years, but is a big part of continuous integration because it is where the source code is kept. Popular source code repository tools include the following:
- Git on the client
- GitLab
- GitHub
- Subversion
- TFS
- CVS
All of these repository tools are explained in detail in Chapter 4, Configuring GitLab from the Terminal, Chapter 5, Importing Your Project from GitHub to GitLab, Chapter 6, Migrating From CVS, and Chapter 7, Switching from SVN.
- Build server: Building software used to be done on the individual developer's workstation, but for the CI pipeline, a dedicated build server is used to compile source code from the source code repository into executable artefacts. Modern build servers do not just build, but also provide advance testing functions. Popular tools include the following:
- GitLab Runners: The build tool for GitLab.
- Jenkins: A fork of the Hudson project and a CI platform. This platform is primarily intended for the repeated execution and monitoring of build tasks, as well as the automated building and testing of applications. The many freely available plugins make it very easy to further expand the functionality of Jenkins. This software is only available as a distributed service to use on the cloud and is tightly integrated with GitHub as a source code repository.
- Configuration management: For CI/CD, you need to control the environment where it takes place. For this, there are configuration management tools that describe and automate large parts of your infrastructure. Popular tools include the following:
- Puppet: Management software which can control large numbers of servers. This concerns both the management of configuration files (the settings of servers) and the management of the installed software (packages). It uses a declarative language and has a steep learning curve.
- Chef: Also configuration management software, Chef supports slightly fewer platforms than Puppet and is not a declarative language. Chef uses pure Ruby code that indicates what you want to do on your servers. You have more freedom to create your own program data structures and functions. It is used by GitLab to manage the omnibus package.
- Virtual infrastructure: Infrastructure on which software runs has always been virtual, and an operating system is already several layers of abstraction. In the cloud, virtual infrastructure is an extra layer of abstraction that represents entire machines (such as networks, nodes, and storage). There is also an orchestration layer that manages the infrastructure. This provides easy up- and down-scaling and can use all resources efficiently. The first real, large-scale virtual infrastructure that became available as a service was Amazon Web Services. The other major tech companies soon followed with Google Cloud and Microsoft Azure. These infrastructures can be managed with their own orchestration tools, but also have APIs that can be used by configuration management tools, including the following:
- Ansible
- Puppet
- Chef
- Google Cloud
Governments often have special requirements regarding their data. This is called data sovereignty and this is why specialist clouds have arisen for governments. According to Garter, these clouds could be the next legacy system, after government infrastructure was moved like-for-like to the cloud without being decomposed into elastic, efficient, and cost-effective cloud components.
Not everyone is able to run their software and data on public clouds, even if they have special agreements. If you run a private or hybrid cloud, for example, you are essentially using abstractions that exist on the internet in your own data center. Even without the elasticity of Amazon or Azure, it can be very beneficial to apply cloud techniques yourself. The accompanying automation tools make integration with existing systems easier, and a lot less people are needed for managing the system. There are also private clouds; for example, VMware has vCloud. It is quite easy to extend your existing VMware infrastructure to create cloud-like environments.
- Test automation: Testing is all about ensuring confidence in your product. When the product reaches deployment time in your pipeline, it should be tested for certain defects automatically before it has reached that point. There are several tools available to perform all kinds of testing and integrate nicely with a lot of other pipeline products; they include the following:
- Selenium: Selenium is an application that allows you to automate browsers. What you do with this depends on your goal. You can automate repetitive administrative tasks, but Selenium is also used for browser testing.
- Cucumber: This is a test tool for Behavior-Driven Development (BDD). The primary goal of BDD is to let people communicate, and close the gap between technical and business people. You can write tests in a human-readable format.
- Apache JMeter: JMeter is an open source tool that performs load, performance, and stress tests. It is a simple but effective application where different types of scripts show exactly what the result of the test is. These scripts are used for HTTP websites and provide a simulated test environment. In addition to applications, JMeter is also suitable for checking services on the web and various databases.
- You can also utilize GitLab Runners and write your own tests.
- Pipeline orchestration: The pipeline refers to an automated number of steps to get your code from inception to production after it has been checked into version control. It's based on the idea of a manufacturing assembly line.
To manage things along the way in a CI/CD process, pipeline orchestration tools were introduced. Some of these tools include:
-
- Kubernetes: Kubernetes is, essentially, a platform for the roll-out and management of containers on a large scale. Kubernetes, Greek for helmsman or pilot, is the second name for the project, which originally saw the light of day in the big halls of Google as Project Seven of Nine. Project Seven of Nine was an external version of Borg, the task scheduler that drives the services of Google, and the operation of which was a Google secret for a long time.
- Built as an extension to the Docker API, orchestration using Swarm also became popular a couple of years ago. It can easily convert a loose group of Docker containers in a managed virtual Docker engine. This makes it very easy to start running container workloads at scale from scratch.
- Mesos/Marathon Apache: Mesos is a distributed kernel and is the backbone of DC/OS. It abstracts CPU, memory, storage, and other computer resolutions. It has APIs for resource management, planning in data centers, and cloud environments. It can scale up to 10,000 nodes. It can therefore be extremely suitable for large production clusters. It supports container orchestration with Marathon.
All of the aforementioned tools can be integrated with GitLab, and you can use GitLab for all parts of a pipeline. You can use runners for testing, building, or deploying your product, and you can utilize Kubernetes to orchestrate your workloads.
It's up to you which part of the pipeline is used in GitLab, but it can support you in all stages of the DevOps life cycle, illustrated as follows:
We have now explained the basic setup of a CD pipeline in DevOps. GitLab offers close to 100% of all the stages, but can integrate with existing components as well.