Best Practices

Thank you for reading this book. I hope you are ready to introduce the continuous delivery approach to your IT projects. By way of a final section for this book, I propose a list of the top 10 Continuous Delivery practices. Enjoy!

Practice 1 – Own the process within the team!

Own the entire process within the team, from receiving requirements to monitoring production. As was once remarked, "A program running on the developer's machine makes no money." This is why it's important to have a small DevOps team that takes complete ownership of a product. Actually, that is the true meaning of DevOps: Development and Operations, from the beginning to the end:

• Own every stage of the Continuous Delivery pipeline: how to build the software, what the requirements are in acceptance tests, and how to release the product.
• Avoid having a pipeline expert! Every member of the team should be involved in creating the pipeline.
• Find a good way to share the current pipeline state (and the production monitoring) among team members. The most effective solution is big screens in the team space.
• If a developer, QA, and IT operations engineer are separate experts, then make sure they work together in one agile team. Separate teams based on expertise result in no one taking responsibility for the product.
• Remember that autonomy given to the team results in high job satisfaction and exceptional engagement. This leads to great products!

Practice 2 – Automate everything!

Automate everything, from business requirements (in the form of acceptance tests) to the deployment process. Manual descriptions, wiki pages with instruction steps, they all quickly become out of date and lead to tribal knowledge that makes the process slow, tedious, and unreliable. This, in turn, leads to a need for release rehearsals, and makes every deployment unique. Don't go down this path! As a rule, if you do anything for the second time, automate it:

• Eliminate all manual steps; they are a source of errors! The whole process must be repeatable and reliable.
• Don't ever make any changes directly in production! Use configuration management tools instead.
• Use precisely the same mechanism to deploy to every environment.
• Always include an automated smoke test to check whether the release was completed successfully.
• Use database schema migrations to automate database changes.
• Use automatic maintenance scripts for backup and cleanup. Don't forget to remove unused Docker images!

Practice 3 – Version everything!

Version everything: software source code, build scripts, automated tests, configuration management files, Continuous Delivery pipelines, monitoring scripts, binaries, and documentation; simply everything. Make your work task-based, where each task results in a commit to the repository, no matter whether it's related to requirement gathering, architecture design, configuration, or software development. A task starts on the agile board and ends up in the repository. This way, you maintain a single point of truth with the history and reasons for the changes:

• Be strict about version control. Version everything means everything!
• Keep the source code and configuration in the code repository, the binaries in the artifact repository, and the tasks in the agile issue tracking tool.
• Develop the Continuous Delivery pipeline as a code.
• Use database migrations and store them in a repository.
• Store documentation in the form of markdown files that can be version-controlled.

Practice 4 – Use business language for acceptance tests

Use business-facing language for acceptance tests to improve mutual communication and a common understanding of the requirements. Work closely with the product owner to create what Eric Evan called the ubiquitous language, a common dialect between the business and technology. Misunderstandings are the root cause of most project failures:

• Create a common language and use it inside the project.
• Use an acceptance testing framework, such as Cucumber or FitNesse, to help the business team understand and get them involved.
• Express business values inside acceptance tests, and don't forget about them during development. It's easy to spend too much time on unrelated topics!
• Improve and maintain acceptance tests so that they always act as regression tests.
• Make sure everyone is aware that a passing acceptance test suite means a green light from the business to release the software.

Practice 5 – Be ready to roll back

Be ready to roll back; sooner or later, you will need to do it. Remember, you don't need more QAs; you need a faster rollback. If anything goes wrong in production, the first thing you want to do is to play safe and come back to the last working version:

• Develop a rollback strategy and the process of what to do when the system is down.
• Split non-backward-compatible database changes into compatible ones.
• Always use the same process of delivery for rollbacks and standard releases.
• Consider introducing blue-green deployments or canary releases.
• Don't be afraid of bugs; the user won't leave you if you react quickly!

Practice 6 – Don't underestimate the impact of people

Don't underestimate the impact of people. They are usually way more important than tools. You won't automate delivery if the IT operations team won't help you. After all, they know the current process. The same applies to QAs, businesses, and everyone involved. Make them important and involved:

• Let QAs and IT operations be a part of the DevOps team. You need their knowledge and skills!
• Provide training to members who are currently doing manual activities so that they can move to automation.
• Favor informal communication and a flat structure of organization over hierarchy and orders. You won't do anything without goodwill!

Practice 7 – Incorporate traceability

Incorporate traceability for the delivery process and working system. There is nothing worse than a failure without any log messages. Monitor the number of requests, the latency, the load of production servers, the state of the Continuous Delivery pipeline, and everything you can think of that could help you to analyze your current software. Be proactive! At some point, you will need to check the stats and logs:

• Log pipeline activities! In the case of failure, notify the team with an informative message.
• Implement proper logging and monitoring of the running system.
• Use specialized tools for system monitoring, such as Kibana, Grafana, or Logmatic.io.
• Integrate production monitoring into your development ecosystem. Consider having big screens with the current production stats in the common team space.

Practice 8 – Integrate often

Integrate often; actually, all the time! As someone once said, "Continuous is more often than you think." There is nothing more frustrating than resolving merge conflicts. Continuous integration is less about the tool and more about the team practice. Integrate the code into one code base at least a few times a day. Forget about long-lasting feature branches and a huge number of local changes. Trunk-based development and feature toggles for the win!

• Use trunk-based development and feature toggles instead of feature branches.
• If you need a branch or local changes, make sure that you integrate with the rest of the team at least once a day.
• Always keep the trunk healthy; make sure you run tests before you merge into the baseline.
• Run the pipeline after every commit to the repository for a faster feedback cycle.

Practice 9 – Only build binaries once

Build binaries only once, and run the same one on each of the environments, irrespective of whether they are in a form of Docker images or JAR packages; building only once eliminates the risk of differences introduced by various environments. It also saves time and resources:

• Build once, and pass the same binary between environments.
• Use an artifact repository to store and version binaries. Don't ever use the source code repository for that purpose.
• Externalize configurations and use a configuration management tool to introduce differences between environments.

Practice 10 – Release often

Release often, preferably following each commit to the repository. As the saying goes, "If it hurts, do it more often." Releasing as a daily routine makes the process predictable and calm. Stay away from being trapped in the rare release habit. That will only get worse and you will end up releasing once a year, having a three month preparation period!

• Rephrase your definition of done to done means released. Take ownership of the whole process!
• Use feature toggles to hide features that are still in progress from users.
• Use canary releases and quick rollback to reduce the risk of bugs in production.
• Adopt a zero-downtime deployment strategy to enable frequent releases.

With the final part of this book, we've covered the most important ideas and tooling around the Continuous Delivery process. I hope you found it valuable, and I wish you all the best in your Continuous Delivery journey!