DevOps Culture
While the majority of this book is about technologies, the key to success with DevOps is people. All of the technology in the world won’t help you be successful if your team doesn’t work together. Below are just some DevOps culture principles.
Demystify deployments
Rolling out a new deployment can often be described by the team as “terrifying” or “scary.” You need to find a way to ensure the entire team knows how to deploy, and that every member of the team is trusted to contribute code to the production environment. The systems put in place to prevent large-scale mistakes should be good enough to ensure that nobody can break the existing experience. To ingrain this in the culture, startups like Lyft have new employees deploy to production on their first day on the job. If something significant breaks, this leads to the next point, the “no blame” rule.
The “No Blame” Rule
No engineer is perfect, and neither are any existing tools or processes. When something breaks, it should never result in finger-pointing and figuring out who will be the scapegoat. Instead, do a post-mortem of the issue, typically called a Root Cause Analysis (RCA), that will objectively explain what happened, why, and how it can be detected and avoided in the future.
Instrument like you need to support it
DevOps often has developers taking responsibility for the uptime of their systems in production. If you’re going to be on call for a production incident, you want to have your system self-heal so you don’t get a call to begin with, and if you do get a call, provide enough information to triage it quickly. Code should be written with this in mind, with thoughtful telemetry, health models, and self-healing as part of the design.
Avoid Tribal Knowledge and Collaborate
Tribal knowledge often occurs in development or operations when there is one member of the team who has sole knowledge about a particular system or process—for example, the one engineer who knows how to deploy a legacy app. To avoid tribal knowledge, as a team you want to embrace a culture of collaboration and knowledge sharing. This can be as simple as starting a team Slack channel, OneNote, or wiki where the team members are responsible for documenting and, most importantly, keeping up-to-date team information, internal processes, checklists, utilities, and more.
Work as a Conveyor Belt
Just like a car factory, small pieces of the product should always be ready to ship to production, going through their own iterations of QA and integration within the larger component scope. Use small, frequent, and atomic check-ins for changes in favor of larger, infrequent code changes.
Automation over Technical Debt
By technical debt, we are referring to a set of tasks or work that are either not completed or are done manually “just this one time” to hit a deadline. Technical debt will always happen, but it’s important that automation be prioritized. An example for operations would be a manual backup of a server’s log files that needs to be automated and set up to run as a scheduled job.