front matter

forewords

When David Farley and I wrote Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Addison-Wesley, 2010), we knew, from years of applying the principles described in the book, that it represented a modern, holistic approach to software delivery that brought substantial benefits to teams and organizations who used it. Multiple research programs (including the one I have been involved in, led by Dr. Nicole Forsgren and described in chapters 8 and 10 of this book) have shown that it leads to higher quality and stability, as well as faster delivery.

Although continuous integration and continuous delivery (often shortened to CI/CD) are now thought of as standard practice, they are still surprisingly hard to implement and get right. There are still too many teams (and customers!) that deal with infrequent, risky releases that happen over evenings or weekends, planned and unplanned downtime, rollbacks, and problems with performance, availability, and security. These are all avoidable problems, but solving them requires ongoing investment in teams, tooling, and organizational culture.

Crucially, many people who are new to the industry aren’t familiar with the fundamental practices and how to implement them. This book does an outstanding job of addressing this problem. Christie Wilson, an expert in continuous delivery who leads the open source Tekton CI/CD project at Google, has written a comprehensive, clear, and thorough book that details both the technology and process sides of implementing a modern software delivery process. She not only covers the principles and the implementation, but also shows why they are important, and provides a step-by-step guide to the hardest problems with which I see teams struggle such as taking an iterative approach to feature development and handling “legacy” codebases.

I hope this book finds a place in every software team’s onboarding list as a primer. It will also prove invaluable as a detailed guide for more experienced software engineers adopting a way of working they’re not familiar with. I am grateful to Christie for creating a resource that I am confident will drive a better understanding of how to implement a modern software delivery process to the benefit of both the industry and the wider public we serve.

—Jez Humble

co-author of Continuous Delivery,

The DevOps Handbook, and Accelerate

The beauty of software is that everything can be improved over time. But it is also the curse of software—because we can change things, we do, essentially all the time. The relentless pressure for new features or other improvements leads to a desire for some kind of high-velocity process for integrating changes, testing them, and getting them out to users.

Christie Wilson has lived this process and watched it from many angles, and created a book about how to get consistent velocity for your software teams. Indeed a team that can achieve a high-velocity process, with lots of automation, has a competitive advantage for their products. Over time, these teams not only gain market share, but they also have higher morale and lower attrition. It is great to be part of a productive team!

A common misconception is that lower-velocity processes, presumably with more barriers to deployment, are safer or more secure. Many teams are averse to change and thus release changes once a quarter, for example. This approach has two serious flaws. First, it typically moves the difficult task of integrating many changes towards the end, but with so many changes to integrate since the last release, it can go horribly wrong and cause large delays. Second, a slow process prevents fast security patches, a critical goal for most teams. The approach described in this book is all about continuous (small) integrations, enabling both fast feedback on problems, and a viable mechanism for security patches.

In the past few years, security challenges have increased dramatically, especially around “supply chain attacks.” Modern software includes components from many places—other teams, other companies, and open source software. It is not crazy to have 1,000 components that need to be integrated together. This requires a different level of automation: we need to know all of the inputs and where they came from, and how they were used together. Christie’s book is one of the first to cover these issues and outline how to add this kind of security to your systems.

Finally, although there are a huge number of tools and options in this space, this book does a great job of covering the key concepts and goals, while also making it real through both examples and discussion of alternatives.

I found the book a breath of fresh air in a complex space, and I hope you will enjoy it, too.

—Eric Brewer

VP of Infrastructure and Fellow at Google

preface

Programming has fascinated me ever since I realized it was a thing. I remember (approximately 300 years ago) a friend telling me about a chess program he wrote; although I had absolutely no idea what he was talking about, I simultaneously realized that (a) I’d never given even a passing thought to how computers worked, and (b) I now absolutely needed to understand as much about them as I could. What followed was alternately amazing and confusing (“variables are like a mailbox” is an analogy that makes perfect sense in hindsight, but as my very first introduction to the idea, it just slid right off my brain). After my first high school class in Turbo Pascal and a lot of self-taught Java, I was hooked.

Although I found programming itself fascinating, I became equally, if not more, intrigued by the processes that are used to organize the work of software development. I’ve spent at least half of my career being disappointed by how little attention these processes get, relative to the impact they have on not only the quality of the software but also the happiness and effectiveness of the people making the software. More than that, I was frustrated when I encountered engineers and managers who trivialized this work. This is often driven by the perception that slamming out code as fast as possible is the best way to maximize return on investment.

Ironically, time and research have shown that speed really is a key indicator of success, but to actually make engineers fast, and make their work sustainable, speed has to be balanced with safety. Maximizing the speed and safety of software development are the heart of continuous delivery, so the concept and the practices involved resonated with me. That being said, I wasn’t aware of continuous delivery itself until fairly recently.

What captivated me first were tests and automation. I still remember the sense of freedom I experienced when I was introduced to tests, and especially test-driven development, and I realized that I could verify the software that I was writing as I was writing it. Being able to check my work as I went felt like a huge weight lifted off my shoulders—specifically, the weight of the voice in my head that sometimes tried to convince me that I didn’t know what I was doing and that nothing I wrote would work properly. Tools and automation further helped me feel confident doing things that had huge consequences and were scary: using them is like having a friend sitting at my side, coaching me through what I’m doing.

Continuous delivery as a concept takes the best of all the testing and automation that has empowered me during my career and packages it as a set of practices that can help anyone improve the way they develop software. I want to help engineers—especially engineers who sometimes doubt themselves or struggle with fear (and I’m guessing that describes most of us at least some of the time)—to feel the same sense of freedom, empowerment, and increased confidence that I did when I wrote a test for the first time.

Thank you for taking the time to read this book. I hope, if nothing else, you can take away from it that most of the bugs and mistakes that happen in software have little to do with the code itself (and certainly not with the person writing the code). What really causes them is software development processes that just need a little TLC—and taking the time to update and fix those processes is well worth the investment.

acknowledgments

First of all, thanks to my unreasonably supportive husband, Torin Sandall (technically we’re both Warwick now, but we’re still getting used to it!), who not only encouraged me through the multiple years I’ve been working on this but also taken so much off of my plate to make sure I’d be able to finish during this most intense time of our lives. (Let’s just say that moving from New York to Vancouver, getting married, and having a baby within just one year is only part of the story!)

Thanks to Bert Bates for forever changing the way I think about teaching and presenting ideas. I hope you’ll feel this book does justice to your compassionate and effective teaching style! I still have quite a ways to go, but I’ll be applying what you’ve taught me for the rest of my life in everything I write and every conference talk I give.

Thanks to my nontechnical friends who provided me with constant encouragement (even though I’m not sure I was ever really able to explain what the book is about) and even sat through my Twitch streams; particularly Sarah Taplin and Sasha Burden, who enjoy an alternate universe existence as start-up founders in chapters 3 and 8.

A quick thank you to the teachers I’ve been lucky enough to cross paths with who have made such a difference in my life: Stuart Gaitt, for encouraging a weird little girl; Shannon Rodgers, for teaching me to really think; and Aman Abdulla, for giving me the practical engineering skills and the high standards that I needed in order to get to where I am today.

Thanks so much to everyone at Manning for giving me the opportunity to write this book; it’s a dream come true! Thanks to publisher Marjan Bace; to Mike Stephens for reaching out to me and starting this wild adventure; to Ian Hough for working closely and patiently with me chapter by chapter; to Mark Elston for reviewing the clunky first drafts (warts and all); to Ninoslav Cerkez for careful technical review; and to the review editor Aleksandar Dragosavlijevic. Thanks also to Sharon Wilkey, who helped me fix and understand my many, many grammatical (and other) errors; to Kathleen Rossland for patiently guiding me through the production process; and to the other behind-the-scenes folks who helped produce this book. To all the reviewers: Andrea C. Granata, Barnaby Norman, Billy O’Callaghan, Brent Honadel, Chris Viner, Clifford Thurber, Craig Smith, Daniel Vasquez, Javid Asgarov, John Guthrie, Jorge Bo, Kamesh Ganesan, Mike Haller, Ninoslav Cerkez, Oliver Korten, Prabhuti Prakash, Raymond Cheung, Sergio Fernández González, Swaminathan Subramanian, Tobias Getrost, Tony Sweets, Vadim Turkov, William Jamir Silva, and Zorodzayi Mukuya—your suggestions helped make this a better book. Thanks also to the marketing team at Manning, especially Radmila Ercegovac for helping me get out of my comfort zone onto some podcasts, as well as Stjepan Jureković and Lucas Weber for my Twitch debut, which was a lot of fun.

Thanks so much to everyone who has patiently reviewed the book as I’ve written it and given me feedback, particularly everyone at Google who took the time to help me jump through all the right hoops, including Joel Friedman, Damith Karunaratne, Dan Lorenc, and Mike Dahlin. Thanks a million times to Steven Ernest for teaching me how important commit messages and release notes are, and opening my eyes to how horribly inconsistent I am with quotation marks. And thanks to Jerop Kipruto for not only reading chapters of this book but getting excited about the content and immediately applying it!

Finally, thanks to Eric Brewer for all of the encouragement and reviews along the way, and for not only believing in the book but also taking the time to craft an inspirational foreword for it. Thanks also to Jez Humble for all the wisdom you shared with me at the start of this journey—which unfortunately, I completely ignored and have now learned the hard way. Better late than never, I guess! To both of you: having your stamps of approval in the forewords of this book is a highlight of my career.

about this book

This book is intended to be the missing manual of how to get started with continuous delivery and apply it effectively: by covering the individual practices that make up continuous delivery and teaching you the building blocks you’ll need in order to create the automation that supports these practices. This is the kind of knowledge you’d have to gather on your own after several years of hard-earned experience. Hopefully, this book will give you a shortcut so you don’t have to learn it all the hard way!

Who should read this book

Grokking Continuous Delivery is for everyone who does the nitty-gritty, day-to-day job of building software. To get the most benefit from this book, you should have some familiarity with the basics of shell scripting, with at least one programming language, and some experience with testing. You’ll also want to have some experience with version control, HTTP servers, and containers. You don’t need deep knowledge on any of these topics; and if needed, you can research them as you go.

How this book is organized: A road map

This book is organized into 13 chapters across four parts. The first two chapters form the introduction to the idea of continuous delivery and the terminology you’ll need for the rest of the book:

• Chapter 1 defines continuous delivery and explains its relation to adjacent terms like continuous integration and continuous deployment.

• Chapter 2 introduces the basic elements that make up continuous delivery automation, including the terminology used throughout the rest of the book.

Part 2 is all about the activities that make up continuous integration and are essential to continuous delivery:

• Chapter 3 explains the vital role that version control plays in continuous delivery; without version control, you can’t do continuous delivery.

• Chapter 4 looks at a powerful but little discussed element of continuous integration: static analysis—specifically, linting—and how you can apply linting to legacy codebases.

• Chapters 5 and 6 both deal with testing, the vital verification piece of continuous integration. Rather than trying to teach you how to test (a wealth of information on that topic already exists in many other books), they focus on common problems that build up in test suites over time—specifically, test suites that become noisy or slow.

• Chapter 7 walks through the life cycle of a code change and examines all the places where bugs can sneak in, and how to set up automation to catch and squash those bugs as soon as they appear.

Part 3 moves past verification of software changes with continuous integration and into releasing that software:

• Chapter 8 takes a look at version control, showing you how it impacts release velocity by looking through the lens of the DORA metrics.

• Chapter 9 demonstrates how to build artifacts safely by applying the principles defined by the SLSA standard, and explains the importance of versioning.

• Chapter 10 returns to the DORA metrics, focusing on the stability-related metrics, and examines various deployment methodologies that you can use to improve your software’s stability.

In part 4, we look at concepts that apply to continuous delivery automation as a whole:

• Chapter 11 looks back at the continuous delivery elements that have been taught in the previous chapters, and shows how to effectively introduce those elements to a greenfield project and to a legacy project.

• Chapter 12 focuses the spotlight on the workhorse often at the heart of any continuous delivery automation: the shell script. You’ll see how to apply the same best practices we use on the rest of our code to the scripts that we rely on to deliver our software safely and correctly.

• Chapter 13 looks at the overall structure of the automated pipelines we need to create to support continuous delivery, and models the features that we need from continuous delivery automation systems to ensure these are effective.

At the end of the book are two appendices that explore specific features of continuous delivery and version control systems that are popular at the time of writing this book.

I suggest starting with reading chapter 1; terms like continuous delivery are used inconsistently in the real world, and understanding its context in this book will help ground your understanding of the rest of the chapters.

Reading part 2 and part 3 in order will be the clearest way to consume the content, since later chapters build on each other. Part 3 particularly assumes that the continuous integration practices described in part 2 are well understood. That being said, you should be able to jump around the chapters if you prefer, and every chapter will reference related material in other chapters when it comes up.

Part 4 is the advanced section of the book. Each chapter refers to concepts covered previously, and some of the content (for example, chapter 12) may make more sense after you’ve gained some experience working with continuous delivery systems in general.

liveBook discussion forum

Purchase of Grokking Continuous Delivery includes free access to liveBook, Manning’s online reading platform. Using liveBook’s exclusive discussion features, you can attach comments to the book globally or to specific sections or paragraphs. It’s a snap to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/book/grokking-continuous-delivery/discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/discussion.

Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the author some challenging questions lest her interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

About the author

Christie Wilson is a software engineer. She is a frequent speaker on CI/CD and related topics at conferences like Kubecon, OSCON, QCon, PyCon and more. Christie started her career in mobile web application development, working on backend services for AAA games where she wrote features that wouldn’t be used until the big launch, when everyone would use them at once. To enable that, she built systems for load and system testing.

Leveraging the experience she gained at these previous companies dealing with complex deployment environments, high criticality systems, and bursty traffic patterns, she moved on to work at Google where she built internal productivity tooling for AppEngine, bootstrapped Knative, and created Tekton, a cloud native CI/CD platform built on Kubernetes (currently contributed to by 65+ companies).