Practical Debugging at Scale
Cloud Native Debugging in Kubernetes and Production
The Apress logo.
This Apress imprint is published by the registered company APress Media, LLC, part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.
Dedicated to Selma, Maya, Tara, and Tao.
It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.
—Sir Arthur Conan Doyle
This isn’t a typical programming book. In a typical book, we build a demo, then slowly enhance it as we go through the chapters to learn a new tool. There’s no single demo we can follow for debugging. When conceptualizing this book, I wanted to create a reference that not only covers the theory but also gets its hands dirty with the practicality. Debugging means chasing a bug wherever it leads you, and as such it requires agility and a wide set of skills.
I wrote this book for developers with some experience. I expect you to know debugging and even be good at it. This book is about making us great at it, getting out of our metaphorical box of debugging tools and practices and familiarizing ourselves with newer approaches.
Debugging is a skill that universities skim over since it isn’t a testable skill. Despite spending a lot of time in the debugger, many developers treat it with the same fondness as one treats taking out the garbage – a necessity that we try to avoid. As a result, we spend more time in the debugger and often reach the wrong conclusions in our debugging sessions.
The book is divided into three distinct parts where we gradually work our way to a uniform skill set.
Part 1 – Basics
Chapter 1: “Know Your Debugger” – We spend 60% of our time debugging problems, and a lot of this time is within the debugger. Yet most of us limit ourselves to “step over” and “continue.” In this chapter, we will discuss dozens of powerful debugging features that most of us are sadly unfamiliar with.
Chapter 2: “The Checklist” – Debugging theory is relatively simple. We encapsulate most of the principles within this chapter.
Chapter 3: “The Auxiliary Tools” – Other than the debugger proper, we need to understand the tools from the operating system and third-party tools for debugging issues. In this chapter, we’ll discuss tools ranging from git bisect to strace to wireshark. We discuss how we can use these tools as part of our investigating journey when tracking some issues.
Chapter 4: “Logging, Testing, and Fail-Fast” – Logging and testing are precognitive debugging. We guess the areas where bugs might present themselves and prepare ourselves for future debugging sessions.
Chapter 5: “Time Travel Debugging” – Time travel debugging isn’t new. It’s a system that records the execution of the program and then lets us play it back and forth similarly to a debug process, but with the ability to view everything taking place.
Part 2 – The Modern Production Environment
Chapter 6: “Debugging Kubernetes” – This isn’t a DevOps book. When a container runs out of memory, the DevOps team is on that and can fix that problem. However, if the code in the container is leaking memory, the DevOps team can’t do anything.
Chapter 7: “Serverless Debugging” – Serverless is notoriously hard to debug. This chapter can’t change that fact. It does lay out the options and tooling we can use to mitigate the situation.
Chapter 8: “Fullstack Debugging” – We discuss the tools and techniques to debug web frontend as well as database backend. This is essential when closing in on an issue that crosses tiers.
Chapter 9: “Observability and Monitoring” – DevOps have had deep insight into the performance and stability of our servers for a very long time. These tools provide a deep level of insight that R&D should be a part of and should use to gauge the “real-world” behavior of our production environments.
Chapter 10: “Developer Observability” – Developers have been left in the dark when it comes to observability. Newer tools are changing the dynamics in this industry and are offering new ways to debug production.
Part 3 – In Practice
Chapter 11: “Tools of Learning” – When we’re new to a project or new to a portion of the code, it’s often hard to understand the dynamic relationship and usage between the various pieces. We can use debuggers to study the way the different pieces fit together.
Chapter 12: “Performance and Memory” – Profilers give us great information, but we’re often left in a lurch by an unclear result. Debuggers can take us the rest of the way and pinpoint the issue.
Chapter 13: “Security” – One of my first interactions with a debugger was as part of a hack. They have long been a tool for hackers; it’s time to use them to improve the security of our systems.
Chapter 14: “Bug Strategies” – In this chapter, we review past issues and lessons learned from the field.
Summary
I hope you find this book helpful. It’s meant as a practical “hands-on” tool that you can reach to when struggling with a difficult bug. A common Israeli military axiom is “training is rough, combat is easy.” I believe that if we train our debugging capabilities to their full extent, we will massively improve overall.
Debugging provides a deep level of insight into the runtime of an application. No modeling software or other tool provides that. It’s a therapeutic system that puts us face to face with our insecurities and doubts. It’s an investigative process that challenges us and rewards us. I hope this book will help you see the bigger picture around this underdocumented task that takes up so much of our time.
A photo of Shai Almog.
A photo of Alejandro Duarte.