Chapter 1. Friction in Software Development
There is still much friction in the process of crafting complex software; the goal of creating quality software in a repeatable and sustainable manner remains elusive to many organizations, especially those who are driven to develop in Internet time.
Is the productivity of your software organization going down? Is your code base harder and harder to evolve every week? Is the morale of your team declining? As with many other successful software endeavors, you are probably suffering from the inability to manage friction in your software development and may have a pervasive case of technical debt.
Why should you care about technical debt? How does it manifest itself? How is it different from software quality? In this chapter, we introduce the metaphor of technical debt and present typical situations where it exists.
The Promise of Managing Technical Debt
Understanding and managing technical debt is an attractive goal for many organizations. Proactively managing technical debt promises to give organizations the ability to control the cost of change in a way that integrates technical decision making and software economics seamlessly with software engineering delivery.
The term technical debt is not new. Ward Cunningham introduced it in 1992 to communicate the delicate balance between speed and rework in pursuit of delivering functioning quality software. And the concepts it encompasses are not new either. Ever since we started creating software products, we have been grappling with this issue under other names: software maintenance, software evolution, software aging, software decay, software system reengineering, and so on.
You can think of technical debt as an analogy with friction in mechanical devices; the more friction a device experiences due to wear and tear, lack of lubrication, or bad design, the harder it is to move the device, and the more energy you have to apply to get the original effect. At the same time, friction is a necessary condition of mechanical parts working together. You cannot eliminate it completely; you can only reduce its impact.
Slowly, over the past ten years, many large companies whose livelihoods depend on software have realized that technical debt, under this or any other name, is very real and crippling their ability to satisfy customer desires. Technical debt has started to translate into financial impact. At some point in the past, companies may have made a trade-off to take on technical debt to deliver quickly or scale quickly, threw more people at the problem when the debt mounted, and never reduced or managed the debt. It is not a proper debt, from an accounting perspective, but the specter of huge costs somewhere on the path ahead will negatively affect the company’s financial bottom line. Government organizations that are large buyers of software also now realize that focusing only on initial development cost obscures the full cost of the software; they have begun to demand justification of all lifecycle costs from the software industry.
Technical debt is pervasive: It affects all aspects of software engineering, from requirements handling to design, code writing, the tools used for analyzing and modifying code, and deployment to the user base. The friction caused by technical debt is even apparent in the management of software development organizations, in the social aspect of software engineering. Technical debt is the mirror image of software technical sustainability; Becker and colleagues (2015) described technical debt as “the longevity of information, systems, and infrastructure and their adequate evolution with changing surrounding conditions. It includes maintenance, innovation, obsolescence, data integrity, etc.” And it relates to the wider concern of sustainability in the software industry—not only in the environmental sense but also in the social and technical senses.
Progress on managing technical debt has been piecewise, and the workforce tends to devalue this type of debt. So it remains a problem. Why do we think that understanding and managing the problem as technical debt will have a different outcome? Software engineering as a discipline is at a unique point at which several subdisciplines have matured to be part of the answer to the technical debt question. For example, program analysis techniques, although not new, have recently become sophisticated enough to be useful in industrial development environments. So, they’re positioned to play a role in identifying technical debt in a way they weren’t a few years ago. DevOps tooling environments that incorporate operations and development further allow developers to analyze their code, locate issues before they become debt, and implement a faster development lifecycle. Developers also now have the vocabulary to talk about technical debt as part of their software development process and practices.
The technical debt concept resonates well with developers, as they look for a well-defined approach to help understand the complex dependencies between software artifacts, development teams, and decision makers and how to balance short-term needs to keep the software product running with long-term changes to keep the product viable for decades. In this way, technical debt can also be seen as a kind of strategic investment and a way to mitigate risk.
Technical Debt A-B-C
Many practitioners today see technical debt as a somewhat evasive term to designate poor internal code quality. This is only partly true. In this book, we will show that technical debt may often have less to do with intrinsic code quality than with design strategy implemented over time. Technical debt may accrue at the level of overall system design or system architecture, even in systems with great code quality. It may also result from external events not under the control of the designers and implementers of the system.
This book is dedicated to defining principles and practices for managing technical debt—defining it, dissecting it, providing examples to study it from various angles, and suggesting techniques to manage it. Our definition of technical debt is as follows:
In software-intensive systems, technical debt consists of design or implementation constructs that are expedient in the short term but that set up a technical context that can make a future change more costly or impossible. Technical debt is a contingent liability whose impact is limited to internal system qualities—primarily, but not only, maintainability and evolvability.
We like this definition because it does not fall into the trap of considering only the financial metaphor implied by the term debt. Although the metaphor carries an interesting financial analogy, technical debt in software is not quite like a variable-rate mortgage or an auto loan. It begins and accumulates in development artifacts such as design decisions and code.
Technical debt also has a contingent aspect that depends on something else that might or might not happen: How much technical debt you need to worry about depends on how you want the system to evolve. We like that this definition does not include defects in functionality (faults and failures) or external quality deficiencies (serviceability), as lumping together defects and technical debt muddies the water. System qualities, or quality attributes, are properties of a system used to indicate how well the system satisfies the needs of its stakeholders. The focus on internal quality is the lens through which these deficiencies are seen from the viewpoint of the cost of change. Technical debt makes the system less maintainable and more difficult to evolve.
Technical debt is not a new concept. It is related to what practitioners have for decades been calling software evolution and software maintenance, and it has plagued the industry ever since developers first produced valuable software that they did not plan to throw away or replace with new software but instead wanted to evolve or simply maintain over time. The difference today is the increasing awareness that technical debt, if not managed well, will bankrupt the software development industry. Practitioners today have no choice but to treat technical debt management as one of the core software engineering practices.
While technical debt can have dire consequences, it is not always as ominous as it may sound. You can look at it as part of an overall investment strategy, a strategic software design choice. If you find yourself spending all your time dealing with debt or you reach the point where you cannot repay it, you have incurred bad debt. When you borrow or leverage time and effort that you can and will repay in the future, you may have incurred good debt. If the software product is successful, this strategy can provide you with greater returns than if you had remained debt free. In addition, you might also have the option to simply walk away from your debt if the software is not successful. This dual nature of technical debt—both good and bad—makes grappling with it a bit confusing for many practitioners.
We will return to the financial metaphor later to investigate whether there are some software equivalencies to the financial ideas of principal, interest, repayment, and even bankruptcy.
Examples of Technical Debt
To illustrate our definition, we offer a few stories about technical debt in software development projects. You will see organizations struggling with their technical debt and software development teams failing to strategize about it.
Quick-and-Dirty if-then-else
A company in Canada developed a good product for its local customers. Based on local success, the company decided to extend the market to the rest of Canada and immediately faced a new challenge: addressing the 20% of Canada that uses the French language in most aspects of life. The developers labored for a week to produce a French version of the product, planting a global flag for French = Yes or No as well as hundreds of if-then-else statements all over the code. A product demo went smoothly, and they got the sale!
Then, a month later, on a trip to Japan, a salesperson proudly boasted that the software was multilingual, returned to Canada with a potential order, and assumed that a Japanese version was only one week of work away. Now the decision not to use a more sophisticated strategy—such as externalizing all the text strings and using an internationalization package—was badly hurting the developers. They would not only have to select and implement a scalable and maintainable strategy but also have to undo all the quick-and-dirty if-then-else statements.
For the Canadian company, the decision to use if-then-else statements spread the change throughout the code, but it was a necessary quick-and-dirty solution from a business perspective to get a quick sale. Doing the right thing at that stage would have postponed the delivery of the system and likely lost them the deal. So even though the resulting code was ugly—as well as hard to modify and evolve—it was the right decision. Now, would you continue down that path and add another layer of if-then-else for each language? Or would you rethink the strategy and decide to repay the original technical debt? Inserting the Japanese version of the quick fix, with its issues of character sets and vertical text, would be too much of a burden and a subsequent maintenance issue. You may argue that a good designer would have set up provisions for internationalization and localization right at the outset, but this is easy to say in hindsight; the demands and constraints at the beginning of development for this small venture were quite different, focused on the main features, and didn’t foresee the need for a multilingual feature.
Hitting the Wall
Two large global financial institutions merged. As a result, two IT systems essential to their business had to merge. The management of the new company determined that a duct-tape and rubber-band system, mixing the two systems in some kind of chimera, would not work. They decided to build a support system from scratch, using more recent technologies and, in some ways, walking away from years of accumulated technical debt in the original systems.
The company organized a team to build the new replacement system. They progressed rapidly because the first major release was to provide an exact replacement of the existing systems. In a few months, they accumulated a lot of code that performed well in demos for each one week “sprint” (or iteration). But nobody thought about the architecture of the system; everyone focused on creating more and more features for the demo. Finally, some harder issues of scalability, data management, distribution of the system, and security began to surface, and the team discovered that refactoring the mass of code already produced to address these issues was rapidly leading them to a complete stop. They hit the wall, as marathon runners would say. They had lots of code but no explicit architecture. In six months, the organization had accumulated a massive amount of technical debt that brought them to a standstill.
The situation here is very different from the first case. This was not an issue of code quality. It was an issue of foresight. The development team neglected to consider architectural and technology selection issues or learn from the two existing systems at appropriate times during development; the team did not need to do all of that up front, but it needed to do it early enough not to burden the project downstream. Refactoring is valuable, but it has limits. The development team had to throw away large portions of the existing code weeks after its original production. Although the organization hoped to eliminate technical debt when it decided to implement a brand-new system after the merger, it failed to incorporate eliminating technical debt into the project management strategy for the new system. Ignorance is bliss—but only for a while.
Crumbling Under the Load
A successful company in the maritime equipment industry successfully evolved its products for 16 years, in the process amassing 3 million lines of code. Over these 16 years, the company launched many different products, all under warranty or maintenance contracts; new technologies evolved; staff turned over; and new competitors entered the industry.
The company’s products were hard to evolve. Small changes or additions led to large amounts of work in regression testing with the existing products, and much of the testing had to be done manually, over several days per release. Small changes often broke the code, for reasons unsuspected by the new members of the development team, because many of the design and program choices were not documented.
In the case of the maritime equipment company, there was no single cause of technical debt. There were hundreds of causes: code imperfections, tricks, and workarounds, compounded by no usable documentation and little automated testing. While the development team dreams of a complete rewrite, the economic situation does not allow delaying new releases or new products or abandoning support for older products. Some intermediate strategy must be implemented.
Death by a Thousand Cuts
One IT-service organization landed several major contracts. Some of this new business allowed the organization to grow its offshore development businesses and enter emerging software development markets. For several years, the organization experienced a hiring boom.
The IT-service projects were similar in nature, and the organization assumed that its new developers were interchangeable across projects. The project managers thought, “The task is customization of the same or similar software, so how different could it be?” But in some cases, the new employees lacked the right skills or knowledge about the packages used. In other cases, time and revenue-growth pressures pushed them to skip testing the code thoroughly or fail to think through their designs. They also did not put in the time to create common application programming interfaces (APIs). The hiring boom created unstable teams, with new members introduced almost every month. It even became an internal joke: “Get a bunch of online Java and Microsoft certifications, and you are a senior developer here.” In no time, the project managers lost control of the schedule as well as the number of defects introduced into the system.
This IT-service organization provides another example in which there is no single source of technical debt. We call this “death by a thousand cuts” because a pervasive lack of competence can result in many small, avoidable coding issues that are never caught. Lack of organizational competency—as in the case of this IT-service organization—easily activates a number of cascading effects. The unplanned and unmanaged hiring boom, the missed opportunity to enforce commonality across the products, and the limited testing all contributed to the accumulating technical debt.
Tactical Investment
A five-person company developed a web application in the urban transportation domain, targeted at users of buses and trains. In this relatively new and rapidly evolving domain, the targeted users could not really tell the company what they would need. “I’ll know it when I see it” was the general response. So, the company developed a “minimum viable product” (MVP) with some core functionality and little underlying sophistication. Members of the company beta-tested it with about 100 users in one city. They had to “pivot” several times until they found their niche, at which point they invested heavily in building the right infrastructure for a product that would be able to support millions of simultaneous users and adapt to dozens of situations and cities.
The initial shortcuts that members of this small company took and the high-level rudimentary infrastructure they initially developed are examples of technical debt wisely assumed. The company borrowed the time it would have spent on the complete definition and implementation of the infrastructure to deliver early. This allowed it to complete an MVP months earlier than traditional development practices, which put the infrastructure first, would have allowed. Moreover, the company learned useful lessons about the key issues (which did not necessarily match its initial assumptions) of reliability, fault tolerance, adaptability, and portability. Building in these quality attributes up front would have created massive rework once the developers understood more completely what their users needed.
All along, members of this company were aware of the deliberate shortcuts they were taking and their consequences on future development. From the perspective of their angel investors, these were good strategies for risk management; if the company found no traction in the market, the developers could stop development early and minimize cost before the company made massive financial investments. Management also made it very clear to everyone, internal and external, that the shortcuts were temporary solutions so that no one would be tempted to keep them, painfully patched, as part of the permanent solution. In this manner, taking on technical debt was a wise investment that paid off. The company repaid the “borrowed time,” but it could also have walked away from the project.
In all these examples, the current state of the software carries code that works, but it makes further evolutions harder. The debt was induced by lack of foresight, time constraints, significant changes in requirements, or changes in the business context.
Your Own Story About Technical Debt?
Now that we have given you a taste of the various flavors of technical debt, maybe you can identify with some of the stories: “Oh, yes, we have some of this here, too!” or “Now this thing we suffer from has a name: technical debt!” You could add your own development (or horror) story here. Over the past few years, the authors of this book have heard similar stories from dozens of companies. These organizations became mired in technical debt from different paths, with different concerns and different consequences. We have heard enough of these stories to classify them into awareness levels about technical debt:
Level 1: Some companies have told us they had never heard the term or the concept technical debt, but it was not difficult for them to see that part of their problem is some form of technical debt.
Level 2: Some companies have heard of the concept, have seen blog posts on the topic, and can provide examples of their technical debt, but they do not know how to move from understanding the concept of technical debt to operationally managing it in their organization.
Level 3: In some organizations, development teams are aware that they have incurred technical debt, but they do not know how to get the management or business side of the company to acknowledge its existence or do anything about it.
Level 4: Some organizations know how to make technical debt visible, and they have some limited team-level strategies to better manage it, but they lack analytical tools to help them decide what to do about technical debt and which part of it to address first.
Level 5: We have not heard from many organizations that respond, “Thank you, all the technical debt is under control.” If this describes your organization, we would love to hear from you about your successful software product.
This feels a bit like the levels of a “TDMM”—Technical Debt Maturity Model—doesn’t it? Regardless of the level you feel you’re at, this book has something for you.
Who Is This Book For?
There are many books and tools that can help you understand how to analyze your software. And there are yet other books that can help you adopt new technology for building microservices, migration to the cloud, front-end web development, and real-time system development. There are also many good books that walk through different aspects of software development, such as software code quality, software design patterns, software architecture, continuous integration, DevOps, and so on. The list is long. But there exists little practical guidance on demystifying how to recognize technical debt, how to communicate it, and how to proactively manage it in a software development organization. This book fills that gap.
We address the roles involved in managing technical debt in a software development organization, from developers and testers to technical leads, architects, user experience (UX) designers, and business analysts. We also address the relationship of technical debt to the management of organizations and the business leaders.
People close to the code should understand how technical debt manifests itself, what form it takes in the code, and the tools and techniques they can use to identify, inventory, and manage technical debt. This is the inside-out perspective.
People facing the customers—the business side of the organization, such as product definition, sales, support, and the C-level executives—should understand how schedule pressure and changes of direction (product “pivot”) drive the accumulation of technical debt. They should be especially conscious of how much the organization should “invest” in technical debt, without repayment, and for how long. This is the outside-in perspective.
Both sides of the software development organization—technical and code-facing or business and customer-facing—should understand the reasoning and decision processes that lead to incurring technical debt and how the consequences of debt result in reduced capacity. They should also understand the decision processes involved in paying back technical debt and getting development back on track. These decisions are not merely technical. For sure, technical debt is embedded in the code base and a few connected artifacts. But its roots and its consequences are at the business level. All involved should understand that managing technical debt requires the business and technical sides of the organization to work together.
Principles of Technical Debt Management
As we progress through the book, we will identify a small number of key software engineering principles that express universal truths related to technical debt. They are rooted in our experience with technical debt in industry and government software projects, and they are accepted or at least acceptable by the software engineering community. The nine software engineering principles follow:
Here is our first principle.
We’ll introduce more principles in the following chapters, and you will find them summarized in the final chapter of the book.
Navigating the Concepts of the Book
The goal for this book is to provide practical information that will jump-start your ability to manage technical debt. The chapters that follow inform the basic steps of technical debt management: become aware of the concept, assess the software development state for potential causes of technical debt, build a registry of technical debt, decide what to fix (and what not to fix), and take action during release planning. The steps draw on the seven interrelated concepts shown in Figure 1.1 that are the basis for managing technical debt.
This book organizes the chapters into four parts.
In Part I, “Exploring the Technical Debt Landscape”—Chapters 1, “Friction in Software Development,” 2, “What Is Technical Debt?,” and 3, “Moons of Saturn—The Crucial Role of Context”—we define technical debt and explain what is not technical debt. We introduce a conceptual model of technical debt and definitions and principles that we use throughout the book. We want to make technical debt an objective, tangible thing that can be described, inventoried, classified, and measured. To do this, we introduce the concept of the technical debt item—a single element of technical debt—something that can be clearly identified in the code or in some of the accompanying development artifacts, such as a design document, build script, test suite, user’s guide, and so on. To keep with the financial metaphor, the cost impact of a technical debt item is composed of principal and interest. The principal is the cost savings gained by taking some initial expedient approach or shortcut in development—or what it would cost now to develop a different or better solution. The interest is the cost that adds up as time passes. There is recurring interest: additional cost incurred by the project in the presence of technical debt due to reduced productivity, induced defects, loss of quality, and problems with maintainability. And there is accruing interest: the additional cost of developing new software depending on not-quite-right code; evolvability is affected. These technical debt items are part of a technical debt timeline, during which they appear, get processed, and maybe disappear.
In Part II, “Analyzing Technical Debt”—Chapters 4, “Recognizing Technical Debt,” 5, “Technical Debt and the Source Code,” 6, “Technical Debt and Architecture,” and 7, “Technical Debt and Production”—we cover how to associate with a technical debt item some useful information that will help you reason about it, assess it, and make decisions. You will learn how to trace an item to its causes and its consequences. The causes of a technical debt item are the processes, decisions, action, lack of action, or events that trigger the existence of a technical debt item. The consequences of technical debt items are many: They affect the value of the system and the cost (past, present, and future), directly or through schedule delays or future loss of quality. These causes and consequences are not likely to be in the code; they surface in the processes and the environment of the project—for example, in the sales or the cost of support. Then we cover how to recognize technical debt and how technical debt manifests itself in source code, in the overall architecture of the system, and in the production infrastructure and delivery process. As you study technical debt more deeply, you’ll notice that it takes different forms, and your map of the technical debt territory will expand to include this variety in the technical debt landscape.
In Part III, “Deciding What Technical Debt to Fix”—Chapters 8, “Costing the Technical Debt,” and 9, “Servicing the Technical Debt”—we cover how to estimate the cost of technical debt items and decide what to fix. Decision making about the evolution of the system in most cases is driven by economic considerations, such as return on investment (for example, how much should you invest in the effort of software development in a given direction, and for what benefits?). For the technical debt items, we will consider principal and interest and associate elements of cost to reveal information about the resources to spend on remediation and the resulting cost savings of reducing recurring interest. We then revisit the technical debt items in the registry collectively and use information about the technical debt timeline to help determine which technical debt items should be paid off or serviced in some other way to ease the burden of technical debt: eliminate it, reduce it, mitigate it, or avoid it. We show how to make these decisions about technical debt reduction in the context of a business case that considers risk liability and opportunity cost.
In Part IV, “Managing Technical Debt Tactically and Strategically”—Chapters 10, “What Causes Technical Debt?,” 11, “Technical Debt Credit Check,” 12, “Avoiding Unintentional Debt,” and 13, “Living with Your Technical Debt”—we provide guidance on how to manage technical debt. A key aspect of a successful technical debt management strategy is to recognize the causes in order to prevent future occurrences of technical debt items. Causes can be many, and they can be related to the business, the development process, how the team is organized, or the context of the project, to list a few. We present the Technical Debt Credit Check, which will help identify root causes of technical debt that show the need for software engineering practices that any team should incorporate into its software development activities to minimize the introduction of unintentional technical debt. The principles and practices you will have learned along the way make up a technical debt toolbox to assist you in managing technical debt.
At the end of each chapter, we recommend activities that you can do today and further reading related to the concepts, techniques, and ideas we discuss.
What Can You Do Today?
Apply the first principal by putting a name to your technical debt. Commit to applying a few basic techniques to your normal development practices as you read each chapter and continue to improve over time.
For Further Reading
The seminal paper that brought us the debt metaphor is the often-cited OOPSLA 1992 experience report, “The WyCash Portfolio Management System,” by Ward Cunningham.
Steve McConnell (2007) provided one of the simplest and most accessible definitions of technical debt: “a design or construction approach that is expedient in the short term but that creates a technical context in which the same work will cost more to do later than it would cost to do now.” Our current definition of technical debt was devised in a week-long workshop in Dagstuhl, Germany, in April 2016 (Avgeriou et al. 2016).
The software crisis was well described in 1994 by Wayt Gibbs, who interviewed many software pioneers and practitioners in industrial organizations, including Larry Druffel, Vic Basili, Brad Cox, and Bill Curtis.
A must-read is Fred Brooks’ “No Silver Bullet” paper (Brooks 1986), which is also a chapter in the 10th anniversary edition of his famous book The Mythical Man-Month (Brooks 1995). Brooks reminds us that “There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.”
A durable software engineering principle should be a simple statement that expresses some universal truth; is “actionable” (that is, worded in a prescriptive manner); is independent of specific tools or tool vendors, techniques, or practices; can be tested in practice, where we can observe its consequences; and does not merely express a compromise between two alternatives. There are two classic books on software engineering principles: 201 Principles of Software Development by Alan M. Davis (1995) and Facts and Fallacies of Software Engineering by Robert L. Glass (2003). In “Agile Principles as Software Engineering Principles,” Norman Séguin (2012) did a thorough analysis of what constitutes a good software engineering principle—as opposed to a mere aphorism, wish, or platitude—and he debunked a few myths about principles.