Chapter 13. Living with Your Technical Debt
In this final chapter, we describe how you can continue exploring the technical debt landscape and how to make the management of technical debt an integral part of your product development activities.
Your Technical Debt Toolbox
By now, you’ve developed a more comprehensive idea of what technical debt is about and how it affects your software development projects. It is likely that your projects suffer from technical debt to some extent. But if managed well, technical debt can be an effective design strategy.
Throughout the book, we have recommended activities that you can do today. But you may still be asking how all these activities fit together in your particular situation. The answer is “It depends!” It depends on your role, on the impact the debt has on the project, on the age of your system, and primarily on where the project needs to go from where it stands today. While the range of actions you can take is wide, here is a generic path:
Become aware: Ensure that all the people involved have a common understanding of what technical debt is and how it affects any project.
Assess the information: Understand the state of the project, what debt you are currently facing, what causes it, and what its consequences are.
Build a registry: Build some form of inventory of technical debt.
Decide what to fix: Look over the technical debt registry as you plan a release for items that will reduce your technical debt and that you will actually tackle.
Take action: Include technical debt identification and management in all software development and business governance practices.
Repeat this process, as it is unlikely that you will escape completely debt-free in one shot.
Does this sound daunting? It does not have to be. At the end of each chapter, we have introduced some ideas for simple undertakings. Here we revisit them so you can think more carefully about living with your debt going forward. Having these practices in your technical debt toolbox will assist you in managing your technical debt.
Depending on the context of your development effort (mostly size, age of the system, and external factors, such as the domain), your software development lifecycle process may be more or less explicit or formal. If you want to explicitly manage the roles, activities, and artifacts of technical debt in your organization, do not make dealing with debt a separate process. Integrate it into your process to complement your current practices.
Proceed in steps, not as a massive change. As the first step, choose what can bring the most immediate benefits. Consider the effect this choice will have on the people involved and any learning required. Keep the technical debt timeline in mind as a guide (see Figure 13.1).
Become Aware
Put a name to your technical debt. Ensure that all the people involved in the project or close to it have a common understanding of technical debt: what it is, what it is not, and how it affects the project. This is important because today many people have heard about technical debt from various sources and developed their own ideas of what it means. Use the definition we give in Chapter 2, “What Is Technical Debt?”
The following are some ways to raise awareness:
Provide a clear, simple definition of technical debt in the context of your project.
Educate the team about technical debt and its causes.
Educate the people in the immediate project environment (management, analysts, and product managers) about technical debt.
Create a “techdebt” category in your issue tracking system, distinct from defects or new features.
Include known technical debt as part of your long-term technology roadmaps.
Extend awareness activities to any external contractors who are part of the project.
Use approaches that increase team communication and help get everyone on the same page conceptually. You might want to organize a lunch-and-learn session with your team to introduce the concept of technical debt. Illustrate this with examples from your own project. A Technical Debt Credit Check can provide a quick look at the overall project and guidance on where to start (see Chapter 11, “Technical Debt Credit Check”).
Assess the Information
Before attempting technical debt remediation, objectively assess the state of the project. Depending on where you are on the technical debt timeline, you may consider the following possibilities:
Establish the goals and criteria against which technical debt will be assessed (see the section “Understanding the Business Context for Assessing Technical Debt” in Chapter 4, “Recognizing Technical Debt”.
Monitor your portfolio by analyzing code, architecture, and production infrastructure to understand the technical debt responsible for the symptoms the team is experiencing (see Chapters 5, “Technical Debt and the Source Code,” 6, “Technical Debt and Architecture,” and 7, “Technical Debt and Production”).
Incorporate lightweight checks to continuously monitor technical debt (see Chapter 12, “Avoiding Unintentional Debt”).
A wide range of activities, including the following, can help you assess the information you gather about technical debt:
Understand the business context to guide the use of source code and analysis tools as input for technical debt analysis.
Create coding, architecture, and production infrastructure standards to serve as the yardstick against which technical debt is measured. Establish thresholds to identify when the debt level is becoming too high.
Develop tests and traceability: requirements, design/code, tests, and test results.
Use tool support to check and enforce some of these guidelines or standards. Deploy a static code analyzer to detect code smells. And do not panic in the face of large numbers of warnings. We gave you some strategies to prioritize these in Chapter 5.
Review the architecture. If it is not documented, glean insights from team knowledge, source code, and the issues being tracked. Use your knowledge of architectural risk to guide automated analysis of the source code.
When fixing a defect or adding a new feature request, look beyond the immediate implementation to see if longer-term design issues could lead to technical debt.
Organize one-hour brainstorming sessions around the question “What design decision did we make that we now regret because it is costing us more than we estimated?” or “If we had to do it again, what should we have done?”
Assessing the information is not a blame game or a whining session; just identify high-level structural issues, the key design decisions from the past that have turned into technical debt today. Later the results will help you determine the impact that technical debt has on your project.
Build a Technical Debt Registry
Introduce a gradual means to build an inventory of technical debt items (see the section “Writing a Technical Debt Description” in Chapter 4):
Refine the “techdebt” category in your issue tracker into a technical debt description. Point at the specific software artifacts involved: code, architecture, or production infrastructure.
Include at least the most common two technical debt subcategories: (1) simple, localized, code-level debt and (2) wide-ranging, structural, architectural debt. Point at the software artifacts involved: code, architecture, or production infrastructure.
Standardize on a single form of “Fix me” or “Fix me later” comment in the source code to mark places that should be revised later. With such comments, they will be easier to spot by using a tool.
Analyze the code and the architecture for the presence of unintentional technical debt and describe the findings in the technical debt registry.
Develop a strategy for prioritizing debt remediation and ensuring that it isn’t starved.
Include technical debt discussions during your iteration reviews and retrospectives and note any technical debt items. Prioritize them as part of your backlog.
As you pursue development, make sure to introduce intentional technical debt items in your registry at the point where you make the decision to incur such debt.
You may have to do a little bit of this inventory work as you gather information to get some concrete examples and input data for your assessment.
Decide What to Fix
If you are facing a large and somewhat varied registry of technical debt, you need to decide what to fix and when. You should base these decisions on your assessment of the situation, including where you want to bring your software product next. To make your decisions, you need to gather additional information about remediation strategies, cost, and the trade-offs involved (see Chapter 8, “Costing the Technical Debt”)
Review items in the technical debt registry to ensure that it contains the appropriate items and that they are prioritized to help with the decision of what to deliver next:
Refine technical debt items to the level of “story cards” on your backlog and make them an integral part of your release planning and iteration planning. Organize your backlog to reflect the four categories of items it contains (see the sidebar “What Color Is Your Backlog?” in Chapter 4).
Estimate not only the cost to pay the technical debt items but also the cost to not pay them: How much will deferring repayment slow current progress? If you are not able to provide an actual cost, use some “t-shirt sizing” strategy.
Allocate time to service technical debt. You might start with 15% of your iteration budget, but you need to keep in mind that one size does not fit all. At times you might need to allocate a whole sprint to technical debt reduction; at other times you might need a lot less time. Monitor your progress and learn from your experience.
Some complex technical debt will have aspects that relate to rework in code, architecture, and infrastructure. Reduction may require more significant re-architecting or systemwide refactoring, which you may have to spread across several iterations.
Put a context-dependent payment plan in place; repaying all debt, except in very small projects, is simply not feasible and is also not the best use of resources.
Show the value of technical debt reduction tasks by how they support high-value change requests for new features or defect resolution. When choosing among refactorings, opt for the change that will offer more flexibility for the future and support more potential evolutions, when economically feasible.
Prioritize technical debt items to fix by doing them first in the parts of your codebase that are the most actively modified. If a subsystem or module will not be modified as a result of a change scenario in the foreseeable future, do not fix any technical debt in it, unless the change is a consequence of fixing the technical debt in a module it depends on.
What you will do now about technical debt can be fully integrated with your release and iteration planning. As you pay down unintentional debt, you may also begin to keep or take on some debt with intention as you seek to proactively manage technical debt (see Chapter 9, “Servicing the Technical Debt”).
Take Action
An integrated technical debt management approach uses your knowledge of your project context to apply practices specific to your situation. It adds proactive measures to understand the causes of technical debt and control the introduction of new debt (see Chapters 10, “What Causes Technical Debt?” and 11, “Technical Debt Credit Check”). Software engineering practices that are essential to any development effort will not only help you address the causes of unintentional technical debt but will also help you avoid it (see Chapter 12).
Take action to identify and manage technical debt in software development and business governance practices. The following are some example of the action you might take:
Aim to reduce technical debt at each development cycle by bringing some technical debt items into your iteration backlog to keep the level of technical debt low.
Develop an approach for systematic regression testing so that fixing technical debt items does not risk breaking the code. (This will remove the objection that “It is not really badly broken, so I won’t fix it.”)
Assess your current practices from the perspectives of business, architecture, development, and organization and identify sources of technical debt to eliminate.
Factor technical debt into business decisions about the opportunity cost of delaying features and reducing risk liability.
Consider taking on intentional technical debt as a short-term or long-term investment, where appropriate, and plan to manage it.
Include indicators of technical debt in any management or overview project dashboard.
Gather key measures of effort or cost associated with technical debt elements to assist in future decision making.
Start simple and iterate through these activities, incrementally improving the process at each iteration and adding sophistication.
Integrating technical debt into your current software development process does not need to change the lifecycle. You may not create new artifacts, except guidelines or standards, if you are missing them. Do not create new roles, apart from a possible “technical debt evangelist” or “technical debt champion” at the beginning. You may, however, deploy new techniques and tools to support some of the activities, and we have enumerated a few throughout the previous chapters.
On the Three Moons of Saturn …
Let’s look at what happened in the three companies we have been using as examples in this book and what we recommend they do now.
Atlas: The Small Startup
The Atlas company became aware of technical debt when it grew to be 15 developers and experienced some difficulties evolving its product for more diverse consumers. The 4 founders had not been aware of the accumulation of debt due to the constant “pivoting” during their early years. More recent hires made the team aware of this, but the debt remained for many months a vague concept, not brought to any concrete action and only a subject of occasional discussion.
One of the founders brought in a consultant who made an initial assessment and delivered presentations to the whole team. The concept of technical debt became clearer for everyone. Atlas acquired a static analysis tool (SonarQube) and a structural analysis tool (Structure 101) and employed a summer student intern to gather data about debt in the project. The results of this side project showed that the team could take some easy actions to mitigate some of the major technical debt. These actions were introduced into two development iterations and involved explicitly putting technical debt story cards on the backlog. But failing to go further led to conflicting priorities later, when the team tried to resolve the debt: Some of the major structural technical debt items are still there, and remediating them seems too daunting to actually tackle.
The Atlas team should now plan to do the following:
Refine its development process to systematically capture technical debt items.
Integrate the big-ticket technical debt items into the planning for future major releases by evaluating the costs associated with remediation and non-remediation. Some of these items will require allocating several sprints of effort to complete.
Systematically reduce technical debt items for code smells in each iteration or most iterations. Train the developers to ensure that they do not inject new code smells.
Phoebe: The Agile Shop with a Successful Product
Phoebe evolved within the “agile” movement, using an improved version of Scrum. Most developers were aware of technical debt, and from the beginning, they systematically included small technical debt items on the backlog. But as the product became successful and the core team began to shrink in size, with much of the development done by external partners, technical debt grew, especially at the structural level. Today the Phoebe team struggles to manage multiple stakeholders with diverse requirements, get ahead of changing technology, and sustain a viable product. As a result, technical debt is accruing, in most cases intentionally.
While the Phoebe team has been trying to repay the debt by prioritizing technical debt reduction in major releases, technology lock-in has become a major hindrance to meeting this goal; decreasing staff size has also been a hindrance. There is also a lot of inconsistency in how the core team identifies and manages the technical debt. For example, the team tried using some tools to look into the code quality but did not sustain their use. Major refactoring releases have eliminated some of the existing technical debt or made it obsolete, but Phoebe has not communicated this broadly to its stakeholders, and it is not clear what the team is handling as the top-priority issues.
The Phoebe team should now plan to do the following:
Raise awareness of technical debt and its impact with partners in its ecosystem.
Explicitly integrate technical debt management in the process used with and by the other partners in the ecosystem, including specified tools.
Add technical debt indicators in the project dashboard.
Integrate the big-ticket technical debt items in the planning for future major releases by evaluating the costs associated with remediation and non-remediation.
Tethys: The Global Giant
For many years at Tethys, technical debt was the elephant in the room. Most senior developers who had been with the project for several years were acutely aware of it, even though they did not call it “technical debt.” They would in private gladly discuss with visitors or newcomers some of the technical debt and when it was intentionally incurred. But from a planning perspective, technical debt and its possible remediation were never on the table and never discussed with the company technical leadership or the business part of the company.
The code is of rather high quality. The company routinely uses various tools to assess the code quality and conformance to its coding and design standards. The technical debt is the result of both intentional structural debt accumulated over the years and a technological shift: Major design choices now look bad due to the evolution of technology over some 15 years.
The Tethys team should plan to do the following:
Agree on a systematic way to identify and capture technical debt, particularly complexity of the architecture and technological shift.
Develop simple means, like t-shirt sizing to start with, to associate remediation and non-remediation cost to major technical debt items.
Make upper levels of management aware of technical debt and its impact on the business.
Involve the product management team in decision making about technical debt reduction and decisions about intentionally taking on more technical debt.
Conduct a Technical Debt Credit Check and include management and product management in the assessment.
Technical Debt and Software Development
Your software development organization will become gradually “technical debt aware.” Ultimately, managing technical debt will become an integral part of your software development.
If your organization is just beginning the journey toward managing technical debt, your startup costs will be higher. To get started, at a minimum, you should plan to do the following:
Understand the state of the development process and its alignment with business goals (see Chapter 10).
Identify technical debt items (see Chapters 5, 6, and 7), including selecting practices and tools to support this activity.
Incorporate technical debt as a major input into software development decisions (see Chapter 9).
Educate all stakeholders inside the development team and at its immediate periphery about technical debt and its consequences (see Chapter 4).
If your organization is already technical debt aware, you can plan to do the following:
Readily identify intentional debt at the point of occurrence and record it (see Chapters 10 and 12).
Regularly monitor the design and the code for new and accumulating technical debt and record any detected technical debt items (see Chapters 5, 6, 7, and 11).
Spread debt reduction across the development lifecycle (see Chapter 9).
Collect metrics on indicators that may point to symptoms of technical debt, such as velocity, lingering defects, and high development estimates (see Chapter 4).
The timeline for a technical debt-aware organization shown in Figure 13.2 shows technical debt being taken on, monitored, and remediated with intention.
An iterative development lifecycle, which is a core feature of agile approaches, offers better opportunities to manage technical debt continuously. The repayment of small technical debt items can be spread over multiple iterations in a single release cycle. However, larger technical debt items—such as architectural ones—may not fit easily in a short iteration cycle. You may be tempted to defer them because they are too disruptive to the rapid pace of development. (This is often the case with architectural activities and is not specific to the architectural debt reduction.) Resist the temptation to incur lots of technical debt in an effort to be more responsive to change requests.
Finale
Technical debt is at the root of the friction we described in Chapter 1, “Friction in Software Development”: the phenomenon that gradually slows down software development teams. Technical debt is simply unavoidable, especially in large and long-lived systems, and even more so in successful systems.
Technical debt has proven to be a useful concept for helping developers and managers approach these issues. Drawing from a financial metaphor, the concept of technical debt shifts decision making from a strict economic standpoint or a pure technical standpoint to a place where various parties can better understand the trade-offs and compromises, assess the current state of development, and determine the way forward.
Technical debt can be an effective tool to sprint to a major short-term milestone—achieving some success very rapidly by borrowing time from the future—and, in this sense, it looks more like debt used as an investment. The problems start to arise later, when the debt is quickly forgotten and not repaid promptly.
In this book, we have identified a small number of principles to help you better understand and deliberately manage technical debt:
Sustaining the pace of innovation while ensuring software quality involves establishing technical debt management as a core software engineering practice. There is growing interest in research, practice, and tool support for managing technical debt. Most of this research can be done only in an industrial environment; the kind of issues we are dealing with cannot be reproduced in a small laboratory experiment. We invite you to join the technical debt community and contribute case studies, stories of technical debt, and practices associated with managing it. A good starting point is the website techdebtconf.org.