Chapter 9. Servicing the Technical Debt

Organizations are often perplexed by questions like “Do we have too much debt?” “Which technical debt items should we remove?” and “Which project should we close out because of technical debt?” In this chapter, we examine the paths you can take to service your technical debt: eliminate it, reduce it, or mitigate it. Using the technical debt descriptions in the registry and the technical debt timeline, we offer an approach to help you decide which technical debt items you should service first and which you can put off for later.

Weighing the Costs and Benefits

At this stage, you have a registry of technical debt items. You know what consequences they could have on the future of your software project in terms of recurring interest and remediation cost as you consider whether to carry or pay the debt.

What should you do about your debt? You might be tempted to answer, “Repay the technical debt items, all of them, one by one and as fast as possible to avoid interest.” This is what you might do with ever-increasing credit card debt. However, there are other options to consider in managing your overall financial health. While it is prudent to eliminate the most severe credit card debt, you would manage a car loan or home mortgage differently. You might be more concerned with cash flow and want to continue making an affordable fixed monthly car payment. Or you might be optimizing your overall financial portfolio. Early in the life of a mortgage, the majority of the payment goes to interest, so you might make additional payments that apply to principal. Later in the life of the mortgage, you might redirect those additional payments to other investments since the majority of payment goes to principal and the incentive to reduce interest is gone. Your goals and the context of your situation will influence your decision.

In software development, you have these and even more options to manage and grow your technical wealth. Unlike with financial debt, you may not have to repay any of your technical debt, or you might have to repay some but not all of it. You can choose.

In deciding what to do, you need to consider the business case for debt reduction, including the costs and corresponding benefits (see Table 9.1). You should evaluate the benefit of reducing risk liability and recurring interest. You should also estimate the opportunity cost of delaying the delivery of new features as you remediate the debt and the cost of paying the current principal and accruing interest. Conversely, the business case to incur or carry debt swaps these factors. The benefit becomes the cost savings of carrying the debt along with earlier feature delivery. The cost becomes the recurring interest and increased liability.

Table 9.1 Costs and benefits of servicing technical debt

Understanding the costs and benefits of carrying versus remediating the debt will give you a sense of where you are on the technical debt timeline introduced in Chapter 2, “What Is Technical Debt?” Have you passed the tipping point so that the cost of interest has become greater than the benefit of incurring the debt in the first place? With the answer to that question, you can examine the technical debt items in your registry and determine which technical debt items you should remediate and which ones you can continue to live with. Weighing costs and benefits of the technical debt items in the registry will enable you to discuss and prioritize actions to take to decide how to remediate the debt (see Figure 9.1).

Risk Exposure and Opportunity Costby Eltjo R. Poort

A realistic business case for technical debt reduction is an important tool to put the risk and cost related to technical debt on the radar of the business stakeholders who can do something about it. On top of recurring maintenance and remediation costs, it should also include less obvious items such as risk exposure and opportunity cost related to a specific debt item.

Risk and opportunity costs often have more impact than the recurring maintenance and direct remediation costs. A technical debt item that might lead to severe security risks will normally be remediated quickly even if the cost of remediation outweighs the maintenance reduction. Think of outdated operating systems with known and unknown vulnerabilities that are no longer patched: Migrating to a new OS might be expensive, but you just cannot afford to risk a security breach. Conversely, an item that would at first sight make very much economic sense to remediate as soon as possible might still have to wait because you need the full development capacity to grasp an opportunity to beat the competition by creating some new functionality.

Risk Exposure

The proper way to calculate the total expected cost of uncertain failure is the well-known risk exposure formula: E(S) = p(S) × C(S), where p(S) is the ­probability of failure scenario S occurring, and C(S) is the cost incurred when S occurs. By summing up the risk exposure E over all possible failure scenarios S caused because of the technical debt, you come as close as statistically possible to an accurate prediction of the expected cost of failure. In practice, technical debt”related risk exposure can often be estimated only as an order of magnitude, but this level of estimation is often enough to make a business case.

I once encountered a situation in which a large transportation company was running some of its core business systems on ancient mini-computers. Spare parts were very hard to get, and the manufacturer had put severe limitations on its maintenance contract. The organization in question had a hard time making the business case for migrating the system to a modern, virtualized, blade-based solution: The cost of the old platforms was so low that the ROI for the migration looked negative. The risk of failure, however, was substantial: A single missing spare part could potentially break the company by disabling its core system for a few days. Including that risk exposure in the technical debt interest leads to a completely different business case; in this case, it pushed the company over the tipping point.

Opportunity Cost

The New Oxford American Dictionary defines opportunity cost as “the loss of potential gain from other alternatives when one alternative is chosen.”

When a development team spends resources and time on reducing technical debt (upgrading, refactoring, repairing), the team will produce fewer end-user stories during that time. Opportunity cost represents the business value that those end-user stories would have yielded, as a way of accounting for the scarcity of the team’s resources.

The literal term opportunity cost is seldom heard during technical debt discussions, but it is often a major factor in deciding when to reduce the debt. Whenever a stakeholder (for example, a product manager) says something like, “Yes, we should do something about this debt, but we cannot afford to do it now,” she is probably referring to the business features that end users are waiting for or that have been promised by a certain deadline. In other words, the opportunity cost of reducing the technical debt—the potential gain from the alternative of delivering the business features on time—is higher than the interest on the technical debt incurred during that period.

This diagram illustrates opportunity cost by comparing two scenarios: in Scenario 1, the technical debt is not paid back, and in Scenario 2, the debt is paid back in Release 1.2. The value curve at the top of the figure makes a little dip in Scenario 2 (dashed line), compared to the continued growth of xScenario 1. The figure shows that in Scenario 1, Release 1.2 introduces five new user stories, while in Scenario 2, there is time for only one user story because a team has spent the rest of the resources on reducing the technical debt. The gap between the dashed line and the solid line represents the opportunity cost of reducing the technical debt. (If you are wondering why the dashed line goes down in Release 1.2, even though the team has added a user story, I use the practical rule of thumb that existing business features in a solution are subject to some type of value decay due to growing expectations and demands from end users.)

A good example of opportunity cost in architectural technical debt reduction was presented to me by architects attending an agile architecture course as part of an exercise. In their organization, a team had been developing business process automation features for 4 years. The organization had kept track of the labor cost savings attributed to that automation effort, which amounted to 9 FTE (full-time equivalent positions) per year on average. The platform the software was running on was due for a major overhaul because it could not easily be made compliant with new European Commission regulations (most notably the EU General Data Protection Regulation). During the overhaul, the team would not be able to develop new features—which meant an opportunity cost equivalent to 9 FTE per year, or 0.75 FTE per month spent exclusively on the overhaul. This was a significant opportunity cost, but in the end, it was determined that the total benefits, including the significant reduction of the risk of noncompliance and reduced maintenance cost, outweighed the total cost (opportunity cost plus the cost of the overhaul itself).

The bottom line is that if you need to draw up a complete business case for servicing a piece of technical debt, make sure you include not only the more obvious principal and interest but also the risk and opportunity cost. This will help facilitate a rational discussion about the impact of running risks and delaying features and, therefore, help you put the decision in its business context.

Let’s use the Phoebe project to demonstrate how to refine issues in the techdebt registry impacting business goals (Table 9.2). Recall that Team Phoebe’s source code analysis surfaced more than 10,000 violations. The team identified two major technical debt items: Phoebe #346: “Remove duplicate code” and Phoebe #345: “Remove empty Java packages.” Reviewing the architecture to complement the code analysis revealed a major risk in the design: Phoebe #420: “Locked-in architectural choices in adapter/gateway separation.” Looking beyond the symptom of a reported defect about crashes to the root cause in the design yielded another techdebt: Phoebe #421: “Screen spacing creates unexpected crashes due to API incompatibility.” Finally, reviewing the production infrastructure surfaced technical debt items related to building and testing: Phoebe#500: “Improve build time” and Phoebe #501: “Improve test infrastructure.” Chapter 8, “Costing the Technical Debt,” provides the means to understand the costs associated with these issues for the architecture, code, and infrastructure, including testing.

Table 9.2 Phoebe techdebt registry

However, there is still more work to be done in sorting out the backlog of issues collectively, setting priorities, weighing costs and benefits, and planning releases that allocate resources among new feature development, necessary design tasks, routine defects to take care of, and technical debt items.

Paths for Servicing Technical Debt

We’ve discussed how technical debt repayment might play a role in what you can deliver, considering a fixed expenditure budget. But the picture becomes more complicated as you establish a roadmap for a project and define the content of future releases. When deciding what to do in upcoming iterations, you need to consider all the items on your backlog of things yet to do and their dependencies, including ­technical debt items.

If you want a system to evolve in a certain direction—for example, by adding a new feature or service—you need to analyze which parts of the system will be affected by this evolution. If those parts of the system contain technical debt items, you need to look at the consequences of those items relative to the proposed evolution. Will they prevent or slow the proposed new development? If yes, then maybe you will need to repay them.

Any plan to repay some technical debt will affect the cost of possible scenarios for evolving the system, which in turn may affect your decision to repay some debt or evolve the system. The choice may well be decided by how much technical debt you must repay before proceeding. This is true for any change, whether it be a request to add a new feature, resolve a problem, invest in architecture, or even consider another technical debt item.

Here is an approach for developing a plan to manage your technical debt while you maintain and evolve your system:

1. Identify the parts of the system that will be affected by a change.

2. Determine whether technical debt items are associated with these parts of the system.

3. Identify the consequences of technical debt on this and possibly other changes.

4. Estimate the cost of the debt repayment and add it to the cost of the change.

5. Estimate the benefit of the debt repayment in enabling the development of this and possibly other changes. (This can be difficult to do!)

This approach is contingent upon having a good grasp of which areas in the system have more technical debt as well as a few maintenance and evolution scenarios to compare potential outcomes. In addition, this approach is most practical when technical debt signals problems with the design.

Remediating hundreds of little code-level “smells” and other code quality issues might involve allocating a fixed percentage of resources to servicing technical debt. This is analogous to adding a buffer of time within a sprint for fixing defects. A fixed percentage gives a team the discretion to deal with code quality issues while controlling spending. In cases of extreme debt, you might allocate an entire sprint or two to work on paying back technical debt.

Experienced teams consider aspects of evolution as they debate design options, backlog grooming, and technology change. Conducting these discussions explicitly for the technical debt items will improve a team’s understanding of the consequences and help members make decisions based on the benefit gained by fixing the related technical debt items.

There are a number of decision points:

• Determining whether the debt is potential or actual

• Deciding to fix or not as new features are developed iteratively

• Deciding whether it is time to mitigate the risk by remediating the debt completely

• Declaring victory by writing off the debt

• Declaring bankruptcy

As your system reaches different points on the technical debt timeline, you will need to revisit whether the path you have selected is continuing to serve you in servicing your technical debt effectively. Let us look at these decision points along the way in more detail.

Is the Debt Potential or Actual?

If you have been thinking of a fixed-rate mortgage as a typical example of financial debt, then this is where the technical debt metaphor breaks down a bit. Your mortgage has defined principal and interest from the outset, and it’s included in the paperwork you signed at the bank. Technical debt does not have defined principal and interest from the outset; it is tied to the current state of the system, and the current principal and interest are tied to your intentions for future changes. You might have a potential for technical debt, but it will be actual technical debt only if you have to evolve your system. You might also decide to walk away from your technical debt by walking away from your system.

The first step for any of the paths you can take is to determine whether the debt is potential or actual. If there is technical debt in parts of the system that do not need to evolve and do not have unintended business impact, you can just ignore it for now. In other words, the (long) list of technical debt items represents only potential debt. The actual debt at any point in time depends on how the system will evolve in the future. The more certain you are about the future evolution and the probability of change, the more confidently you can identify the actual debt and the payback strategies.

Principle 9: Technical Debt Depends on the Future Evolution of the System

The value of debt as a strategic investment and its cost of remediation depend on the changes to be made on the system from now on. It is because of this principle that technical debt assessment and management are not one-time activities. They are strategic software management approaches that you should incorporate as ongoing activities for the development and sustainment of all software-reliant systems.

Should You Work on Debt, Feature Delivery, or Both?

Technical debt is an attribute of the state of a system at some point in time. While you may have identified technical debt items, you cannot associate to them any meaningful metric of cost until you look into the future and consider ­dependencies—dependencies among the technical debt items and dependencies of future features on them.

Team Atlas has some experience with this type of thinking:

The Atlas project met their goal of time to market with a successful product launch. There is growing demand to evolve the system by adding a new feature. Designing the feature points out some dependency on an element of technical debt in the system.

The value of the feature is the same, regardless of how it is implemented. But the project may incur a different cost based on whether the team eliminates or mitigates the technical debt. In this case, the system had some potential debt, but because it is affected by a prospective evolution, it now has actual debt. Team members can determine the cost of repaying the debt and the cost of carrying each technical debt item, as described in Chapter 8. They should include the opportunity cost and risk liability in the decision to mitigate business risk.

The decision is determined by the cost”benefit trade-off of servicing the debt. The cost to fix includes the cost of repaying the technical debt and the opportunity cost of delaying features. The benefit includes the reduced recurring interest cost and reduced risk liability. You can compare these costs to see where the project is on the technical debt timeline and make a decision about repayment. Is the project still getting value from carrying the debt and the recurring interest, and is the risk liability still low? Or have you passed the tipping point, beyond which you are suffering from technical debt? If the cost to fix is reasonable for the given benefit, then it is sensible to proceed with the fix. If not, the debt may have passed the point where it is feasible to remediate, and you should consider other paths, such as declaring bankruptcy.

It is rare that a system would evolve one feature at a time. A feature could also depend on multiple technical debt items; many features together would therefore implicate many technical debt items. When you potentially have multiple features with different values, you may be able to combine them in different “packages” over a release timeline and maximize value over time at a given cost.

There is some invisible internal value generated by refactoring areas of high technical debt in the system, if it makes future evolutions of the system easier and therefore cheaper. This forethought takes a longer view into the future of the system. In planning the future of a project, you should include technical debt in the economic reasoning of the business, not just the value of features visible to users. And you must look at the benefit and cost of repaying or not repaying the debt that affects the possible evolution scenarios.

Is It Time to Mitigate Risk?

If your system is already deployed in the field, your backlog very likely contains defects that must be fixed urgently. These have negative value because they decrease the system’s usability and they make customers unhappy, and customers or users today can express their unhappiness widely through social media, which may affect future sales. So, your business people want at least some of the defects fixed in the upcoming release, and they may even be ready to compromise with some of the features, maybe to the detriment of the architecture, which you know you really need but for which the business people may not see much value. The Atlas project ­demonstrates this problem:

The Atlas project is slowing in productivity, and team members spend more and more time fixing defects at the expense of adding new features. The time to fix the defects is increasing as well.

Over time, risk liability may be the most important factor in the case for debt reduction. Instead of pulling a feature off the backlog, you might consider a defect, an architecture investment, or a technical debt item to be of higher priority to mitigate risk. That backlog item might depend on refactoring parts of the system to eliminate other elements of technical debt.

The path is influenced by the cost”benefit trade-off of servicing the debt. The total cost includes the cost of the backlog item, the cost of repaying the technical debt, and the opportunity cost of delaying features. This approach provides the benefit of eliminating the recurring interest cost of the technical debt and reducing risk liability. This also supports the goal of lowering the probability and impact of failure in a system that is growing in complexity.

Is It Time to Write Off the Debt?

In the case of debt amnesty, you write off the accrued technical debt and do not have to repay it. Contexts for such a decision include developing a throwaway prototype or concluding that a feature or product is a failure and no longer needed for various reasons, such as a lack of customer interest or value. The written-off technical debt item will continue to appear in your registry, but it doesn’t matter anymore because there is no longer any recurring interest and hence no benefit in reducing its cost.

Is It Time to Declare Bankruptcy?

Bankruptcy happens when the part of the software system that contains the technical debt item is no longer viable to support future development, and a complete rewrite is needed. In some extreme cases, the whole system may have reached a point where a complete rewrite is the only option. Bankruptcy is justified when the cost of rewriting the system is lower than the cost of maintaining it (that is, the sum of recurring interests).

After restructuring the software and emerging from bankruptcy, a project may elect to monitor technical debt more closely by implementing checks and tests that must pass, and if they don’t, they break the build.

The Release Pipeline

The different paths to servicing technical debt can be used individually and in ­combination to sort out and prioritize the product backlog and assign the backlog elements to iterations or releases. Figure 9.2 shows an example of a plan for the next three releases. Each release contains issues from the product backlog that are a mix of desired features, architectural elements, defect fixes, and technical debt remediation. There will be more detail for the first release or two and less detail further out into the future as part of a long-term release plan showing what features, improvements, defect fixes, and technical debt payments will be part of each release. The arrows depict dependencies both within and across releases. In many cases, the architecture must be developed ahead of the features and technical debt remediation that depend on it.

When you have the mechanisms in place to service technical debt, you can run some what-if scenarios to adjust the technical debt remediation timeline:

• What if we want to be debt free? What is the cost of paying off all the debt now? Or what is the cost of not letting the debt increase while we figure out how to pay it down?

• What if we postpone a payment? What is the cost of living with the debt, and how will the repayment increase for each subsequent release?

• What if we need to conserve cash flow now, but we also want to be debt free by the end of the development phase? Given that the product will go into sustainment in three years, how can we structure payments so we will be debt free by then and ready to shift resources to new product development rather than support unnecessary maintenance?

While running what-if analysis on scenarios and comparing their implications will give you more information to make choices, you might still have the resources to select only a few of the debt repayment refactorings. How do you choose? Systems differ widely, and different quality attributes (security, fault tolerance, usability, performance, evolvability) matter more in different contexts than in others. This is especially true for major structural changes. It also applies to tackling some of the scattered code imperfection.

Returning to the Phoebe project, the team chose different payment strategies for the technical debt items in its registry. Team Phoebe prioritized fixing the defect in the user screen feature that was causing a very visible crash and had implications for security.

Team members implemented a patch to address their most immediate concern, and then they fixed the design in the next release to pay the debt fully. The issue with duplicate code concerned adapters and was related to the locked-in architecture choices in the adapter/gateway separation, so these issues were treated together.

To mitigate this major risk, team members better defined the responsibilities of the adapter and gateway and refactored the code to better separate the two components. They required new code to conform to the new design while they updated the existing duplicate code incrementally. Improving build time was the next issue to address.

A partial payment improved maintainability without sacrificing performance. Removing the empty Java packages was a more localized fix, and the team addressed this issue as part of the fixed buffer of time for dealing with defects and technical debt. The consequences of technical debt in the legacy test framework were diminishing as the framework was being used less and less, so the team deferred action on remediation for this issue.

The Business Case for Technical Debt as an Investment

We have sounded all the ills technical debt brings to software endeavors. However, when properly managed, technical debt can be a wise investment. This aligns with the basic financial metaphor of taking out a mortgage or borrowing money from a bank to start a new venture, which can be smart ways to build assets. When managed well, design choices bearing technical debt can be fruitful strategic investments and opportunities to investigate the market and learn new technologies if they are monitored for incurring interest.

Let us illustrate the combination of paths a project team can take in its journey to make a wise software investment, this time using actual dollars and Atlas. As we map the team’s choices, we will use the financial concept of net present value (NPV) to evaluate the implication of taking one path or another as Atlas decides whether to select another feature, mitigate risk, pay the debt, or declare bankruptcy. The NPV is the hypothetical value (estimated today) of an investment made today compared to its future returns—that is, its possible value in the future. We will use real options to model this. Real options include the decision—but not the obligation—for a business to pursue, defer, or abandon a capital investment.

Atlas envisions building a new software product, called alphaPlus, to put on the market. The initial plan is to invest $2 million to develop alphaPlus and ship Version 1. Market analysis shows a reasonable demand for alphaPlus, but in such a dynamic world, success is not guaranteed. The business analysts estimate that the product has a 50% chance of success, defined as the market loving it and bringing the company $4 million in revenue. There is also a 50% chance of a mediocre result: that the market hates it, and the return is only $1 million. As you can see in Figure 9.3, the NPV of the investment is $0.5 million. This is still positive overall, so the alphaPlus project is worth launching.

But what if Atlas were to take on an enormous amount of technical debt to bring a simpler prototype to market much earlier and use it to “test” the market? Then Atlas would invest only $1 million and take on $1 million of ­technical debt (mostly architectural, in limited scalability, in a single geographic locale, and with some internal ugliness).

If the market loves the product, then and only then would Atlas invest the other $1 million to complete alphaPlus. If the market hates it, Atlas will not pursue alpha­Plus and will just walk away. As you can see in Figure 9.4, the NPV is now better: $1 ­million. So, taking on this technical debt is a more valuable investment!

But wait a minute. The decision is not this simple. Atlas will have to pay some interest on its technical debt. Estimating 50% interest on the debt to get to ­Version 2, Atlas must invest not the $1 million spared but $1.5 million. However, by releasing Version 1 much earlier, the company also increases its chance of success by leapfrogging the competition, from the business analysts’ estimation of 50% to 67%. If the product is not successful, again, Atlas will not invest another dollar. The company will declare bankruptcy and walk away. As you can see in Figure 9.5, the NPV is still $1 million.

Even after the Atlas product has met success, the company is not bound to repay the technical debt and refactor the system for a clean Version 2. Team members still have the option of living with the debt and piling up more features. They can use the same reasoning again and again at each decision point in the future, based on what they know at that point in time (see Figure 9.6).

Is this “real options” strategy practical? Not quite yet. It might look good and might provide some rationale for making decisions. However, it requires many numbers about probabilities of events in the future, about which most software development organizations have no clue, so they have to make wild guesses. This approach could work in theory but not yet in practice. Nevertheless, the thought process and the act of building a simple decision tree can assist you in uncovering critical decision points on the technical debt timeline when you take on and plan to remediate debt. It also shows clearly that technical debt can be an asset—a good thing.

What Can You Do Today?

At this point, it is important to incorporate the following basic rules of thumb into your decision making during iteration and release planning:

• Ensure sustainable team velocity by allocating time to servicing technical debt. Start by allocating 15% of your iteration budget. But know that there is no one-size-fits-all strategy. You might, for example, need to allocate a whole sprint to reducing technical debt; at other times, you may be able to tolerate more debt. Monitor your progress and learn from your experience.

• Put a context-dependent payment plan in place because repaying all debt, except in very small projects, is simply not feasible and also not the best use of resources.

• Show the value of technical debt reduction tasks by specifying how they ­support high-value change requests for new features or defect resolution.

• When choosing among refactorings, opt for the change that will offer more flexibility for the future and support more potential evolutions, when ­economically feasible.

• Prioritize technical debt items to fix by starting with the parts of your code that are the most actively modified. If a subsystem or module will not be modified as a result of a change scenario in the foreseeable future, do not fix any technical debt in it unless the change is a consequence of fixing the technical debt in a module it depends on.

• Recognize the time when the project is so far past the tipping point that future maintenance or evolution is no longer viable. That is the time to declare bankruptcy.

• Don’t be afraid to take on technical debt strategically to your advantage when there is value in achieving a business objective and the servicing cost is predictable.

For Further Reading

Klaus Schmid first articulated the distinction between potential debt and actual debt (2013b), and then formally described it in mathematical terms (2013a). Eltjo Poort (2014, 2016) eloquently articulated the business case for technical debt reduction and architecture’s role in risk management. Bankruptcy and amnesty have been identified by Eric Ries (2011) and Edith Tom and colleagues (2012). Highsmith (2010) showed the financial implication of technical debt manifesting itself as increased cost of change.

Understanding the value of software—in particular, the value of the design of software—is not trivial. Baldwin and Clark (2000) describe how modular designs create value in the form of future flexibility. Kruchten (2011) wrote about the value of software architecture in his blog.

Deciding whether to pay back technical debt is related to making solid software and business trade-offs. If you need a starter book for financial concepts applied to software, see Reifer’s Making the Software Business Case (2001), which will help you work through them.