Chapter 4. Recognizing Technical Debt

In this chapter, we describe the causal chain of technical debt: causes and ­consequences. We expand on the concept of a technical debt item as a simple mechanism to identify and record the technical debt in a system. Then we explain how a software evolution strategy provides a starting point for analyzing the costs associated with technical debt.

Where Does It Hurt?

On any project that has run for a while, development teams might begin to observe signs that trouble is brewing, that something is not quite right or not working as well as it was before. The system becomes prone to certain types of defects, more bugs, or more crashes. Customers make more change requests, and it takes developers longer to satisfy them. Some customers even walk away from the system in frustration. ­Project managers are amazed by the estimated effort to implement what at first looks like a small improvement. These are consequences of technical debt, but they represent only the emerged part of the iceberg; they are symptoms of a more daunting condition in the system.

In Chapters 1, “Friction in Software Development,” and 2, “What Is Technical Debt?” we characterized actual technical debt as mostly invisible, except to the developers. But technical debt has consequences—sometimes even a chain of consequences—some of which transpire outside the system. Some of these consequences may be visible as symptoms of the underlying technical debt.

This chain of causes and effects looks like this:

Causes → Technical debt → Consequences → Symptoms

Let’s look at this chain in more detail with a simple example, based on a story introduced in Chapter 1, about a Canadian company that first developed a product for customers who speak English and then needed to make the product multilingual. This company is Atlas, the small startup, one of the three representative examples introduced in Chapter 3, “Moons of Saturn—The Crucial Role of Context”.

Early in its existence, Atlas produced a demo version of its product almost overnight to show to a group of venture capital investors. A rudimentary scaffolding, called L10N (localization) and I18N (internationalization), was used in the code in lieu of proper localization and internationalization software. To appeal to another part of the Canadian population, the developers next wrote ugly code to support one other language, French, in addition to English. A few weeks later, the CEO of Atlas assured prospective Japanese customers that a Japanese version could be completed just as quickly as the French version. In fact, adding this third language proved extremely cumbersome. It required major changes in the way the code was developed. It also meant removing and redoing all the changes made to accommodate French. And it took quite some time to achieve and necessitated putting other developments on hold.

The chain of cause and effect in this case looks like this:

• Causes: Developers completed the first version in time for the demo under schedule pressure. They were also unfamiliar with I18N and L10N software.

• Technical debt: Code snippets to handle a second Latin-alphabet language were scattered all over the codebase because internationalization had not been ;considered as a key architectural driver and had to be retrofitted.

• Consequences: The code was error prone and could not support other languages, especially non-Latin-alphabet languages.

• Symptom: The visible consequence was a long delay to add support for a third (non-Latin-alphabet) language, when the team finally recognized the impact of the issue.

But the news was not all bad. Another consequence was that the investors were impressed by the bilingual version, and they moved forward with funding:

• Consequence: The company got a third round of investment from a venture capital firm (yeah!).

If members of the Atlas development team had been aware of the technical debt as it was incurred, they would have identified the risk early and could have made some contingency plans to deal with it. It is likely the schedule would not have given them the flexibility to accommodate French debt free, but they could have begun proactively managing the debt while supporting future languages. This would have helped set expectations for a longer release when negotiating resources for the Japanese version. More often than not, technical debt is unintentional and does not become visible until much later, when consequences surface.

The first step toward recognizing technical debt is to investigate the chain of cause and effect in reverse:

Symptoms → Consequences → Technical debt → Causes

Using the analogy of a health issue, a physician would start from the symptoms to diagnose the problem. Similarly, you should aim to detect more consequences of technical debt, possibly less visible ones, by looking inside the system, and they will eventually point you to the development artifacts containing the debt. We call these artifacts and their associated principal and interest a technical debt item (refer to Chapter 2). Identifying these items aids in resolving the problem at its source rather than treating the symptoms and then seeing the problems resurface.

Pursuing this path of analysis, you next ask, “Why do we have this debt item?” The answers to this question will help you locate the causes of technical debt—even its root cause. While understanding causes isn’t strictly necessary for resolving the technical debt, it may provide insight into how the development environment is creating conditions for incurring technical debt. It may lead to changes in the organization to avoid generating more technical debt. We will explore causes of technical debt in more depth in Chapter 10, “What Causes Technical Debt?”

In the technical debt timeline we introduced in Chapter 2, the first goal is to reach the point of awareness, or knowing what technical debt you have in your system (see Figure 4.1). The technical debt item will enable you to track the debt you become aware of within your software development process so that you can estimate, discuss, and prioritize actions to take.

What Are the Visible Consequences of Technical Debt?

Some symptoms—such as the release delay faced by Team Atlas for its Japanese-language version—emerge after the entire system has been affected. They surface late in the development cycle and manifest as increased testing time, problems integrating subsystems into the rest of the system, and projects hitting major impediments that stop the release of new features. Other symptoms surface later still, during maintenance, and are reflected in increased maintainability and sustainment costs.

These symptoms are consequences of technical debt that manifest directly in the system. But consequences can reach further into the environment of which the system is a part. These consequences include an overall decline in quality that is visible to the end users and results in an increase in customer change requests or a decrease in market share as usage declines. When the consequences of debt are visible, they become easier for development teams to communicate to decision makers. Visible consequences also make it easier to get management buy-in for fixes, as Joe, a developer from Tethys, the global giant introduced in Chapter 3, summarizes:

I think that it is fairly easy to convince management when performance is really bad, when they are experiencing latency, when systems stop working, and when they see exceptions on the user interface.

But it is also a risk because by the time consequences become visible, the debt may also have a higher cost of remediation.

Some symptoms of technical debt surface earlier during software development, before they affect the entire system. These include an increasing number of issues and bugs, a decreasing rate of development productivity (for example, velocity) or cumulative flow, and increasing code quality concerns (for example, cyclicity, McCabe complexity). Development teams often are aware of these symptoms, even the debt itself, but do not have the mechanisms or incentives to communicate the issue. This is where the technical debt item can help.

Principle 4: Technical Debt Must Trace to the System

To reason about technical debt, estimate its magnitude, and offer information on which to base decisions, you must be able to anchor technical debt to explicit technical debt items that identify parts of the system: code, design, test cases, or other artifacts. A development organization also needs to recognize other forms of friction related to processes, people, and the development infrastructure. But these sources of friction are causes of technical debt; they are not the debt themselves.

When you trace technical debt to the system, start with your business context, assess artifacts across the technical debt landscape, and record the results as a technical debt description.

Writing a Technical Debt Description

A technical debt description captures where in the system the debt is located (the concrete system artifact) and the associated state of consequences that it causes in the system.

Recall that Atlas’s small startup team has just released the second version of its product, with the addition of French to a system that already supports English. The project team is now contemplating adding support for a third language.

A user story to describe the new feature request might take this form:

As a <stakeholder>, I want to <action with system> so that <benefit>.

For Atlas, this looks as follows:

As the Atlas Company, we want a Japanese-language version of our product so that we can increase market share and profit.

However, you need more than a user story to describe a technical debt item. You need to enhance the basic story by documenting some of the who, what, when, where, and why (also known as the five Ws, or 5W) to describe the problem so that you can make it visible to the entire project team and deal with it as you would any other issue on the backlog. A technical debt description is a user story that includes the five Ws that explain the associated technical debt.

Here is the 5W version of the Atlas team’s technical debt description:

As a developer (who), I need to pay down the debt where internationalization (what) is scattered over the code (where). The accumulating cost to add support for additional languages will soon outweigh the initial benefit of implementing the ad hoc solution of if-then-else statements for the first two releases to obtain another round of funding (when). There will be a long delay to support the next language, and the code will soon no longer support additional languages, especially languages using non-Latin characters (why).

You will need to collect your technical debt descriptions in what we call the technical debt registry, or registry for short. But you can use the same repository and tool that you are already using to manage work—your backlog.

Table 4.1 lists the essential fields to capture a technical debt item. They can easily be incorporated into your issue tracking process and technical debt registry.

Table 4.1 Technical debt description

Typically, to track technical debt, software development teams use whatever tool they routinely use to manage the project, such as an issue tracker system or a defect database. Most issue trackers include capabilities to create custom types and fields. We strongly recommend creating a type for technical debt items and tagging technical debt descriptions with a label, such as “techdebt,” if they are stored with user stories, defects, and other tasks.

If your team is disciplined, members can easily document the discussion of consequences and change requests as part of the detailed description field of an existing issue type. However, we often observe that software developers explain the what and the where as they incur or become aware of technical debt, but they fail miserably to highlight clearly the consequences of not fixing it, how the debt might grow over time, and a reasonable time to pay the debt if the fix must be deferred. Therefore, we recommend at a minimum creating a custom field and building the discipline to record the consequences of accumulating debt. That will help you assess how high the interest of the debt is growing. Such a simple practice has powerful operational benefits, such as retrieving all outstanding and possibly closed techdebt issues and assessing their importance and priority against the team’s resources.

Table 4.2 shows the technical debt description for Atlas after the second release of its product, with the addition of French to a system that already supports English. The project team is now contemplating a third language.

Table 4.2 Techdebt on internationalization

Understanding the Business Context for Assessing Technical Debt

Clearly understanding your business goals is essential for your team to establish criteria for selecting suitable techniques and tools to analyze your software, identify the technical debt, and document the technical debt items that matter to you. It will give you the proper starting point for managing technical debt.

The process to follow in uncovering technical debt is the same as for uncovering any other issue in your system. The challenge is to be disciplined enough to trace the concerns about the business goals to the relevant technical debt item and anchor it in the concrete system artifacts. We recommend starting with the business goals and concerns and anchoring the rest of the activities in those goals accordingly:

1. Understand your key business goals.

2. Identify key concerns/questions about the system related to your business goals.

3. Define observable qualitative and quantitative criteria related to your questions and goals.

4. Select and apply one or more techniques or tools to analyze your software for the criteria defined.

5. Document the issues you uncover as technical debt items.

6. Iterate through activities 2 to 5.

Understanding technical debt starts with enumerating the key business goals and the context of the business. The Atlas, Phoebe, and Tethys projects, like most if not all other software development projects, share a similar goal of reducing development costs, but their different contexts require executing this business goal in different ways.

The business goals have immediate bearing on key concerns related to the systems and, consequently, their source code, architecture, development, deployment, and delivery infrastructure. A clear enumeration of the business goals will help identify the criteria that you need to measure the concerns against. For example, if an organization has a business goal of reducing maintenance costs, some questions to ask about the source code can include “What is the degree of ease and speed required to enhance the software?” and “Does it make more sense to evolve the current system or develop a new one from scratch?” These questions should also take into account the team’s position on the technical debt timeline as that will influence the analysis strategies. For example, did the system acquire the debt recently? Or has the debt been accruing for a while, with yet unknown impact beyond the tipping point?

The next step then becomes a matter of defining measurement criteria to assess the answers to those questions. If the criteria are not met within reason, technical debt starts accumulating. The larger the gap in meeting these criteria, the greater the consequences. Technical debt increases the costs of change and rework, so these criteria should be input for assessing the impact of rework and cost of change.

Moreover, they allow the development team to select and apply analysis methods and tools to assess the artifacts accordingly. The final activity is to document the uncovered technical debt in the form of a technical debt item, while consolidating or linking to related issues, where possible.

After conducting these activities summarized in Figure 4.2, you will have a solid basis to reason about your technical debt. This brings you to the awareness point. You will also probably be able to determine whether you are beyond the tipping point on the technical debt timeline.

Ideally, the process of uncovering and managing technical debt is not a distinct, independent, one-time ceremonial activity but iterative and continuous. As development continues, you will incrementally revise and improve on identifying your technical debt items. Your business goals are not likely to change rapidly, but when they do change, check whether the traceability from goals to questions still holds or whether you need to add new questions and measurement criteria. And, most importantly, listen to team members and understand their concerns about where significant technical debt resides in the system to guide the assessment process and serve as a sanity check on the results. In addition, use the debt assessment process to avoid confusing the causes of technical debt with your current debt. Any sound approach to establishing solid technical debt management practices assumes that you are willing to assess the context and state of your software development project to identify the causes of your debt. In Chapter 11, “Technical Debt Credit Check,” we provide a technique to guide your efforts.

Assessing Artifacts Across the Technical Debt Landscape

Using business goals to drive the approach to identify technical debt items will take you throughout the technical debt landscape, to the code, to the architecture, and to the production infrastructure.

Technical Debt and Code

Technical debt is closely associated with the code itself, usually resulting from schedule pressure, lack of a documented programming standard, lack of tools, and developers’ errors. You will find plenty of resources on the Web about assessing your code for technical debt and its symptoms, including code quality standards, code smell examples, static code analyzers, security compliance checkers, and the like.

We will devote some time to code-related issues and technical debt in Chapter 5, “Technical Debt and the Source Code,” where we describe how to look beyond external quality issues such as defects and recognize when there are internal code quality issues that may require you to deal with technical debt. We explain how using static code analysis techniques can help you discover accumulated issues in source code that could result in technical debt and how to filter and prioritize the results to more effectively avoid unintentional buildup of technical debt.

Technical Debt and Architecture

Technical debt associated with the architecture results from key early decisions made in the design of the software product, such as choices of technology, programming languages, platforms, frameworks, middleware, and how to partition the system. The key difference between technical debt at the code level and technical debt at the architecture level is that the code is much more concrete, tangible, and visible. It can be easily explored and manipulated by using software tools.

Many architectural issues surface during deployment or run-time, even if the structure of the system and its code appear to be satisfactory. Not only is it typically more difficult to detect and assess architectural technical debt with tools, but also the cost and value associated with repaying the debt are larger and deeply intertwined in a complex network of structural dependencies.

Changing major architectural decisions is hard because these decisions have wide-ranging consequences for a software system—its functionality, present and future; its key quality attributes, such as modifiability, performance, security, and availability; and its code, which will need to change to support changes in these decisions. Such changes are usually made in large and long-lived systems, where the payoff will be significant.

Paradoxically, architectural debt is what happens to successful systems and companies: The size and scope increase, the business targets shift, companies merge, and acquisition leads to the merging of incompatible systems or systems that were built using different premises.

We’ll examine architectural technical debt more closely in Chapter 6, “Technical Debt and Architecture.”

Technical Debt and Production

Not all technical debt is strictly associated with the code or the architecture. Technical debt can also occur in production infrastructure. Current DevOps trends are increasing automation capabilities and tool support, blurring the boundaries between development and operations, and exposing deficiencies in the production process used by the development organization. As a consequence, the delivery environment is becoming a key software development artifact (also referred to as infrastructure as code). The production infrastructure contains significant code and has an architecture as well. If the build, test, deployment, or delivery strategy and accompanying tools do not align, evolving the system is harder and riskier.

We discuss technical debt that stems from the delivery process and production infrastructure in Chapter 7, “Technical Debt and Production.”

What Color Is Your Backlog?

Step back now and look at what we have added to the work of the software development team: a technical debt registry! At any point in time, the software development team is facing a set of “things” it has to do to make progress and tracking the backlog in one or more development tools. We sort the things to do into four categories, as shown in Figure 4.3.

There are things to do that are directly visible to the outside world:

• Adding functionality or features

• Fixing defects

Features add value to the product, whereas defects reduce the value of the product. These activities drive the organization, the sales, the success, and the customer satisfaction, and they are visible on the release schedule. But there are also things to do that are not directly visible and that are not directly driven by the outside world:

• Defining a software or system architecture and establishing and refining a production infrastructure

• Repaying technical debt

These activities have a cost to implement, though they do not directly add value. Technical debt is in this category: invisible to the outside world but indirectly adding negative value to the software product.

Figure 4.3 summarizes the four kinds of things to do that are managed in the software development product backlog. You can make them easier to distinguish by color coding the items.

• Features: Visible and positive value (green)

• Defects: Visible and negative value (red)

• Architecture and infrastructure: Invisible and positive value (yellow)

• Technical debt: Invisible and negative value (black)

Whether your backlog is managed in a single tool or not is another choice.

What Can You Do Today?

Simply providing a means to document known technical debt as technical debt items can be an eye-opening experience for a development team. It also creates a technical debt awareness mindset for the team, which helps reduce the rate of unintentional technical debt going forward. You can use this as a starting point for your registry and step back from there, articulating the overall business goals and deciding what further analysis may be needed. Use the following activities to start documenting your technical debt:

• Refine the “techdebt” category in your issue tracker into a technical debt description. Point at the specific software artifacts involved—code, architecture, or production infrastructure.

• Going further, possibly reorganize your backlog to explicitly “tag” the four categories of work shown in Figure 4.3.

• Create coding, architecture, and production infrastructure standards against which to measure technical debt.

• Standardize on a single form of “Fix me” or “Fix me later” comment in the source code to mark places that should be revised later. They will be easier to spot by using a tool.

Software developers can easily incorporate technical debt management into their daily tasks. For example, for issues that have related technical debt items, developers should incorporate remediation strategies as part of their “done” criteria. We have observed that creating such a technical debt management practice changes developer behavior. Developers disclose their technical debt in the form of discussions and comments by either explicitly referring to issues as technical debt or adding comments such as “fix me,” “workaround,” or “this is a hack.” Creating explicit technical debt items is an opportunity to enable a more proactive management strategy.

For Further Reading

The process we introduce in this chapter to identify technical debt items is strongly influenced by the Goal Question Metric (GQM) approach defined by Vic Basili, Gianluigi Caldiera, and Dieter Rombach (1994).

The motivation and benefits of writing a good technical debt item are similar to those related to writing a good bug report. A lot of research, especially by Microsoft, has underlined the importance of clearly written bug reports, which are likely to get more attention than poorly written ones. Work by Tom Zimmermann and colleagues (2010) provides empirical evidence in this regard. Much of this evidence also supports the importance of writing a good technical debt item. Li and colleagues (2015) have proposed a scenario-based approach to identifying actual architectural debt items.

Developer discussions and code comments include technical debt item ­discu­ssions. Bellomo and colleagues (2016), Bavota and Russo (2016), and Potdar and Shihab (2014) give some examples of such technical debt items. The technical debt description we discuss in this chapter systematizes this practice. Many authors have identified the concept of a technical debt registry, but in particular, see Narayan Ramasubbu and Chris Kemmerer (2017) at the University of Pittsburgh.

Shane Hastie interviewed Philippe Kruchten for InfoQ in 2010 on the four colors tactic for your backlog illustrated earlier in Figure 4.3.