On the Road

We’re kicking off our roadtrip in Montreal, where we currently live. Of the hundreds of towns and cities speckled along that shore, we have to pick just one to aim for. So, after a bit of research, we decide to aim for Vancouver.

Next, we need to familiarize ourselves with the highways leading directly into the city, and decide where we might want to stay upon arrival. We reach out to friends who’ve either lived in Vancouver or travel there frequently for recommendations. We land on Yaletown, a neighborhood known for its old warehouse buildings by the water. Now that your trip has a well-defined destination, we can start figuring out precisely how to get there.

At Work

To illustrate the many important concepts in this chapter, we’ll be using an example of a large-scale refactor at a fifteen-year-old biotechnology company we’ll call Smart DNA, Inc. Most of their employees are research scientists, contributing to a complex data pipeline cromprised of hundreds of Python scripts across a few repositories. The scripts are deployed to and executed on five distinct environments. All of these environments rely on a version of Python 2.6. Unfortunately, Python 2.6 has long since been deprecated, leaving the company susceptible to security vulnerabilities and preventing them from updating important dependencies. While relying on outdated software is inconvenient, the company has not prioritized upgrading to a newer Python version. It’s a massive, risky undertaking given the very limited testing in place. Simply put, this was the biggest piece of technical debt at the company for many years.

The research team has recently grown concerned about their inability to use newer versions of core libraries. Given that the upgrade is now important to the business, we’ve been tasked with figuring out how to migrate each of the repositories and environments to use Python 2.7.

The research team manages their dependencies using pip. Each repository has its own list of dependencies encoded in a requirements.txt. Having these distinct requirements.txt files has made it difficult for the team to remember which dependencies are installed on a given project when switching between projects. It also would require the software team to audit each file and upgrade them to be compatible with Python 2.7 independently. As a result, the software team decided that while it was not necessary, it would make the Python 2.7 upgrade easier for them (and simplify the researchers’ development process) to unify the repositories and thus unify the dependencies.

Our execution plan should clearly outline all starting metrics, target end metrics, with an optional, albeit helpful additional column to record the actual, observed end state. For the Python migration, the starting set of metrics was clear: each repository had a distinct list of dependencies, with each environment running Python 2.6. The desired set of metrics was equally simple: have each of the business’s environments running Python 2.7, with a clear, succinct set of required libraries managed in a single place. Table 4-1 shows an example where we’ve listed Smart DNA’s metrics.

On the Road

So, for our roadtrip, we do a quick search to see what the most direct route between Montreal and Vancouver looks like. Taking into account minimal traffic, it appears to take 47 hours if we were to leave Montreal and drive non-stop westward.

Figure 4-1. The most direct route between our address in Montreal and the Yaletown neighborhoood in Vancouver

We can determine a more reasonable lower bound for our trip by deciding how many hours we’re comfortable driving per day, and splitting that up evenly over the approximate 47 hours. If we want to commit to 8 hours of driving, it’ll take us just about 6 days.

Now that we’ve mapped the shortest possible path between the two points, we can start to pick out any major complications or over-arching strategies we want to change. One peculiarity with the drive in Figure 4-1 is that the vast majority of it travels across the United States, not Canada. If we want to restrict our drive to the area north of the 49th parallel, we’d be adding an extra hour or two to the trip. However, because it does reduce overall complexity of the trip (no need to carry around passport or worry about time wasted at a border crossing), we’ll opt to stay in Canada.

Figure 4-2. A slightly slower route restricted through Canada

At Work

Unfortunately, Google Maps for software projects doesn’t exist quite yet. So how do we determine the shortest path from now to project completion? We can do this a few different ways:

• Open up a blank document and for 15 to 20 minutes (or until you’ve run out of ideas), write down every step you can come up with. Set the document aside for at the very least a few hours (ideally a day or two), then open it up again and try to order each step in chronological order. As you begin to order the steps, continue to ask yourself whether it is absolutely required in order to reach the final goal. If not, remove it. Once you have an ordered set of steps, reread the procedure. Fill in any glaring gaps as they arise. Don’t worry if there are any steps that are terribly ill-defined, the goal is only to produce the minimum set of steps required to complete your project. This won’t be the final product.

• Gather a few coworkers who are either interested in the project or you know will be contributing. Set aside an hour or so. Grab a pack of sticky notes and a pen for each of you. For anywhere between 15 to 20 minutes (or until everyone’s pens are down), write down every step you think is required, each on individual sticky notes. Then, have a first person lay out their steps in chronological order. Subsequent teammates go through each of their own sticky notes and either pair them up with their duplicates, or insert them into the appropriate spot within the timeline. Once everyone’s organized all of their notes, go through each individual step and ask the room whether they believe that the the step is absolutely required in order to reach the goal. If not, discard it. The final product should be a reasonable set of minimal steps. (You can easily adapt this method for distributed teams by combining all ideas individually brainstormed steps into a joint shared document. Either way, the final output of the exercise should be a written document that is easy to distribute and collaboratively improve.)

If neither of these options work for you, that’s alright! Use whatever method you find most effective. So long as you are able to produce a list of steps you believe model a direct path to achieving your goal, no matter how ill-defined they might be, then you’ve successfully completed this critical step.

Our team at Smart DNA gathered into a conference room for a few hours to brainstorm all of the steps required to get everything running a newer version of Python. On a whiteboard, we started out by drawing a timeline. On the far left was their starting point, and on the far right their goal. Teammates alternated listing important steps along the way, slotting them in along the line. A subset of the brainstormed steps are as follow:

• Build a single list of all of the packages across each of the repositories manually.

• Narrow-down the list to just the necessary packages.

• Identify which version each package should be upgraded to in Python 2.7.

• Build a Docker container with all of the required packages.

• Test the Docker container on each of the environments.

• Locate tests for each repository; determine which tests are reliable.

• Merge all of the repositories into a single repository.

• Choose a linter and corresponding configuration.

• Integrate linter into continuous integration.

• Use linter to identify problems in the code (undefined variables, syntax errors, etc.)

• Fix problems identified by the linter.

• Install Python 2.7.1 on all environments and test.

• Use Python 2.7 on a subset of low-risk scripts.

• Roll out Python 2.7 to all scripts.

We can see from our subset that some can be parallelized, or reordered, and others should be broken down into further detail. At this point in the process, our focus is on getting a rough sense of the steps involved; we’ll refine the process throughout the chapter.

On the Road

For the stretch of the trip between Winnipeg and Vancouver, we ask some friends and family for recommendations of sights to see and things to do. After weighing their suggestions with our own interests, we come up with a rough itinerary which includes everything from camping, museum visits, tasty pitstops, and a few visits to extended family. But at no point do any of these points of interest take us radically off-course.

Figure 4-3. Our rough itinerary

At Work

We can apply similar tactics to narrow-in on our milestones for our refactoring effort. For each of the steps we brainstormed previously, we can ask ourselves these questions:

1. Does this step feel attainable in a reasonable period of time?

Let’s refer back to our previous example, outlined in “At Work”. A logical, feasible milestone might be to combine each of the distinct repositories into a single repository for convenience. The software team at Smart DNA anticipate it’ll take six weeks to properly merge the repositories without disrupting the research team’s development process. Given the software team is accustomed to shipping at a quicker pace, and they’re concerned about morale if they set out to merge the repositories too early-on in the migration, they decide on a simpler initial milestone: generating a single requirements.txt file to encompass all package dependencies for each of the repositories. By taking the time to reduce the set of dependencies early, they are simplifying the development process for the research team, taking a substantial step towards enabling the merging of the repositories, and all of that well before the migration to Python 2.7 is complete.

2. Is this step valuable on its own?

When choosing major milestones, we should optimize for steps that demonstrate the benefits of the refactor early and often. One way to do that is to focus on steps that, upon completion, derive immediate value for other engineers. This should hopefully increase the morale of both your team and other engineers impacted by your changes.

When scoping out the Python migration, we noticed that none of the repositories used any continuous integration to lint for common problems in proposed code changes. We know that linting the existing code could help us pinpoint problems we risk encountering when executing it in Python 2.7. We also know that enabling a simple, automatic linting step could promote better programming practices for the entire research team for years to come. In fact, it seems so valuable that under different circumstances, instituting an automatic linting step might’ve been a project all on its own. This indicated to us that it was a meaningful, significant intermediate step.

3. If something comes up, could we stop at this step and pick it back up easily later?

In a perfect world, we wouldn’t have to account for shifts in business priorities, incidents, or re-orgs. Unfortunately, these are all a reality of working, regardless of the industry. This is why the best plans account for the unexpected. One way of accounting for disruptive changes is by dividing our project into distinct pieces that can stand alone in the unlikely event we need to pause development.

With our Python example, we could comfortably pause the project after fixing all errors and warnings highlighted by the linter, but before beginning to run a subset of scripts using the new version. Depending on how we tackled the refactor, pausing half-way through could risk confusing the researchers actively working in the repository. If the refactor needed to be paused for whatever reason, pausing immediately before we started running a subset of scripts using to Python 2.7 would be safe; we would still have made considerable progress towards our overall goal and have a clean, easy place to pick things back up when we are next able to.

After taking the time to highlight strategic milestones, we reorganized our execution plan to highlight these steps and grouped subtasks accordingly. The more refined plan is as follows:

• Create a single requirements.txt file.

  • Enumerate all packages used across each of the repositories.

  • Audit all packages and narrow-down list to only required packages with corresponding versions.

  • Identify which version each package should be upgraded to in Python 2.7.

• Merge all of the repositories into a single repository.

  • Create a new repository.

  • For each repository, add to the new repository using git submodules.

• Build a Docker image with all of the required packages.

  • Test the Docker image on each of the environments.

• Enable linting through continuous integration for the mono-repo.

  • Choose a linter and corresponding configuration.

  • Integrate linter into continuous integration.

  • Use linter to identify logical problems in the code (undefined variables, syntax errors, etc.)

• Install and roll out Python 2.7.1 on all environments.

  • Locate tests for each repository; determine which tests are reliable.

  • Use Python 2.7 on a subset of low-risk scripts.

  • Roll out Python 2.7 to all scripts.

Hopefully after you’ve identified key milestones, you have a procedure that feels balanced, achievable, and rewarding. It’s important to note, however, that this isn’t a perfect science. It can be quite difficult to weigh required steps against one another according to the effort they involve and their relative impact. We’ll see an example of how we decided to weigh each of these considerations when strategically planning a large-scale refactor in both of our case study chapters, Chapter 10 and Chapter 11.

Finally, once we have our end state and our key milestones, we want to interpolate our way through the intermediate steps between our end state and each of our strategic intermediary milestones. This way, we maintain focus on the most critical pieces, all the while building out a detailed plan.

This is where we can spend some time figuring out whether certain portions of the refactor are order-agnostic; that is, they can be completed at any point, with very few to no prerequisites. For example, let’s say you’ve identified a few key milestones for your project; we’ll call them A, B, C, and D. You notice that you need to complete A before tackling B or C, and B needs to be completed before you tackle D. You have three options with regards to C: you could parallelize development on C at the same time as D, complete C then D, or complete D followed by C.

If you have a hunch that B is going to be a difficult, lengthy milestone and D looks just as challenging, you might want to break things up by putting milestone C between B and D. This should help boost morale and add some pep to the team’s momentum as you work through a long refactor. On the other hand, if you think that you can comfortably parallelize work on milestone C and D, and wrap up the project a little bit sooner, then that might be worthwhile option as well.

It all comes down to balance: balancing time and effort associated with each requisite step, all the while considering their impact on your codebase and the well-being of your team.

Dark Mode/Light Mode

We can compare pre-refactor and post-refactor behavior by employing what we’ve coined at Slack as the light/dark technique. Here’s how it works.

First, implement the refactored logic separately from the current logic, with the goal being to remove the current logic altogether. Example 4-1 depicts this step on a small scale.

// Linear search; this is the old implementation
function search(name, alphabeticalNames) {
  for(let i = 0; i < alphabeticalNames.length; i++) {
    if (alphabeticalNames[i] == name) return i;
  }
  return -1;
}

// Binary search; this is the new implementation
function searchFaster(name, alphabeticalNames) {
  let startIndex = 0;
  let endIndex = alphabeticalNames.length - 1;

  while (startIndex <= endIndex) {
    let middleIndex = Math.floor((startIndex+endIndex)/2);
    if (alphabeticalNames[middleIndex] == name) return middleIndex;

    if (alphabeticalNames[middleIndex] > name) {
      endIndex = middleIndex - 1;
    } else if (alphabeticalNames[middleIndex] < name) {
      startIndex = middleIndex + 1;
    }
  }

  return -1;
}

Then, as shown in Example 4-2, relocate the logic from the current implementation into a separate function.

// Existing function now calls into relocated implementation
function search(name, alphabeticalNames) {
  return searchOld(name, alphabeticalNames);
}

// Linear search logic moved to a new function.
function searchOld(name, alphabeticalNames) {
  for(let i = 0; i < alphabeticalNames.length; i++) {
    if (alphabeticalNames[i] == name) return i;
  }
  return -1;
}

// Binary search; this is the new implementation
function searchFaster(name, alphabeticalNames) {
  let startIndex = 0;
  let endIndex = alphabeticalNames.length - 1;

  while (startIndex <= endIndex) {
    let middleIndex = Math.floor((startIndex+endIndex)/2);
    if (alphabeticalNames[middleIndex] == name) return middleIndex;

    if (alphabeticalNames[middleIndex] > name) {
      endIndex = middleIndex - 1;
    } else if (alphabeticalNames[middleIndex] < name) {
      startIndex = middleIndex + 1;
    }
  }

  return -1;
}

Then, transform the previous function into an abstraction conditionally calling either implementation. During dark mode, both implementations are called, the results are compared, and the results from the old implementation are returned. During light mode, both implementations are called, the results are compared, and the results from the new implementation are returned. As can be seen in Example 4-3, repurposing the existing function definition allows us to modify as little code as possible. (Though not depicted in our example, to prevent performance degradations as part of the light/dark process, both the old and new implementations should be executed concurrently.)

// Existing function now an abstraction for calling into either implementation
function search(name, alphabeticalNames) {
  // If we're in dark mode, return the old result.
  if (darkMode) {
    const oldResult = searchOld(name, alphabeticalNames);
    const newResult = searchFaster(name, alphabeticalNames);

    compareAndLog(oldResult, newResult);

    return oldResult;
  }

  // If we're in light mode, return the new result.
  if (lightMode) {
    const oldResult = searchOld(name, alphabeticalNames);
    const newResult = searchFaster(name, alphabeticalNames);

    compareAndLog(oldResult, newResult);

    return newResult;
  }

  return search(name, alphabeticalNames);
}

// Linear search logic moved to a new function.
function searchOld(name, alphabeticalNames) {
  for(let i = 0; i < alphabeticalNames.length; i++) {
    if (alphabeticalNames[i] == name) return i;
  }
  return -1;
}

// Binary search; this is the new implementation
function searchFaster(name, alphabeticalNames) {
  let startIndex = 0;
  let endIndex = alphabeticalNames.length - 1;

  while (startIndex <= endIndex) {
    let middleIndex = Math.floor((startIndex+endIndex)/2);
    if (alphabeticalNames[middleIndex] == name) return middleIndex;

    if (alphabeticalNames[middleIndex] > name) {
      endIndex = middleIndex - 1;
    } else if (alphabeticalNames[middleIndex] < name) {
      startIndex = middleIndex + 1;
    }
  }

  return -1;
}

function compareAndLog(oldResult, newResult) {
  if (oldResult != newResult) {
    console.log(`Diff found; old result: ${oldResult}, new result: ${newResult}`);
  }
}

Once the abstraction has been properly put in place, start enabling dark mode, (i.e. dual code path execution, returning the results of the old code). Monitor any differences being logged between the two result sets. Track down and fix any potential bugs in the new implementation causing those discrepancies. Repeat this process until you’ve properly handled all discrepancies, enabling dark mode to broader groups of users.

Once all users have been opted into dark mode, starting with the lowest-risk environments first, begin enabling light mode to small subsets of users (i.e. start returning data from the new code path). Continue logging any differences in the result sets; this can be useful if other developers are actively working on related code and risk introducing a change to the old implementation that is not reflected in the new implementation. Continue to opt braoder groups of users into light mode, until everyone is successfully processing results from the new implementation.

Finally, disable executing both code paths, continuing to monitor for any reported bugs, and remove the abstraction, feature flags, conditional execution logic, and, once the refactor has been live to users for an adequate period of time (whatever that might be for your use case), remove the old logic altogether. Only the new implementation should remain where the old implementation once was. See Example 4-4 for an example.

// Binary search; this is the new implementation
function search(name, alphabeticalNames) {
  let startIndex = 0;
  let endIndex = alphabeticalNames.length - 1;

  while (startIndex <= endIndex) {
    let middleIndex = Math.floor((startIndex+endIndex)/2);
    if (alphabeticalNames[middleIndex] == name) return middleIndex;

    if (alphabeticalNames[middleIndex] > name) {
      endIndex = middleIndex - 1;
    } else if (alphabeticalNames[middleIndex] < name) {
      startIndex = middleIndex + 1;
    }
  }

  return -1;
}

Smart DNA’s Rollout

With the refactor at Smart DNA, the greater risk was in migrating each of the repositories’ many dependencies to versions compatible with Python 2.7, not with running the existing code itself using the newer Python version. The software team decided that they would first a few preliminary tests, setting up a subset of the data pipeline in an isolated environment, installing both versions of Python, and running a few jobs using the new dependency file in the 2.7 environment. When they were confident with the results of their preliminary tests, they would slowly, carefully introduce usage of the new set of dependencies in production.

To limit the risk involved, the team audited each of the jobs that make up the researchers’ data pipeline and grouped them according to their importance. Then the engineers chose a low-risk job with the fewest downstream dependencies to migrate first. They worked with the research team to identify a good time to swap the configuration to point to the new requirements.txt file and new Python version. Once the change had been made, the team planned to monitor logs generated by the job to catch any strange behavior early. If any problems crept up, the configuration would be swapped back to its original version while the software team worked on a fix. When the fix was ready, the team would repeat the experiment. As part of their rollout plan, the team required that the configuration change sit in production for a few days, allowing for the job to run successfully on a dozen different occasions before moving onto a second job.

After the second job was successfully migrated, the software team would opt-in all low-risk jobs into the new configuration. They would then repeat the process for the medium-risk jobs. Finally, for the most critical jobs, the team decided to migrate each of these individually due to their importance. Again they would wait a few days before repeating the process for the next job, and so on. In all, the team determined it would take nearly two months to migrate the entire data pipeline to the new environment. While this might sound like a grueling process, both the software and research teams agree that it is necessary so as to sufficiently reduce risk. It gives everyone adequate opportunity weed out problems at small increments early, ensuring that the pipeline remains as healthy as possible throughout the entire process.

Interpolating Goal Metrics to Intermediate Milestones

Recall our Table 4-1 where we showed our starting metrics alongside our final goal metrics. For each of our milestones, if the start and end metrics are applicable to our intermediate stages, we can add an entry highlighting which metrics we expect to change and by how much if our metrics lend themselves well to intermediate measurements during the refactor.

End goal metrics that might lend themselves better to intermediate measurements include complexity metrics, timings data, test coverage measurements, and lines of code. Be warned, however, that your measurements might trend worse before they trend better again! Consider the “Dark Mode/Light Mode” approach, for instance; having two code paths, both of which do the same thing, will definitely lead to a tangible uptick in complexity and lines code.

Unfortunately, with our Python migration example, the language version remains the same throughout most of the project. Only once the team has reached the stage of rolling out the new version to each of the company’s environments we start to see our metrics change. To measure progress, we will need to come up with a different set of metrics to track throughout development.

Distinct Milestone Metrics

As the previous section showed, not all end goal metrics will lend themselves well showing intermediate progress. If that happens to be the case, we’ll still need at least one helpful metric to indicate momentum. The metrics we choose might not directly correlate to our final goal, but they’re important guideposts along the way.

There are a number of simple options. Say at Smart DNA we’ve set up continuous integration and enabled the linter to warn on undefined variables. We can use the number of warnings remaining as a metric to measure their progress within the scope of that step. Table 4-2 shows each of the major milestones we brainstormed in ??? with their corresponding metric. (Note that the starting value for the linting milestone is an approximation. The team provided an estimate here by running pylint with the default configuration across the three repositories and summing up the number of warnings generated.)

Transparency

Transparency helps build trust across teams. If you’re up front with other engineers at the company, then they’re more likely to be engaged and invested in your effort. It should go without saying, but if your team drafts up a plan and starts executing on a refactor that impacts a number of groups without warning, you risk dangerously eroding that relationship.

You must be mindful of the fact that your proposed changes could drastically change code that they own, or affect important processes they maintain. With Smart DNA’s Python migration, we’re combining three different repositories into one. This is a significant change for any developers or researchers working in any of these repositories. The teams impacted should be adequately forewarned that their development process is going to change.

The refactor also risks impacting other teams’ productivity. For instance, if we’re proposing to combine all required packages into a single, global requirements.txt file, we may need other teams’ help getting their changes reviewed and approved. We might even inquire about borrowing engineers from other teams to help out with the refactor (see Chapter 6 for a more in-depth look at how to recruit teammates).

Similarly, you have to make sure that your plans align with impacted teams. If you’re planning to modify code owned by another team just as they are planning to kick off development on a major feature (or perhaps their own hefty refactor), you will need to coordinate to make sure you aren’t stepping on each others’ toes.

Perspective

The second reason to share your plan with other teams is to get their perspective. You’ve done the research to define the problem and draft a comprehensive plan, but are the teams that risk being impacted by your proposed changes supportive of your effort? If they do not believe that the benefits of your refactor outweigh the risks and inconvenience to their team, then you may need to reconsider your approach. Perhaps you could convey the benefits in a more convincing manner, or find a way to reduce the level of risk associated with the current plan. Work with the team to figure out what would make them more comfortable with your plan. (You can use some of the techniques outlined in the next chapter to help out.)

If you’re working to refactor a complex product, there are likely a number of edge cases you haven’t considered. Just getting that second (and third and fourth) set of eyes can make a huge difference. Let’s say that while auditing the packages used by the research team at Smart DNA, we fail to notice that some researchers have been manually updating a requirements.txt file on one of the machines directly, rather than making their changes in version history and deploying the new the code. When we share our plan with the researchers, they’ll point out that they typically update their dependencies on the machine itself and that the software team should verify the version there rather than the one checking into their repository. That insight would have saved our software team a great deal of pain and embarrassment had we started executing on the project without consulting the researchers first.

Remember that while it’s important to get stakeholders’ opinions about your plan before kicking off execution, nothing is set in stone at this stage. Your plan will likely change throughout the duration of the refactor; you’ll run into an unexpected edge case or two, maybe spend more time than anticipated solving a pesky bug, or realize part of your initial approach simply won’t work. At this stage, we are seeking out others’ perspectives mostly as a means of ensuring transparency with others and weeding out the blatantly obvious problems early. We’ll discuss how to keep these stakeholders engaged and informed as your plan evolves in Chapter 7.