Experts of Many Trades

Next, highlight any names that pop up more than once. There isn’t much overlap in our example set but we notice that Jesse might be a good candidate for two of the five items. Depending on a number of different factors including the size of your engineering organization, and the average tenure of an engineer, there can be quite a few senior individuals who have a breadth of expertises, many of which could be helpful to your refactor. Conferring with someone who happens to be an expert on multiple relevant topics can be helpful on many fronts.

First, it can help us decrease the total number of people we’ll need to coordinate with in order to complete our project. Coordinating a large project on a single team can be difficult, never mind coordinating a large project involving multiple developers across a number of different teams. Each contributor not only has to be pitched on the effort and brought up to speed, but they must also adapt to your team’s development process (i.e. weekly or daily stand-ups, monthly retrospectives, etc.) It can take a considerable amount of time and effort before everyone is well-aligned and operating at a good pace.

Second, experts who happen to have a deep understanding of multiple important aspect of the project likely have a strong perspective on how these pieces work together. This can be very valuable insight shared by very few other engineers at the company. Given our sample list of experts in Table 6-1, Jesse is likely one of those individuals. From our conversations with them, we know that they’ve worked closely with the Release & Deploy Team over several months to help them build a percentage-based release system for two important services at company. We also know that after that project, they moved to the internal tools team where they’ve worked to improve the availability of automated testing environments. Jesse is just one of those engineers who’s been at the company for a while, worked on a laundry list of projects, and has keen insight into how each of these pieces work together.

Unfortunately, people like Jesse can be quite busy (probably because they’re providing input on a number of different projects as an SME in addition to leading a few of their own). If they are not available to help in a regular capacity but you believe their unique knowledge is critical to the refactoring effort, offer to have them review your execution plan. I’ve found their input particularly helpful with regards to verifying my least confident time estimates. If you’re looking for an expert to be actively involved in your project, they’ll be able to suggest another expert or two to replace them.

If very few names (or none at all) overlap and your list of required expertises is quite lengthy, not to worry! You can still successfully execute on a large-scale refactor with just a handful of resourceful individuals.

Revisiting Active Contributors

For me, a good rule of thumb is to limit the number of active contributors to the size of team you’ve been most comfortable working with in the past. If you’ve been on successful product engineering teams of six, then limit your team to six active contributors. Everyone’s experience working with different teams at different companies is a little bit different; you know yourself and your preferred working conditions best, so go with what you know to be most effective. Large refactoring projects are plenty complex enough from both a process and technical standpoint; don’t let your team be yet another potential curveball.

If your list of active contributors feels too long, review your list and see if there are any expertises for which you can instead seek out the help of a subject matter expert. Coordinating with SMEs come at a much lower coordination cost because they are only consulted on an ad-hoc basis. We’ll cover some strategies for effectively communicating with SMEs in Chapter 7.

Biases in Our Expert List

If we happen to know someone who could be a valuable expert on one or more of the items on our list, we might ask them for their help directly. Chances are, they’ll be more than happy to help out. After all, asking someone you know is probably the most convenient option. If you’ve worked together previously, you’ll be able to establish a cadence that works well for both of you pretty quickly and begin making some salient progress early.

Asking a colleague for help directly can have its drawbacks, however. Software engineers are notoriously bad at estimating how much time and effort a task will take. This is often a consequence of the relentless optimism that being a software engineer requires. When something seems like a small ask, sometimes our colleagues can be a little too quick to say yes, not taking much time to properly scope the commitment. They may only realize well after the project’s kicked off that they’ve said yes to a few too many things and are now struggling to juggle it all. (I’ve been that person and trust me when I say that saying yes to too many things is just as unhelpful as saying no to everything.)

Another problem with asking a colleague for help directly is that you might overlook others who are better suited for the role. We all suffer from a number of biases we must consciously work to counteract. One such bias is recency bias, where we tend to recall things we’ve seen more recently more quickly. We are more likely to list a colleague as a good potential expert if we’ve heard their name or spoken to them more recently. We need to mindful of that bias before we finalize our expert list and take a minute to question whether each expert truly is the best one for the job or if we just happened to see their name CC’ed on an email a few days ago. If we think there’s a chance there might be a more qualified candidate, we should do our research and consider reaching out to a team rather than an individual. Managers of expert teams can vet your request for help to each of their developers and gauge interest. Great managers will identify those on their team who could contribute meaningfully but also would benefit the most from the visibility and career growth of contributing to your refactoring effort.

It’s also important not to confuse expertise with seniority. While Frankie might not be the engineer with the most industry experience nor longest tenure at the company, but they’ve made significant contributions over the past few months and you’re confident they can answer your questions and offer valuable insights in code reviews. Sometimes, the most senior person might not be the best co-conspirator; oftentimes these developers are very busy leading demanding projects of their own and their time is more valuable elsewhere. Your project might also be a prime opportunity for someone to get valuable exposure and visibility beyond their immediate team. Refactoring (particularly refactoring at scale) can be a tricky endeavor but it’s not one that engineers with just a years (or even months) of experience can’t meaningfully contribute to and learn from.

Owners

This kind of team owns a particular piece of the product and are refactoring code that they primarily own or are responsible for. This code interfaces with other teams’ code at some number of boundaries. At those boundaries, they must figure out whether to make the changes themselves, or coordinate with the engineers whose code they are interfacing with to make the necessary changes.

Say, for instance, you work at a company with three broad engineering groups: developer productivity, infrastructure, and product engineering. You are on the team responsible for testing libraries and tooling for your application within the developer productivity group. While it’s always great that engineers across the organization are writing more unit tests, you’re worried that the amount of time required to run them all has begun to hinder everyone’s ability to ship code quickly. With performance in mind, you start tracking timings for individual unit tests, gathering metrics on how long certain operations like setting up a complex mock state take. Your team decides to kick off a refactor focusing their efforts on speeding up the mock set-up process. While benchmarks for the new version show a drastic improvement, existing unit tests will need to be migrated to use the new set-up logic in order to benefit from the speed-up. There’s two main ways to go about the migration:

Proposed Approach

1. Have your team identify a few simple tests which might benefit the most from the migration. Reach out to the prodct engineering teams to get additional context on which tests they deem to be the most high-impact.

2. Start with Option 1 (see “Option 1: One team does migrates all tests” above). Begin migrating the tests manually and document the process thoroughly. (If the tests are clearly owned by a specific team, either give that team a heads up or work with them to complete the migration.)

3. For the migrated test files, run benchmarks to clearly demonstrate the performance impact. Document those, too.

4. Develop a code modification tool to automatically migrate a few simple cases. Run the modification tool on small, logical subsections of the testing suite until all candidate tests have been migrated.

5. Kick off Option 2 (see “Option 2: Teams update their own tests” above). Evangelize the new mocking system by highlighting the benefits and pointing engineers to sample migrations. Spin up office hours to answer questions and troubleshoot with engineers in person. Consider organizing regular “jam sessions” where engineers across organization can join in with your team to crank out a few migrations.

6. Work with teams to set quarterly objectives around improving the performance of their tests; if they’ve committed to being evaluated on participating in the effort, there’s a better chance it’ll get done.

Clean-Up Crews

Some larger engineering organizations have teams dedicated to improving developer productivity. The range of the kind of work that these teams take on can be quite wide: they provision and manage development environments; they write editor extensions and scripts to automate repetitive tasks; they build tooling to help developers better understand the performance implications of their proposed code changes; they maintain and expand upon the core libraries all product engineers depend on (including logging, monitoring, feature flags, etc). More often than not, the developer productivity teams that continue to work alongside product developers within the boundaries of the application end up taking on the role of the “clean-up crew”.

“Clean-up crews” take on the important (but often thankless) work of identifying and shedding cruft and antipatterns from a codebase, and establishing better, more sustainable patterns in their stead. These teams are usually made up of engineers who care deeply about code health and want their fellow product engineers’ to have an easy time developing, testing, and ultimately shipping new features. Seeing other developers at the company use (and appreciate) their libraries and tools is what gives them the greatest satisfaction.

Typically, these teams take on hefty refactors for two reasons. First, the team’s breadth of knowledge of the codebase is unparalleled. Because these crews are owners of core, functional libraries, they tend to have at least some exposure to almost every single corner of an application. This is especially true of teams that work inside monolothic codebases. Second, the team values developing ergonomic solutions that are accessible to everyone, regardless of team or seniority; they have valuable experience thinking about what kinds of interfaces strike the right balance between extensible and practical. If the main driving motivation behind the project is to boost developer productivity (and keep it there), then this is the perfect team. A third, implicit reason, is that by having the “clean-up crew” map out and execute the refactor, product development teams can continue to focus on feature development relatively undisturbed.

Unfortunately, clean-up crews are not sustainable. When these groups are productive, other engineering teams, typically feature-development teams, feel less of a responsibility to commit to doing important maintenance work. Over time, the clean-up crews accrue an insurmountable amount of work, slowly burning out their team members. As a result, these teams are usually short-lived or have high turnover. Furthermore, the teams shrugging maintenance work gradually lose the muscle memory associated with supporting features long-term. Throwing another large-scale refactor their way might not be a viable option.

Metrics

In Chapter 3 we explored a variety of different ways in which we could quantify the current state of the application before embarking on our refactoring journey. Chapter 4 discussed how to develop a thorough plan of action, complete with solid set of success metrics determined from the initial measurements taken using the methods outlined in Chapter 3. These metrics can help you build a convincing argument for getting help with your refactoring endeavor.

Typically, these kinds of pitches are most effective with the more skeptical experts and those that are most data-driven in their regular work. These are the engineers that are always asking questions; they actively monitor the p95 response times of APIs their team is responsible for maintaining; they’re the first ones to notice an uptick in the average number of database operations hitting a specific shard. Appeal to their analytical side with your own metrics and you might secure yourself a new expert.

First, articulate why the metrics you’ve chosen are good indicators of the problem. Take the time to carefully explain the relationship between the problems you hope to fix, how you chose to quantify them, and the initial statistics you’ve gathered. Choose simple metrics first, and then augment your case with additional supporting data points. If you’ve acquired or generated any visuals that help illustrate the problem, reference them; even those we think of as “numbers people” appreciate an explanatory graph or chart every so often.

Then, juxtapose the starting metrics with your defined success metrics, staring with the desired end state. Afterwards, you can walk the expert through the evolution of the metrics throughout the effort, from start to finish. Justify that your success metrics decisively show that the refactor was successful and that they are sufficiently ambitious but achievable.

Generosity

There’s an odd cognitive dissonance known as the Benjamin Franklin effect: you have a better chance at getting someone to like you if you ask them for a favor than by doing a favor for them. To give an example, say Charlie asks a favor of Dakota. Dakota happily obliges. The phenomenon follows that Dakota is more likely to do another favor for Charlie than if Charlie had done one for them. The idea is that people help others because they like them, even if they actually don’t, because their minds struggle to maintain logical consistency between their actions and perceptions.

Engineers working closely with the code you aim to improve are more likely to understand its pain points. They probably know at least a handful of other engineers (either on their immediate team or in the organization at large) that experience these same pain points regularly. If this expert is the kind of coworker that has a finger on the pulse when it comes to the health of the codebase and the engineering morale surrounding it, then there is a strong chance that they have a great deal of empathy for their teammates and you can successfully appeal to their inner altruist.

Ask the expert about the things they’ve heard their teammates complain about. Make a mental (or written) note of the specific pain points that the refactor intends to fix. Once you’ve commiserated on the difficulties of the code in its current state, list each of the problems they mentioned and walk through your proposed solution. There might be a few problems that you don’t yet have an explicit solution for, and that’s perfectly ok! In fact, this is precisely why you reached out to this expert; you seek their perspective on the problems you’re trying to solve. Make it clear to them that these are the kinds of insights that they could provide to the project. Finally, emphasize that their contributions would concretely make their coworkers’ lives (at least a little bit) more pleasant and more productive. Point to the expected benefits of the refactor and summarize the success metrics (because a multi-faceted pitch is ultimately a stronger pitch).

Opportunity

If the expert you’re pitching is looking for a good career advancement opportunity or a chance to be more visible to other parts of the engineering organization, a large-scale refactoring project can be the perfect line item on their resume. Earlier in the chapter, we mentioned that some managers might want to identify team members who might both be an asset to the project while also gaining valuable visibility within the broader engineering organization; if they’ve provided you with a few names, make sure to have a conversation with them about what kind of growth and visibility these individuals need in order to get to the next level.

When you sit down with the expert, have a conversation about what types of growth opportunities they’re looking for. Hopefully the engineer and their manager are aligned on what behaviors they need to exemplify or projects they need to drive to grow in their career, but that is not always the case. If you want to decisively convince the engineer to join you, all the while setting them up for success, taking the time to coalesce the manager’s expectations with those of the engineer is the best approach. From the combined input, take the time to identify a few key portions of the refactor that this expert could contribute to in a way that demonstrates the key characteristics they’re looking for. When you meet with them, walk them through each of the milestones and highlight the contributions they can make. Describe how you hope each of these contributions can help them achieve their goals. Be careful to keep an open dialogue, and be open to their input. You’re not in their shoes, nor are you their manager, so their perspective on how they can be best set up for success might differ from your own.

Bartering

If all else fails, be ready to barter. Bartering can be a great way to acquire the resources you need to successfully finish your project, with some sort of commitment in return. Typically, bartering doesn’t happen between yourself and another engineer but rather between your own manager and the manager of the team you’re seeking help from. The promise you make in return can vary; it’s all about finding what the other manager values most, and finding an adequate alternative you’re happy to provided in exchange. Here are just a few examples:

• Say your team has open headcount and the team you want to recruit experts from is in desperate need for additional headcount. If your organization allows it, and you are comfortable giving up some of your available headcount, you could provide the team with the headcount it needs in exchange for one or two engineers to contribute actively to the refactoring effort.

• On the off chance that your teams have compatible feature ownership, you could barter taking additional ownership of some components the other team has been wanting to shed. Oftentimes when teams have unclear or debated boundaries, there tends to be ares that become entirely unowned or tossed between the two teams frequently (which essentially leads them to be unowned). In exchange for help, your team could agree to decisively own those features or components for a set period of time (a few quarters or a year).

• If your engineering organization has communal responsibilities (completing a certain number of hours of customer support or participating in interviews), you can offer that your team take on some (or all) of the expert team’s responsibilities for a defined period of time after the refactoring effort has wrapped up. (Ideally, you agree for the exchange to kick off only after the project has finished or when it is near completion because any time taken away from it will only make it drag on at the detriment of everyone involved.)

Be aware that with bartering, either party can fall through on their promise if priorities shift within the duration of the refactoring effort. Re-orgs at companies of any size can render these agreements void due to shifts in management or feature ownership. Managers or engineers leaving the company or switching teams can also impact any pre-arranged agreements. The longer the refactor is ongoing, the greater chance the agreement might fall through for whatever reason.

Repeat

If you’re not able to convince the first name for each expertise, don’t worry! This is why brainstorming multiple names early-on is important. Ideally you’re able to secure an expert for each expertise, and if you have trouble coming up with more candidates, consider reaching out to those who’ve turned down the opportunity for any additional recommendations; they might be able to give you a name or two.

If you’re unable to secure an expert for an expertise that you won’t need initially, consider pausing the search and picking it back up once you reach the stage where it becomes necessary. Experts who were previously on the fence might be convinced to join if they are able to see sufficient progress and maybe the hint of a positive shift in the initial metrics. Refactoring can be a little bit like snowballs rolling down a snow hill; as it gains momentum, it affects a greater and greater surface area, gathering up more and more resources as it nears completion.

Realistic Scenario

The most realistic scenario is one where you end up with a small handful of committed experts and teammates. At smaller companies experiencing a great deal of growth, everyone wears more than one hat and every engineer has a full plate, so it’s unlikely you’ll be able to get an expert to fill each of your desired expertises. At larger, more stable companies, you might have a difficult time getting folks from other teams to commit to helping you out simply due to organizational boundaries and priorities; just because someone is an expert in something you’ll need context on to successfully complete your refactor doesn’t mean that it is that expert or that expert’s management chain’s top priority.

Regardless of who you were able to convince before kicking off development, you’re in a good spot if you’ve managed to gather a core team of at least a few engineers for the earliest portions of the project. After all, the team you start with might not be the team you end with because the support and expertise you need to complete the first few milestones isn’t necessarily the support you’ll need for the remainder of the project. You might very well be able to encourage others to join you once you’ve shown some tangible progress and the benefits of the refactor become more visible to fellow engineers.

Worst-Case Scenario

The absolute worst-case scenario is if you aren’t able to secure any additional help and need to execute on the project alone. Now before we start exploring how to make the best of this situation, I want to take a moment to acknowledge that if your only option is to execute on a large, cross-functional refactor alone, you may want to consider not doing it at all. If the engineering organization is not sufficiently convinced by your proposal to properly allocate staffing and the expert engineers you’ve reached out to are unconvinced as well, maybe it’s time to go back to the drawing board and strengthen your case. Otherwise, maybe it’s time to consider that perhaps now is not the right time to execute on this project.

In the event that your manager, teammates, and a number of other engineers believe in the importance of the effort, but there simply aren’t enough resources to go around, then you may consider moving forward alone. Be forewarned, however, that it is not an easy path. Working alone can be terribly isolating. Because it’s just you, slowly making progress one step at a time, it can feel like you aren’t making significant progress. You rarely have the chance to bounce ideas off of other people who have substantial context on the state of the project and don’t need to be brought up to speed every time you need a second opinion.

On the plus side, you don’t have to coordinate with anyone else; you hopefully know the sequence of steps you need to take, and you can execute on them serially. Not needing to coordinate with anyone else can also be a serious downside. You have to keep very, very good track of everything that you are doing and make that information available publicly so that others who are invested in your effort but unable to contribute are able to gage where you are on the project.

One or two incidents are nearly inevitable when making expansive changes to a codebase. While postmortems should be blameless, when there is only a single individual responsible for a given project, it can feel as though the burden of responsibility and subsequent remediation falls solely on you instead of on a group of folks.

I recommend you find a buddy; maybe someone else who has also been tasked to be the sole owner of a significant project. This person is there to keep you accountable and motivated, similarly to how you might regularly meet up with a friend for yoga: you know that they’ll be there because you’ll be there, and vice versa. You can establish a regular cadence for meeting up and talking through the progress you’ve made to date on your respective projects. You can help each other brainstorm solutions to the tough problems, and, on occasion, review each others’ code. Either way, having someone there to keep you company on the tough road ahead is absolutely critical to staying on track.