I believe that every human being endowed with intelligence, memory, and strength of character bears within him a little of the supernatural as well. The highest purpose of the conductor is to release this superhuman potential in every one of his musicians.
—Charles Munch
In this chapter, we will gain a glimpse into the roles and responsibilities of a manager.
Our aim is to present an overview of the bigger picture, illustrating the difference between a manager and a leader. This allows the reader to gain an understanding of what a manager does and an appreciation for some of the challenges they face.
By reading this chapter, we hope that you will recognize just how important the subsequent topics in this book (such as hiring or feedback) are, and how they fit into the big picture. This will allow you to become a good manager of people by recognizing managers for what they are.
Managers Are Like Conductors
A conductor’s role and relationship with their orchestra is a good analogy for a manager and their team (we are not the first to draw this parallel1, 2).
Jeremy Cuebas,3 a Colorado-based conductor, greatly illustrates the role of a conductor. According to him:4
The conductor acts as a time-beater, a teacher, and an artistic leader.
As a Time-Beater, they show the orchestra when to start and how fast to go.
While conducting, they do other things like the meter [measurement], dynamics [relationship], character [growth], or articulation of the music[project management]. Once an orchestra [team] gets started, it can pretty much keep playing even if the conductor decides to walk off the stage.
It’s important to understand that an orchestra can play without a conductor. This is especially true for a professional orchestra [experienced team], but even a middle or high school orchestra doesn’t need a conductor as long as the music isn’t too difficult or complicated.
Once the conductor sets the tempo [standards and processes], the orchestra can mostly continue on by themselves.
If there are changes in tempo, stops, and starts in the music, or if the orchestra accidentally slows down or speeds up, the conductor will need to step in again to fix the tempo.
In the role of Teachers, they need to educate the students and are responsible when things go wrong.
Orchestras don’t just get together for the first time and play a concert. They have rehearsals first.
To learn new or difficult music, the conductor must act as a teacher. At a higher level, the orchestra is not just playing the music, but actually preparing it for a concert. The conductor will have only a few rehearsals to make sure everything is put together correctly.
Just as a piano student practices to teach their hands how to play a song well, so a conductor “practices” the orchestra to teach them how to play the music well.
In the role of Artistic Leaders, conductors turn disconnected things into something of beauty.
Through hours of deep and intense study, a conductor will learn the music and make interpretive decisions about how it should be played to create the most sublime experience for the orchestra and the audience.
Each piece of music has nearly infinite decisions that have to be made about it, no matter how simple or complex it is. The composer always gives us the notes and rhythms, and they usually give us the dynamics (changes in volume), but many other things are left up to the performers.
These are the sorts of decisions that a conductor must investigate while preparing to conduct a piece. Then, through words and physical gestures in rehearsal and performance, the conductor communicates their decisions to the orchestra.
Much like painting a picture, the conductor begins with a vision and then executes their craft to make it real. The orchestra can play without them, but every orchestra member will play it differently.
Managers and Leaders
At the beginning of the previous chapter, we mentioned that people often use the terms Leader and Manager interchangeably, and while they’re different, they’re also intimately related.
A General is a leader in the Military.
A Conductor is a leader in an Orchestra.
An Engineering Manager is a leader in a software development organization.
The skills ascribed to leadership are required to be an effective leader. Still, each profession has its own set of necessary skills, which requires that the leadership traits be attuned and specialized to each job—that’s what we call managers.
For the rest of this chapter, whenever we mention Managers, we’ll talk specifically about the traits required to be a successful Engineering Manager on top of those that make people good leaders.
Management in Practice
There are four Ps in the life of Engineering Managers, and so far, we have discussed three: People, Projects, and Processes.
The fourth and most tangible of them is Practices—the actual application of all the other Ps, and the P that Managers are judged by.
As a manager, you use practices to set a baseline for your team; they set the tone for interaction and the dynamics between people, helping them grow, solve problems and conflicts, and achieve their goals. Here, we’re going to cover some of the most common practices Managers perform almost daily.
Picking the Right Management Framework
Arguably, the most important tools in a manager’s toolbox are their management process.
A common mistake inexperienced managers make is adopting the first management framework they worked with as the standard for running software development teams.
While they might have had a good experience with a particular set of processes, knowing why that set of processes was picked in the first place allows them to adapt.
A good and typical example of this kind of mistake is using SCRUM-like processes for Support or Platform teams. SCRUM, its sprint model, and product-oriented planning processes are designed to work best for Product Development teams, building features iteratively, alongside users and product managers, with somewhat long lifecycles. This context is slightly predictable, and the pace of adjustments and changes is on the scale of days to weeks. Meanwhile, Support and Platform teams’ work is dictated by external factors that change much faster than product development, on a scale from hours to days.
In this case, using a Kanban-like model is advisable. Kanban helps balance demands with available capacity. It’s a framework that handles well the immediate future and is very flexible to change the mid-term while not too concerned with the long term. Kanban and SCRUM are just two examples of many management frameworks. Still, there are many more out there, and a good manager should be aware of the alternatives and select the one that makes sense to their context.
Are you in a regulated environment like banks or healthcare?
How many external dependencies, like teams and clients, do you have?
How much freedom do you have to experiment?
How much bureaucracy are you subject to?
How much autonomy does your team have to make decisions?
How controlled is your budget?
Kanban
Extreme Programming
Lean
SAFe
SCRUM
Crystal
One last caveat, no matter what framework you pick: don’t think you have to follow things to the letter.
One of the core tenants of agile development is “people over processes.”
Collaborating with people around you, learning from their feedback, and experimenting can provide great insights that, if carefully applied, can lead you to a process that works to your company, project, and team’s peculiarities more than anything you can get out-of-the-box.
Facilitation
Managers bridge gaps and allow people to cooperate under resource-constrained environments, be it time, talent, technical difficulties, etc.
As facilitators, they help to create interactions between people, and when it comes to engineering, one of the most important aspects of facilitation is shielding your team.
In a demanding, high-paced environment that constantly rains demands and questions down upon members of the development team, a good manager acts as a shield, keeping their team out of the rain so they can focus on the tasks at hand.
They act as the team’s guardians without imposing their presence—the team knows that the manager is there without feeling their weight on them.
Keeping track of what is being worked on so they’re able to communicate the state of things
Maintaining a pulse on the overall workload of your team, setting a healthy work pace
To be present at meetings and know how to balance stakeholder requirements with both your team’s workload, abilities, and technical needs
Collecting feedback about the team’s and project’s performance to drive personal and business improvements
Finding and negotiating the resources the team needs to succeed
Shielding
Development requires focus. When you pull someone away from their work, you lose the time spent in the actual meeting and momentum. It takes time to pause, switch context, and change tracks to be a productive contributor to a meeting.
It also takes time to focus and settle back into your code. Losing this time can wipe out ideas, thoughts, and motivation. As Ryan Singer wrote:5 “Losing the wrong hour can kill a day. Losing a day can kill a week.”
Meetings can become emails.
Standup updates can be done through Slack.
Support can be done through rotation.
Tasks can be reprioritized to reduce interruptions.
Managerial politics and unnecessary meetings can be skipped.
Tools can be purchased to automate specific tasks.
Managers must understand what should be done, when, and why to make the right decisions to help the team become as productive as possible.
Without barriers, stakeholders will rain requests and demand down upon the team whenever they see fit. Therefore, a good manager almost acts like a border guard: they vet what comes through to the engineers and when.
Vetting requires an intake process to help the team improve interaction across the company: Stakeholders will know what to expect and what they are required to do to get their ideas accepted. Product management will maintain better product oversight since ad-hoc work becomes less likely. And finally, engineers will be less anxious and more able to focus on the tasks.
Lastly, managers sometimes have to shield teammates from each other. Human dynamics are not always perfect, even in the best teams; sometimes, people won’t be aligned. Just because two people do not get along does not mean they are bad professionals or bad apples, and sometimes life will lead even the best friends to a clash. They might simply have a personality conflict, and while they might still try to find common ground and work toward the same objective, they will be drained by being forced to interact when their relationship is not at its best.
We all spend more time with our colleagues than anybody else, and having to interact day after day with people we’re not on good terms with is tiring and bad for everybody involved.
As a manager, one of your jobs is to help people keep good, civil relations even when their personalities are not a match.
That might mean temporarily reducing exposures while relationships are being repaired, helping people find common ground, or softening the impact via other means.
Overseeing
Oversight does not mean keeping track of the time a team spends in front of a screen or seeing whether they respond to messages in minutes. Nor does it involve installing spyware on people’s computers under the guise of “managing their productivity.” Instead, oversight means letting people do their jobs once they have justified their decisions.
Unjustified decisions may result in problems that can have lasting cascading effects. For example: choosing a certain technology that will be used across the entire project just because you are comfortable with it due to some previous experience, without accounting for whether it is the right tool for the job or not.
Decisions taken now may be difficult to revert in the future, example: choosing not to add written tests at the beginning of the project can promote code that is harder to test, making it much harder to introduce some tests later.
A decision may increase complexity, which affects other developers and the quality of the software as a whole.
It is important to note that requiring people to justify their decisions does not imply removing their autonomy to do their work. On the contrary, people should operate under autonomy (more on this later), but not as lone wolves. You do not want a situation where autonomy is used as a trump card to avoid discussions and well-thought-out solutions to the problem under a team’s purview. Every good decision should be defendable from its specific standpoint.
Technical Oversight
In order to vet, negotiate, and prioritize technical decisions, managers need to have an in-depth knowledge of the system’s current state. The best way to gain this knowledge is to maintain oversight and be part of the evolution of the system your team is building.
Software systems are complex and tend to become more complicated by the day.
You are not actively writing code daily.
Whenever weighing priorities, the technical aspect often loses.
Listen during standups. Managers tend to focus on processes and people, which can lead them to not focus on the technical aspects during standups. Make sure you’re following the technical conversations with high attentiveness.
Silently participate in code reviews. Reviewing the code alongside engineers will give you a rough idea of what-is-what and the more granular technical decisions. But be careful not to be overbearing: Depending on your level of technical knowledge, you may or may not have anything useful to contribute or take away ownership that should belong to them. Best leave actual reviewing up to the individual engineers, though. Nothing is worse than a micromanaging manager who comments on code without sufficient technical knowledge. If you comment, make sure you know what you are talking about.
Don’t fake it. A common theme in self-help books is “fake it until you make it.” This might work for some areas, but it won’t work when dealing with competent people and will lead you to lose the credibility you need as an effective leader.
Keep yourself informed. If you are unfamiliar with a particular design pattern, concept, or technology used in your project, go out of your way to understand it. Please read up on it, exercise your skills, and build something. You won’t be able to contribute effectively if the technology you’re discussing is a mystery to you.
Don’t micromanage. Managers trying to maintain technical oversight often fall into the trap of trying to do so by micromanaging. Constantly asking people for updates, questioning every decision, or inquiring about why they didn’t try X or Y alternatives. This can quickly overburden the team. Even, and especially if the team didn’t make the decision you would, you have to accept it unless there’s a glaring problem. It is important to understand that managing people often requires you to take a step back and let them do their job.
Keeping a sufficient level of technical knowledge cannot be stressed enough, as this is the key to helping your engineers execute. You only have to be careful to do so, making sure you keep your focus on management and maintaining your team’s autonomy.
Autonomy
The word autonomy comes from the Greek word “autonomíā,” meaning “freedom to use its own laws, independence,” and this is exactly what autonomy is. That is, being able to execute independently but responsibly, without needing constant approval and reassurance that you are acting within a given framework. One of the quickest ways to kill a project, or an entire company, is via “death by committee.” If every minute decision needs to be discussed by the team or needs various stamps of approval by superiors, then your product will go nowhere. Hire the right people and let them do their job. And once these people can justify major decisions, technical or otherwise, they should be given the space and freedom to execute.
It’ll consume most of their time, as they’ll have to oversee everything around them constantly.
Along the same lines, it will stop them from scaling their impact, which will hold their careers back.
It also kills motivation, which leads to frustration and eventually people quitting.
Autonomy is critical in remote teams. With limited face-to-face time, and the challenges posed by remote communication, the best way for a manager to ensure a team is making progress is to allow people autonomy and the sense of pride that comes with it.
Good managers use their oversight to build and maintain autonomy within their reports, a notion that’s best summarized by a popular management quote often attributed to Jack Welch (CEO of General Motors): “Communicate your ideas, distribute resources, and get out of the way.”
Managing Deadlines
There are two types of deadlines: soft deadlines, those that can be negotiated, and hard deadlines, those that cannot.
Hard deadlines arise due to circumstances outside the team’s or company’s control—for example, a new Tax and Reporting feature is required due to new IRS guidelines and is required by a certain date to avoid fines.
Soft deadlines, on the other hand, are those set internally and, as such, should be defined with team participation.
- 1.
They help us to align other workloads and plans.
- 2.
They give the team direction and a sense of priority.
- 3.
They help hold us accountable.
A big mistake inexperienced managers make is confounding soft with hard deadlines.
That is, they treat soft deadlines as hard deadlines as a way to “motivate” engineers to achieve more. They set deadlines in stone to increase output without needing them to be set in stone. However, all that this does is create unnecessary pressure, and when scope and time can’t be changed, the cost comes in the form of poor quality software and technical debt.
The inverse can also be true; another common mistake is treating hard deadlines as soft. Not communicating the expectations correctly and not doing more aggressive prioritization can lead to an exponential increase in pressure toward the end date.
Project Management
One of the core expectations of Engineering Management is being able to manage projects and the features that come with them, which heavily impacts managing the team itself. For one, the nature of your work will determine the skills the team will need, the pace of work, expectations across stakeholders, and so on.
Managing a project requires you to work as an interface between engineering and the rest of the business. Without facilitating the exchange between ideas and requirements with the business’ needs, you will not be able to manage a team effectively.
It is important to highlight that Project Management is its own discipline, with an extensive Body of Knowledge (BoK), and is very complex, a complexity that is increased further depending on the nature of the challenges. It is also important to note that most big and small companies don’t recognize this complexity and don’t prepare, screen for, or train people to succeed in this role.
There are an uncountable number of books about project management, and its nuances are beyond the scope of this book. Still, we’re now going to try to give you a glimpse into its expectations for an Engineering Manager (EM) when working in an organization building products for consumers.
Isn’t Project Management the Role of a Product Manager?
While PMs have a deeper understanding of the product, the business, the product vision, and the users, the EMs have deeper technical knowledge.
As the people responsible for building the software, the EM is much more intimately involved with the technical aspects of the product’s user experience.
The EM tends to have much greater visibility into the granularity around execution.
There’s no clear industry standard around the role of EMs and PMs and where their responsibilities and boundaries lie.
When an EM does not understand how to manage a project, the end result is simple but impactful: The team will end up building the wrong thing the wrong way.
Building the wrong feature or project for extended periods can lead to a waste of time and money at best, and bankruptcy at worst.
Either way, this will lead to the team’s dissatisfaction. In the end, people want to feel that what they do matters. Building the wrong thing is a concrete, even if indirect, statement that all the effort and passion put into their work was for nothing, which can make their manager’s life hard to nigh impossible.
Building the Right Thing
An understanding of your company’s business model, strategies, and plans
Domain and technical knowledge of the business and its industry
A Bird’s Eye view of what’s going in on in the company, especially if you have internal dependencies
Identifying and mitigating risks
Stakeholder management
Understanding who your users are and what they value
- 1.
Being too technical and focusing more on the solution (technology) rather than the problem
- 2.
Being too hypothetical, and trying to picture solutions instead of being grounded in how and why users use the product
What is valuable to the customers is not always helpful to the business—not aligning the two perspectives amounts to rowing against the tide. Remember the maxim: No plan survives contact with the enemy.
In the same way, having the people you’re trying to help tell you, directly or not, that a feature truly addresses their needs is the only way to be sure you’re building what matters (incidentally, also the name of a great book on the topic6).
Surveys – From Net Promoter Score (NPS, a way of gauging customer satisfaction) to targeted surveys around a specific topic
Interviews – Structured, with a script trying to dig for more information, to unstructured like blind usability tests
A/B tests – Building variations of the same thing to see what performs best in real life
Regardless of whether performed by the EM or PM, it is critical to identify and filter out features that serve the user (and hence add value to the product itself).
A company’s mission can be changed, reformed, and adapted. However, the same cannot be said about your customers. Once your product no longer provides value to the customer, your product and business will fail.
Managers have a responsibility toward their teams, peers, and the company to ensure the focus on what’s valuable at the time and be constantly reviewing the context so they can adapt when things change.
Planning and Prioritization
An unintuitive but crucial aspect of project management is managing complexity. All but the smallest projects (one person, one day) are webs of interdependent relationships of many factors—teams, users, processes, technologies, strategies, opinions, etc.
To successfully run a project, managers need to see the forest for the trees while not losing sight of each twig.
The way to do that without losing control of the project is by creating a cadence of execution where each step takes you closer to the goal by carefully choosing what step is foundational to the next. This is what we call prioritizing.
What? What needs to be done—what’s the scope?
Why? Why does it have to be done—what are the motivations and impacts?
Where? Where will it live—where in the system, and how does the user get there?
When? When will it be done—do we have what we need by the time we start it?
Who? Who’s going to do it? Who is going to be affected? Who’s going to approve?
How? How will it be done? Tools, tech, processes?
How much? How much will it cost to make? How many resources will it consume?
If you’re committed to building the right thing and are listening to the users, you’re already getting much of the information necessary to answer these questions. Your relationship with the internal stakeholders should provide the rest, and you can start prioritizing.
Sounds complicated? It’s not when done right.
Planning and Prioritization in Most Companies
Let’s discuss what project planning typically looks like in software development.
If you’re already working in the field, you likely saw some of this in action—they’re rituals and artifacts commonly referred to as sprints, backlogs, grooming, epics, stories, or any variation of these.
A typical way companies run projects goes like this: After deciding on the goals, people sit down to break the work down into milestones, called Epics, hoping each epic will get you closer to the project’s goal.
Then they discuss each epic, break it into components called Stories, and select which stories will go into the epic.
With the list of stories, they go through an estimation session and discuss each story, write specs, and do planning-poker or other types of estimation.
With all the stories estimated, they use the team’s velocity, a measure of how much progress a team can make in a fixed period, to estimate how long the project will take.
In the end, they hope to know, among other things, how long the project will take, who needs to be involved, what the product will look like, and how it will impact the company’s objective.
There are many problems with this process, and they often stem from companies and managers failing to acknowledge the inherent complexity of projects of any meaningful size—as discussed, almost all of them.
Trying to plan out an entire project upfront is an exercise in futility, often better served to soothe concerns, and rarely works to make it successful.
Here’s a non-extensive list of themes and issues with this standard way of planning:
You Can’t Predict the Future
There are too many variables we can’t control that can significantly impact a project.
A key person might leave the company or team, or a new one might join. A critical technology, like an external API, might change without notice. A government, or even your company, can create or change policies.
Everyday one is incorporating new information and learnings arising during execution.
Opinions may also change during the project’s life, and the assumptions based on them can mean that a small or big part of the original plan can, too, often have a ripple effect.
Even a pandemic can happen. One can only imagine how many projects were changed or canceled because of that.
Our Brains Are Limited
Society, and the corporate world, often push us to play smart. The truth is that our brains have limited capacity.
Even when working together, there’s a limit to what we can see, especially when the web of complexity is weaved so widely, as in the case of project management. Trying to grasp all the parts of an entire project and create an entire map (roadmap) upfront is virtually impossible due to these limitations.
Myriads of neurological research describe our minds’ boundaries and how it affects our perception of the world. Still, a particular set of these limits are more prominent when planning projects: biases.
Bias happens when there’s disproportionate weight in favor of or against an idea, thing, or person. One can have a bias all their life or learn new ones at any point. We often perceive them more quickly when they’re against an individual, group, or belief, but people can also have a bias toward ideas and attitudes. Those are harder to spot and arguably most impactful when collaborating to build things.
Let’s look at one of the most common: confirmation bias. Simply put, we look for, interpret, and favor things we already believe in or value. When facing a choice and presented with two sets of information, one aligned with our thinking and one that does not, most people pick the former without even thinking about the latter.
A classic example in software engineering is when a backend and a frontend developer discuss implementing a feature. Assuming that that feature could be done using either expertise, it’s unlikely they’ll even consider the other person’s perspective. At least when deciding in isolation.
In essence, confirmation bias tells us that “if all you have is a hammer, everything looks like a nail.”8
People can fail to discuss the trade-offs.
Blindspots.
Making strategic decisions without proper vetting.
And many, many more.
The list of biases and their impacts is extensive and beyond the scope of this book, but here are a few more often seen in project planning.
Anchoring Bias
This bias describes how the first fact we’re exposed to influences (or anchors) our subsequent decisions.9
For example, if you tell a group that 90% of users liked feature A and ask them the percentage that will like similar feature B, each person will answer with a number close to 90%.
If you tell them that only 10% liked feature A and ask the same question, they’ll give you a number closer to 10%.
Availability Heuristic
This is the notion that the first thing that comes to a person’s mind when evaluating a specific topic, concept, or decision is the right one.10
Bandwagon Effect
The adoption of behaviors, or the making of decisions, because a group behaves in a certain way.11
Choice-Supportive Bias
The tendency to think the decisions we made are the right ones and the ones we didn’t are the wrong ones.12 For example, if a person chooses option A instead of B, they will likely ignore or downplay option A’s faults while amplifying the bad aspects and drawbacks to option B.
We invite the reader to learn more about biases, and we can humbly say you’ll become a better professional if you do.
Communication Is Inherently Fraught
People tend to think of communication as something trivial as it’s something we do all the time, and while casual communication can be easy, corporate communication can become rather complex.
When communicating with stakeholders, peers, or even within our teams, our human tendency is to assume things. This is the consequence of our many biases.
This is not out of malice; it’s just a consequence of our complex biologies, social norms, and psych. We rarely think about these things, but they permeate how we address the world outside. A meeting after a good night of sleep can go wildly different than if you had a bad one.
Other factors, like the volume of information we have to process and our level of attachment to certain decisions, can severely impact what we say and hear.
Trying to capture and communicate all project steps in the first go maximizes all the issues we have with communications. At the same time, this is too much to digest in a short period (relative to the size of the project) and not enough time to digest all the complexities of a project.
Another important fact to consider is the practice of capturing all the communication in written form. We do that to manage complexity and align expectations around what we’re trying to do, but we forget this is inherently flawed. Let’s see why by looking at a trivial example of a feature called “User selects state for tax purpose”:
As a US Citizen, I want to select my state of residency in a dropdown when buying a product so that I pay the correct taxes.
- This is mandatory
- Taxes are applied according to the selected option.
This “User Story” follows the “As a [persona], I [want to], [so that]” with the intent to describe who’s going to be impacted by this feature (persona), what needs to be done, and why this is important for them. It also contains a list of “Acceptance Criteria” or a list of items you need to check to validate if a story can be considered done.
Although they can vary in format and content, software development often happens through building a series of these user stories where each unit and increment leads to a finished product.
There are many problems with capturing dozens or hundreds (sometimes thousands) of these stories upfront, coming up with a “list of requirements,” and expecting it to be a reliable roadmap to build something.
A user interface for Alabama, Alaska, Arizona, Arkansas, California, Colorado, Connecticut, Delaware, Florida, Georgia, Hawaii, Idaho, Iowa, Kansas, Kentucky, Louisiana, Maine, and Maryland.
The “User selects state for tax purpose” story
Limited Perspective
By definition, the scope of a story will be limited by the knowledge and bias of the people involved in defining said scope.
In this story, the expectation could easily be set to use the 50 US states, but what about the US territories, of which there are 14? If the story targets the “US citizen” persona, those territories should also be considered as each has an equivalent of a state tax. What about federal taxes? Should these stories also account for them, as some territories fall into different regulations?
What about the seven US states that don’t have taxes? Should we hide these states? Should we set zero taxes? What about US citizens living abroad?
This is a simple feature with some relatively obvious edge cases. Still, it’s easy to see many possible permutations and edge cases arising from people.
One could argue that each of these cases could be captured on its own story, and it’s a failure of the people responsible for “capturing the specs” not to have taken one, or many, of these cases into account. In reality, even for the smallest of features, there is a small set of views and insights a group can come up with, no matter how varied and smart they are.
Incidentally, this combinatorial explosion, the rapid growth of complexity due to the multiple combinations of all the variables in a problem, is why projects often go over time and budget.
Focusing on the Solution, Not the Problem
Another common issue with capturing specs with texts is the inclination to define what needs to be done instead of what we’re trying to solve.
Our example states that we need to provide a dropdown to the user so they can select their state, but the core problem we’re trying to address is having the user pay the correct taxes based on where they live. When developers receive that story, they won’t stop to think about the edge cases we discussed; they focus on doing what they’ve been asked to do.
What if, instead of the original text we proposed, the story had a more succinct description like “As a US Citizen, I want to pay the due taxes based on where I live?”
This approach is an improvement on the original story. Because we’re not providing developers with a set of things to do, they would be forced to think through the problem.
They would have to talk to stakeholders and understand what the problem entails at the same time of development, which would lead to them having to deal with fresh information (more on that soon). They’d have to use the PM as a knowledge base and lean on their expertise while also developing critical thinking and expertise of their own on the domain they’re working on.
This would also give the problem (story) extra pairs of eyes and give developers a higher degree of freedom, accountability, and pride due to being responsible for deciding what to do. In this example, they could use a third-party service to [automagically] get the user tax based on their geolocation and have all the corner cases addressed by a specialized service.
One of the main reasons people don’t take this stance when creating stories is because the trade-off is losing the imaginary predictability people think they get by upfront planning everything.
Our intuition makes us falsely believe that if we break down some work in all its constituent units, we can estimate how long the work will take by adding up each unit’s cost and effort.
As we can see from some previous points, this is futile for all but the smallest of projects.
Static Views of a Dynamic World
Let’s ignore all the complexities of projects and their myriads of interconnected relationships. Let’s also dream up a perfect team capable of writing the best, most cohesive, brilliant stories.
Would this allow this hypothetical team to plan and correctly estimate a project upfront? Unlikely.
Planning an entire project that can last many months is akin to taking a snapshot of what’s known about the project at the beginning of it at a fixed point in time. Even smaller chunks of work (milestones) suffer from the same problem.
One reason for that is that life is dynamic, not static. Things change—all the time.
What if a new tech, that could make a feature easier, comes up? Would you stick to the plan instead of considering it? What if a critical stakeholder leaves your project? What if a competitor comes up with the same feature making the current strategy irrelevant?
Too many factors change every day. Some small and some big, but all affect the project and change its scope in meaningful ways.
A Brief History of Software Development
So, should we stop planning projects altogether? Should we stop writing specifications and requirements? Is every problem too complex to handle? Should we throw away all the norms?
Well, not quite. As we’re about to see, all these processes have their merit, and to better appreciate it and understand when and why they sometimes fail, it’s good to understand where they come from and how they evolved.
First, let’s take a step back and recognize that Project Management is a highly developed discipline. It became a formal discipline barely over a hundred years ago when Frederick Winslow Taylor published his seminal work The Principles of Scientific Management.13 It has been instrumental in bringing forth our modern world ever since.
Taylor’s focus was to improve manual-labor-based production and resource management. Eventually, it inspired the rise of new sets of processes responsible for managing increasingly complex projects, like taking people to the moon.
Today, anything from cars to computers to MRI machines is conceptualized and brought to reality using management principles derived from that, which are not unlike the one described in the previous section.
It’s safe to say that every product produced at scale these days had a successful project behind them. Some of them even became famous on their own, such as the one leading to iPhone’s creation: Project Purple—look it up.
The range of Software Development Life Cycle (SDLC) processes we have today is an adaptation of the methodologies developed over decades to build everyday products. In the same fashion, SLDC projects have been evolving too.
In the Beginning
The first SDLC methodologies reflected Henry Ford’s 1913 world-changing innovation—the moving assembly line, which was inspired by Taylor’s scientific management theory and had his direct support after Ford hired him as a consultant.
With the car’s design on paper and necessary resources ready, the exact sequence the car will be assembled is planned.
Then, a person starts working on a part and, after they finished, they move it to the next step in the line, and handle it to next person, so they can build the next part by integrating it with the previous one.
This continues with all the planned steps until the car is done.
The car is then quality controlled, and it’s deemed ready to be sold if no problem is found.
Knowing upfront what to do, having a defined set of measurable steps, and making the workers pure executors of what the managers decided were some of the crucial factors that made the assembly line so effective in cutting production costs, which led to cars’ price reduction and making it affordable to the masses.
It’s also important to notice that these were big improvements over the lack of established processes in that era, but they came with obvious drawbacks like, ironically, turning workers into “replaceable cogs.”
The assembly line parallel in the software development world is known as waterfall development. It was developed in 1956 to construct SAGE, NORADS’s Cold War computer network that coordinated data from multiple radar locations, one of the largest scale projects at the time.
Its name comes from the process flowing in one direction, downward like a waterfall, and has a fixed set of phases: conception, initiation, analysis, design, construction, testing, deployment, and maintenance.
The similarity with the assembly line process is self-evident, but back then, it was easy and natural for people to believe that this was the best way to build software. After all, it was a derivation of the most successful process of building things.
An Evolution
Manufacturing processes evolved, and the most significant change since the Taylor-Ford assembly line, funnily enough, also came from the automotive industry.
The Toyota Production System (TPS) was developed and perfected by Taiichi Ohno and Eiji Toyoda but was only made public in the early 1990s and was inspired by Taylor and Ford’s observations on production inefficiencies.
As a testament to its merit, it is arguably recognized as the single factor that turned Toyota into the automotive industry leader for most of the last half-century.
TPS’s effectiveness is attributed to its focus on eliminating or minimizing “the 3Ms”; Muri, Mura, and Muda, the Japanese words for overburden, inconsistency, and waste, respectively. Its goal is to have a dynamic process that can deliver results with the least effort, in a way that feels natural, by constantly tweaking it as you learn about its existing or newly acquired inefficiencies.
Continuous improvement.
Respect for people.
The right process will produce the right results.
Add value to the organization by developing your people and partners.
Continuously solving root problems drives organizational learning.
- 1.
Everyone involved and impacted by the process has a say in it and are invited to help improve it.
- 2.
The process is intentionally designed to be dynamic. There’s no recipe. Each product demands its own variation and should adapt as frequently and fast as inefficiencies are found.
This was a clear break from the do-as-you’re-told policy of Ford’s time, as people were now given autonomy to influence the results they were accountable for.
The “Toyota Production System,” while not perfect and subject to a fair amount of criticism, was a big improvement on the methods of the time. It was so impactful that it led to a second manufacturing revolution and became the new standard.
Its more generalized form, known as “Lean Manufacturing,” trickled down to many sectors and morphed into several industry-specific versions like “Lean Construction” and “Lean Higher Education,” among others.
It also gave birth to a new SDLC called “Lean Software Development,” popularized by the eponymous book “Lean Software Development: An Agile Toolkit14” by Mary and Tom Poppendieck. It was also one of the key inspirations for the creation of methodologies like Agile Software Development, often just called Agile, SCRUM, and more.
TLC’s influence in Today’s software development practices is pervasive. Concepts like Kaizen, roughly translated to continuous improvement, led to the creation of rituals like “retrospectives” and practices like Continuous Integration. Kanban, also the name of an SDLC methodology, led to the adoption of Backlogs.
The Root of the Problem
At this point, the reader might see a clash of opinions—on the one hand, we argue that the complexity of projects makes upfront planning a futile exercise. On the other hand, we state the same set of processes, with over a hundred years of successful track record, works very well.
Where does the difference lie? It lies in what one tries to build: software vs. physical products.
We previously discussed companies’ typical planning and prioritization approaches and the issues around them.
It’s fair to highlight that the issues mentioned often stem from companies using processes and practices conceived for manufacturing goods (turning raw materials into finished physical products)—particularly outdated ones.
At first glance, this might seem non-consequential, but there are some fundamental breaks between the two.
Product Development
In a manufacturing process, the word “development” does not have the same meaning as it does for software. Development means creating the initial designs, modeling each component, and verifying that all the parts work together to prove that the results are within the customer’s specified requirements. In software engineering, we call this prototyping.
Because physical goods depend on physical resources, the consequent prohibitive costs of producing the wrong product at a scale that comes from it lead to much more time and investment spent in this stage on physical things than on software.
Verification and Validation (V&V)
Verification: The process of checking that a product works as it’s expected to, without technical issues (bugs)
Validation: The process of checking whether the product attends the user needs it’s supposed to (requirements)
Verification asks, are we building the product right?
Validation asks, are we building the right product?
The key property of digital products that is harder to test for V&V than physical ones is called falsifiability, or the complexity of proving that something is wrong.
A plane has to fly, a ball has to roll, and a door needs to open and close. While those are simplifications, the nature of the tests one can apply to something physical is somewhat limited. Even subjective properties of products like taste, aroma, and texture can be subjected to tests that are still deterministic. These characteristics make defining if a physical item is built “correctly” or it is “working” straightforward. With software, things are far from being this simple.
Beyond non-functional requirements, like safety, security, performance, etc., the definition of “correct” is open to interpretation.
Who’s to say that a feature truly addresses a user’s need or is the best way to solve a problem? Or even harder, that it’s the best implementation given the context?
It also doesn’t require a physical resource, so it’s harder to pinpoint waste.
It’s comparatively more uncomplicated to put in front of users than physical products, which makes the work put into prototyping and validation a relatively higher cost of its total lifecycle.
Software malleability also enables higher competition, which in turn increases the pressure for time-to-market, which in turn can force people to cut corners.
All these factors, as unintuitive as they are, are the reason and the root of so much frustration felt by developers all around the world as the concept of project management is so poorly understood and its complexity understated.
An efficient engineering manager, one that gets things done, needs to appreciate these complexities because they have a direct impact on the life of the people they lead and can make a difference between their careers being successful or not.
A Lean Project Management
So, here’s the Billion dollar question: How to reconcile the fluidic nature of software, its inherent complexity, and the challenges of development processes while making a project more likely to be successful?
A clue can be found in the TPM originally called just-in-time production, or JIT, as in “making only what is needed, only when it is needed, and only in the amount that is needed.”
As the story goes, Taiichi Ohno, who’s considered the father of JIT, got the inspiration from visiting an American supermarket. Then he observed how the shelves were replenished at the same pace customers were picking items from them.
In his view, this ensured that there was no excess or shortage of products which, consequently, led to maximizing sales, conserving resources, and setting the pace for resupplying the store. In summa, the elimination of waste.
The high complexity of software development, exhaustively explored previously, can be tackled with the diligent application of JIT.
While Lean Software Development is one of the closest SDLC methodologies to JIT, and we encourage the reader to learn more about it, the core concept is what matters—just do what you need right now.
Let’s now look at a JIT-like project management approach for developing software.
The Lean Inception
Create alignment around goals.
Define what needs to be done in the current iteration.
Set the ground rules of collaboration.
Discuss the current state of things and adjust expectations.
Make changes in the scope or processes to maximize the chances of success.
The Lean Inception (LI) is a set of practices developed for inception targeting software development. It was created by Paulo Caroli and described in his excellent book: Lean Inception: How to Align People and Build the Right Product.15
LI was created as a tool to quickly define a project’s Minimum Viable Product, commonly referred to as an MVP.
Eric Ries, author, and creator of the “Lean Startup” movement, defines MVPs as “that version of a new product which allows a team to collect the maximum amount of validated learning about customers with the least effort.”
Applying JIT to software development with successful project management is achieved by building your product as a series of MVPs and adjusting the product and process as you learn until you reach the project’s goals.
This approach has many benefits, with the most obvious being making it easier to tackle complexity.
As Desmond Tutu, Nobel Peace Prize laureate for his work on anti-apartheid, once wisely said: “there is only one way to eat an elephant: a bite at a time.”
Complex problems can be dealt with if we break them down into smaller, more manageable parts.
- 1.
Increased engagement
- 2.
Minimization of issues coming out of misunderstanding
- 3.
Maximization of the delivery of value to the business
Engagement is one of the least discussed aspects of this process but the most crucial in our experience. The second and third benefits might not be achievable without engagement or at least become severely compromised.
That said, the best way to achieve engagement is by continually releasing value to the business, which we often do through the release of features.
Most people organically commit to doing their best work, supervised or not, when they see a frequent flow of changes gradually but steadily, making the product better, the users happier, the business achieving its goals, and getting constant recognition.
This “happy state” makes all the complexities of building software projects, amplified in remote environments, much more manageable.
We now offer the readers suggestions of how to apply these concepts in real life. You can use this as a starting point, but keep in mind that this is a condensed version, therefore not complete and ignoring many caveats.
In Practice
Before Starting Projects
Before a project starts, there are three activities that, when done correctly, help to steer the course of the project toward success.
First – Select the Core Team
- 1.
A Sponsor: The person who requested or approved the project, usually someone involved in strategy
- 2.
A Lead: The one responsible for the project’s execution, usually the Engineering Manager
- 3.
A Validator: The one responsible for making decisions around the project from a business perspective, often the Product Manager
While the same person can do those roles, we advise against it. Conflict of interest arising from a person wearing multiple hats is well-documented.
Second – Define Stakeholders
Beyond the core team, we need to map and understand clearly who the stakeholders are because there’s very little chance of this, or any process, working well without that.
Map the people holding crucial information, subject matter experts, and those impacted by the execution of the project directly or not.
Third – Set Your Success Criteria
Identify the success metrics for the project and make them visible. This doesn’t have to be measurable (although it helps a lot), but it needs to be something that can work as a compass to help tell if the project is going in the right direction.
It can be as objective as a Key Performance Indicator, or KPI, or as subjective as user happiness.
The Liftoff Meetings
Involving Everyone Impacted
The more people you have in the room, the more chances for opinions to clash. Assigning the roles we mentioned helps to address that, as they can work as the voice for others.
There’s no need for consensus, just agreement on the general objectives. In case of hard disputes, use the Sponsor or Lead to work as mediators and tie breakers.
- Define clearly the difference between who’s involved and who’s committed. This helps during discussions and decision-making.
Rule of thumb: People committed are those responsible for the execution and success of the project and are accountable if it fails. Everyone else is just involved.
Set regular checkpoint meetings, or mini-inceptions, if you will. Use those for temperature checks and, if necessary, course correction. Do that as soon as possible so people can work it into their schedules.
Lean inception was originally conceived to be a weeklong workshop, and while this is reasonable for large projects, it’ll be overkill for the ones of small or medium size.
Instead, try to compile LIs core sessions into two or three partial day sessions to keep with its spirit but maximize everyone’s time.
We suggest the following flow:
Meeting 0 – With the Team
Have a high-level overview of the project’s goals, motivations, and impacts on the business and team.
Start motivating everyone.
- Set initial expectations.
For the team as a group
For each individual in private
Meeting 1 – With the Team and Stakeholders
Explain the process to everyone involved without too many details.
Set the ground rules for collaboration, and have it documented.
Invite the client or someone speaking on their behalf, often the validator, to explain the problem the project is trying to solve.
Ask for people tangentially involved to add to the discussion.
Have each team member take individual notes.
Meeting 2 – With the Team
Milestone and Epics breakdown. Try splitting the project into small deliverables with high business values.
- Sticky notes session with everyone raising possible solutions and then grouping the different ideas into themes.
Common post-its usually indicate stories, but those should not be discussed during this meeting.
Common themes eventually become Epics.
Connect themes can be grouped into Milestones.
Finish the meeting with a team agreement around Milestones; no need to agree on the Epics (or even spend much time on them).
Prioritize first the things (Epics, Milestones) with high certainty.
For the one with less certainty, shelve the discussions around them, unless they are foundational to the project, then schedule timeboxed research stories.
Prioritize then things with high value for the business and low complexity unless they’re foundational.
- Treat each Milestone as an MVP in the sense it delivers the max business value with the least effort.
If necessary, break the milestones into multiple versions to allow for a fast-paced release cycle.
Meeting 3 – Team and Stakeholders
Present the result of the previous meeting in a condensed way.
Focus on Milestones and some focal Story/Epics to give some sense of depth.
Get an agreement and sign-off from those you need.
Assuming all’s well, you’re ready to start the project.
If not, rinse and repeat Meeting 2–3 (or 1) until you’re ready.
Keep the milestones as simple as possible and with clear goals.
Even if the project has many clear milestones, choose just a few (2–4) to be your project’s Version 1. This way, you create a clear checkpoint to review, regroup and reprioritize things based on the learnings captured along the project.
Starting the Project
Start reviewing the material from the previous meetings and having a free-form conversation about what’s required to achieve the Milestone goals with minimum effort. It goes without saying, but I’ll say it anyway, compromise on scope, not quality.
Go over the sticky notes of the initial story and milestone breakdown, and use it as a starting point but focus on the first milestone only.
The way we’re working, we create stories for milestones only when we’re about to start working on them. This ensures fresh context for everyone involved, and you can leverage the learnings of a previous milestone to have a more constructive conversation. That said, not every team might be apt or enjoy working this way. Play to your strengths (and weaknesses) to decide the format, but I suggest giving it a try.
Writing Stories
When creating stories, try to follow a linear and incremental flow with stories complementing each other. Start with any foundational stories and go from there.
This model can lead to bottlenecks, especially in the beginning.
- Some suggestions to deal with that are:
Pairing. Helps with knowledge sharing, which is a great side-benefit
Leaving a buffer of stories that people could pick from to account for that
Tech debts or off-sprint activities
This can lead to good parallelization if played well.
What’s the problem we’re trying to solve
Who’s going to benefit from it
Why are we doing it
List of criteria for a story to be accepted
Other information can be provided but trying to write the solution on the same story tends to lock it down while leaving it open allows for some level of freedom for those implementing it.
When discussing the stories, allow each person to write what they think the story should be after the team discusses its overall goals. This allows them to think from a different perspective, think about the business implications, and be more critical of what’s required for it to be completed (especially when writing down acceptance criteria), and it’s more engaging to everyone.
One last thing worth noticing is to try separating stories from tasks. Tasks are specific things that need to get done, stories are small, self-sufficient steps we need to take to achieve something bigger.
Running the Sprint
The stories should be prioritized before the sprint starts. The planning meeting should take care of it, and grooming sessions, a moment to review stories and check on the state of things, allow for changes that might have happened meanwhile.
Try setting sprint goals whenever possible and ask the team what they are willing to commit to. This creates a psychological deadline that will help people to be more critical of unplanned work, scope changes, or external noise. It should at least be helpful to make those things more visible, especially when coupled with a “Sprint Goal check” during stand-up.
As the team members pick stories to work on and they get done, deliver in a badge fashion to your validator or lead so they can try it out and give feedback. Unbadge after they approve or the time of roll out for the feature is right.
Some teams create private slack rooms to ask for this kind of feedback. This has the side effect of increasing engagement around everyone involved in the project, which can lead to great shoutouts.
Immediate feedback – things that need to be changed in a current story or meaningful things on stories recently released.
Must-haves – things that impact the Milestone currently being worked on or upcoming Milestone.
Nice-to-haves – good ideas in need of more research that could feedback into the business pipeline. We call that our Icebox.
Review all those feedback with our stakeholders during your checkpoint meetings to get more information and adjust expectations (except the immediate ones that might have been tackled already by the time we meet with them).
Closing Milestones
During checkpoint meetings, get official signoff from stakeholders until you acknowledge that the milestone is completed, by reviewing the goals and checking if they were achieved.
Closing the Project
After each milestone, check if the project goals have been achieved. If it hasn’t, pick the next milestone that takes you closer to the goal.
If it has, your project is done.
Things Left Unsaid
We believe that the topics discussed in this chapter can give you a good understanding of the challenges of an Engineering Manager. Still, while we think we provide a comprehensive view of what the role requires, there are two big topics that we’re purposefully not covering: Career Development and Diversity, Equity, and Inclusion.
Career Management encompasses everything from professional development, to behavioral education, to promotions. It can vary wildly from company to company (just as the expectations around EM can be), and it’s yet another important area without clear standards. We advise the reader to delve into the topic, discuss different approaches with seasoned managers, get a better understanding of how the process works in their organization, and start forming their critical thinking and framework to do career development when it comes effectively.
Last but definitely not least, comes Development and Diversity, Equity, and Inclusion, or DEI.
An (inevitably) crude overview of the topic:
Diversity is the presence of differences like race, gender, sexual orientation, nationality, and much more. A significant source of conflict, frustration, and, sometimes, commercial failure arises from not seeing and understanding these differences, therefore not accepting and including said differences in the conversation.
Equity is enabling fairness and giving different people similar chances. A fair misunderstanding around equity is thinking everyone should be treated the same, but in reality, it means people should have the same opportunities. An anecdote to illustrate this can be thought of as having a marathon and giving everyone the same shoes as giving everyone a shoe that fits.
Inclusion is making sure people feel welcomed in a community, not despite their differences but because of them, and are able to participate in the decision-making process equitably.
A great read on DEI is the book “Just Work: Get Sh*t Done, Fast and Fair” by Tim Scott. It focuses on six dimensions of bias that might be present in a work setting (three at an individual level, three at a systems level) while discussing her personal experiences with biases and that of her peers.
Career Development and DEI are likely some of the most exciting and challenging parts of the life of an EM, and we can’t stress enough that you’d be in a league of your own if you get good at them.
Conclusion
Managing people and projects are difficult tasks.
As a developer, showing a certain level of frustration or dissatisfaction might be tolerated, permitted, or, at times, even encouraged. Once you are responsible for facilitating interactions between people, you will set the tone.
Consequently, your work life will become more political, and your impact will take a while to show, albeit it’ll likely be more meaningful and long-lasting.
You will be switching from a “hands-on job” (direct impact) to a “hands-off” job (working through others). That does not mean that you will be doing less work; on the contrary. But it does mean that the nature of your work changes considerably. A large part of your job becomes behavioral rather than practical. Being present, listening to people, protecting your team, and creating and shaping an atmosphere conducive to collaboration require you to adopt new habits and develop new skills.
These might come naturally to some people, but most of us need to train these skills. For example, not becoming involved in minute details (i.e., not micromanaging) can become difficult since “minute details” have been your bread and butter as an individual contributor throughout your entire past.
There is no silver bullet to becoming a good manager. Like in any profession, it takes time and experience, and the best way to gain this experience is to first work under others who know their trade well.
This way, you can gradually take over responsibility, safely try things, and practice your new skills with support and experience to guide you so that when the time is right, you’re well prepared to formally leap into management.
So, Should You Become a Manager?
After reading all this, you might be discouraged from taking on this role. The complexity, the challenges, and the discomfort might look like too much.
But maybe you’re exactly the right person for the job.
In software engineering, bad managers often are those lacking practical experience in the field, never working under others doing the job they’re now managing. They never saw the other side of the fence.
Managers without technical experience never have to hone the skills that would later allow them to maintain technical oversight, manage expectations and deal with the complexity of hard problems.
It also meant that they never truly understood what it takes to build solid systems or what the effort entails, which, in turn, can make them difficult people to work for and with.
Explaining basic concepts to the people you report to, can be frustrating, making it harder to relate, discuss solutions, and reason with them. It also makes it extra hard for you to justify decisions and can lead to frustrating disagreements as they are inefficient at mediating technical discussions.
On the other hand, having worked as a developer, you can recognize technical teams’ challenges, empathize uniquely, and be more successful than most in guiding groups to success.
Also, non-technical managers of technical teams never lived through the result of making bad technical decisions as engineers. They were never bitten by the problems that they created. Consequently, they struggle to identify wrong choices and flawed technical justifications and are not as willing to innovate as they incur risks they don’t fully grasp.
All of this has a very personal consequence. It means that a team is no longer ruled by meritocracy as the most competent people are no longer free to advance the product in the best way possible. It’s now a team governed by politics. The friendliest, most eloquent, or political engineer would win an argument where introverts would likely lose their voices.
Every field needs capable people in leadership positions. In the software engineering world, this could be you. Throughout the rest of this book, we’ll help you to understand how to go from an outstanding engineer to an exceptional engineering manager.