Bureaucrats derive their power from their position in the structure, not from their relations with the people they are supposed to serve. The people are not masters of the bureaucracy, but its clients.
— Alan Keyes
To understand how profoundly bureaucrats shaped modern human existence, we need to travel back in time to the end of the last ice age (11,000 B.C.), which saw the dawn of a new era. It was then that humans began slowly settling down, leaving behind their scattered hunter-gather existence, in exchange for a more sedentary lifestyle. Across Eurasia and Africa, farming, herding, mining, and metallurgy led to massive increases in population size, eventually turning small tribes into large, complex, political organizations (such as kingdoms). This increase in complexity required the rise of two powerful new technologies: writing and abstract thought. While archaeologists suggest that the small groups that composed early hunter-gatherers had relatively limited mathematical capabilities that usually did not exceed counting past 10, larger groups of people are more difficult to coordinate, and hence require more complex ways of accounting and spreading ideas. The Babylonians and Egyptians, who are among the early pioneers of mathematics and writing, developed advanced numeric systems as well as methods for multiplication, solving equations, and working with fractions which, among other things, they applied to advancing construction and taxation. The latter allowed these ancient civilizations to amass great wealth, turning them into flourishing empires.
How is this relevant to managing agile projects? Well, regardless of what ancient civilization we look at—from Babylon to Rome—we see that record keeping was at the heart of their complex political organization. The administration of their vast empires would not be possible without it, as the establishment of accountability, as well as the formulation, regulation, and communication of information and activities, is what allowed for coordination and hence expansion and complex organization in the first place.
These fundamental activities of documentation and coordination were undertaken by specialists, whom we today refer to as “bureaucrats” and who, unfortunately, often have a bad reputation. The word “bureaucrats” itself is a derivation from French and translates to “government by people at desks”1—a notion that is accurate, albeit hints at a problematic, distanced, and inefficient form of government. It is important to remember then, as we talk about bureaucracy throughout this chapter, that this “desk governance” forms the fabric of our society. As such, we consider bureaucracy to be the fundamental building block for coordination, and once you realize this, you also realize that most things that we know of today would not be possible without it: Just like bureaucrats built ancient empires, they build modern ones—be it political ones like the US or Chinese governments, or business empires—like those run by Jeff Bezos or Tim Cook.
But just like a building block gives a building its form, it can also be the cause for collapse if stacked unevenly, too high, or without the support of steel and cement. This is the central message that we are trying to convey in this chapter, and if you stop reading now, then you already know most of what we are about to elaborate on: Bureaucracy in itself is an essential force that acts as the lifeblood of any sizable business. Too much of it, however, will stifle and eventually kill productivity.
Types of Bureaucracy
External: Bureaucracy that we do not control and simply need to conform to, government imposed processes like tax declarations, ISO certifications, SEC registration statements, privacy policies, vendor agreements, or company bylaws.
Internal: Bureaucracy is defined within a company and that the company itself has autonomy to change. Examples include Product Requirements Document (PRD), Request for Comments (RFC), and Objectives and Key Results (OKR) documents.
As external bureaucracy is (for the most part) outside of an organization's control, this chapter is solemnly concerned with internal bureaucracy. Unless specifically stated, assume that any reference to bureaucracy refers to the latter.
From Process to Bureaucracy
As we discussed in Chapter 5, processes define clear steps and requirements on how to achieve certain objectives. Over time, as our operating environment changes, these processes need to adapt and incorporate mechanisms to remain useful (or be replaced altogether). As changes accumulate, the complexity of processes tends to grow, which affects the number of required actions to complete them (i.e., processes grow and require more effort to both maintain and execute. An increase in bureaucracy tends to result in an increase in total effort expended to accomplish a goal). The latter typically refers to the amount of the “inputs” or information it needs. That is, when processes evolve and grow, they increase the amount of required paperwork. In other words, they become more expensive and time intensive. Ideally, processes should remain malleable and adaptive so they can continue to be relevant and evolve to deal with new scenarios. However, unless this particular aspect is properly cared for, what we see is the “solidification” of processes, where they become less malleable over time and stop re-evaluating the necessity of all its requirements resulting in what we actually perceive as unneeded bureaucracy.
In general, it is advisable to be careful when judging processes. Processes often have hidden depths, and upon first seeing a process, we tend not to be aware of all scenarios that the process is capable of dealing with. A process that demands a lot of information that might seem unnecessary at a first glance has possibly faced a situation where that information was needed before but (at the time) was not available. This hidden depth might affect the adopter's perception and make the person assume that a given process is simply unneeded bureaucracy when in fact there is a lot of complexity that is simply not visible to the naked eye.
That said, the fact that a certain process demands a lot of energy or information from its adopters is not always a sign that it is extremely capable and ready to deal with different situations. Just as any other product of human action, it might be poorly optimized or a victim of misaligned interests.
A poorly optimized bureaucratic process would be one that although it might cover a lot of different scenarios, many of those scenarios are unlikely, overestimated, or even totally outdated, despite that, the user needs to comply with a long or time-consuming list of requirements just to avoid those unlikely scenarios. Like one of your meetings that could easily be an email, although simple and apparently innocuous they tend to happen often, killing the team productivity without delivering the expected result.
A bureaucratic process with misaligned interests however can be very damaging to the organization, those types of processes can be qualified as such when the person controlling/executing the process wants to shift the burden of his work onto the adopter of the process or wants to prevent certain objectives from being accomplished in the first place. One common example of such conflict of interest in engineering often occurs between the verticals within that department. For example, when dividing teams into separate units of backend and frontend rather than full-stack units that act in cohesion, the interests between these two units may conflict. One common approach for conflicting teams to “get their way” and work on what they consider a priority (but which may not be a priority for the company as a whole) is to introduce roadblocks in the form of paperwork. Instead of collaborating with the team requesting a certain feature or implementation, they first require this team to jump through as many hoops and approval processes as possible: unnecessary RFCs that take a long time to review, duplicate PRDs that may require approval from other sectors of the organization, and so on so forth. Instead of cooperating coherently and transparently, different teams create paperwork as shields to protect their own interests.
Internal Organization
The larger an organization, the more people and activities there are to coordinate. Seeing how the latter (organization of key activities) is the key objective of bureaucracy, the bureaucracy system of an organization naturally grows as the organization increases in size.
More people also naturally implies more interests, and as the number of distinct individual interests grows, so do the chances for malalignment. In other words, damaging bureaucratic processes that arise due to conflict of interests may increase exponentially to the amount of new hires (each new person leads to a n*n-1 new set of relationships.). It is important to understand that this does not need to be the case if a company (i) pays careful attention to its internal organization, (ii) instills a culture of collaboration rather than competition, and (iii) gives autonomy to teams to allow them to make decisions without, or with minimal, external dependencies. Based on our experience, the former is the most important to get right, as even if a company's culture may not be universal across the organization as a whole, organizing people effectively allows the company to remedy cultural shortcomings.
Effective organization in essence means creating groups of people that share a common interest that aligns with the interests of the company as a whole. Vertical vs. horizontal organization provides a good example of this. Engineering departments that are organized in verticals have a top-down command structure where responsibilities are “sliced” by technical capability. For example, a backend team is controlled by a director and subordinates on one hand, and a frontend team is controlled by a different director and subordinates on the other hand. A horizontal organization is a synonym for worker autonomy and, within our context, involves creating full-stack teams that all take ownership of certain products or parts of products. In the case of the full-stack team, interests are aligned: Their objective is the success of the team, and hence the product as a whole. In the case of the vertical teams, cohesion does not come so naturally: faced with one product, two teams essentially compete to achieve objectives set forth by different directors. If the vision, philosophy, or approach to the engineering of these directors differ in any way, conflict is likely to arise as a result. As such, preventing misaligned interests is much more difficult with vertical organizations as leadership needs to always be closely aligned in order for the machine to function well.
Another example of misalignment, now on a strategic level, is the incorrect use of Key Performance Indicators (KPIs) and Objectives and Key Results (OKRs). Companies use these as tools to set the goals that will lead them to reach strategic objectives but often fall into the trap of setting too many of them on vertical instead of horizontal levels. An example of this is team-level OKRs. Although there’s nothing inherently wrong with the practice, they’re often set in a way that discourages collaboration. Every team wants and is accountable for achieving its goals, but if the goals are not set correctly or aligned with those of other teams, groups will actively avoid helping each other because they have their own urgencies.
Verticals also bring with them a natural increase in process work: Different teams need to coordinate, and coordination naturally means some form of bureaucracy.
Inefficiencies
Today, a large body of research and literature exists examining the relationship between organizational size and bureaucracy. One indicator of the level of bureaucracy within an organization is to use what is called “administrative intensity.” That is, “the number of managers, professionals, and clerical workers divided by the number of craftsmen, operatives, and laborers employed by the organization.”2 When studying administrative intensity, researchers frequently reference two prevalent theories: Parkinson's Law and Public Choice Theory. The latter, when applied to bureaucratic processes, essentially argues that bureaucratic processes introduce inefficiencies as organizations grow. That is, “economies of scale are eventually counter-acted by bureaucratic congestion.”3 Similarly, Parkinson's Law “states that bureaucracy grows regardless of the quantity of work to be undertaken and even when the overall size of an organization is shrinking” (Boyne 1986; Parkinson 1957).4 In other words, Parkinson's Law argues that bureaucracy is an animal in itself and not naturally coupled to productivity. Instead, bureaucratic processes can take on a life of their own, generating work and consuming resources for the sake of generating work and consuming resources. Bureaucracy, proponents of Parkinson's Law argue, is an end in itself, not a means to the end of other activities.
Both arguments have merit, and indeed we have seen their real-world manifestations many times. For example, if you set a 1-hour meeting, people will fill up that hour and rarely try to finish it sooner. Bureaucracy does indeed increase with organization size and can become a burden as it does. Bureaucratic processes can also be carried out for the sake of it, without serving coordination or record keeping (although a certain number of bureaucrats may always be required due to external bureaucratic constraints, like laws or legal compliance due to industry regulations like PCI, HIPAA, etc.) With emphasis, the word can in the previous two sentences, as these inefficiencies don't’ always need to hold true. They can be avoided by (i) understanding what problems internal bureaucratic processes should be solving, (ii) understanding what healthy bureaucracy looks like, (iii) identifying the attributes unhealthy bureaucracy exhibits, and (iv) knowing how to apply the understanding of bureaucracy in order to strike a balance and solve for inefficiencies. We will discuss each of these points in the following sections.
Characteristics of Healthy Bureaucracy
Justifiable – They have a good reason to exist and that reason is still relevant today as it were when the process was created;
Periodically Reviewed – People responsible for this process review it periodically in order to ensure that it is still relevant, making adjustments as needed. An important part of this revision process is to ensure that cases which were required in the past are still relevant today. Requirements that are now obsolete present opportunities to decrease the complexity of the process and should be abandoned.
Documented – The required information, steps, and estimation time to complete are all relevant information that should be available on some common repository so people that need to go over this bureaucratic process may plan accordingly.
Malleable – In order to avoid the “solidification” we described previously it is important to keep the process flexible enough to adapt to new scenarios while efficient to deal with the existing ones, this balance of course is not trivial, ensuring that all the involved stakeholders can present their views (usually during period reviews) is a good way to ensure that everyone is satisfied or at least understands the requirements for a heavier process.
Healthy bureaucratic processes are also usually self-evident: that is, their reason for existence is clear to all involved. When executing a healthy process, team members should not be asking themselves what the purpose of a given activity is. The activity should be a natural part of their job, not a hurdle or hoop that people jump through.
Characteristics of Excessive Bureaucracy
Misaligned interests – When the person responsible for the process does not have the same interests as the adopters of a process the end goal diverges from what it should be, resulting in unneeded overhead. Unnecessary meetings, paperwork that is written once and never read are the most common side effects.
Inertia – The bureaucratic process exists for its own sake. That is, “is the way it is simply because it always was this way.” The responsible for the process is not interested in making changes or the system that is in place does not support a change that would make it better. The side effects tend to be unneeded overhead just like with misaligned interests but the reason behind it is different.
MBA-Syndrome – The creators of the process read somewhere that this bureaucratic process is what they should implement, without properly understanding why and how. They decided to copy procedures without really understanding what these procedures are intended to solve. As a result, the people executing the nonsensical process do not know why they are executing it. Nobody can provide a clear objective, nor does anybody know what problem the process is solving for.
Striking a Balance
Successful bureaucracy means striking a balance between productivity and record keeping. The best tip that we can give to this end is: use your common sense. If you don't know why you are performing a given bureaucratic process, if its activities are duplicated in some shape or form, or if executing the bureaucratic process impacts your productivity or timelines, then it's time to revisit (and probably discard) the entire process.
Furthermore, pay attention to your internal organization. How you structure teams and departments, and the egos that play a role in creating this structure, will have a big impact on whether or not “desk governance” forms a healthy part of your organization, or whether it drains resources.
Last but not least, remain vigilant. Just because a process that was created in the past made sense then, does not mean that it makes sense today. Review and revise. And do so periodically.
Conclusion
Staying true to the underlying message in this chapter that record keeping should be efficient, concise, sensical, and goal-oriented, we terminate this brief chapter abruptly. Ancient bureaucrats laid the foundations of our modern civilization. They ushered in the dawn of writing and abstract thought, casting our planet into a new bright light. It is up to our common sense to ensure that this light is not blocked by stacks of paper. Whether executing existing processes, or defining new ones, keep in mind that (i) a certain amount of record keeping is always required due to external bureaucratic requirements, (ii) misaligned interests, inertia, and “the unknown” are all characteristics of unhealthy processes. If your process exhibits any of the latter attributes, it’s time to review and revise.