Chapter 2. Roles for the Scalable Technology Organization

When the general is weak and without authority; when his orders are not clear and distinct; when there are no fixed duties assigned to officers and men, and the ranks are formed in a slovenly haphazard manner, the result is utter disorganization.

—Sun Tzu

A common cause of failures in scalability and availability is lack of clarity in responsibilities of people. In this chapter, we start by taking a look at two very real examples of what might happen without role clarity and clearly defined responsibilities. We also discuss executive roles, example organizational responsibilities, and the roles of various individual contributors. We conclude the chapter by introducing a tool that can aid in responsibility definition and help to reduce role-based (or affective) conflict.

This chapter is meant for companies of all sizes. For large companies, it can serve as a checklist to ensure that you have covered all of the technology and executive roles and responsibilities as they relate to scale. For small companies, it can help jumpstart the process of ensuring that you have your scalability-related roles properly defined. For technology neophytes, it serves as a primer for how technology organizations should work; for seasoned technology professionals, it is a reminder to review organizational structure to validate that your scalability-related needs are covered. For all companies, it clearly defines why everyone in a company has a role in creating scalable products.

In this case, the absence of a team or person responsible for matching expected demand to existing capacity and determining appropriate actions would be disastrous in an environment where demand is growing rapidly. Nevertheless, this failure happens all the time—especially in young companies. Even companies that have a person or organization responsible for capacity planning often fail to plan for their newest system additions.

At the other end of the spectrum from the “missing an owner” problem is the case where multiple organizations or people are given the same goal. Note that this is not the same thing as “sharing a goal.” Sharing a goal is good, because “sharing” implies cooperation. Rather, what we are describing here is the existence of multiple owners without clarity as to who owns which actions to achieve that goal and sometimes without understanding that another group or person is working on the same thing. If you work in a smaller company where everyone knows what everyone else is doing, this concern may seem a bit ridiculous to you. Unfortunately, this problem exists in many of our larger client companies—and when it happens, it not only wastes money and destroys shareholder value, but can also create long-term resentment between organizations and destroy employee morale. Few things will destroy morale more quickly than a group believing that it is fully responsible but failing to deliver on a task or project because the group members assumed someone else owned a part that did not complete. The multiple-owner problem is often at the heart of the role-based (“affective”) conflict that we described in Chapter 1, The Impact of People and Leadership on Scalability.

As an example, let’s assume that an organization is split between an engineering organization, which is primarily responsible for developing software, and an operations organization, which is primarily responsible for building, deploying, and operating databases, systems, and networks. Let’s further assume that we have a relatively inexperienced CTO who has recently read a book on the value of shared goals and objectives and has decided to give both teams the responsibility of scaling the platform to meet expected customer demand. The company has a capacity planner who determines that to meet next year’s demand, the teams must scale the subsystem responsible for customer contact management to handle at least twice the number of transactions it is capable of handling today.

Both the engineering and operations teams have architects who have read the technology section of our book, and both decide to make splits of the database supporting the customer contact management system. Both architects believe they are empowered to make the appropriate decisions without the help of each other, as they are unaware that multiple people have been assigned the same responsibility and were not informed that they should work together. The engineering architect decides that a split along transaction boundaries (or functions of a Web site, such as buying an item and viewing an item on an ecommerce site) will work best. In contrast, the operations architect decides that a split along customer boundaries makes the most sense, such that various groups of customers will reside in separate databases. Both architects go about making initial plans for the split, setting their teams to doing their work and then making requests of the other team to perform some work.

This example may sound a bit ridiculous to you, but it happens all the time. In the best-case scenario, the two teams will stop there and resolve the issue having “only” wasted the valuable time of two architects. Unfortunately, what usually happens is that the teams become polarized and waste even more time in political infighting, and the final, probably delayed result isn’t materially better than if a single person or team had the responsibility to craft the best solution with the input of other teams.

These examples of executive and individual contributor responsibilities are not meant to restrict you to specific job titles or organizational structure. Rather, they are intended to help you outline the necessary roles within a company and define certain responsibilities or professional focus areas. We’ve chosen to define these roles by the organizations in which they have traditionally existed to make them easier to understand for a majority of our audience. For instance, you may decide that you want operations, infrastructure, engineering, and quality assurance (QA) to exist within the same teams, with each team dedicated to a specific product line (in fact, as you will see in the next chapter, we strongly suggest such an organization structure).

You may recall from our introductory discussion that there is no “right or wrong” answer when designing an organizational structure—there is simply a set of benefits and drawbacks for any decision. The important point is to remember that you should include all of the appropriate responsibilities in your organizational design and that you should clearly define not only who is a responsible decision maker but also who is responsible for providing input to any decision, who should be informed of the decision and actions, and who is responsible for making the decision happen. Perhaps the most critical element in making the best decisions consistently is having the best decision-making process in place, as that ensures the right information is gathered by the right people to inform the ultimate decision maker. That applies to anyone who must make decisions in the company. We’ll discuss this last point in a brief section on a valuable tool later in this chapter. The appropriate organizational structure can help limit some of the conflict that may otherwise result from a lack of clarity, although conflict can never be completely eliminated. At the very least, as a leader, you should be clear about the responsibilities and desired outcomes for any group within your organization.

It is hard to imagine that someone could rise to the position of CEO and not understand how to read a balance sheet, income statement, or statement of cash flow. That same person, unless he or she has an accounting background or is a former CFO, is not likely to understand the intricacies of each accounting policy—nor should this individual need to. The CEO’s job is to ask the right questions, to get the right people involved, and to solicit the right outside help or advice to arrive at the right answer.

The same holds true in the technical world—the CEO’s job is to understand some of the basics (the equivalent of the financial statements mentioned previously), to know which questions to ask, and to know where and when to get help. Here is some advice for CEOs and other managers responsible for technical organizations who have not been the chief technology officer or chief information officer of a company, do not have technical degrees, or have never been an engineer.

This executive interrogation is a key ability shared by many successful leaders. The Bill Gates version of this skill was known as the “BillG Review.”¹ Knowing when to probe and where to probe and probing until you are satisfied with answers need not be limited to the CEO. In fact, managers and individual contributors should all hone this skill—and start early in their careers when doing so.

1. See Joel Spolsky. “My First BillG Review.” http://www.joelonsoftware.com/items/2006/06/16.html.

More than likely, the CEO will decide to delegate authority to several members of his or her team, including the chief financial officer, individual business unit owners (also known as general managers), and the head engineering and technology executive (referred to as either the CTO or the CIO in our book).

The CFO is also not likely to come from a very technical background, but can benefit from asking the right questions and creating an appropriate network, just as the CEO can. Questions that the CFO might ask regarding scalability include which other scale alternatives were considered in developing the proposed budget for scale and which tradeoffs were made in deciding upon the existing approach. The intent here is to ensure that the team considered more than one option. A bad answer would be “This is the only way possible,” as that is rarely the case. (We want to say it is never the case, but on rare occasions only one route may really be possible.) For example, if the company is facing challenges in lowering its expenses, a CFO might decide that older equipment such as servers and networking gear can still be used because its depreciation has fallen off the books. In reality, however, older equipment is usually much less efficient and much more prone to failure, which will cause its total cost of ownership to increase significantly. A good answer, then, might be “Of the options we evaluated, this one allows us to scale horizontally at comparatively low cost while setting us up to scale even more cost-effectively in the future by creating a framework from which we can continue our horizontal scale.”

Very often, we run into situations in which the business unit owner claims that demand simply can’t be forecasted. Here, demand means the number of requests that are placed against a system or product. This is a punting of responsibility that simply should not be tolerated within any company. In essence, the lack of ownership for forecasting demand by the business means that this responsibility gets moved to the technology organization, which is very likely less capable of forecasting demand than the business unit owner is. Yes, your forecasts will probably be wrong, especially in their infancy, but it is absolutely critical that you start the process early in the life cycle of the company and mature it over time.

Finally, like the other senior executive staff of the company, the business unit owner is responsible for helping to create a culture of scalability. This individual should make sure to ask the right questions of his or her peers (or subordinates) in the technology organization and try to ensure that those technology partners receive the funding and support to properly assist the business units in question. Both efforts are essential to the success of scalability within the company.

In essence, “the buck stops here.” Although it is true that the CEO can’t really “delegate” responsibility for the success of the platform scalability initiatives, it is also true that the chief technology executive inherits that responsibility and shares it with the CEO. A failure to properly scale will likely at least result in the termination of the chief technology executive, portions of his or her organization, and potentially even the CEO.

The CTO/CIO must create the technical vision for the company overall. For the purposes of our discussion, within a growth company that vision must include elements of scale. The chief technology executive is also responsible for setting the aggressive, measurable, and achievable goals that are nested within that vision and for ensuring that his or her team is appropriately staffed to accomplish the associated scalability mission of the organization. The CTO/CIO is responsible for the development of the culture and processes surrounding scalability that will help ensure that the company always stays ahead of end-user demand.

The CTO/CIO will absolutely need to delegate responsibilities for certain aspects of decision making around scalability as the company grows. Of course, as we pointed out previously, such delegation never eliminates the executive’s own responsibility to ensure that scalability is done correctly, on time, and on budget. Additionally, in hyper-growth environments where scale is critical to company survival, the CTO should never delegate the development of the vision for scale. The term “lead from the front” is never more important than in this context, and the vision does not need to be deeply technical.

Although the best CTOs we have seen have had technology backgrounds varying from once having been an individual contributor to having been a systems analyst or technical project manager, we have seen examples of successful CTOs without such backgrounds. With a nontechnical CTO/CIO, it is absolutely critical that this individual has some technical acumen and is capable of speaking the language and understanding the critical tradeoffs within technology, such as the relationship of time, cost, and quality. Calling upon a technology neophyte to lead a technical organization is akin to throwing a non-swimmer overboard into a lake; you may be pleased with your results assuming the person can swim, but more often than not you will need to find a new partner in your boat. Equally important with respect to a neophyte CTO is gaining trust from the technical staff; without this trust from below, the CTO cannot hope to lead effectively.

Equally important is that the CTO have some business acumen. Unfortunately, this can be as difficult to achieve as finding a chief marketing officer with a Ph.D. in electrical engineering (not that you’d necessarily want one)—such people exist, but they can be challenging to find. Unfortunately, most technologists do not learn about business, finance, or marketing in their undergraduate or graduate courses. Although the CTO does not need to be an expert on capital markets (that’s likely the job of the CFO), he or she should understand the fundamentals of the business in which the company operates. For example, the CTO should be able to analyze and understand the relationships between the income statement, the balance sheet, and the statement of cash flow. In a technology-centric company like an Internet firm, the CTO will often have the largest budget in the company. Such an executive will often be charged with making very large investments in capital, telecommunications carriers, data center leases, and software licensing—and his or her lack of understanding of financial planning may expose the company to large risks in capital misappropriation, overpaying vendors, and unplanned and unforeseen expenses (such as periodic software license “true-ups”). The CTO should also understand marketing basics at the level of at least a community college– or company-sponsored course on the subject. This is not to say that the CTO needs to be an expert in any of these areas; rather, a basic understanding of these topics is critical to making the business case for scalability and to being able to communicate effectively in the business world. We’ll discuss these areas in later chapters.

Our goal here is not to define strict organizational or functional boundaries. As mentioned earlier, organizing by function is one way of structuring an organization; in particular, it is an effective approach when employing classical waterfall approaches to development. As we will describe in Chapter 3, Designing Organizations, when companies focus on developing products and need to do product discovery, we prefer organizations that are aligned with the products a company produces and processes that enable product discovery.

Architects are responsible for designing a system and having designs ready to solve any scale-related issue. In Part II, “Building Processes for Scale,” we identify a key process that the architecture team should adopt to help identify scale-related problems across all of the technology disciplines.

Architects may also be responsible for forming information technology (IT) governance, standards, and procedures, and for enforcing those standards through such processes as the Architecture Review Board discussed in Chapter 13, Joint Architecture Design and Architecture Review Board. When architects perform these roles, they do so at the request of the chief technology executive. Some larger companies may create process-engineering teams responsible for procedure definition and standards enforcement.

Companies use many different titles to designate the individuals who are focused on designing the technology architecture, including software architect, systems architect, scalability architect, enterprise architect, and even Agile architect. These roles, no matter the title used, tend to focus on one of two areas. The first is software architecture, which is primarily concerned with how the software is designed and structured. These individuals focus on areas such as service-oriented architecture frameworks or coding patterns that provide guidance and governance to developers. The second is system architecture, which is primarily concerned with how the software is deployed and supported on the hardware (or virtual machines). These individuals focus on areas such as eliminating single points of failure, fault isolation, and capacity planning. From the customer’s perspective, the scalability of a system is mostly concerned with the system architecture. This important role is covered in more depth in the discussion of individual contributor responsibilities later in this section.

In the Software as a Service (SaaS) and Web 2.0 worlds, DevOps is usually responsible for deploying, running, and monitoring the systems that create the company’s revenue. This team should be part of the Agile development efforts described in detail later in this book. If you do not employ either our Agile organizational structure or Agile development practices, you might still at least consider embedding DevOps personnel within engineering teams to help them better understand the deployment architectures of the solutions they create. Given that the team interacts with how the system runs every day and has daily insight into system utilization data, these team members are uniquely qualified to identify bottlenecks within the system and to design software with deployment in mind.

Often, DevOps is responsible for development/testing environments, build scripts, deployment scripts, monitoring solutions, log file aggregation, and development or testing tools. DevOps personnel might be responsible for creating reports that show trended availability over time, bucketing root cause and corrective actions, and determining mean time to resolution and mean time to restoration for various problems.

Regardless of the composition of the team, the organization responsible for monitoring and reporting on the health of systems, applications, and quality of service plays a crucial role in helping to identify scale issues. The processes that this group employs to manage issue and problem resolution should feed information into other processes that help identify scale issues in advance of major outages. The data that the operations organization collects is incredibly valuable to those performing capacity planning as well as to those responsible for designing away systemic and recurring issues such as scale-related events. The architecture and engineering teams rely heavily on DevOps to help them identify what should be fixed and when. We discuss some of these processes in Part II—more specifically in Chapter 8, Managing Incidents and Problems, and Chapter 13, Joint Architecture Design and Architecture Review Board.

Within companies hosted in a cloud environment (e.g., Infrastructure as a Service [IaaS] providers like Amazon Web Services) the infrastructure team is often responsible for managing the virtual machine instance, the network (i.e., load balancers), and security (i.e., firewalls). Infrastructure engineers also aid in identifying capacity constraints on the systems, network devices, and databases that they support and help determine appropriate fixes for scale-related issues. Finally, they are responsible for making sure the systems they put in place can be managed and monitored at scale.

You might notice that we use the word emergency when describing the purchase of a larger system. Many companies take the approach that “scaling up” is an effective strategy. Our position, as we will describe in Chapters 21 through 25, is that if your scaling strategy relies on faster and bigger hardware, your solution does not scale. Rather, you are relying on the scalability of your providers to allow you to scale. Stating that you scale by moving to bigger and faster hardware is like stating that you become fast by buying a bigger, faster car. You have not worked to become faster, and you are only as fast as anyone else with similar wealth. Scalability is the ability to scale independently of bigger and faster systems or the next release of an application server or database.

The tool we use most often is called RASCI. It is a responsibility assignment chart, and the acronym stands for Responsible, Accountable, Supportive, Consulted, and Informed.

• R: Responsible. This is the person responsible for completing the project or initiative.

• A: Accountable. This is the person to whom R is accountable and who must approve the work before it is okay to complete. When using RASCI as a decision-making tool, the A is sometimes referred to as the approver of any key decision.

• S: Supportive. These people provide resources to complete the project or initiative.

• C: Consulted. These people have data or information that can be useful in completing the project.

• I: Informed. These people should be notified, but do not need to be consulted or provide input to the project.

RASCI can be used in a matrix, where each activity or initiative is spelled out along the y-axis (vertical axis) of the matrix and the individual contributors or organizations are spelled out on the x-axis of the matrix. The intersection of the activity (y-axis) and the organization (x-axis) contains one of the letters R, A, S, C, or I, but may include nothing if that individual or organization is not part of the initiative.

Ideally, in any case, there will be a single R and a single A for any given initiative. This helps eliminate the issue we identified earlier in this chapter—that is, having multiple organizations or individuals believe that they are responsible for any given initiative. By holding a single person or organization responsible, you are abiding by the “one back to pat and one throat to choke” rule. A gentler way of saying this is that distributed ownership is ownership by no one.

This is not to say that others should not be allowed to provide input into the project or initiative. The RASCI model clearly allows and enforces the use of consultants or people within and outside your company who might add value to the initiative. An A should not sign off on an R’s approach until such time as the R has actually consulted with all of the appropriate people to develop the right course of action. Of course, if the company has the right culture, not only will the R want to seek those people’s help, but the R will make them feel as if their input is valued and value is added to the decision-making process.

You can add as many C, S, and I personnel whom you would like and who add value or who are needed to complete any given project. That said, guard against going overboard in terms of who exactly you will inform. Remember the earlier point about people getting bogged down with email and communication that does not concern them. Young companies often assume that everyone should feel involved in every decision or be informed of every decision. This information distribution mechanism simply does not scale, however—and results in people reading emails rather than doing what they should be doing to create shareholder value.

A partially filled-out example matrix is included in Table 2.1. Based on our discussion thus far regarding different roles, let’s see how we’ve begun to fill out this RASCI matrix.

Earlier, we indicated that the CEO absolutely must be responsible for the culture of scalability, or the scale DNA of the company. Although it is theoretically possible for the CEO to delegate this responsibility to someone else within the company from a practical perspective, and as you will see in the chapter on leadership, this executive must live and walk the values associated with scaling the company and its platform. As such, even with delegation and as we are talking about how the company “acts” with respect to scale, the CEO absolutely must “own” this responsibility. Therefore, we have placed an R in the CEO’s column next to the Scalability Culture initiative row. The CEO is obviously responsible to the board of directors and, as the creation of scale culture has to do with overall culture creation, we have indicated that the board of directors is the A.

Who are the S people in the Scalability Culture initiative? Who should be informed and who needs to be consulted? In developing your answer to this question, you are allowed to have people who are supportive (S) of any situation also be consulted (C) in the development of the solution. It is implied that C and S staff will be informed as a result of their jobs, so it is generally not necessary to include an I wherever you feel you need to communicate a decision and a result.

We’ve also completely filled out the row for Technical Scalability Vision. Here, as we’ve previously indicated, the CTO is responsible for developing the vision for scalability for the product/platform/system. The CTO’s boss is very likely the CEO, so that executive will be responsible for approving the decision or course. Note that it is not absolutely necessary that the R’s boss be the A in any given decision. It is entirely possible that the R will be performing actions on behalf of someone for whom he or she does not work. In this case, however, assuming that the CTO works for the CEO, there is very little chance that the CTO would actually have someone other than the CEO approve his or her scalability vision or scalability plan.

The CTO’s peers are consultants to the scalability vision—the people who rely on the CTO for either the availability of the product or the back-office systems that run the company. These people need to be consulted because the systems that the CTO creates and runs are the lifeblood of the business units and the heart of the back-office systems that the CFO needs to do his or her job.

We have indicated that the CTO’s organizations (Architecture group, Engineering team, Operations team, Infrastructure team, and QA) are all supporters of the vision, but one or more of them could also be consultants to the solution. The less technical the CTO, the more that executive will need to rely upon his or her teams to develop the vision for scalability. In Table 2.1, we have assumed that the CTO has the greatest technical experience on the team, which is obviously not always the case. The CTO may also want to bring in outside help in determining the scalability vision or plan. This outside help may take the form of a retained advisory services firm or potentially a technology advisory and governance board that provides the same governance and oversight for the technology team that a board of directors provides at a corporate level.

Finally, we have indicated that the board of directors needs to be informed (I) of the scalability vision. This might be a footnote in a board meeting or a discussion centered on what is possible with the current platform and how the company will need to invest to meet the scalability objectives for the coming years.

The remainder of the matrix has been partially filled out. Important points with respect to the matrix are that we have split up the tasks/initiatives to avoid overlaps in the R category. For instance, the responsibility for infrastructure tasks has been split from the responsibility for software development or architecture and design tasks. This allows for clear responsibility in line with our “one back to pat and one throat to choke” philosophy. In so doing, however, the organization might tend to move toward designing in a silo or vacuum, which is counter to what you would like to have in the long term. Should you structure your organization in a similar fashion, it is very important that you implement processes requiring teams to design together to create the best possible solution. Matrix-organized teams do not suffer from some of the silo mentality that exists within teams built in silos around functions or organizational responsibility, but they can still benefit from RASCI. You should still have a single responsible organization, but you want to ensure that collaboration happens. RASCI helps enforce that philosophy through the use of the C attribute.

You should spend time working through the rest of the matrix in Table 2.1 until you feel comfortable with the RASCI model. It is a very effective tool in clearly defining roles and responsibilities and can help eliminate duplicated work, unfortunate morale-deflating fights, and missed work assignments.

One final note on roles and responsibilities: No team should ever cite roles and responsibilities as a barrier to getting things done. The primary responsibility of any person in the company is to create value for customers and stakeholders. There will certainly be times when people must perform roles and execute tasks that are outside of their defined responsibility. To the degree that such efforts help the company achieve its mission, personnel can and should be willing to step outside their boundaries to get the work done. The important message is that when those situations occur, they should work with leadership to identify where there are consistent gaps, and leaders should commit to getting those corrected for the future.

We also introduced a tool called RASCI to help define roles and responsibilities within the organization. Feel free to use RASCI for your own organizational roles and for roles within initiatives. The use of RASCI can help eliminate duplicated work and make your organization more effective, efficient, and scalable.

• Overlapping responsibility creates wasted effort and value-destroying conflicts.

• Areas lacking responsibility create vacuums of activity and failed scale initiatives.

• The CEO is the chief scalability officer of the company.

• The CTO/CIO is the chief technical scale officer of the company.

• RASCI is a tool that can help eliminate overlaps in responsibility and create clear role definition. RASCI is developed in a matrix in which

R stands for the person responsible for deciding what to do and running the initiative.

A is accountable for the initiative and the results or the approver in a decision-making process.

S stands for supportive, referring to anyone providing services to accomplish the initiative.

C stands for those who should be consulted before making a decision and regarding the results of the initiative.

I stands for those who should be informed of both the decision and the results.