Chapter 13. Joint Architecture Design and Architecture Review Board

Thus it is that in war the victorious strategist only seeks battle after the victory has been won, whereas he who is destined to defeat first fights and afterwards looks for victory.

—Sun Tzu

In this chapter, we will introduce two proactive processes that are both cross-functional in nature and interwoven within the product or software development life cycle: joint architecture design (JAD) and the Architecture Review Board (ARB). JAD is a collaborative design process wherein all engineering personnel necessary to develop some new major functionality work together to define a design consistent with the architectural principles of the organization. The ARB is a review board of select architects from each of the functional or business areas, who ensure that, prior to the final signoff of a design, all company architectural principles have been incorporated and best practices have been applied.

In dysfunctional organizations, the implementation of a cross-functional JAD is challenging but absolutely necessary to help cure the dysfunction. As we discussed in Chapter 3, Designing Organizations, function-based organizations often experience affective conflict (the bad kind of conflict, as described in Chapter 1). Team members associate their social identity with their functional team rather than with the cross-functional team that delivers the product or service. This naturally creates an “us versus them” mentality. The different teams might also suffer from the “experiential chasm,” as discussed in Chapter 6, Relationships, Mindset, and the Business Case. Likewise, personnel handling different technology functions can find it difficult to communicate and may have widely differing skill sets, creating a gap that separates them just like the one between the business and technology teams. When a company’s subdivisions are organized by function, the implementation of a process to bring teams together to jointly design the product is essential. (We’ll talk in the next chapter about how the Agile organization helps resolve this issue without using process.)

A JAD group should include the software engineer responsible for ultimately coding the feature, an architect, at least one but possibly multiple operations people, and—as needed based on the feature—the product manager, a project manager, and a quality assurance engineer. As mentioned earlier, each person on such a team brings unique knowledge, perspectives, experiences, and goals that complement as well as counter-balance the others. With a cross-functional team, the parties recognize more needs than just their own. For example, although the operations engineer still has the goal from her own organization of maintaining availability, she now also has the goal of designing a feature that meets the business requirements. This retention of the original goal combined with the embrace of a cross-functional goal helps ensure that she is vigilant as ever about what goes into production.

The JAD approach is not limited to waterfall development methodologies, where one phase of product development must take place before another phase begins. JAD can and has been successfully used in conjunction with all types of development methodologies, such as iterative or Agile approaches, in which specifications, designs, and development evolve as greater insights into the product feature are gained. Each time a design is modified or extended, a JAD can be called to help with it. The type of architecture does not preclude the use of JAD, either. Whether it is a traditional three-tier Web architecture, service-oriented architecture, or simply a monolithic application, the collaboration of engineering, operations, and architects to arrive at a better design is simply taking advantage of the fact that solutions arrived at by teams are better than those produced by individuals. The more diverse the background of the team members, the more holistic the solution is likely to be.

The actual structure of the JAD team is very informal. After the team has been assigned to the feature, one person takes the lead on coordinating the design sessions; this individual is typically the software engineer or the project manager, if one is assigned. Usually multiple design sessions are held; they can last between one and several hours depending on people’s schedules. For very complex features, multiple design sessions for various components of the feature should be scheduled. For example, a session focusing on the database and another addressing the cache servers should be set up separately.

Typically, the sessions start with a discussion covering the background of the feature and the business requirements. During this phase, it is a good idea to have the product manager present and then on call for any clarifications as questions come up. After the product requirements have been discussed, a review of the architectural principles that relate to this area of the design is usually a good idea. Next, the team brainstorms and typically arrives at a few possible solutions. These options are written up at the end of the meeting and sent around for people to ponder until the next session. Usually only one or two sessions are required to come to an agreement on the best approach for the design of the feature. The final design is written down and documented for presentation to the ARB.

The entrance criteria for JAD are the following:

• Feature Significance. The feature must be significant enough to require the focus of a cross-functional team. The exact nature of significance can be debated. We suggest measuring it in three ways:

Size. For size, we use the amount of effort needed for the feature development as the measurement. Features requiring a sum of more than 10 total engineering days are considered significant.

Potential impact on the overall application or system. If the feature touches many of the core components of the system, it should be considered significant enough to design cross-functionally.

Complexity of the feature. If the feature requires components that are not typically involved in features such as caching or storage, it should go through JAD. A feature that runs on the same type of application server as the rest of the site and retrieves data from the database is not complex enough to meet this requirement.

• Established Team. The feature must have representatives assigned and present from engineering, architecture, and operations (database and system administrators, possibly a network administrator). If needed, personnel from the quality assurance, project management, and product management areas should be assigned to the JAD team. If these required team members are not assigned and made available to attend the meetings, the feature should not be allowed to undergo JAD.

• Product Requirements. The feature must have product requirements and a business case that warrant use of JAD. Tradeoffs are likely to be made based on different architectural solutions, and the team will need to be able to distinguish the critical requirements from the nice-to-have ones. Understanding the revenue generated will also help the team decide how much investment should be considered for different solutions.

• Empowerment. The JAD team must be empowered to make decisions that will not be second-guessed by other engineers or architects. The only team that can approve or deny the JAD design is the ARB, which performs the final review of the architecture. In RASCI terminology, the JAD team is the R (Responsible) for the design and the ARB is the A (Accountable).

The exit criteria for a feature coming out of JAD are the following:

• Architectural Principles. The final design of the feature must follow all architectural principles that have been established in the organization. If there are exceptions to this rule, they should be documented and questioned by the ARB, resulting in a possible rejection of the design. We will talk more about the ARB process in the next chapter.

• Consensus. The entire team should be in agreement and support the final design. The time to voice dissent in regard to the design is during the team meetings, not afterward. If someone from the JAD team begins second-guessing team decisions, this should be grounds for requiring the JAD process to be conducted again, and any development of the feature should be stopped immediately.

• Documentation of Tradeoffs. If any significant tradeoffs were made in the design with respect to the requirements, cost, or principles, they should be clearly articulated to the ARB and for any other team member to reference when reviewing the design of the feature.

• Documentation of the Final Design. The final design must be documented and posted for reference. It may or may not be reviewed by the ARB, but the design must be made available for all teams to review and reference in the future. These designs will soon become system documentation as well as design patterns that engineers, architects, and operations folks can reference when they are participating in future JAD processes.

• ARB. The final step in the JAD process is to decide whether the feature needs to go to the ARB for final review and approval. We will talk in more detail in the next chapter about which features should be considered for ARB review, but here are our basic recommendations for appropriate criteria:

Noncompliance with architectural principles. If any of the architectural principles were violated, the feature should go through ARB review.

Projects that cannot reach consensus on design. If the team fails to reach consensus, the project can be either reassigned to a new JAD team or sent to the ARB for a final decision on the competing designs.

Significant tradeoffs. If tradeoffs had to be made in terms of product requirements, cost, or other non-architectural principles, this compromise should flag a feature as needing ARB review.

High-risk features. We will discuss how to assess risk in much more detail in Chapter 16, Determining Risk; for now, simply recognize that if the feature is considered high risk, its design should go through ARB review. A quick way of identifying high-risk features is to look at how many core components the feature touches or how different it is from other features. The more core components that are touched and the greater the difference from other features, the higher the risk.

People who hold a particular position can be important to the ARB to provide the gravitas to uphold the board’s decisions. The ARB needs to include the right people to make the right decision and to be given the final authority over making that decision. In turn, the ARB may need to have executive participation. If the VPs delegate the ARB responsibilities to managers or architects, then the VPs need to support and not second-guess their subordinates. The ARB, in these matters, would be considered the A (Accountable) within the RASCI process.

Many leaders within an organization do not hold a manager or executive title. These are the people to whom the team looks in meetings to sway opinions and provide guidance. This trait of peer leadership is one that you want to look for when selecting people to place on the ARB.

Expertise, whether in engineering disciplines, architecture, or business, is a characteristic that people should display if they are participants on the ARB. Such individuals usually are in roles such as architects, senior engineers, or business/product owners, though they can be found elsewhere as well. These people’s input is sought when a tough question must be answered or a crisis resolved. Their expertise comes in a variety of subjects; for example, it might include expertise on the platform itself because of a long tenure working with the product, expertise with a specific technology such as caching, or even expertise with a specific large customer of the business.

Ideally, then, ARB members will be leaders who are respected and who have domain expertise. Some members may have a greater amount of one characteristic than another. For instance, a senior director of engineering may be well respected and display great leadership, yet not have true domain expertise. This senior director may still be an excellent candidate for the review board. Here are some roles within the organization that you should consider when assessing candidates for membership on the ARB:

• Chief architects

• Scalability architects

• Infrastructure architects

• VP or directors of engineering

• VP or directors of operations or infrastructure

• Senior systems administrators, database administrators, or network engineers

• Senior engineers

• CTO or CIO

• General manager of business unit

• Product or business owners

This list is not inclusive but should provide you with an idea of where to look for individuals who display the three key traits of respectability, leadership, and domain expertise. As with most topics that we have discussed, the real test is whether the ARB approach works for and within your organization. The number of members on the ARB can vary depending on the organization, the number of people available, and the variety of skill sets required. We recommend the board consist of between four and eight members.

Membership on the ARB should be considered an additional responsibility to an individual’s current role. It is always considered voluntary, so if necessary a member can ask to be excused. Ideally, we would like to see the ARB remain in place for a significant period of time so that this team can establish tenure in terms of assessing projects and designs. You can modify the permanent or nonpermanent nature of this membership in many ways, however, and you may want to consider several factors when deciding who should be a permanent member versus who should fill a rotating seat.

One factor to take into account is how many suitable members are present in your organization. If there are only four people who display the traits necessary for serving on this board, you will probably have to insist that these individual occupy permanent positions on the ARB. Another factor that will determine how permanent, semi-permanent, or rotational this role should be is how often you have features that need to proceed through ARB review. If your organization has enough engineers and enough JAD projects that the board must meet more than once per week, you may need to rotate people on the board or even consider having two different ARBs that can alternate in fulfilling this responsibility. A third factor, besides the number of viable candidates and the number of ARB meetings, is the specificity of the members’ expertise. If there are multiple technologies or technology stacks or separate applications, you should consider rotating people in and out of the board depending on the characteristics of the feature being discussed.

Several different methods of rotation for the ARB positions may be used. One straightforward method is to change the constituency of the board every quarter of the year. Depending on how many people are fit for this service, they could rotate on the board every six months or once each year or even beyond. Another approach is to have some individuals be permanent members of the board, such as the architects, and rotate the management representative (e.g., VP of engineering, VP of operations, CTO) and the team members (e.g., senior engineer, senior systems administrator, senior network engineer). Any of these methods will work well as long as there is consistency in how each team approaches its decisions and the team is given the authority of approving or rejecting JAD proposals.

Each meeting should include the following elements:

1. Introduction. Some members of the design team may not know members of the ARB if the engineering organization is large.

2. State Purpose. Someone on the ARB should state the purpose of the meeting so that everyone understands what the goal of the meeting is. We suggest you point out that the ARB will be making a judgment on the proposed architecture and not on people on the design team. If the design is sent back with major or minor revisions requested, the decision should not be taken as a personal attack. The agenda for everyone in the organization should be to ensure the proper governance of the IT systems and support the scalability of the system.

3. Architecture Presentation. The design team should present the design to the ARB. A well-structured presentation should walk the ARB members through the thought process starting with the business requirements; it should follow this with an outline of the tradeoffs, alternative designs, and finally the recommended design, including its strengths and weaknesses.

4. Q&A. The ARB should spend some time asking questions about the design to clarify any points that were vague in the presentation.

5. Deliberation. The ARB members should dismiss the design team and deliberate on the merits of the proposed design. This review can take many forms. For example, the board members might cast an initial vote to determine where each member stands, or they might choose someone to lead a point-by-point discussion of the pros and cons of the design before casting ballots.

6. Vote. The ARB should have an established process for determining when product features are approved and when they are rejected. We have often seen ARBs reject a design if a single member votes “no.” You may want to adopt a three-fourths rule if you believe reaching 100% agreement is unlikely and will be unproductively arduous. In such a case, we recommend that you reconsider who makes up the constituency of the board. Members should most strongly advocate for what is best for the company. Someone who abuses his or her power and consistently hassles design teams is not looking out for the company and should be replaced on the ARB.

7. Conclusion. When a decision is made, the ARB should recall the design team and explain its decision. This decision could be one of four courses:

• Approval. The ARB could approve the design to move forward as outlined in the proposal.

• Rejection. The ARB could reject the design and request a completely new design be developed. This second choice is an absolute rarity. Almost always, there is something from the proposed design that can be salvaged.

• Conditional Approval. The ARB could approve the design to move forward, albeit with some changes. This choice would indicate that the team does not need to resubmit the design to ARB but can proceed under its own guidance.

• Rejection of Components. The ARB could reject the proposed design but make specific requests for either more information or redesigned components. This fourth option is the most common form of rejection, and the specific request for more information or a change in the design usually comes from specific members of the ARB. Such a decision does require a resubmission of the design to the ARB for final approval prior to beginning development on the feature.

These steps can be modified as necessary to accommodate your team size, expertise, and culture. The most important item to remember (and you should remind the team of this point) is that ARB members should first and foremost put what is best for the company before their personal likes, distastes, or agendas, such as something causing more work for their own organizational team. Of course, what is best for the company is to get more products in front of the customers while ensuring the scalability of the system.

The entry criteria for a feature coming out of JAD into ARB review are the following:

• Established Board. The Architecture Review Board must be established based on the criteria mentioned earlier in terms of roles and behaviors that the members should demonstrate.

• Design Complete. The feature should meet the exit criteria outlined for JAD:

Consensus. The design team should be in agreement and all members should support the final design. If this is absolutely not possible, a feature may be submitted to the ARB with this fact acknowledged and the two or more proposed designs each presented.

Documentation of Tradeoffs. If any significant tradeoffs were made in the design with respect to the requirements, cost, or principles, they should be documented.

Documentation of Final Design. The final design must be documented and posted for reference.

• Feature Selection. The feature having completed design should be considered as a candidate for ARB review. If this feature has any of the following characteristics, it should proceed through ARB review:

Noncompliance with Architectural Principles. If any of the architectural principles were violated, this feature should go through ARB review.

Projects That Cannot Reach Consensus on Design. If the team fails to reach consensus, the feature can be either reassigned to a new design team or sent to the ARB for a final decision on the competing designs.

Significant Tradeoffs. If tradeoffs had to be made in terms of product requirements, cost, or other non-architectural principles, this compromise signals that the feature should proceed to ARB review.

High-Risk Features. We will discuss how to assess risk in much more detail in Chapter 16, Determining Risk; for now, recognize that any high-risk feature should go through ARB review. A quick way of determining if the feature is high risk is to look at how many core components it touches or how different it is from other features. Either of these factors increases the amount of risk associated with the feature.

Depending on the decision made by the ARB, different exit criteria exist for a feature. Here are the four possible decisions on these criteria and what must be done following the ARB session:

• Approval. Congratulations—nothing more is required of the team by the ARB. Now the tough part begins, as the team must develop and implement the project as it has been designed.

• Rejection. If the design is completely rejected, the ARB should provide a rationale for its decision. In this case, the same design team may be asked to redesign the feature or a new team may be formed to create the second design. The team should remain in place to provide the design if the current team has the right expertise and if it is still motivated to succeed.

• Conditional Approval. If the ARB has conditionally approved the design, the team should incorporate the board’s conditions into its design and begin to produce the feature. The team may return to the ARB in person or via email if any questions arise or if it needs further guidance.

• Rejection of Components. If the ARB rejects the design of certain components, the same design team should come up with alternative designs for these components. Because the ARB review is most often treated as a discussion, the design team should have a good idea of why each component was rejected and what would be needed to satisfy the board. In this case, the design team needs to schedule a subsequent presentation to the ARB to receive final signoff on its design.

All too often, dysfunction in technology organizations causes features to be designed in silos. Such problems are especially likely to arise in function-based organizations as well as when an experiential chasm is present. The fix is forcing the teams to work together for a shared goal—something that can occur through the JAD process.

The JAD process includes both mandatory participants and optional team members. The design meetings should be structured based on components, and it is important to start by making sure every team member is familiar with the business requirements of the feature as well as the applicable architecture principles of the organization.

A JAD checklist can help you and your organization get started quickly with the JAD process. We recommend that you start with our standard steps as outlined on the checklist, but fill it out as necessary to make it part of your organization. Of course, you should also document this process so it becomes fixed in your organization’s culture and processes.

The entry criteria for JAD focus on the preparation needed to ensure the JAD effort will be successful and the integrity of the process remains intact. Letting features slip into a JAD without all the required team members being on board is a surefire way to cause the process to lose focus and effectiveness. The exit criteria for JAD address the need for all members of the team to agree on the feature design and, if necessary, to present the design for ARB review.

Three traits are essential for members of the ARB: respect, leadership, and expertise. The amount of formality that characterizes the ARB members’ review depends on the culture of the organization, but we recommend keeping it as informal as possible to avoid intimidating design team members and to foster a healthy, productive discussion about the design.

The entry criteria for ARB review are focused on ensuring that the right feature is being sent to the ARB, that the right ARB team is formed, and that the feature is prepared as well as possible to proceed through ARB. Selecting the right feature is not always easy. Recognizing this fact, we recommend four tests for whether a feature should proceed through ARB review: noncompliance with architectural principles, significant tradeoffs having to be made to the business requirements, inability of the JAD team to reach consensus, and high-risk features.

Here is one final thought about how to implement JAD and ARB. The company Shutterfly, an Internet-based image publishing service based in Redwood City, California, implemented JAD and ARB review. It utilized JAD for features that “some other team would notice.” If the developers could implement a feature without anyone noticing, it was deemed not worthy of JAD. If another team would notice the feature (e.g., the network team would notice a spike in traffic), then it might be considered JAD-worthy. Similarly, if a feature was big enough that most or all teams would notice its implementation, then it was considered ARB-worthy. Such simple criteria are easy for everyone to understand in regard to when to implement each process, but strict enough to ensure the needed features get the attention they deserve.

Through the use of JAD and the ARB, your organization can be ensured of better designs that are purposefully made for improving scalability.

• The JAD is the best way to pull together a cross-functional team whose members may not otherwise be incented to work together.

• The JAD team must include members from engineering, architecture, and operations (database administrators, systems administrators, or network engineers).

• JAD is most successful when the integrity of the process is respected and entry and exit criteria are rigorously upheld.

• A final review of the application’s architecture ensures buy-in and acknowledgment as well as prevents finger pointing.

• The proper constituency of the ARB is critical if it is to uphold the purpose of a final architecture signoff and if its decisions are to be respected.

• Members of the ARB should be seen as leaders, be well respected, and have expertise in some area of the application or architecture.

• ARB membership can be assigned on a rotational basis, but the position is always seen as incremental to each member’s current duties.

• All ARBs should start with a discussion of the purpose of the ARB—to ensure designs support the organization’s business needs, including scalability.

• Entry into ARB review should be granted only to features that are sufficiently prepared.

• Decisions by the ARB should be considered final. Some organizations may include an appeals process if it is deemed necessary.