Chapter 12. Establishing Architectural Principles

He wins his battles by making no mistakes. Making no mistakes is what establishes the certainty of victory, for it means conquering an enemy that is already defeated.

—Sun Tzu

Corporate “values” statements are often posted in workplaces and serve to remind employees of the desired behavior and culture within their companies. In this chapter, we present the scalability analog to these corporate values—architectural principles. These principles both guide the actions of technology teams and form the criteria against which technology architectures are evaluated.

Figure 12.1 Quigo Goal Tree

Ideally, our architectural principles will be based on our high-level goal tree topics and not on specific goals. Our goals are likely to change over time, so our principles should broadly support both future goals and current ones.

Our clients often find it useful to begin a principle-setting session by brainstorming. For example, you might start by taking the goal tree from Figure 12.1 and writing each theme on a whiteboard. Ensure that the goal tree stays prominently displayed throughout the session for reference. For each topic (e.g., quality, availability, cost, TTM, and efficiency), focus on finding broad principles that can serve as a “test” for any given initiative—that is, principles with which you can determine whether a suggested architectural design change meets the selected criteria. Principles that are so lofty and nebulous that they cannot be used practically in a design or architecture review (e.g., “design to be scalable”) won’t assist in driving cost-effective, scalable, and highly available architectures. Likewise, principles that are aspirational in nature but are often cast aside in practice in favor of expediency or for other reasons are not useful.

When we facilitate these sessions, we keep our clients focused on principle exploration—writing down as many principles as possible under each topic. Discussion is valuable, because each person should be able explain the meaning of a proposed principle. Oftentimes these discussions improve the phrasing of the principle itself. For instance, a principle that is presented as “horizontal scalability” may start a discussion about whether that principle is meant to keep the team from implementing “vertical scalability” (the ability to use a larger solution or server). The team may then decide to change the title “horizontal scalability” to “scale out, not up” to show the desire to limit vertical scalability wherever possible in favor of horizontal scalability.

When conducting an exercise similar to the one described here, an experientially diverse team is important. Recall from Chapter 3, Designing Organizations, that experiential diversity increases the level of cognitive conflict, and hence the innovation and strategic value of the results. Brainstorming sessions, then, should include members with backgrounds in software development, general architecture, infrastructure, DevOps, quality assurance, project management, and product ownership. Questions that appear initially driven by a session participant’s lack of experience in a particular area frequently uncover valuable insights.

It is important that each of the candidate principles embody as many of the SMART characteristics discussed in Chapter 4, Leadership 101, as possible. One notable exception is that principles are not really “time bounded.” When speaking of architectural principles, the “T” in the SMART acronym means the principle can be used to “test” a design and validate that it meets the required intent. Lofty nebulous principles don’t meet this criterion.

One often-used lofty—and therefore unfortunate—architectural principle is “infinitely scalable.” Infinite scalability is simply not achievable and would require infinite cost. Furthermore, this principle does not allow for testing. (How would you test against a principle that by definition has no limit?) No deployment architecture can meet the defined goal. Finally, we must consider the role of principles in guiding organizational work habits. Principles should ideally help inform design and indicate a path toward success. Lofty and aspirational principles like “infinite design” don’t offer a map to a destination, making them useless as a guide.

At the beginning of the second day, review the principles you wrote on the whiteboard and have the team recap the discussions from the previous day. One way to do so is to have participants present each of the principles on the board and invite a brief discussion of its merits. Kicking off the second day in this fashion will stimulate the collective memory of the team and jumpstart the value creation effort for the remainder of the day.

When selecting principles (during the latter portion of the second day), we like to use a crowd sourcing technique. Our approach is to allow each of the participants in the meeting to vote for a constrained number of principles. Using this technique, each participant walks to the whiteboard and casts up to his or her quota of votes. Principles are best kept to an easily memorized and easy-to-use number (say, 8 to 15); we generally limit votes per person to roughly half the number of desired principles. Having fewer than 8 principles isn’t likely to cover the entire spectrum necessary to achieve the desired effect. Conversely, having more than 15 principles tends to be too cumbersome for reviews and makes it too hard to remember the principles as means to drive daily efforts. Choosing half the number of principles as a vote quota usually makes it easy to narrow the number of candidate principles to a manageable size.

After the team voting is complete, you will likely end up with clear winners and a number of principles that are “ties” or are close in terms of total votes. It is common to have more than your desired number of principles, in which case you might decide to invite some discussion and debate (cognitive conflict) to narrow the list down to your desired number. Once the voting is finished, it’s time to think of your “principle go to market” strategy.

We like to present the principles in a fashion that is easily distributed on a single page and that makes explicit the causal roadmap between the principle and what it is expected to achieve. One way to do so is to bundle the principles into the areas they are expected to support in a Venn diagram. We are big fans of Venn diagrams for presenting principles because so many of the principles are likely to affect multiple areas.

Returning to the earlier example, Quigo’s goal tree topics include efficiency, time to market, availability, cost, and quality. When developing the principles at Quigo, we decided that efficiency and cost were tightly correlated and could be presented within one region of the Venn diagram (higher levels of efficiency are correlated with lower costs). While scalability was not part of the goal tree, it is highly correlated with availability and deserved a place in our diagram. Quality is also clearly highly correlated with availability; thus, quality, availability, and scalability are presented together. With these relationships, we reduced our Venn diagram to three regions: availability/scalability/quality, cost/efficiency, and TTM (time to market). Figure 12.2 shows a potential outcome of such a Venn diagram plot.

Figure 12.2 Architecture Principles Venn Diagram

For instance, at the time of design of any system, you know which services and systems it will need to interact with. Perhaps the service in question repeatedly makes use of a database or some other data store. Potentially, the service makes a call, preferably asynchronously, to another service. You also know that from time to time you will be writing diagnostic information and potentially errors to some sort of volatile or stable storage system. You can apply all of this knowledge to design a system that can give you more information about future scale needs and thereby increase the system’s availability.

We argue that you should build systems that will help you identify potential or future issues. Returning to our example system and its calls to a database, we should log the response time of that database over time, the amount of data obtained, and maybe the rate of errors. Rather than just reporting on that data, our system could be designed to show “out of bounds” conditions plotted from a mean of the last 30 Tuesdays (assuming today is Tuesday) for our five-minute time of day. Significant standard deviations from the mean could be “alerted” for future or immediate action depending on the value. This approach leverages a control chart from statistical process control.

We could do the same with our rates of errors, the response time from other services, and so on. We could then feed this information into our capacity planning process to help us determine where demand versus supply problems might arise. In turn, we can identify which systems are likely candidates for future architectural changes.

Ensuring that your product designs allow services to be hosted in multiple locations and operate independently of one another is critical to achieving rapid deployment. Such designs also allow companies to avoid the constraints of power and space within a single facility and enable cloud infrastructure deployments. You may have an incredibly scalable application and platform, but if your physical environment and your operating contracts keep you from scaling, you are just as handicapped as if you had nearly infinite space and a platform that needs to be re-architected for scale. Scalability is about more than just system design; it is about ensuring that the business environment in which you operate, including contracts, partners, and facilities, is also scalable. Therefore, your architecture must allow you to make use of several facilities (both existing and potentially new) on an on-demand basis.

In many cases, you may be tempted by the competitive edge promised by a new vendor technology. Be cautious here: As an early adopter, you will also be on the leading edge of finding bugs with that software or system. If availability and reliability are important to you and your customers, try to be an early majority or late majority adopter of systems that are critical to the operation of your service, product, or platform. Implement newer technology for new features that are less critical to the availability of your solution, and port that technology to mission-critical areas once you’ve proven it can reliably handle the daily traffic volume.

Asynchronous systems are far more tolerant of slowdowns. As an example, consider the case where a synchronous system can serve 40 simultaneous requests. When all 40 requests are in flight, no more can be handled until at least one completes. In contrast, asynchronous systems can handle the new request and do not block for the response. They have a service that waits for the response while handling the next request. Although throughput is roughly the same, they are more tolerant to slowness, as requests can continue to be processed. Responses may be slowed in some circumstances, but the entire system does not grind to a halt. Thus, if you have only a periodic slowness, an asynchronous system will allow you to work through that slowness without stopping the entire system. This approach may buy you several days to “fix” a scale bottleneck, unlike the immediate action required with a synchronous system.

In many places, however, you are seemingly “forced” to use a synchronous system. For instance, many database calls would be hampered and the results potentially flawed if subjected to an asynchronous passing of messages. Imagine two servers requesting similar data from a database—for example, both of them asking for the current bid price on a car, as depicted in Figure 12.3.

Figure 12.3 Asynchronous Ordering of Bidding Events

In Figure 12.3, system A makes a request, after which system B makes a request. B receives the data first and then makes a bid on the car, thereby changing the car’s price. System A then receives data that is already out of date. Although this seems undesirable, we can make a minor change to our logic that allows this to happen without significant impact to the entire process.

We need merely change the case in which A subsequently makes a bid. If the bid value by A is less than the bid made by B, we simply indicate that the value of the car has changed and display the now-current value. A can then make up her mind as to whether she wants to continue bidding. In this way, we take something that most people would argue needs to be synchronous and make it asynchronous.

Whenever possible, avoid developing stateful products. Where necessary, consider storing state with the end user rather than within your system. If that is not possible, consider a centralized state caching mechanism that keeps state data off of the application servers and allows for its distribution across multiple servers. If the state needs to be multitenant for any reason, attempt to segment the state information by customer or transaction class to facilitate distribution across multiple data centers, and try to maintain persistency for that customer or class of transaction within a single data center, with only the data that is necessary for failover being replicated.

Could Google, Amazon, Yahoo, or eBay run on a single system? Could any of them possibly run on a single database? Many of these companies started out that way, but the technology of the day simply could not keep up with the demands that their users placed on them. Some of them faced crises of scale associated with attempting to rely upon bigger, faster systems. All of them would have faced those crises had they not started to scale out rather than up.

The ability to scale out, rather than up, is essential when designing to leverage the elasticity and auto-scaling features present within many Infrastructure as a Service (IaaS) providers. If one cannot easily add read copies of databases, or additional Web and application servers, the benefits of renting capacity within “the cloud” (IaaS) cannot be fully realized. There may be no other principle as important to scaling a product as the ensuring that one can always scale horizontally (or out, rather than up).

“Scaling out, not up” speaks to the implementation of the first set of splits. Perhaps you are splitting transaction volume across cloned systems. You might have five application servers with five duplicate read-only caches consisting of startup information and nonvolatile customer information. With this configuration, you might be able to scale to 1 million transactions per hour across 1,000 customers and service 100% of all your transactions from login to logout and everything in between. But what will you do when you have 75 million customers? Will the application’s startup times suffer? Will memory access times begin to degrade? More worrisome, can you even keep all of the customer information within memory?

For any service, you should consider how you will perform your next type of split. In this case, you might divide your customers into N separate groups and service customers out of N separate pools of systems, with each pool handling 1/Nth of your customers. Or perhaps you might move some of the transactions (e.g., login and logout, updating account information) to separate pools if that will lower the number of startup records needed within the cache. Whatever you decide to do, for major systems implementations, you should think about it during the initial design even if you implement only one axis of scale.

The principle of isolating faults (i.e., creating fault isolation zones) is analogous to building an electrical system with circuit breakers. Segment your product such that the failure of a service or subservice does not affect multiple other services. Alternatively, segment your customers such that failures for some customers do not affect failures for your customer base in its entirety.

Automation, by comparison, is comparatively cheap and will always make the same mistakes or have the same successes in exactly the same way each time. As such, it is easy to tune and easy to rely upon automation for simple repetitive tasks. For these reasons, we believe all systems should be built with automation in mind at the beginning. Deployment, builds, testing, monitoring and even alerting should be automated.

• Principles should be broad enough that they are not continually revised, but should also be SMART and thematically bundled or grouped.

• To ensure ownership and acceptance of your principles, consider having your team help develop them.

• Ensure that your team understands the RASCI of principle development and modification.

• Keep your principles to a number that is easily memorized by the team to increase utilization of the principles. We suggest having no more than 15 principles.