Chapter 20. Designing for Any Technology

Success in warfare is gained by carefully accommodating ourselves to the enemy’s purpose.

—Sun Tzu

Have you ever heard someone describe an architecture or design by using the names of the third-party or open source solutions used to implement that platform or architecture? It might sound something like this:

We use Apache Web servers and Tomcat application servers. We use MySQL databases and NetApp storage devices. Our network gear is Cisco routers and switches.

Nice product plugs. If this were a movie, you could sell the product placements. Each of the solution providers mentioned will certainly appreciate this description of an implementation. And yes, we meant to say “implementation,” because the preceding statement is neither a design nor an architecture.

This chapter describes the difference between architecture and implementation. We further make the argument that the best architectures are not accomplished through vendor or open source implementations, but rather through vendor and technology-agnostic designs.

The best architects of buildings and houses do not describe trusses, beams, and supports using the vendor’s name, but rather by their size, load capacity, composition, and so on. This is because the architect realizes that in most cases the items in question are commodities and that the vendor solution will likely be selected based on price, reputation, and quality. Similarly, electrical engineers do not typically reference vendor names when describing a design; instead, they refer to a resistor by its level of resistance (in ohms) and its tolerance (variation to actual resistance). The house architect and the electrical engineer understand that describing something with a vendor’s name is an implementation, whereas describing something through specifications and requirements is a design or an architecture.

Implementations represent the choices you have made due to cost considerations, build versus buy decisions, returns on investment, skill sets within your team, and so on. The use of C++ or Java or PHP as a coding language is not indicative of your architecture; rather, these are choices of tools and materials to implement components of your architecture. The choice of a Microsoft database over Sybase, MySQL, or Oracle as a database is not an architecture, but rather an implementation of a database component of your architecture. The decision to go open source versus using a vendor-provided solution is another example of an implementation decision, as is the decision to use a Microsoft operating system rather than some variant of Linux. Referring back to the discussion on architects and electrical engineers, implementation decisions are made by technicians (builders and electricians) and are made to be consistent with the architectural design.

The implementation is a snapshot of the architecture. Imagine an architecture with a write database and a read database. All writes go to the write database, and all reads are directed to one or several read databases in a load-balanced fashion. For a very small site, it may make sense for a single database to accomplish all of these things (with an additional failover database for high availability, of course). The implementation in this case would be a single database, whereas the site is potentially architected (and implemented in the software) for more than one database. Now consider the case where the architecture calls for nearly any database to be used with the application connecting through an abstracted data access layer (DAL) or data access object (DAO). The specific implementation at a point in time might be a database from Microsoft, but with some modifications to the DAL/DAO it could ultimately become an open source database or database from IBM, Oracle, or Sybase. While the implementation (choice of solution provider) may change over time, the architecture stays somewhat static.

We define technology-agnostic design (TAD) and technology-agnostic architecture (TAA) as the design or architecture of a system that is agnostic toward the technology required to implement it. The aim of TAD and TAA is to separate the design and architecture from the technology employed and the specific implementation. This separation decreases both cost and risk while increasing scalability and availability of your product, system, or platform. Some of our clients even incorporate TAD or TAA into their architectural principles.

Technology solutions, much like building materials, experience the effects of commoditization over time. A good idea or implementation that becomes successful in an industry is bound to attract competitors. The competitors within the solution space initially compete on differences in functionality and service, but over time these differences decrease as all competitors adopt useful feature sets. In an attempt to forestall the effects of commoditization through increased switching costs, providers of systems and software try to produce proprietary solutions or tools that interact specifically and exclusively with their systems.

Avoiding getting trapped by extensive modification of any provider’s solution or adoption of tightly integrated provider tools allows you the flexibility of leveraging the effects of commoditization. As competitors within a solution space begin to converge on functionality and compete on price, you remain free to choose the lowest cost of ownership for any given solution. This flexibility results in capital outlay, which minimizes the impact on cash flow and lowers amortized costs, which positively affects profits on a net income basis. The more your architecture allows you to bring in competing providers or partners, the lower your overall cost structure.

Several times during your career, you are likely to find a provider of technology that is far superior in terms of features and functionality to other providers. You may determine that the cost of implementing this technology is lower than using the other providers’ technology because you have to build less to implement the product. In making such a decision, you should feel comfortable that the “lock-in” opportunity cost of choosing such a provider exceeds the option cost of switching to another provider in the future. In other words, recognize that other providers will move quickly to close the functionality gap, and do your best to ensure that the integration of the service in question can be replaced with products and services from other providers down the road. Recognize further than in using proprietary tools today, you reduce your leverage in negotiating lower prices from other providers in the future.

Imagine that you have built a wonderful business producing specialty vans for handicapped people from Ford vehicles. One hundred percent of your business is built around the Ford Econoline Van, and you can’t easily retool your factory for another van given the degree of specialization required in tooling and skills. What do you do if Ford goes out of business? What happens if Ford stays in business but increases its prices, significantly changes the Econoline Van family, or increases the interest rate on the loans you use to purchase and customize the vans?

Now, apply a similar set of questions to your implementation technologies (again, not an architecture). Maybe you have used a proprietary set of APIs for some specific asynchronous functionality unique to the database in question. Alternatively, maybe you have leveraged a proprietary set of libraries unique to the application server you’ve chosen. What do you do when one or both of those providers go out of business? What if the provider of either technology finds itself being sued for some portion of its solution? What if the viability and maintenance of the product relies upon the genius of a handful of people within the company of the provider and those people leave? What if the solution suddenly starts to suffer from quality problems that aren’t easily fixed?

Technology-agnostic design reduces these risks by increasing your ability to quickly move to other providers. By reducing your switching costs, you have reduced not only your own costs, but also the risk to your customers and shareholders.

A common misperception is that by implementing a certain solution, you become reliant upon that solution. Just because you use GoldenGate’s database replication technology does not mean that you are dependent upon it alone for scale. True, on any given day, you will rely upon the application to work for proper functioning of your site, but that is not the same as saying that the architecture relies upon it for scale. Again, we separate architecture from implementation. Architecture is a design and should not rely upon any given vendor for implementation. Implementation is a point-in-time description of how the architecture works on that day and at that moment. The proper architecture in this replication scenario would call for a replication mechanism with requirements that can be fulfilled by a number of vendors, of which GoldenGate is but one. If you have done a thorough analysis of the provider landscape and know that you can switch either databases or replication technology providers (again, not without some work), you have a scalable solution that is independent of the provider.

You should not get caught in the trap of saying that you must personally build all components to be truly independently scalable. Recall our discussion in Chapter 15, Focus on Core Competencies: Build Versus Buy. Scalable design allows you to drop in commodity solutions to achieve an implementation. Furthermore, nearly all technologies ultimately move toward commoditization or attract open source alternatives. The result is that you rarely need to build most things that you will need outside of those things that truly differentiate your product or platform.

The second way that TAD supports scalability is actually embedded within the four questions from Chapter 15. Can you see it? Let’s ask two questions to get to the answer. First, do you believe there will be a rapid convergence of the functionality in question? Second, do you need to deeply integrate the solution in question to leverage it? If you believe that competitors will rapidly converge on functionality and you do not need deep integration to the selected provider’s solution, you should consider using the solution. In doing so, you avoid the cost of building the solution over the long term. The key, as hinted at by the second question, is to avoid deep integration. You should not be building logic deep within your system to benefit from a provider’s solution. Building such logic ties you into the solution provider and makes it more difficult for you to benefit from commoditization—that is, to switch vendors in an effort to reduce price.

We argue that deep integration of a third-party provider’s solution is almost never critical to scale if you’ve properly architected your system, platform, or product. In our experience, we have never come across such a situation that absolutely demands that a company be deeply tied with a third-party provider to scale to the company’s needs. In most cases, this misconception is fueled by a poor decision made somewhere else in the architecture. You may, however, find yourself in a situation where a pending or existing crisis can be resolved more quickly by leveraging unique functionality provided by a third party. Should you find yourself in that situation, we recommend the following course of action:

1. Abstract the integration into a service so that future changes are limited to the service and not deeply integrated within your systems. Such an abstraction will limit the switching costs after your crisis is over.

2. Make a commitment to resolve the dependency as soon as possible after the crisis.

The approach for TAD is straightforward and consists of three easily remembered rules. The first is to think and draw only “boxes,” like those that you might find in a wire diagram. For a physical architecture, these boxes depict the type of system employed but never the brand or model. Router, switch, server, storage, database, and so on are all appropriate boxes to be employed in a physical architecture. Logical architectures define the activities, functions, data flow, and processes of the system and should also avoid the inclusion of vendor names. For this step, it does not matter that your company might have contracts with specific providers or have an approved provider list. This step is just about ensuring that the design is agnostic, and contracts and approved providers are all implementation-related issues.

The second rule is to remove all references to providers, models, or requirements that demand either a provider or model. It’s appropriate to indicate requirements in a design, such as a switch that is capable of 10-gigabit throughput, but it is not appropriate to indicate a switch capable of running a Cisco proprietary protocol. Again, interface requirements have to do with implementation. If you have something running in your production environment that requires the same brand of system in other places, you have already locked yourself into a vendor and violated the TAD approach.

The final rule is to describe any requirements specific to your design in agnostic terms. For example, use the number of SQL transactions per second instead of a proprietary vendor database. Use storage in terms of IOPS, bytes, spindles, or speeds rather than anything specific to any given vendor like a proprietary storage replication system. When in doubt, ask yourself whether your design has forced you into a deal with a vendor for any given reason, or whether the description of the design is biased by any given vendor or open source solution.

Teams that employ the JAD and ARB processes should ensure they are following TAD during architecture design and review sessions by enforcing these three rules. If a design shows up for an ARB review with the names of vendors written in the boxes, it should be rejected and the team asked to redesign it appropriately.

A very simple test to determine whether you are violating the technology-agnostic design principles is to check whether any component of your architecture is identified with the name of a vendor. Data flows, systems, transfers, and software that are specifically labeled as coming from a specific provider should be questioned. Ideally, even if for some reason a single provider must be used (remember our arguments that this should never be the case), the provider’s name should be eliminated. You simply do not want to give any provider a belief that you have already chosen its solution—doing so will handicap you in negotiations.

Technology agnosticism is as much about culture as it is a process or principle during design. Engineers and administrators tend to be very biased toward specific programming languages, operating systems, databases, and networking devices. This bias is very often a result of past experiences and familiarity. An engineer who knows C++ better than Java, for instance, will obviously be biased toward developing something within C++. An engineer who understands and has worked with Cisco networking equipment her entire career will obviously prefer Cisco to a competitor. This bias is simply human nature, and it is difficult to overcome. As such, it is imperative that the engineers and architects understand the reasons for agnosticism. Bright, talented, and motivated individuals who understand the causality between agnosticism and the maximization of flexibility within scalability and the maximization of shareholder wealth will ultimately begin parroting the need for agnosticism.

TAD is as much a cultural initiative as it is a process or principle. The biggest barrier to implementing TAD will likely be the natural biases of the engineers and architects for or against certain providers. Ensuring that the organization understands the benefits and reasons for TAD will help overcome these biases. Review the section in Chapter 4, Leadership 101, covering the causal roadmap to success.

• TAD and TAA reduce costs by increasing the number of options and competitors within a competitive selection process.

• TAD and TAA reduce risk by lowering switching costs and increasing the speed with which providers or solutions can be replaced in the event of intellectual property issues or issues associated with business viability of your providers.

• TAD and TAA increase scalability through the reduction of cost to scale and the reduction of risk associated with scale. Where technology solutions are likely to converge, TAD and TAA can help you achieve rapid and cost-effective scale by buying rather than building a solution yourself.

• TAD and TAA increase availability by allowing you to select the highest-quality provider and the provider with the best availability at any point in time.

• TAD and TAA are as much cultural initiatives as they are processes or principles. To effectively implement them, you need to overcome the natural human bias for and against technologies with which we are more or less conversant, respectively.