Traditional Delivery Fog in the Service World

Let’s look back at BigCo and FastCo to see what exactly is causing them to both fail to establish the necessary situational awareness and improvement loops that are essential to succeed.

There are few surprises in the failures at BigCo. They have simply replaced the customer’s IT operations with their own. Despite possibly having more advanced datacenters, tools, and dedicated staff, all the old split-responsibility habits that shaped team ways of thinking over so many years as a traditional software company still linger. Many of the same organizational structures remain in place, fragmenting situational awareness and any chance of developing shared responsibility for ensuring the service stays attuned to what the customer desires.

If this is not bad enough, the fragmentation caused by the organizational structure encourages teams to assess and reward teams based upon more traditional and locally focused metrics. This creates little incentive to close any awareness gaps, let alone collaborate on work outside the immediate remit of the team that is unlikely to be rewarded by the team performance assessment mechanism.

While FastCo has no legacy organizational structures to disrupt information flow, there are still signs that some traditional mental models endure. For starters, developers are often assessed on delivery speed and output, not on understanding and achieving customer outcomes. Nobody intentionally tries to build anything to meet outcomes they are not incentivized to achieve.

The problems do not end there. Few developers are motivated to learn how to manage the operational side of a service environment any more than the minimum absolutely necessary. What many fail to realize is that knowing your own code and having significant technical knowledge does not automatically make you the best person to both operationalize and operate it. This lack of operational knowledge and available expert guidance can lead to poorly crafted or dangerously unsuitable software being released with little more than a view that it seemed to work fine in the development environment.

The lack of operational awareness not only creates challenges with service design and coding but also can have a real impact on the effectiveness of service hosting configuration design, instrumentation, and operational management mechanisms. Many mistakenly believe that operational details, and even the whole of Technical Operations, can be easily made to go away by pulling in enough operational tools and solutions from outside cloud platform and infrastructure providers. But such thinking is as flawed as believing that you can become as talented as a famous basketball player by wearing the same brand of shoes as them.

For starters, having great tools does not necessarily mean you know how to use them effectively. In fact, the increased depth of complexity and potential lack of transparency they can create in the service stack can make it harder to understand the nuances and interactions within and across the stack, even when you are actively trying to look for them. This increases the risk that some element critical for service health will be missed.

You can also introduce otherwise avoidable problems by simply trusting that external providers who deliver operational functions will automatically match the service quality levels your customer expects. FastCo discovered this the hard way by committing to service level guarantees far higher than those of their upstream provider with little forethought on how to close the gap.

Uptime and reliability are far from the only potential mismatches. There can also be others in the performance, scalability, defect remediation, and security offered by upstream providers that, without sufficient operational awareness, can remain hidden until it is painfully expensive, disruptive, or time consuming to mitigate. This can lead to unexpected behavior and embarrassing incidents that can put your relationship with your customer base at risk.

To deliver services that meet the objectives and expectations of your customer, you not only need to overcome the various obstacles that can get in the way of your awareness of service ecosystem dynamics, but you also need to be sure your service aligns with the “ilities” customers require.

The Challenge of the “ilities”

It is far too easy for service delivery teams to become unclear about or pay too little attention to the specific service qualities that the customer expects to be in place to help them achieve their target outcomes. Customers often do not help matters. Some focus on the solution or functional/procedural details for how it is delivered (as discussed in Chapter 2, “How We Make Decisions”) rather than on how various service qualities might affect them. Still, it doesn’t take a genius to know that not delivering what the customer expects is not a strategy for success.

There can be a lot of service qualities that a customer might find important. As most of these qualities end in “ility,” I prefer to refer to them in the same shorthand way as Tom Gilb, a well-known systems engineer and early Agilist, by the term ilities. Ilities tend to fall into two categories:

• Service performance requirements (such as availability, reliability, and scalability) to meet customer needs.

• Risk mitigation (things like securability, recoverability, data integrity, upgradability, auditability and any other necessary legal compliance capabilities) to meet customer, legal and regulatory requirements.

Ilities are often overlooked for a number of reasons. One of the larger challenges is that when they do surface, they are often labeled as “nonfunctional requirements.” The problem with this term is that it is easy to believe that developers deliver functional requirements, and that any “nonfunctional” ones are either already naturally present or are delivered by someone else. Proving that a non-functional requirement is not present is often left to the testing team, who must then prove unequivocally that it is both missing and that remedying the situation requires otherwise unplanned work. Not only is it tough to prove something doesn’t exist, especially when there can be enough differences between test and production environments to create plausible deniability, but also convincing others that work must be done to satisfy the non-functional requirement for delivery must also fight the bias against creating unplanned work that can delay the overall release.

Delivering the ilities that your customers need also faces the problem that their importance and how they are perceived can differ depending on the nature of the domain where the customer uses the service.

Just as air requirements differ between a runner and a deep sea diver, the ilities needed for a service used by an investment bank trading desk are very different from those needed for a free consumer email service, and both are entirely different from those required for a service that manages hospital medical records. Even within a service type, the ilities might vary widely depending upon service hosting and consumption locations (United States vs. Europe vs. China) and customer type (consumer vs. business vs. government).

Such variables can make what is suitable for one customer (like a US hospital group) completely unfit for another (such as a Chinese government-run hospital). As a result, service providers who know little about an industry, market, or customer can easily either fall foul of customer needs or find themselves limited in what/where/how they can deliver or otherwise uneconomically exposed.

Another challenge with ilities is that they require understanding what delivering them effectively means within your service ecosystem. What elements contribute to delivering them? Are they built into the service or require specific actions to maintain? What is your ability to track how well you are delivering them and counteract any events or conditions that might put their suitability at risk? Do the means required to deliver them limit your ability or options available to respond to new problems or opportunities?

All of these can be surprisingly difficult to do, even in less complex, ordered operational domains. Ilities like availability, scalability, security, and data integrity usually depend upon multiple delivery aspects working together in a known and complementary way. Any one problem or misunderstanding, whether from a technical limitation, broken interaction, or even an environmental hygiene issue, can create a cascade of failures that can both hide the cause and obfuscate the severity of the real and damaging impact on the customer.

With all this lack of poor information flow, missing skills, and poorly understood ilities, how can an organization build up enough ecosystem and operational awareness to confidently deliver services that will meet the needs and expectations of customers?

The Role of Managers in Eliminating Service Delivery Fog

Many mistake the role of manager as one geared toward telling people what to do. Some even go so far as thinking it is the manager’s responsibility to tell staff how to perform their work.

Anyone who has been effective at management knows the reality is necessarily quite a bit different. For one, managers are not all-knowing entities. Unless the delivery ecosystem is in the predictably ordered Cynefin Clear domain (as described in Chapter 5, “Risk”), they are far less likely to have both more-detailed subject matter expertise and more-up-to-date details of the service technology stack than their staff. Even if they somehow managed to have both the awareness breadth and expertise depth, not only would the manager be incapable of scaling sufficiently to handle a team of a reasonable size and set of responsibilities, they would create an unstimulating working environment that needlessly wasted the capabilities of their own staff while simultaneously limiting everyone’s ability to grow and improve.

Such a combination is, at best, a recipe for unhappiness and failure.

While managers might not know everything, their position does provide them with three important and unique capabilities:

• They are part of the organization’s leadership structure, which gives them a seat at the table that can be used as a conduit to capture, convey, and provide clarity around the intent of the business vision.

• They act as a chief advocate for their team that can help articulate any capability gaps or obstacles that the team needs leadership help to overcome.

• They can help leadership map delivery work to customer ilities, and communicate any important team findings that might affect the larger strategy.

There are a number of great ways that managers can perform this, one of which is the commander’s intent pattern, as detailed in Chapter 3, “Mission Command.”

Another unique capability a manager has is that they have the ability to get the attention of and interact with peers across the organization. This enables the manager to see across the organization, facilitating information flow by helping remove communication barriers that might distort context or limit team members’ ability to learn. The manager can also provide outreach and foster collaboration between teams.

Finally, managers are well placed to see the overall dynamics of their own team. This allows them to act as a mentor and coach to help the team and its members learn and improve. Mentoring is one important way, some of which is discussed in the “Coaching Practice” section in Chapter 7, “Learning.” Part of this mentoring also includes providing team members a space and air cover to step back from the dynamics of the delivery ecosystem from time to time to experiment and fail in a safe and informative way.

Allowing for failure may sound anathema to management. That might be true if carelessly high risks are allowed. But managers can help define and defend experimentation boundaries that allow teams to safely try new technologies and approaches and allow team members to better understand the delivery ecosystem, innovate, and ultimately to improve. By allowing failures to be tolerated, everyone can openly discuss what happened and why something did not work so that they can come up with a potentially better approach in the future.

Identifying What You Can or Cannot Know

Even though they play an important role in the process, managers obviously cannot singlehandedly eliminate all service delivery fog. The key elements in your service delivery ecosystem that contribute to those ilities that are meaningful or important to your customer not only will span systems, software, people, skills, and suppliers, they also will be affected by the interplay between them. Some, from the software and systems themselves to configuration, access control subsystems, network services, and the data itself, contribute directly. Others, from troubleshooting and recovery tools, skilled staff, and instrumentation systems, are supporting mechanisms that aid finding and fixing ilities that start to drift outside acceptable parameters.

Understanding what these elements are, the relative importance of the ilities they deliver, and how they contribute to them will help you build a map that you can use to learn and improve your ability to deliver them. What is particularly important about building this map is that you will soon realize how much control you have with managing the ilities these elements provide. Elements will fall into the following three distinct categories: the known “manageables,” the known “unmanageables,” and the unknown.

Let’s look at how each category can help you and your team gauge and ultimately shape your organization to best manage your operational risk.

What You Know and Can Manage

Known “manageables” consist of all the elements in your service ecosystem that you know you can demonstrably control that directly contribute to the operational ilities of your service. This is the category where, in a perfect world, all the elements necessary for all critical or important ilities would optimally live.

However, being known and “manageable” is only useful if you are able to monitor, respond to, and recover from any events that jeopardize the delivery of those ilities in a timely and effective way. Let’s take the example of transaction responsiveness. Most customers will have expectations regarding what length of delay is acceptable for a transaction. Having known “manageable” elements that you can adjust or change to shorten growing delays is not terribly useful if you are not able to make those adjustments at a speed or in a way customers can tolerate.

Similarly, it is not enough to simply have the element that is directly and knowingly “manageable.” You also need the supporting mechanisms that find any ility delivery problems with that element and correct them to also be responsive enough. This is one of the most common ways that organizations that try to manage ilities fail. Some will put in place elaborate tools and technologies for managing such ilities as scalability and recoverability. Yet I have seen where they will either not have sufficient in-house skills to use them effectively, or split responsibilities across the organization and include teams that are unable or unwilling to contribute effectively to meet customer needs.

It is important to know such elements and how they contribute to the success of the service, know and test their thresholds, track their current state in production, and ensure that any groups that must act when thresholds are approached are capable of doing so in an acceptable timeframe with an acceptable level of accuracy.

The Known Unmanageables

Known “unmanageables” are all the elements in your service ecosystem that you know contribute to the operational ilities of your service but are not under your direct control. The most common of these today are cloud and PaaS services, such as AWS and Azure, that are relied upon for critical infrastructure and platform services. Just as common as these services but often overlooked are components delivered by commercial hardware (such as a provider of network, compute, or storage equipment) or software (such as database, payment, or security COTS packages) providers, smaller but often just as important caching and gateway service providers, as well as providers of externally run build, test, and deployment services.

Knowing their importance and what they contribute is extremely helpful. But the big thing with known “unmanageables” is to minimize their risk to your ecosystem. With systems and software, this might involve building robust monitoring, troubleshooting, and recovery procedures, along with using more standard or proven versions and configurations in places where they provide critical capabilities.

Another even more effective approach is to engineer and operate your services in a way that is intentionally resilient to any ilities issues those “unmanageables” might create. This is very much the thinking behind Netflix’s Simian Army and Chaos Engineering. For Netflix, they knew that hosting significant components of a streaming service on AWS instances could expose them to unpredictable performance characteristics that might cause lags and frame drops that could hurt the customer’s viewing experience. They also knew that engineers have a tendency to ignore or downplay the threat of ecosystem problems, creating software that struggles to deal with the unexpected.

By using tools like Chaos Monkey and Latency Monkey, engineers knew that “once in a blue moon” sorts of problems would definitely occur, forcing them to engineer their software to resiliently deal with them. This results in lower risks to the business and ultimately the more predictable delivery of the ilities the customer expects.

The Unknowns

Even if you have the ability to manage an ility, it is going to be of little help if you do not know its constituent elements, their state, or how to best operate them to deliver what the customer needs. Ilities of this sort are the unknowns. These unknowns introduce a level of unmanaged risk that can damage or even destroy your business.

Unfortunately, with all the lack of information flow and suboptimal grasp on the service part of delivery, being an unknown is the default state for most ilities. It is also where most service delivery organizations find themselves at the beginning of their service engineering journey. Even if you solve the problem with determining what the customers need, you know that you still have a lot of work to do to close the awareness gap within your delivery ecosystem to deliver it.

Ways the Team Can Eliminate Service Delivery Fog

There are several other ways that teams can, regardless of your organizational structure, enhance their shared situational awareness, all while continuing to learn and improve. Organizing work to make it clearer of what is going on (covered in Chapter 12, “Workflow”), having regular good but lightweight ways of sharing information and ideas (covered in Chapter 14, “Cycles and Sync Points”), and having intelligent and ility-meaningful instrumentation (covered in Chapter 11, “Instrumentation”) are obvious ways to help. There are also ways to restructure governance processes (covered in Chapter 15, “Governance”) and automation practices (covered in Chapter 10, “Automation”) so that they enhance rather than hinder awareness.

Before getting into specific patterns and practices, it is good for you and your team to capture and close any maturity gaps that can hinder information flow and delivering effectively. This is discussed in the next chapter, along with the first of the “duty” roles that a team can use to facilitate operational information flow and spot improvement areas within and between teams. I have used the pattern myself in a number of organizations and have found it a great way of breaking down silos while simultaneously improving cross-team technical coordination and expertise.

The second “duty” role, the Queue Master, is far more critical. It is covered in Chapter 13, “Queue Master,” and is a means to allow each team member a chance to step back from their normal work to improve team situational awareness and spot learning and improvement areas by monitoring the work flowing through the team.