Chapter 10. Change Management
Change management is a broad function that helps us plan, review, and communicate many different types of planned and emergency (unplanned) system modifications. Changes may be bugfixes or new features and can range from a trivial configuration modification to a huge infrastructure migration. The goal of change control is to manage all changes to the production (and usually QA) environments. Part of this effort is just coordination, and that is very important. But part of this effort is also managing changes to the environment that could potentially affect all of the systems in the environment. It is also essential to control which releases are promoted to quality assurance (QA) and production. Change control can act as the stimulus to all other configuration management–related functions as well. Throughout this chapter we will discuss how to apply change management in the application lifecycle management (ALM).
10.1 Goals of Change Management
The goal of change management is to identify and manage risk. Change management can help drive the entire build, package, and deployment process by identifying the potential downstream impact of making (or not making) a particular change. There are seven different types of change management, including managing changes to the software development process itself, which we will explain further in Section 10.17. Change management also has the essential goal of facilitating effective communication and traceability. Done well, change management can be an essential part of the software and systems delivery process, accelerating the rate of successful upgrades and bugfixes while avoiding costly mistakes. However, the lack of change management can lead to serious outages, lead to reduced productivity, and endanger the organization.
10.2 Why Is Change Management Important?
Change management is important because it helps drive the entire configuration management process while identifying technical risk inherent in making (or not making) a change. Risk is not always bad if it is identified and fully understood. An effective change management process helps identify, communicate, and then mitigate technical risks. Communication and traceability are key aspects of the effective change management process. In fact, change management can drive the entire ALM effort. When process changes occur, change management helps ensure that all stakeholders are aware of their new roles, responsibilities, and tasks. Organizations that lack effective change management are at increased risk for costly systems outages. Change management drives the ALM.
In some organizations, change management simply focuses on identifying potential scheduling conflicts that can adversely impact the application build, package, and deployment process. Managing the calendar is certainly important, but that is only the beginning. It is common for a particular deployment or configuration change to depend upon the same resource as another change. The effective change management process also identifies the affected assets and the experts who can best understand and communicate the potential downstream impact of a change. With complex technologies, many IT professionals are specialists in a particular area, resulting in organizational silos that may not fully understand how changes that they may make could potentially impact other systems. The change management process should identify all stakeholders who should be involved with evaluating a proposed change and then communicate the potential risk of making (or not making) a particular change.
DevOps principles and practices are particularly effective in improving the change management process, which often fails to include all of the required stakeholders and subject matter experts.
10.3 Where Do I Start?
Getting started with change management usually begins with assessing the existing processes and identifying what is being done well and what could be improved. We find that many organizations focus narrowly on the calendar and then fail to adequately review and evaluate technical risk. This is usually because the technical experts associated with assets that are the subject of a particular change request have not been identified. We also find that many change management functions need to improve their approach to communicating planned (and unplanned emergency) changes.
It is common for the change management function to be conducted as a very long meeting, often lasting two hours or more, which is considered a painful bureaucratic experience of little actual value. We usually start by identifying routine change requests (CRs) that can be handled in a separate meeting, thereby shortening the change management meeting. Long painful meetings rarely motivate participants to implement effective change management.
We try to improve change management by having less of it.
Identifying changes that are routine and can be designated as preapproved is an effective first step in improving the change management process. We are often able to separate out routine changes, which still need to be communicated but do not require the same assessment of technical risk as application deployments or configuration changes. Closely related is the practice of creating specialized change control boards (CCBs). We have seen organizations that had separate change control boards for information security, which focused on authorization, entitlements, and key infrastructure such as firewalls. The specialized change control board (CCB) may go into great detail, which could simply be a single line item in the centralized change management function. One key aspect of effective change management is ensuring traceability, which is actually a regulatory requirement in many organizations.
10.4 Traceability for Compliance
Change management provides a robust framework for documenting changes that are essential for successful operations as well as regulatory compliance. Publicly traded firms, including banks, trading firms, and many other companies, are required by federal law to provide accurate records of all changes to production, which helps to answer the question “what did we do last time” to solve a problem or to provide a new feature. The change management function provides this traceability through documentation, usually via a robust workflow automation tool. We discussed workflow automation tools in Chapter 7. Closely related are the functions of incident and problem management, which we will discuss later in this chapter. When bad things happen, it is not uncommon for outside agencies to ask for documentation to demonstrate that the company took prudent steps to avoid the risk associated with common mistakes. You may hear folks discussing what a “prudent” approach would be. Having your steps well documented, including approvals, tests, and steps taken to avoid human error, can go a long way to demonstrating that your team did everything possible to avoid a mistake. Change management plays a key role in providing that documentation and traceability. It also helps enable the assessment and management of risk.
10.5 Assess and Manage Risk
Risk is not intrinsically bad. But risk must always be evaluated and understood by all stakeholders. The first step is to understand the risk and communicate with all other stakeholders. The logical next step is to create a plan to mitigate and manage the risk. In practice, change management helps identify the risk of making a change versus the risk of not making a change. Large-scale systems actually have considerable inherent risk, often because of their complexity and decisions that have been made over time, which may have compounded the risk. Making pragmatic short-term decisions is sometimes referred to as creating technical debt. The idea here is that short-term decisions made for the sake of expediency build up a type of “IOU”—the technical debt. Too many short-term decisions increase risk and may eventually lead to the system being very brittle.
Technical debt is often unavoidable.
The most important aspect of managing risk is ensuring that there is excellent communication.
10.6 Communication
The greatest risk that we see in most organizations is poor communication. Large organizations in particular are often siloed, and many technology professionals may not even know who to reach out to for help or who should be informed when there are incidents and problems. This is precisely where DevOps principles and practices can be most valuable. We often find ourselves acting as a catalyst for better communication and collaboration by ascertaining who should be involved with a particular call. Organizational dynamics can sometimes be challenging—especially when managers model dysfunctional behavior by allowing their subordinates to view themselves as being in competition with another group. In this context, change management has the goal of keeping everyone informed and reducing the noise that is often associated with dysfunctional—and poorly implemented—change management. Change management is a valuable function, but unfortunately, we often observe ineffective or even counterproductive efforts.
Sometimes we even see behavior that somewhat resembles job actions as if IT professionals were part of a union. We also see situations where managers play politics to the detriment of the organization.
Managing changes is essential. Unfortunately, there is often a lack of coordination in change management within the ALM itself.
10.7 Change in Application Lifecycle Management
Change management, within the context of the ALM, must evolve and adapt in order to be effective. The ALM also should evolve as part of continuous process improvement. This effort is often managed by a software engineering process group (SEPG), a group that is responsible for managing changes to the process itself. We first learned about the value of the SEPG from the Software Engineering Institute’s (SEI) Capability Maturity Model (CMM), which later evolved into the Capability Maturity Model Integrated (CMMI). Many folks found the full CMM to be difficult to implement, but the value of having a dedicated group manage changes to the software development process cannot be underestimated. This is particularly true when organizations themselves undergo significant changes, which could happen due to mergers and acquisitions.
Managing change must be handled in a holistic way. It is essential to consider the entire ecosystem.
10.8 The Change Ecosystem
Change management must always be understood within the context of an ecosystem. Developers have their priorities—generally to deliver new features and bugfixes at a rapid pace. We usually hear developers discuss what might happen if a particular change does not occur. Operations has a difficult perspective and should always be focused on maintaining reliable systems. The information security (InfoSec) group helps keep our systems secure, and QA and testing guide us along the path to producing code that is as free of defects as possible. End users and the business have their own perspectives, which may include dealing with strong competitive pressures, which in turn, puts pressure on the ecosystem to evolve and meet business demands. The change management function benefits greatly from DevOps by ensuring that we have effective communication and collaboration. We need to always consider the risk of making a change versus the risk of not making that change, and each of these stakeholders has an essential role in making these decisions. Only when you consider change management in a holistic way can you make the right decisions to ensure that your systems are reliable, secure, and effectively supporting the business.
Understanding change management in a holistic way is essential. Ensuring quality is also a fundamental requirement for effective change management.
10.9 QA and Testing
Change management benefits greatly from integration with the QA and testing process. Unfortunately, far too many folks fail to consider the requirements of QA and testing in the change management function. When we participate in change management, we often ask how we can verify that the change request was completed successfully. Equally important is establishing criteria for QA and testing should the change need to be backed out. The change request should document these steps and establish acceptance criteria, both for the successful implementation of a change and in the case of a change needing to be backed out.
We see many circumstances where QA and testing are limited, but other verification and validation criteria can still be successfully implemented. For example, you might want to ensure that systems have rebooted successfully before the testing process begins.
Environment monitoring helps maintain reliable systems by providing an early warning that an incident may be about to occur.
10.10 Monitoring Events
Event monitoring is an essential capability that is often overlooked in many organizations. Many types of events could potentially be monitored in large enterprise systems. The ITIL v3 framework describes events as a “change of state” that may require some action and could be indicating that an incident is about to or has already occurred. The most common events are alerts or notifications regarding thresholds such as disk and memory limits. These constraints must be defined and monitored, providing sufficient time to address any potential problems before they result in a systems outage. The change management process should always identify events that should be monitored and that might be potentially affected by the change request. One problem is that we often do not know what to monitor, and this is exactly where DevOps can help us.
Runtime dependencies can be complicated and difficult to monitor, as well as to interpret. Too often, the operations team starts to identify events that should be monitored late in the software and system development lifecycle. This approach often does not work because the experts who wrote the code and are most knowledgeable about dependencies may have moved on to other projects by the time that the operations team is starting to create runbooks and establish support procedures. The best approach is to begin to identify events that should be monitored early in the process and establish automated monitoring for test environments as part of the deployment pipeline.
Monitoring events is essential. When major incidents occur, it is essential to establish an organizational structure to facilitate communications and collaboration. This is usually called the command center.
10.11 Establishing the Command Center
Large-scale technology systems always have challenges. Outages can occur due to hardware failure thought to be reliable, third-party products, or any number of other dependencies. What matters most is how the support team responds to and deals with any challenges that occur. We have discussed how monitoring events can sometimes help by providing an early warning system. We have sometimes identified serious issues about to occur and quickly scrambled to address problems, communicating with our users as needed. We find that fast and effective response is essential, but excellent communication is far and away the most important factor when it comes to ensuring that our customers and end users are satisfied with our service and support. When serious outages occur, communication must be managed from an incident command center (ICC).
The command center provides a structure to coordinate the response and especially the communication between all of the stakeholders engaged in addressing the incident. We view the command center from a DevOps perspective; resources from different teams must be able to communicate and collaborate effectively. In practice, we find these teams are often highly structured and accustomed to operating from what actually may be a very rigid interface. Other team members may experience these teams as being highly siloed. We often see dysfunctional behavior, which hampers the response and resolution. Establishing a command center provides an independent structure to ensure that the required groups are each represented and working together effectively. The measure of success is how effectively the team addresses and resolves incidents when they occur.
The command center is essential for establishing effective communications. Yet, most teams are judged based upon their initial response and escalation when incidents occur.
10.12 When Incidents Occur
When incidents occur, the team’s success is often measured by their initial response. Identifying the scope of the incident—and especially all of the stakeholders who should be involved—is a fundamental requirement. Too often, the scope of the problem is not well understood and the necessary resources are not successfully identified and engaged. Sometimes this is because technology professionals are either afraid to speak up or they give their input but are ignored by those in positions of power when they do. In practice, we see professionals who want to do the right thing, but their cognitive processes may be impaired by the stress and shock experienced when they fear that they may be blamed for a catastrophic outage.
Incidents are sometimes unavoidable, but once their negative aspects have been addressed, they should be used to improve our processes. From a DevOps perspective, incidents are feedback loops that serve to help us identify where we need to establish, or at least modify, IT controls intended to avoid mistakes. Incidents can also help identify test cases that should be created to avoid defects in future releases. Smart technology leaders embrace every opportunity for the team to continuously improve existing processes. The way in which you manage incidents will determine how your team responds to any future incidents that occur.
When incidents occur, we often have the opportunity to identify new test cases and events that should be monitored. Incidents may identify defects in the code or sources of human error, which could occur again in the future. DevOps teaches us to view these challenges as feedback loops and opportunities for improvement. This is also where we may need to establish IT controls, which we discuss in Chapter 16, “Audit and Regulatory Compliance.” Sometimes, incidents may be difficult to diagnose, especially in terms of their root causes. It is common for serious issues to require escalation, a process known as problem management.
10.13 Problems and Escalation
Problems are generally distinguished from incidents by the need for root-cause analysis and often an escalation to include additional resources. We see many situations where the incident response team just does not have sufficient expertise and resources to address serious problems. Escalation may require that the developers who wrote the code participate in the problem resolution process, assuming they are still available. There are also many situations where third-party providers might need to be involved as well. Generally, we like to record problem investigation sessions so that we can review steps taken and consider ways to improve our processes later.
Significant outages may identify serious defects that need to be addressed, as well as new features and tools that should be developed to help prevent future problems. It is important to identify events that should be monitored in the future to avoid similar situations. Problem management and escalation also require effective communication to all stakeholders. We find that in the middle of a serious problem, stress and fatigue can impair the performance of many team members. This is where incident and problem management teams can help improve the effectiveness of the overall escalation effort.
In IT, no less than in politics, remember always that “no good crisis should ever go to waste.”
Problem management and escalation are essential processes that can harness the opportunity to minimize the impact of a serious outage. Generally, we find that the change management process itself must be continuously reviewed and improved.
10.14 The Change Management Process
The biggest impediment to effective change management is often the change management process itself. Too often, change management is a verbose bureaucratic process that wastes everyone’s time while doing little to avoid human errors, along with other types of risk associated with implementing changes. The key to success is to choose the best tools to support the change management process.
The first thing that we usually do is work to get the change management sessions to be shorter and better organized. Creating an agenda and ensuring that subject matter experts (SMEs) are only required to join in when change requests (CRs) they are involved with are being discussed streamlines meetings and minimizes resistance. Routine changes should be identified and handled separately as “pre-approved” change requests.
The ITILv3 framework describes a change advisory board (CAB), which acts as an expert resource to the change management function. We have frequently seen confusion regarding the difference between the CAB and the CCB. Some organizations combine these two groups, often maintaining the name of CAB, which sounds more official to many. We believe that the CCB and the CAB should be kept separate, inviting CAB members into the CCB meeting when they are needed to act in an advisory capacity. The CCB should be more narrowly focused on the change process and traceability, whereas the CAB helps understand the downstream impact of a change. CAB members are the SMEs associated with specific assets that they know well. CCBs members are often willing to sit through the entire change management meeting. The SMEs in the CAB should only be consulted for the CRs for which they are involved. CAB members are certainly welcome to stick around for the entire change control meeting, but are not mandated to do so. We see CAB members self-select to leave as soon as the CR they are involved is discussed and approved (or not).
We also find that creating specialized change request boards can be helpful to streamline the process. For example, we had one specialized change management meeting focused on storage and capacity planning across Unix, mainframe, and Windows. This meeting took over an hour and covered technical details of interest only to those deeply involved with managing the complex storage devices, including storage-attached networks (SANs) and network-attached storage (NAS). The main change management meeting covered this item in a one-line summary, which advised everyone regarding which weekends would be focused on upgrading storage, effectively prohibiting any application upgrades on the same day. No one objected to getting a weekend off, and we avoided the scheduling conflict of trying to upgrade a system on the same day that the storage systems would be offline, effectively rendering the underlying systems inaccessible. One effective way to optimize the change management process is to establish entry and exit criteria up-front.
10.14.1 Entry/Exit Criteria
When changes are complicated and require a technical review, we have found that providing documentation of the change up-front can help make the change management process much more efficient. We have had considerable success with this approach, and shortening the time for the change management meeting can be reinforcing. Most importantly, we want to identify assets that may be impacted by a change and the subject matter experts who should be consulted for each change.
Similarly, establishing exit criteria up-front, including test cases and criteria for verification and validation, can also help make the process more effective. This is one area where having a post-implementation review can help improve the change management process.
10.14.2 Post-Implementation
Post-implementation reviews help us understand what went well and what could be improved in the change management process. In Chapter 13, we discuss retrospectives, which have a broader scope than post-implementation reviews. What is common to both is the need to get honest and open feedback on what went well and what could be improved. One clear way to improve the change management process is to preapprove routine changes.
10.15 Preapproved Changes
Some changes are well understood and may occur on a frequent basis. ITIL v3 refers to these changes as standard changes, and they are generally separated out to allow more time to focus on normal changes, which require full review by the change advisory board. Unfortunately, we see organizations where the change management function simply groups all changes together. This is usually because the folks running the change control process lack the technical expertise to really understand the nature of the changes. Mixing all of the changes together into one long meeting tends to be highly dysfunctional and really undermines the effectiveness of the change control process. Taking preapproved changes out of the formal change control meeting is a good first start to help make your change management function more efficient and effective. Pre-approved changes still need to be communicated, but should be handled separately, leaving the actual change control meeting to focus on assessing and managing the technical risk inherent in any complex software upgrade, configuration modification, or even bugfixes.
Focusing on the changes and the process is important. But establishing an effective change management function is even more essential for successfully managing changes.
10.16 Establishing the Change Management Function
Establishing an effective organizational structure to champion change management is a critical success factor for any agile ALM. Change management must work successfully within the culture of the organization, while ensuring compliance with any required regulatory and compliance requirements. The CM group facilitates and drives the process, but in many ways, they must align with the corporate structure in order to be successful. Change management ensures effective communication, collaboration, and documentation for traceability. Change management must be continuously reviewed and improved, while also ensuring that processes are well understood, followed, and work effectively to identify and manage risk. Change management should also effectively help change to occur. We find that the structure of the change management process is essential.
10.16.1 Change Control Board
As discussed previously, we view the change control board (CCB) as being the process-oriented representatives from each of the essential organizational functions. In practice, we see representatives from development, operations, the business, information security, and QA and testing being included. The CCB must ensure that all affected parties are informed and, when necessary, brought in for consultation. This is where we view the change advisory board as being quite helpful.
10.16.2 Change Advisory Board
The change advisory board provides expertise on the potential downstream impact of a change. It has become common practice to combine the change control board with the change advisory board, calling the resulting group the “CAB.” However, we believe that these two entities are best kept separate, with the CCB focusing on process and the CAB advising on the potential downstream impact of a change.
10.17 Change Control Topology
There are many different types of change control, and you will likely need to have two or more different CCBs, with each handling one or more of the following seven types of change control. I am not suggesting that you need to implement all of these change control functions. It has been our experience that change control usually involves a combination of these functions.
Implementing change management usually just begins with a couple of these functions.
Controlling changes to production is a key place to start.
10.17.1 A Priori
Some organizations establish a disciplined process whereby permission for a change is requested before any actual change to the code is made. I have seen defense contractors who had to describe the changes that they wanted to make and then await approval from a government agency before actually writing the code that implemented the change. In this process, requests for change (RFCs) are usually created and reviewed by the respective change control board. A priori change control usually refers to changes in the code and most often consists of defining requirements followed by the actual designing of the system. The role of configuration management in this case is to track requirements throughout the lifecycle and confirm that all requirements were included in a specific release. Many organizations have a regulatory requirement for tracking requirements, and that often includes a change control function. Tracking source code changes to requirements is important, but changes are often related to environment configuration as well.
10.17.2 Gatekeeping
The most common type of change control, and usually the first to be implemented, is “gatekeeping” change control where the CCB reviews RFCs that will impact production or QA. Usually this involves giving authority to promote a new release of the code into production or QA. Similarly, patches to existing releases are also reviewed by the CCB. This function generally evaluates whether or not there is a risk that the RFC could potentially affect the production or QA environments (which is common for the members of the CCB to have questions about). The CCB is responsible for reviewing the RFC and approving or rejecting it. Traditionally, the CCB will require that all necessary technical experts be present at the CCB meeting, although, in practice, this is often not practical. The ITIL framework has made popular the use of a CAB that consists of experts who can advise on the downstream impact of a particular change. I will discuss how to set up a CAB and why it may need to be separate from the CCB in a later section of this chapter. Closely related are configuration changes, which will be discussed in the next section.
10.17.3 Configuration Control
When the request for change involves a configuration change, the CCB reviews and considers its downstream impact. Configuration changes can have the same impact as a new release. In practice, understanding the interface dependencies often requires specialized knowledge, and these should be reviewed by a board that contains members who possess this expertise. In this case, I believe that the governing body should be called a configuration control board. However, there is some confusion in the terminology commonly used today. Many of the industry standards describe the configuration control board as governing the configuration of a system in terms of the configuration of the source code itself instead of environment configuration. In these standards, a configuration of the code refers to a specific set of versions of the source code. I believe that this usage is confusing and a relic of days past when configuration control referred to controlling the version of a Cobol program that was being promoted on a large IBM mainframe computer. Today, most organizations promote a packaged release that may contain thousands of configuration items, including binaries, XML, and many other artifacts. I believe that it makes more sense to use configuration to refer to environment configuration and to use terms like “baseline” or “release” to refer to a specific set of code versions that are promoted as a release. There are many reasons for this. Most releases are packaged, and the entire release package is deployed as a complete package. The last thing that the administrator deploying the release wants to know about is the specific versions of each of the configuration items that make up the packaged release. However, in these same situations, environment configurations such as interprocess communication ports are still managed through the change control process, as they should be. So, if you want port 9444 opened on an application server, then you need to complete a change request and, once approved by the configuration control board, the data security team will modify the iptables to allow interprocess communication on port 9444. In my opinion, true configuration control should refer to interface (runtime) dependencies only.
10.17.4 Emergency Change Control
There will always be times when emergencies require immediate changes. It is likely that the CCB cannot meet at any hour of the day or night to authorize needed emergency changes, and focusing on strict adherence to the regular approval process may result in the company production system being down for an extended period. Any successful change control function must include a well-defined process for managing emergency changes. I recommend that a very senior manager’s approval be required for emergency changes and that there be discussion after the event to understand why an emergency change was required in the first place. I have seen situations in which technology professionals abused the emergency change control process to bypass the regular change control process. In such cases, you will be more successful if you have the support of senior management to ensure that everyone follows the process in the best way possible.
10.17.5 Process Change Control
Organizations establish processes to run their operations on a day-to-day basis. These processes are established, and the teams affected are expected to comply. When circumstances require that the processes need to be adjusted and the changes can have wide-ranging impacts upon the entire organization, these changes should be reviewed by a specialized CCB, often called a software engineering process group (SEPG). In this case, the process engineering should be placed in the hands of a change control board that is responsible for reviewing requests for changes to the process. The CCB for process engineering is also tasked with communicating process changes to all affected parties and stakeholders. I believe that the best response to a mistake is to reexamine existing processes and ascertain whether or not additional process steps are warranted. Process improvement should be an organized continuing collaborative effort, and the process CCB can help manage the process engineering effort on an ongoing basis.
10.17.6 E-change Control
E-change control refers to using e-mail or a change control management system to electronically authorize changes. This approach is often used for emergency changes, as described in Section 10.17.4, or in routine changes that do not require review.
10.17.7 Preapproved
We strongly recommend separating out preapproved, or what ITILv3 refers to as standard, changes. These changes typically occur on a regular basis and are well understood by the organization. They still need to be traced and communicated, but should ideally be separated from the rest of the normal changes.
We often see a management oversight function, which may be its own change control board. This group, responsible for ensuring that the change control process is being followed, performs an important function; it is equally important to ensure that change management is handled consistently across each of the platforms in use throughout the organization.
10.18 Coordinating across the Platform
It is common for change control to focus on one or more platforms and, in many ways, that is actually quite appropriate. Change control for a mainframe follows a particular pattern and certainly a particular culture. Change control in a Linux or Unix environment may be quite different. We prefer to create change management functions that align with the culture of the organization, and often that requires that we align with multiple platforms as well. But the truth is that many releases affect one or more of the organizational tiers. You may have changes on the mainframe that are a dependency for corresponding changes on the Linux or Unix environment. Similarly, changes on the Windows platform may be dependent upon changes on the mainframe, Unix, or Linux machines. Sometimes, you will need to coordinate changes across, as well as within, each of the platforms.
Equally important is coordinating changes across the entire enterprise.
10.19 Coordinating across the Enterprise
Many large organizations have divisions that act as if they are separate companies. In fact, sometimes they are the result of mergers and acquisitions and functionally remain separate for all practical purposes. In banking and other highly regulated industries, there may actually be a requirement to keep specific divisions completely separate. We find that sometimes we are creating change management functions in one division of the company that operates significantly differently from another completely separate division. Typically, we set the same goals for communication and traceability, while establishing procedures that work in alignment with the culture and goals of each specific group. When working across a large organization, you may feel very much like you are working for one or more separate companies. One occupational hazard is trying to operate across organizational structures that have become accustomed to working as separate entities. This can sometimes be challenging, and often we struggle to work within the separate silos, which effectively act like fiefdoms.
10.20 Beware of Fiefdoms
Fiefdoms refer to the feudal lords who acted in a totalitarian fashion in medieval England. We apply this term to organizational structures that operate in a similarly totalitarian fashion. We are all too familiar with this dysfunctional structure and keep a suit of armor to don for just such occasions. Fiefdoms in business settings tend to self-destruct, and you may need to be there to help pick up the pieces and repair the damage. We find that these groups are so well entrenched that you may have little success dealing with them directly. Generally, everyone else is fed up with them, too, and may actually help you bypass them so that you can get things done. Sadly, we have seen fiefdoms in our work with government agencies, groups that put more effort into maintaining their authority than into accomplishing the objectives for which they are responsible.
Fiefdoms present a strong defense when trying to establish effective communication and collaboration. Fortunately, such dysfunctional behavior is usually the exception and not the rule. We do find that sometimes we need to establish specialized change control.
10.21 Specialized Change Control
We have come across technical functions that required specialized change control in order to operate effectively. One example is within the security realm around firewalls and other security appliances. Sometimes, there is a limit to transparency to ensure that information around existing intrusion detection practices stays within the information security function. When appropriate, we work with stakeholders to establish specialized change control functions with enough interface to the main change control board to ensure sufficient transparency where it is warranted and necessary.
10.22 Vendor Change Control
There are times when you may need to visit vendor sites to ensure that they have sufficient software methodology and IT controls in place to avoid errors and meet your own regulatory compliance requirements. Banks and financial institutions often have to be reminded that using a vendor does not absolve them from having to ensure that their software is written in compliance with all regulatory and audit requirements. Health insurance vendors, for example, must be reviewed to ensure that their controls meet the requirements for Health Insurance Portability and Accountability Act (HIPAA) and Code of Federal Regulations (CFR) 21. We have attended and audited change control meetings at vendor sites to assess existing practices. It is not uncommon to include such compliance as a contractual obligation. Government contractors are often required to achieve and maintain CMMI level 2 (and some level 3) process capabilities. This is often referred to as subcontractor maintenance. Always bear in mind that outsourcing your work does not necessarily outsource your obligations.
Ensuring vendor compliance is essential. We also find ourselves needing to be very aware of service providers’ change control policies and procedures.
10.23 SaaS Change Control
Software, Platform, and Infrastructure as Service providers all should maintain clear and transparent change control procedures, especially with regard to backwards capability. We have seen considerable bad behavior in this area with service providers forcing upgrades that caused downstream impact and yet provided no backwards capability.
Beware of service providers who do not provide transparency into their change control processes. You need to identify this as a risk and plan accordingly. We have run into the same issue with upgrading software on electronic devices such as a GPS.
Change management requires a consistent focus on continuous process improvement.
10.24 Continuous Process Improvement
Change management is an organizational function that should, and usually does, evolve over time. We try to start with a very light approach to managing change and then only add more controls as necessary. The process itself should be managed, and effective communication is essential for its successful evolution. Try to avoid too much rigidity, which always motivates the team to bypass the process. Constantly evaluate your own effectiveness and focus on identifying, communicating, and mitigating risk.
10.25 Conclusion
Change management is an essential part of the agile ALM. Too often, this function is associated with verbose, painful, and ineffective processes. To foster colleague buy-in and increase compliance, focus your change management approach on both agile and Lean, maximizing its value as a feedback loop and a key driving force for an effective ALM.