CHAPTER 16
Configuration Management
Configuration Management includes the control processes that allow managers to have confidence that all system components are managed effectively. The components include source code, production hardware, and the production environment itself. Whether vendor packages or internally customized systems, effective configuration management lays out well-defined, disciplined processes that identify who is accountable for each step.
Decades ago, this subject was demanding, but now the subject is daunting. Highly complex and sophisticated development of source code components leads to increasing confusion and chaos. Systems can run on multiple platform types. Many versions of a system can be in production at any given time, and all of them need to be supported. The growing trend in server consolidation drives multiple systems to run on a single server with shared common components (database drive), and so one system change affects many unsuspecting system owners.
This chapter presents concepts that will allow you to diligently control the systems and the production infrastructure on which the systems run. Without clear control, system integrity is lost. We will not attempt to evaluate the use of automated tools because new automated Configuration Management tools appear continuously in the marketplace and any product evaluations are soon outdated. But you will be able to evaluate vendor-supplied tools yourself when you have understood the concepts presented in this chapter and know how to implement an overall Configuration Management process.
This chapter divides Configuration Management into the following five distinct controls:
• Version Control
• Change Control
• Migration Control
• Production Control
• Environment Control
Other sources may treat Configuration Management differently. Some traditional approaches view Version Control as the only component of Configuration Management. Others combine these controls. Keep an open mind. These five are separated in order to get the points across. Grasp what Configurations Management encompasses and ensure that all aspects are implemented; whether you or your IT department wants to combine these controls or label them differently is of lesser importance.
The controls of Configuration Management must be repeatable. Each change should be handled the same. The processes for each control must be well documented, and the specific changes must also be documented. The entire process must be able to pass an audit, including the retained evidence of changes and their approvals. Your company’s production environment deserves this level of rigor to ensure the future viability of the business.
Before we describe each of the five controls, let’s map Configuration Management into the context of the traditional SDLC and our Work Tracking Tool approach from Chapter 11, “Work Tracking.” Figure 16-1 provides a relative time line that maps several of these areas together. Start by focusing on the Work Tracking Life Cycle in the third row. These are the steps presented in Figure 11-1 of Chapter 11, “Work Tracking.” The steps start from SUBMIT and end at CLOSE.
Figure 16-1: Configuration Management Timeline
Above the Work Tracking Life Cycle, the SDLC illustrates how these processes are similar. To complete the mapping, above the SDLC in the first row, are the typical software development environments. The software engineering discipline recommends having separate environments from production to accomplish the different life cycle stages.
The shaded area in Figure 16-1 shows four of the five Configuration Management Controls. The two dashed lines represent major approvals that affect the different controls. There are a total of four approvals listed on this figure, shown by an “A” inside a diamond with a reference number besides the diamond. The following defines each of these approvals.
A1: Represents approval to proceed with coding the change based on estimate and impact analysis
A2: Represents approval of all system/system testing
A3: Represents Change Control approval that all criteria to proceed with the production change are met
A4: Represents final timing approval to proceed with migrating the change into production
Figure 16-2 provides more details on the five Configuration Management Controls’ interrelationships with one another. The five Controls are separate processes. Within each Control Process are the components making up the process. The following five sections will explain each control area in depth. Environment Control is added to Figure 16-2 but is not included in Figure 16-1.
Version Control
The traditional view of Configuration Management encompassed only Version Control. Version Control concentrates on the need to maintain confidence that the system objects (pieces) are managed effectively and that the application/system can be rebuilt to match production versions. This rebuilding ability is vital, especially when making changes to system objects that exist as multiple versions in production.
Figure 16-2: Configuration Management Details
Version Control is the process, and most likely the tool, used for tracking version changes, documenting them, recording who changed what and why it was changed, and ensuring appropriate builds in all environments (development, test, training, staging, and production).
The following items have to be maintained in order to build the current version of a system:
• Source code
• Compilers
• Vendor executables
• Vendor documentation
• System configurations
• Hardware requirements related to different software versions
The Problem
Software development and maintenance is a complex process. A system or applications will consist of many objects, each of which can have multiple versions. A risk exists that you will not be able to control the process effectively and that there will be an adverse effect on the customer. There are not too many situations worse than introducing a defect into production due to lack of Version Control. Such an error shows your customer that your team doesn’t know how to manage the software objects or even know where the correct versions are located.
Figure 16-3 shows one potential problem. Production Version A is made up of Version 1 of the four objects. If someone alters Object 2 but did not save the original version, there would be no way to reproduce Production Version A.
This situation may seem harmless, but if Production Version A remained in production and developed a defect, there would be no easy way to confidently reproduce it in a test environment and debug. Similarly, Object 2 Version 2 could be corrupted, and the older version would be needed. It may even take time to determine that Object 2 Version 1 is missing. This circumstance can lead to confusion and increased time for problem solving.
The complexity increases when multiple versions of a product exist, as shown in Figure 16-4. The various types of dashed lines show how each separate version is built. Note that in this example some object versions are used for more than one production version, while other object versions are not used at all.
Figure 16-4: Multiversion Example
Figure 16-4 may take some time to understand, but what if you don’t have a figure like this to accurately show your configuration? How long would it take to investigate and produce this level of understanding? Dealing with this kind of a problem is what version control is all about.
The Solution
Version Control is the disciplined process for ensuring that all software versions proceed under systematic control. Such control preserves traceability and integrity, which reduce risk to the customer. The goal is consistency and reproducibility so that older versions can be reproduced if needed. The versions may include a combination of your team’s software and vendor software. As shown in Figure 16-1, Version Control involves a process to control object sources by allowing a check-in and check-out process, by providing the ability to build (compile) the components, and by ensuring that all documents are updated.
Version Control can be implemented at any level of sophistication. Greater rigor should be applied when the complexity of the build is greater. Version Control can be a manual process that relies on the developers to adhere to the procedure, but more commonly it is a Version Control tool available in the marketplace that enforces the procedure. Contrary to the opinion of some developers, the purpose of Version Control is not to slow down the developer.
At any given time, your team has to be able to simultaneously manage the past, present, and future versions, as shown in Figure 16-4. Your team has to be able to:
• Reproduce a build with the correct components and deploy the current version, even if previous team members have left the company.
• Start an enhancement effort to modify, control conflicts, and build (compile) the next version.
• Build (compile) all older versions that are still in production in order to debug a defect.
• Improve quality and reduce defects caused by building products with the wrong components.
• Facilitate movement of team members to other projects without losing the team’s knowledge base or productivity.
• Track which versions are in production.
Version Control Process
Figure 16-5 shows the Version Control process flow. The process starts with an approved request; the request should clearly map to a specific enhancement or defect fix. The team member then follows the procedure for checking out the objects. The software objects, build instructions, and documentation are checked out together. The check-out step records who has taken out the objects, and no one else can check them out until they are checked back in. This mechanism prevents changes from interfering with one another or overwriting each other.
The process allows team members to view the objects without officially checking them out, which is appropriate when a team member is analyzing a potential change.
After the objects and documents are checked out, coding can begin in order to modify the objects or create new objects. Then the build instructions can be used to compile and link the objects in order to create the necessary executable programs. After testing is complete and the documentation is updated, the objects and documentation can be checked back in. The check-in step should also include recording what changed and why it was changed. Nothing fills this requirement as well as comments embedded in the module’s source code itself that are clear and well thought out.
Figure 16-5: Version Control Process Flow
A Version Control process that manages your configuration offers the following benefits:
• Fewer mistakes when rebuilding executables
• Lower cost, due to decrease in fixes of preventable mistakes
• Reduced number of deployment errors
• Happier customers having greater system availability
• Defects quickly reproduced in any version for debugging
Under normal Version Control rules, only one person can check out a source code module at any given time. One programmer’s changes are thus prevented from interfering with another programmer’s changes. However, there are cases in which multiple projects are initiated concurrently, which may require both project teams to have access to the same modules. In these types of cases, the Version Control methodology will have to carefully address how each team’s changes will be retained. Significant time is involved in diligently tracking and analyzing the changes so that one team’s changes do not interfere with the other team’s changes. In many cases, the first team’s change will be retained, and the second team will have to ensure that the first team’s change is incorporated into the second team’s future changes. This could create a situation where the second team misses including a change. Someone needs to be tasked with oversight responsibility to ensure that all changes are included appropriately.
Change Control
Change Control governs what changes can be made to the production environment. This process is closely related to the Production Control process, but these controls have their own individual focus, so we present them separately. The Change Control Process is the guardian of production and balances the need for changes with the potential negative impacts on the business.
At this point in the Configuration Management Timeline (Figure 16-1), the ticket to have the change programmed and tested has been approved. Change Control takes this ticket to the next step to obtain approval for moving the change into production. In order to do this, other factors need to be considered.
An effective Change Control Process should track and document each production modification request. The process should be systematic, known by all involved, and auditable. The process must not introduce unnecessary delays in executing changes and should be sensitive to business impacts. The request for the production change should be submitted with an appropriate lead time before the change will be made to production.
The essence of the Change Control Process is to verify and obtain necessary approvals. The people running the Change Control Process should be knowledgeable about the production environment and the degree to which the business relies on the systems. Their function isn’t just to complete a checklist; they must protect the business owners and users from unnecessary impacts. The process should:
• Verify that the appropriate business owner approved the change to be coded. Even though it’s a bit late to catch this problem, approval should still be verified.
• Verify that testing is completed and approved by the appropriate people.
• Verify that risks are addressed and that a back-out plan was documented.
• Assess the business impact of the change and any interdependencies. The timing of the change could be a major factor in any business impact.
• Review what resources are needed to implement the change, and check their availability.
• Verify that the migrator will be a different person than the one who made the coding changes. This requirement enforces segregation of duties, which is a best practice in Sarbanes-Oxley.
• Obtain all appropriate approvals. The Production Control Process should have the lists of who needs to approve and who should be notified.
• Ensure appropriate communications are sent before and after the change.
When all verifications and approvals are obtained, and the time to make the change is at hand, the message for the migrator to proceed can be sent. If the change does not involve moving executables to production, such as a reboot, this process should be evaluated to see if steps could be skipped. But since even a reboot has the potential of impacting the business, it may still be appropriate to complete all the steps.
The final step for the Change Control Process is to verify that the Migration Control Process successfully completed the change. Note that the Migration Control Process makes the actual change and is separate from the Change Control Process. If the migration is successfully completed, the Change Control Ticket can be closed. If not, the ticket will have to be updated with a new date or other appropriate status notes.
Migration Control
Migration Control ensures that a disciplined procedure is followed that allows only approved changes with the correct components to be made to production. Migration Control must follow a systematic method that is repeatable so that it can be duplicated in a test environment if a problem is encountered.
Migrating systems to production should occur only when the Change Control Process approves the change—with the one exception of emergencies, which we will cover shortly. But there are still other tasks to complete before the actual migration can take place. Think of these tasks as the ones to perform in the 30 minutes before migrating.
The time at which the change is to take place will have been set weeks ahead of time. But unforeseen factors could have surfaced that might delay the migration. The timing thus needs to be verified to confirm that no business showstoppers or holds are in place. Then you can obtain final business approval to migrate.
After all of these approvals and verifications, the next step is to make the production change. After the change has been made, you need to verify that no adverse effects were experienced by the system. If necessary, execute the back-out plan.
The person actually migrating the change should be a different person than the one who made the coding changes. This requirement is intended to prevent any inappropriate changes from taking place that could compromise the production environment, and it is also a best practice in Sarbanes-Oxley. The system security access should be set up to group people by roles, with the developer group restricted from migrating and the migrator group restricted from developing.
Emergency production changes are the last item to consider for this control. The process must account for variations to the normal process flow. The appropriate person, under specified conditions, should be authorized to make production changes in the event of an emergency without Change Control approval. But by the next business day after the emergency is over, the person should submit a Change Control Ticket to document what happened. The Change Control Procedures should articulate the acceptable deviations for emergencies and should define what an emergency is and is not (e.g., how poor planning does not constitute an emergency).
Production Control
The Production Control Process ensures the stability of the production environment and minimizes business interruptions. Production Control involves many aspects of Operations, which is accountable for upholding the terms of the Service Level Agreements (SLAs). Production Control must ensure that changes are understood by all the Operations staff so that they can address any unforeseen instabilities resulting from the change. A subset of Production Control addresses the subject of Configuration Management. For purposes of Configuration Management, we focus on the information needed from Production Control to assist in Change Control.
With the growing complexity of the IT environment and the trend of consolidating systems on fewer servers, it is increasingly difficult to know who is affected by a single production change. Preparing for a simple server reboot can become a major research effort—trying to determine who needs to be notified and who needs to give approval.
A process and database should be set up to keep track of the following information, whose integrity must be maintained on an ongoing basis to guarantee its usefulness:
• All hardware (servers) and hardware owners
• All systems and system owners
• All databases and database owners
• What systems and databases are on what hardware
Keeping track of what systems and databases are on what hardware provides the answers to “Who needs to be informed of a change?” and “Who needs to approve?” This information can be used to automate most of the Configuration Management Process (approval requests, communications, etc.).
Environment Control
Thus far we have focused on Change Control for the production environment. But there can be many other environments that maintenance and project teams use to develop new functionality and investigate defects. These can include:
• Development Environment
• Test Environment
• Staging to Production Environment
• Training Environment
Using the other Change Control processes to migrate changes in non-production environments can be overkill. A simpler approach could be justified. The decision on which approach to take should be based on solid needs within IT. The other Change Control processes should be used for non-production only if they add value to IT by solving arbitration issues between team members or eliminating corrupted environment. Do not use the other Change Control processes if these issues can be mitigated by simpler means.
The Change Control Process can be modified to implement a simple method for controlling the non-production environments. This simplified process should have an owner and should accomplish the following:
• Provide a policy to manage many team members working in the same environment at the same time. A dedicated person to fill this role may be required for two major concurrent projects.
• Develop a schedule for who will own the environments at any give time. The owner will have authority to make changes. Other team members wanting to work in that environment at that time would have to coordinate with the current owner or wait their turn as owner.
• Specify what condition the environment should be in when transferred to a new owner according to the schedule.
• Identify who will administer access to the environments. Only IT team members will typically need access; however, business users may need access to execute testing or to attend training.
Overall, these non-production environments should have no links to production. Programmers must be able to make changes in a development mode without having the change affect production or other environments.