CHAPTER 40

MANAGING SOFTWARE PATCHES AND VULNERABILITIES

Peter Mell and Karen Kent

40.1 INTRODUCTION

40.2 MOTIVATION FOR USING AUTOMATED PATCHING SOLUTIONS

40.3 PATCH AND VULNERABILITY MANAGEMENT PROCESS

40.3.1 Recommended Process

40.3.2 Creating a System Inventory

40.3.3 Monitoring for Vulnerabilities, Remediations, and Threats

40.3.4 Prioritizing Vulnerability Remediation

40.3.5 Creating an Organization-Specific Remediation Database

40.3.6 Testing Remediations

40.3.7 Deploying Vulnerability Remediations

40.3.8 Distributing Vulnerability and Remediation Information to Administrators

40.3.9 Verifying Remediation

40.3.10 Vulnerability Remediation Training

40.4 PATCH AND VULNERABILITY MANAGEMENT ISSUES

40.4.1 Enterprise Patching Solutions

40.4.2 Reducing the Need to Patch through Smart Purchasing

40.4.3 Using Standardized Configurations

40.4.4 Patching after a Security Compromise

40.5 CONCLUSION AND SUMMARY OF MAJOR RECOMMENDATIONS

40.6 FURTHER READING

40.7 NOTES

40.1 INTRODUCTION.

Vulnerabilities are flaws that can be exploited by a malicious entity to gain greater access or privileges than it is authorized to have on a computer system. Patches are additional pieces of code developed to address problems (commonly called “bugs”) in software. Patches enable additional functionality, or they address security flaws such as vulnerabilities within a program. Not all vulnerabilities have related patches, especially when new vulnerabilities are first announced, so system administrators must be aware not only of applicable vulnerabilities and available patches, but also of other methods of remediation (e.g., device or network configuration changes, or employee training) that limit the exposure of systems to vulnerabilities.

Patch and vulnerability management is a security practice designed to proactively prevent the exploitation of IT vulnerabilities that exist within an organization. The expected result is to reduce the time and money spent dealing with vulnerabilities and the exploitation of those vulnerabilities. Proactively managing system vulnerabilities will reduce or eliminate the potential for exploitation and involve considerably less time and effort than responding after an exploitation has occurred.

This chapter is designed to assist organizations in implementing patch and vulnerability remediation programs. It first explains the impact that vulnerability exploitation can have and then illustrates how automated patching solutions can reduce that impact. Next, the chapter discusses in detail how to create an organizational process, and how to test the effectiveness of the process. It also seeks to inform the reader about the technical solutions that are available for vulnerability remediation.¹

For more general information about related topics, see selected chapters in this Handbook:

• Writing secure code—Chapter 38
• Vulnerability assessments—Chapter 46
• Operations security—Chapter 47
• Application controls—Chapter 52
• Monitoring and control systems—Chapter 53

40.2 MOTIVATION FOR USING AUTOMATED PATCHING SOLUTIONS.

From January through December 2007, the total number of published computer vulnerabilities recorded in the U.S. National Vulnerability Database was 6, 691, or 557 per month and 18 per day.² Even a small organization with a single server can expect to spend time reviewing a handful of critical patches each month. This stream of vulnerabilities has resulted in systems constantly being threatened by new attacks.

The level of damage caused by an attack can be quite severe. A number of Internet worms (self-propagating code that exploits vulnerabilities over the Internet), such as Code Red, Nimda, Blaster, and MyDoom, have been released in recent years. There are some common data points for these worm outbreaks. First, as the authors of worm code have gotten more sophisticated, the worms have been able to spread faster than their predecessors. Second, they each hit hundreds of thousands of computers worldwide. Most important, each one of them attacked a known vulnerability for which a patch or other mitigation steps had already been released. Each major outbreak has been preventable.

Benjamin Franklin once said that “an ounce of prevention equals a pound of cure.” Patch and vulnerability management is the “ounce of prevention” compared to the “pound of cure” that is incident response. The decision on how and when to mitigate via patching or other remediation methods should come from a comparison of time, resources, and money to be spent. For example, assume that a new computer worm is released that can spread rapidly and damage any workstation in the organization unless it is stopped. The potential cost to not mitigate is described by this equation:

Cost not to mitigate = W*T*R

Where:

W = Number of workstations

T = Time spent fixing systems or lost in productivity

R = Hourly rate of the time spent

In addition to the costs identified through this formula, a security incident could also cause damage to an organization's reputation. This is most significant for organizations that are entrusted with sensitive information or operations. When determining the potential cost of not mitigating, an organization should consider the possible impact to its reputation, its stock price, and even to its marketing.

For an organization where there are 1,000 computers to be fixed, each taking an average of 8 hours of downtime (4 hours for one worker to rebuild a system, plus 4 hours during which the computer owner is without a computer to do work) at a rate of $70/hour for wages and benefits:

1, 000 computers * 8 hours * $70/hour = $560,000 to respond after an attack

Compare this to the cost of manual monitoring and prevention. Assume the vulnerability exploited by the worm and the corresponding patch are announced in advance of the worm being created. This has been true for most past exploits, as true zero-day attacks (those with no advance knowledge) are not frequent. Manually monitoring for new patches for a single workstation type takes as little as 10 minutes each day, or 60.8 hours/year. Applying a workstation patch generally takes no more than 10 minutes. This makes the cost equation:

For any single vulnerability for which a widespread worm will be created, manual monitoring and patching is much more cost effective than responding to a worm infection. However, because patches are constantly released, manual patching becomes prohibitively expensive unless the operating environment consists of only a few software packages (thus decreasing the total number of patches needed) or the organization relies on end users to patch their systems (thus distributing the patching workload, but also introducing a need for patch installation oversight). Since few organizations use a small number of software packages or can rely on end users to effectively patch systems, widespread manual patching is not a cost-effective organizational approach.³

A third option is to invest in an automated patching solution. These solutions automatically check for required patches and deploy them. Both free and commercial solutions are available. Assume that a commercial solution costs $15,000 and charges $20 per computer for annual maintenance. This approach will be much cheaper than the manual solution, even though it will be necessary to dedicate possibly an entire person to maintaining, updating, and patching with the automated solution.

It is not possible to save money by neglecting patch installation. It is extremely expensive to employ manual patching efforts, and it is difficult to do it effectively. Therefore, it is strongly recommended that all organizations make effective use of automated patching solutions.

40.3 PATCH AND VULNERABILITY MANAGEMENT PROCESS.

This section discusses a systematic approach to patch and vulnerability management. The approach is provided as a model that an organization should adapt to its environment as appropriate. Implementing such an approach is necessary to respond cost effectively to the ever-growing number of vulnerabilities in IT systems.

40.3.1 Recommended Process.

Organizations should create a group of individuals, called the patch and vulnerability group (PVG), who are specially tasked to implement the patch and vulnerability management program. The PVG is the central point for vulnerability remediation efforts (e.g., patching and configuration changes). Since the PVG must actively work with local administrators, large organizations may need to have several PVGs. These PVGs could work together in a confederation or could be structured hierarchically with an authoritative top-level PVG. The remainder of this chapter is based on the assumption that there is only one PVG per organization.

As much as possible, the burden of implementing and testing remediations should be shifted from local administrators to the PVG. This should save money by eliminating duplication of effort (e.g., multiple system administrators testing the same patch on similar computers) and by enabling automated solutions, thereby avoiding costly manual installations. The easiest way to accomplish this is by implementing enterprise patching solutions that allow the PVG, or a group they work closely with, to automatically push patches out to many computers quickly. If automated patch management tools are not available, the PVG should coordinate local administrator efforts.

For the PVG to be able to test automatically deployed patches adequately, organizations should use standardized configurations for IT devices (e.g., desktop computers, routers, firewalls, and servers) as much as possible. Enterprise patch management tools will be ineffective if deployed in an environment where every IT device is configured uniquely, because the side effects of the various patches will be unknown.

To implement a cost-effective PVG, the scope of the PVG must be well defined. The PVG will monitor for and address only vulnerabilities and remediations applicable to IT technologies that are widely used within the organization. Some organizations might choose to have their PVG monitor for vulnerabilities and remediations for all IT technologies used within them. This is most feasible when the organization has a relatively small variety of IT technologies in use, or when the PVG uses an external vulnerability monitoring service (as described in Appendix C of NIST SP800-40v2)¹ that can monitor for all the necessary IT technologies on behalf of the PVG.

The list of IT technologies will be carefully formulated and made available to all local administrators. The local administrators will be responsible for securing IT technologies that are not within the PVG scope. The PVG will provide assistance and training to local administrators in how to perform this function. The remainder of this section provides details on the roles and responsibilities of the PVG and system administrators.

40.3.1.1 Patch and Vulnerability Group.

The PVG should be a formal group that incorporates representatives from information security and operations. These representatives should include individuals with knowledge of vulnerability and patch management as well as of system administration, intrusion detection, and firewall management. In addition, it is helpful to have specialists in the operating systems and applications most used within the organization. Personnel who already provide system or network administration functions, perform vulnerability scanning, or operate intrusion detection systems are also likely candidates for the PVG.

The size of the group and the amount of time devoted to PVG duties will vary broadly across various organizations. Much depends on the size and complexity of the organization, the size and complexity of its network, and its budget. The PVG of smaller organizations may be comprised of only one or two members, with a focus on critical vulnerabilities and systems. Regardless of the organization's size or resources, patch and vulnerability management can be accomplished with proper planning and process.

The duties of the PVG are outlined next. Subsequent sections discuss certain duties in more detail.

1. Create a system inventory. The PVG should use existing inventories of the organization's IT resources to determine which hardware equipment, operating systems, and software applications are used within the organization. The PVG should also maintain a manual inventory of IT resources not captured in the existing inventories.
2. Monitor for vulnerabilities, remediations, and threats. The PVG is responsible for monitoring security sources for vulnerability announcements, patch and nonpatch remediations, and emerging threats that correspond to the software within the PVG's system inventory.
3. Prioritize vulnerability remediation. The PVG should prioritize the order in which the organization addresses vulnerability remediation.
4. Create an organization-specific remediation database. The PVG should create a database of remediations that need to be applied to the organization.
5. Conduct generic testing of remediations. The PVG should be able to test patches and nonpatch remediations on IT devices that use standardized configurations. This will avoid the need for local administrators to perform redundant testing. The PVG should also work closely with local administrators to test patches and configuration changes on important systems.
6. Deploy vulnerability remediations. The PVG should oversee vulnerability remediation.
7. Distribute vulnerability and remediation information to local administrators. The PVG is responsible for informing local administrators about vulnerabilities and remediations that correspond to software packages included within the PVG scope and that are in the organizational software inventory.
8. Perform automated deployment of patches. The PVG should deploy patches automatically to IT devices using enterprise patch management tools. Alternatively, the PVG could work closely with the group actually running the patch management tools. Automated patching tools allow an administrator to update hundreds or even thousands of systems from a single console. Deployment is fairly simple when there are homogeneous computing platforms, with standardized desktop systems and similarly configured servers. Multiplatform environments, nonstandard desktop systems, legacy computers, and computers with unusual configurations may also be integrated, although not as easily.
9. Configure automatic update of applications whenever possible and appropriate. Many newer applications provide a feature that checks the vendor's Web site for updates. This feature can be very useful in minimizing the level of effort required to identify, distribute, and install patches. However, some organizations may not wish to implement this feature because it might interfere with their configuration management process. A recommended option would be a locally distributed automated update process, where the patches are made available from the organization's network. Applications can then be updated from the local network instead of from the Internet.
10. Verify vulnerability remediation through network and host vulnerability scanning. The PVG should verify that vulnerabilities have been successfully remediated.
11. Vulnerability remediation training. The PVG should train administrators on how to apply vulnerability remediations. In organizations that rely on end users to patch computers, the PVG must also train users in this function.

40.3.1.2 System Administrators.

System administrators are responsible for making sure that applicable IT resources follow the organization's standard configuration and that those resources are participating in the organization's automated patching system. If the organization is not using an automated patching system, system administrators must use the PVG as a primary resource for vulnerability remediation and work with the PVG on time frames for remediation application. For IT resources that are outside of the PVG scope, system administrators are responsible for monitoring for vulnerabilities and remediations, for applying remediations, and for testing those remediations

40.3.2 Creating a System Inventory.

The PVG should use existing inventories of the organization's IT resources to determine which hardware equipment, operating systems, and software applications are used within the organization and then to group and prioritize those resources. The PVG should also maintain a manual inventory of IT resources not captured in the existing inventories. Having a system inventory and priority listing will enable the PVG to determine which hardware and software applications they will support by monitoring for vulnerabilities, patches, and threats, and will enable them to respond quickly and effectively.

40.3.2.1 IT Inventory.

An inventory of all IT resources contained within the system should be created. This inventory should be updated regularly as part of the system's configuration management process. All IT resources within an organization must be assigned to a particular system such that the set of all systems covers all IT resources.

Creating and maintaining a separate inventory for each system may not be cost effective. Therefore, organizations may prefer to maintain an organization-wide inventory containing all IT resources. This is perfectly acceptable (and in many cases recommended) as long as each IT resource is labeled so that it is associated with one and only one system. The capability to output the list of IT resources associated with each system must exist. Organizations often have multiple inventories of IT resources. For example, some organizations use automated asset management software that inventories devices and the software each device runs. Organizations might also have inventories performed as part of business continuity planning and other efforts.

Each organization must determine the proper level of abstraction for its inventory. For example, one organization may track what software is installed on each computer, while another organization may also track software version numbers. Organizations should carefully and deliberately choose their level of abstraction because sometimes collecting too much information is just as bad (or worse) than collecting too little. Organizations should determine what uses they will make of their inventory, in addition to patch management, and collect only the information needed for those uses.

A sample list of items that an organization could include within its inventory follows (not all items will apply to all IT resources).

1. Associated system name
2. Property number
3. Owner of the IT resource (i.e., main user)
4. System administrator
5. Physical location
6. Connected network port(s)
7. Software configuration
   1. Operating system and version number
   2. Software packages and version numbers
   3. Network services
   4. Internet Protocol (IP) address, if it is static
8. Hardware configuration
   1. Central processing unit
   2. Memory
   3. Disk space
   4. Ethernet addresses (i.e., network cards)
   5. Wireless capability
   6. Input/output capability (e.g., Universal Serial Bus, Firewire)
   7. Firmware versions

It is usually impractical to require people to enter this information manually for each IT resource. Organizations that try this approach may end up with inventories that contain large sets of IT resources that are inaccurate and updated infrequently. A more effective approach is to use commercially available automated inventory management tools whenever possible. These tools typically require organizations to install an agent on each computer. The agent then actively monitors changes in the computer's configuration and reports to a central database, thereby providing the PVG and management an accurate picture of a system's IT resources. Unfortunately, as good as the automated tools are, some information will always need to be manually keyed (e.g., physical location). An automated tool should provide the option to gather this information periodically by presenting users with forms to fill out.

40.3.2.2 Grouping and Prioritizing Information Technology Resources.

The resources within the inventory should be grouped and assigned priority levels to facilitate remediation efforts. These prioritizations can be stored in the inventory itself. Resource grouping and prioritization is helpful in assessing risk to systems, and should be used to help identify which systems may require the special attention of the patch and vulnerability management program. The primary grouping should be by system name and by the level of impact that a compromise of each system could have on the organization. An example of impact categorization for systems is FIPS PUB 199, Standards for Security Categorization of Federal Information and Information Systems. It defines low-, moderate-, and high-impact categories that are based on the potential impact of a loss of confidentiality, integrity, or availability of a system. The categories should be used to prioritize multisystem vulnerability remediation efforts.⁴ For example, systems used as servers or for security management, and systems containing information of higher importance, typically have a higher impact category than standard users' systems. It may also be useful to group resources by network location. This is particularly important for those resources that are directly exposed to the Internet and those that reside within internal high-security areas.

If this grouping and prioritization is not performed, organizations may embark on unnecessarily costly remediation strategies. For example, when a new vulnerability is discovered within an organization that does not do remediation prioritization, system administrators might be instructed to patch all vulnerable computers immediately. This could result in a major disruption as system administrators stop all other work so they can patch computers. Even worse, the patch may be applied quickly without thorough testing, resulting in actual damage to the organization's systems. With prioritization, the organization may realize that a majority of the vulnerable computers could be patched over a period of time using the organization's standard configuration management process and patch-testing procedures. The organization could then focus its immediate patching efforts on the vulnerable computers that are most at risk, possibly those directly exposed to the Internet.

40.3.2.3 Use of the IT Inventory and Scope of Related Duties.

The inventory is the foundation on which the PVG will conduct its operations, since it is the PVG's window into understanding the organization's IT configuration. The inventory will be used primarily to create a list of PVG-supported hardware equipment, operating systems, and software applications. It will also help the PVG and administrators to respond quickly to threats, and provide system personnel information to help them secure their systems.

The PVG should define a set of hardware equipment, operating systems, and software applications that will be supported. The PVG will then be responsible for monitoring information regarding vulnerabilities, patches, and threats corresponding to the supported hardware, operating systems, and applications. The PVG should clearly communicate the supported resources to system administrators so that the administrators know which hardware, operating systems, and applications the PVG will be checking for new patches, vulnerabilities, and threats. The list of supported resources should be created by analyzing the inventory and identifying those resources that are used. Hardware equipment, operating systems, and software applications used on high-priority or sensitive systems, or on a large number of systems, should be included on the list. By publishing this list, the PVG will enable system administrators to know when, or if, they have an unsupported resource. System administrators should be taught how to monitor independently and how to remediate unsupported hardware equipment, operating systems, and software applications.

The PVG should also give system owners, system security officers, and system administrators access to the inventory information. Typically, these parties already have access to existing inventories, but the PVG inventory might contain additional inventory information that is otherwise unavailable. This will help them better to secure the organization's systems. However, system personnel should have access only to their own system inventory, since system inventory information is sensitive in nature. Giving system personnel access to their own inventory is also important because maintaining the inventory will require the PVG to work closely with system personnel.

40.3.3 Monitoring for Vulnerabilities, Remediations, and Threats.

The PVG is responsible for monitoring security sources for vulnerability announcements, patch and nonpatch remediations, and threats that correspond to the software within the organizational software inventory. A variety of sources should be monitored to ensure that the PVG is aware of all newly discovered vulnerabilities.

40.3.3.1 Types of Security Concerns.

The PVG is responsible for monitoring for vulnerabilities, remediations, and threats.

• Vulnerabilities. Vulnerabilities are software flaws or misconfigurations that cause a weakness in the security of a system. Vulnerabilities can be exploited by a malicious entity to violate policies—for example, to gain greater access or permission than is authorized.
• Remediations. There are three primary methods of remediation: installation of a software patch, adjustment of a configuration setting, and removal of affected software. Refer to Section 40.3.7 of this chapter for further details regarding methods of remediation.
• Threats. Threats are capabilities or methods of attack developed by malicious entities to exploit vulnerabilities and potentially to cause harm to a computer system or network. Threats usually take the form of exploit scripts, worms, viruses, rootkits, and Trojan horses.

System administrators should also monitor for vulnerabilities, remediations, and threats to systems under their control but running software not contained in the organizational inventory.

40.3.3.2 Monitoring Vulnerabilities, Remediations, and Threats.

There are several types of resources available for monitoring the status of vulnerabilities, remediations, and threats. Each type of resource has its own strengths and weaknesses. We recommend using more than one type of resource to ensure accurate and timely knowledge. The most common types of resources are:

• Vendor Web sites and mailing lists
• Third-party Web sites
• Third-party mailing lists and newsgroups
• Vulnerability scanners
• Vulnerability databases
• Enterprise patch management tools
• Other notification tools

Vendors are the authoritative source of information for patches related to their products. However, many vendors will not announce vulnerabilities in their products until patches are available; accordingly, monitoring third-party vulnerability resources, as well, is recommended. Enterprise patching tools usually provide lists of all patches available from supported vendors, which relieves the PVG from constantly having to monitor a large number of vendor security Web sites and mailing lists.

Organizations should monitor for vulnerabilities, remediation, and threats using these resource types at a minimum:

• Enterprise patch management tools, to obtain all available patches from supported vendors.
• Vendor security mailing lists and Web sites, to obtain all available patches from vendors not supported by the enterprise patch management tool.
• Vulnerability database or mailing lists, to obtain immediate information on all known vulnerabilities and suggested remediations (e.g., the National Vulnerability Database).
• Third-party vulnerability mailing lists that highlight the most critical vulnerabilities (e.g., the US-CERT Cyber Security Alerts). Such lists will help organizations focus on the most important vulnerabilities that may be overlooked among the myriad of published vulnerabilities.

After initial assessment of a new vulnerability, remediation, or threat, the PVG should continue to monitor it for updates and new information. For example, a software vendor might release a new patch in place of a software reconfiguration it originally recommended as a temporary remediation measure. By ongoing monitoring for new information, the PVG would be aware of the new patch and could determine if it would provide a better solution than the software reconfiguration. Ongoing monitoring is also important because additional analysis of vulnerabilities might determine that they are more or less severe than previously thought.

40.3.4 Prioritizing Vulnerability Remediation.

The PVG should consider each threat and its potential impact on the organization when setting priorities for vulnerability remediation. This evaluation would include these steps:

• Determine the significance of the threat or vulnerability. Establish which systems are vulnerable or exposed, with a focus on those systems that are essential for operation, as well as other high-priority systems. Evaluate the impact on the systems, the organization, and the network if the vulnerability is not removed and it is exploited. The organization's security architecture may automatically mitigate certain threats, thus reducing the urgency to apply relevant patches. For example, if the organization disables certain functionality within its browsers (e.g., scripting languages), then applying patches that fix vulnerabilities within those scripting languages is not a priority.
• Determine the existence, extent, and spread of related worms, viruses, or exploits. Ascertain whether malicious code has been published, and the level of distribution. Determine the damage caused, such as system access, information disclosure, arbitrary code execution, or denial of service. Organizations should assume that malicious individuals are in possession of exploit code for any vulnerability for which there is a patch, since patches are often reverse engineered quickly.
• Determine the risks involved with applying the patch or nonpatch remediation. Identify, through research and testing, whether the fix will affect the functionality of other software applications or services. Establish what degree of risk is acceptable.

The PVG is not expected to perform this evaluation on its own. System and network security officers and administrators might assist the PVG by assessing the impact of threats to individual systems, based on vulnerability, remediation, and threat information provided by the PVG.

The PVG should be aware of the resource constraints of local administrators and should attempt to avoid overwhelming them with a large number of patches or other remediations for identified vulnerabilities. With the exception of small IT deployments, it is a complex and difficult endeavor for local administrators to perform all remediations in a timely manner. This is attributed not only to time and resource constraints, but also to the greater complexity and heterogeneity of systems in larger environments. Thus, setting priorities for which systems, to patch in what order, is essential for an effective patch process.

40.3.5 Creating an Organization-Specific Remediation Database.

The PVG should create a database of remediations that need to be applied within the organization. Enterprise patch management tools usually supply such a database, but the PVG may need to manually maintain a separate one for IT technologies not supported by the patch management tool. Manually maintained databases should contain instructions on removing vulnerabilities by installing patches or performing workarounds as well as the actual patches when applicable.

Organizations might also find it helpful to have the PVG write a threat assessment summary for the most significant vulnerabilities and patches, then distribute the summary to local administrators and management, or make it available through the remediation database. The summary should be helpful in ensuring that people understand the importance of performing the remediation and the possible consequences of not doing so.

Whether automated or manual, databases should contain a copy of each patch for situations when the Internet may not be accessible or when the vendor's Web site may have been compromised. In addition, it is likely easier for local administrators to apply a patch using the PVG database as opposed to a vendor site that might overwhelm administrators with a large array of available patches. Although the creation of a database is recommended, resource constraints may limit an organization to listing only Web sites or specific Uniform Resource Locators (URLs) for each patch. Such a solution should be feasible when each hyperlink to a patch is associated with documented advice and time frames from the PVG. Manually maintaining databases may be possible, but purchasing automated patching products that inherently contain such databases is strongly recommended.

40.3.6 Testing Remediations.

If an organization uses standardized host configurations, the PVG will be able to test patches and nonpatch remediations on those configurations. This will avoid the need for redundant testing by each local system administrator. System administrators are responsible for testing patches and nonpatch remediations to mitigate vulnerabilities and threats identified for software not monitored by the PVG.

Precautions should be taken before applying the identified patch or nonpatch remediation. Remediation testing guidelines may include these points:

• Most vendors provide some type of authentication mechanism. The downloaded patch should be checked against any of the authenticity methods the vendor provides, including cryptographic checksums, Pretty Good Privacy (PGP) signatures, and digital certificates. Some of these methods, such as verifying digital signatures, are highly automated, requiring little user interaction. Others, such as SHA-1 or MD5 checksums, require the user to visit the vendor's Web site to compare the checksum listed there against the checksum for the downloaded patch. Although these methods add another level of authentication, they are not foolproof.
• A virus scan should also be run on all patches before installation. Before running the scan, the PVG or system administrator should ensure that the virus signature database in the antivirus program is up to date. Again, this system is not foolproof. For example, if an attacker has created an entirely new Trojan horse and included it with the patch, it might not be detected by the virus scan.
• Patches and configuration modifications should be tested on nonproduction systems since remediation can easily produce unintended consequences. Organizations should use existing change management procedures when possible for testing patches and configuration modifications. Also, using images of standard configurations on test systems, or within virtual machines on test systems, can expedite the testing process. Many patches are extremely complicated and can affect many portions of a system, since they often replace system files and alter security settings. Examples include enabling default user accounts that had been disabled, resetting the passwords for default user accounts, and enabling services and functions that had been disabled.
• Patches may also include fixes for multiple vulnerabilities or contain nonsecurity changes, such as new functionality. In addition, patches and configuration changes are often released in haste to repair a vulnerability quickly, and therefore they often receive less testing than the original software. Installing patches, modifying configurations, and uninstalling software may change the system behavior so that it causes other programs to crash or otherwise fail.
• Installing one patch might also inadvertently uninstall or disable another patch. If there is a dependency, there is the need to ensure that patches are installed in a certain sequence. Also, it is important to determine whether other patches are uninstalled when a particular patch is installed.
• Testing should be performed on a selection of systems that accurately represent the configuration of the systems in deployment. So many possible system configurations exist that the vendor cannot possibly test against all of them. Thus, the remediation may have unintended consequences only on one particular configuration. After the remediation is performed, the PVG should check that all related software is operating correctly.
• Before performing the remediation, and especially if there is a lack of time or resources to perform a test on the patch before employing it on a production system, the PVG may wish to learn what experiences others have had in installing or using the patch. For instance, others' experiences could indicate whether the patch or configuration adjustment corrects the vulnerability, opens an old vulnerability, creates a new vulnerability, degrades performance, or is incompatible with other required applications. It is important to remember that others' experiences might vary due to environment-specific factors, implementation differences, and other reasons.
• If one or more of the described problems apply, the PVG will need to consider whether the disadvantages of installing the patch outweigh the benefits. If the remediation is not critical, it may be better to wait until the vendor releases a newer patch that corrects the major issues. (This is a common occurrence.) Also, the ability to “undo” or uninstall a patch should be considered; however, even when this option is provided, the uninstall process does not always return the system to its previous state.

40.3.7 Deploying Vulnerability Remediations.

Organizations should deploy vulnerability remediations to all systems that have the vulnerability, even for systems that are not at immediate risk of exploitation. For example, if a system has a vulnerable service disabled, the service is not immediately exploitable, but it could be enabled inadvertently or intentionally at any time, which would cause the system to be vulnerable. Vulnerability remediations should also be incorporated into the organization's standard builds and configurations for hosts. Three primary methods of remediation can be applied to an affected system:

1. Applying a security patch (also called a “fix” or “hotfix”) repairs the vulnerability, since patches contain code that modifies the software application to address and eliminate the problem. Patches downloaded from vendor Web sites are typically the most up to date and are likely free of malicious code.
2. Configuration adjustment. Adjusting how an application or security control is configured can effectively block attack vectors (the paths by which an exploit can penetrate a computer) and reduce the threat of exploitation. Common configuration adjustments include disabling services and modifying privileges as well as changing firewall rules and modifying router access controls. Vulnerable software applications can be modified by adjusting file attributes or registry settings.
3. Software removal. Removing or uninstalling the affected software, or vulnerable service, eliminates the vulnerability and any associated threat. This is a practical solution when an application is not needed on a system. Determining how the system is used, removing unnecessary software and services, and running only what is essential for the system's purpose is a recommended security practice.

The mitigation of vulnerabilities and threats may be as simple as modifying a configuration setting or as involved as the installation of a completely new version of the software. No simple patch application methodology applies to all software and operating systems. Before performing the remediation, the administrator may want to conduct a full backup of the system to be patched. This will allow for a timely restoration of the system to a previous state if the patch has an unintended or unexpected impact on the host.

Applying patches to multiple systems is a constant administrative challenge that may seem especially daunting when implementing patches on hundreds or thousands of servers and desktop systems. This task can be made less burdensome with the use of applications that automatically distribute updates to end user computers. These enterprise patch management tools are included with network operating system software and distributed by third-party vendors. The capabilities of these tools vary greatly. Some focus on the distribution of patches, relying on the system administrator to identify a necessary patch and to arrange for the tool to deliver and install the patch. Other tools actively search for necessary patches and automatically notify the system administrator of the available ones; the administrator can then approve the tool's installation of the patches on the appropriate hosts. Enterprise management tools can vary greatly in their support of different operating systems and applications. Those that are bundled with an operating system tend to support the fewest operating systems and applications. Those from third-party vendors are generally compatible with the widest range of systems. Automated patch distribution tends to work best for organizations with a relatively homogenous environment and standardized configurations. Refer to Section 40.4.1 for further information on enterprise patching solutions.

Organizations need to apply patches manually for operating systems and applications that their enterprise patch management tools do not support. Also, many appliance-based devices cannot be updated by patch management tools, even if the appliances use operating systems and applications that the patch management tools support. This is because appliances often use customized limited-functionality versions of operating systems and applications, which are not intended for administrators to access directly. Because the appliances' customized operating systems and applications are based on the same code as the standard programs, they are typically susceptible to many of the same vulnerabilities. However, often the appliances cannot be patched as quickly as standard devices, because patches for appliances typically can be applied only through updates provided by the device's manufacturer. In many organizations, substantial effort is needed to apply patches manually for appliances, and for operating systems and applications, not supported by patch management tools.

Regardless of whether remediation involves automated patching or manual updates, system administrators may believe that the disadvantages of a suggested remediation outweigh its benefits. They may not wish to install the patches or perform the configuration modifications at all. The reasons behind these decisions should be documented and communicated back to the PVG and then to the appropriate management for approval.

The risk of delaying remediation must be weighed carefully. Consider these issues:

• Threat level. Does the organization or system requiring remediation face numerous or significant threats? For example, public Web servers often face high threat levels. In general, timely remediation is critical for these systems. In contrast, for an intranet site that is inaccessible from the Internet, remediation can often be delayed because such a site usually faces a lower threat level.
• Risk of compromise. What is the likelihood that a compromise will occur? If the vulnerability is easy to exploit, then remediation should be applied swiftly.
• Consequences of compromise. What are the consequences of compromise? If the system is critical or contains sensitive data, then the remediation should be performed immediately. This holds true even for noncritical systems if a successful exploitation would lead to an attacker gaining full control of the system.

Unfortunately, neither decision—to apply or not to apply a remediation—is risk free. The correct decision is not always clear. The PVG, system administrators, and management must work together to create a systematic process for evaluating risks and determining the appropriate decision within the context of their organization. It is desirable to integrate the remediation process with the existing configuration management procedures, in order to secure IT devices without causing unintended damage.

40.3.8 Distributing Vulnerability and Remediation Information to Administrators.

The primary way in which the PVG should distribute patches is through enterprise patch management software. However, it is sometimes necessary for the PVG to communicate remediations directly to local administrators. E-mail lists provide an effective method for distributing information regarding the priority of vulnerabilities, particulars about corresponding patches, configuration modifications, and other details. However, to decrease the chance of a spoofed e-mail containing a Trojan horse patch, actual patches should be distributed by the PVG to administrators from an internal secured Web site. (Ideally patches are distributed using automated patching tools.) Additional controls, such as using digital signatures, may be used to support the integrity of the patches and of the e-mail lists themselves. Several e-mail lists may be maintained for administrators who are responsible for various types of systems (e.g., UNIX administrators and Windows administrators). Alternative methods of patch and information distribution, such as on removable media, should be considered if the network or the secured Web site is unstable or unusable.

40.3.9 Verifying Remediation.

The PVG and system administrators should verify that they have remediated or mitigated vulnerabilities as intended. There are understandable benefits in confirming that the remediations have been conducted appropriately, possibly avoiding the experience of a security incident or unplanned downtime. This can be accomplished by several methods:

• Verify that the files or configuration settings the remediation was intended to correct have been changed, as stated in the vendor's documentation.
• Scan the host with a vulnerability scanner that is capable of detecting known vulnerabilities.
• Verify whether the recommended patches were installed properly by reviewing patch logs.
• Employ exploit procedures or code, and attempt to exploit the vulnerability (i.e., perform a penetration test).
• Organizations should consider having the PVG verify remediations on new servers before they are deployed to production, if the PVG has sufficient resources.

Only an experienced administrator or security officer should perform exploit tests, since they involve launching actual attacks within a network or on a host. Generally, this type of testing should be performed only on nonproduction equipment and only for certain vulnerabilities. The tests should be conducted only by qualified personnel who are thoroughly aware of the risk.

The next sections provide more details on using vulnerability scanners, reviewing patch logs, and checking patch levels when computers attempt to join an organization's network.

40.3.9.1 Performing Vulnerability Scanning.

Vulnerability scanners are commonly used in many organizations to identify vulnerabilities on their hosts and networks. A vulnerability scanner identifies not only hosts and open ports on those hosts, but also associated vulnerabilities. Running vulnerability scanners frequently can be helpful in identifying new hosts on a network as well as their vulnerabilities. A host's operating system and active applications are identified and then compared with a database of known vulnerabilities. Vulnerability scanners can be of two types: network and host.

Network scanners are used to map an organization's network and identify open ports, vulnerable software, and misconfigured services. They can be installed on a single system on the network and can quickly locate and test numerous hosts. Network scanners are generally ineffective at gathering accurate information on hosts using personal firewalls, unless the personal firewalls are configured to permit the network scanning activity.

Host scanners must be installed on each host to be tested. These scanners are used primarily to identify specific host operating system and application misconfigurations and vulnerabilities. Host scanners have high detection granularity and usually require not only local host access but also a root or administrative account. Some host scanners offer the capability of repairing misconfigurations.

Vulnerability scanners vary widely in capability and performance. Some perform optimized searching and can scan a host or network much faster than other systems. Some provide detailed reports and information about the remediation of each discovered vulnerability, while others provide only the most basic information about which vulnerabilities were found.

Vulnerability scanners employ large databases of vulnerabilities to identify those associated with commonly used operating systems and applications. The vulnerability database must be updated frequently, so that the scanners can identify the newest vulnerabilities. When a match is found, the scanner will alert the operator to a possible vulnerability. Most vulnerability scanners also generate reports to help system administrators fix the discovered vulnerabilities. Unfortunately, vulnerability scanners are not completely accurate; some vulnerabilities may be missed, and others that do not exist may be identified. Organizations should consider using multiple vulnerability scanning products so that false positives generated by one scanner could be validated by another.⁵

Vulnerability scanners provide these capabilities:

• Identify active hosts on networks.
• Identify active and vulnerable services on hosts.
• Identify vulnerabilities associated with operating systems and applications.
• Test compliance with host application usage and security policies.

Vulnerability scanners can help identify out-of-date software versions and applicable patches or system upgrades. Generally, host-based scanners are more effective at doing this than network-based scanners. Although scanners can be helpful at finding outdated software, scanners may identify deliberately deployed settings as vulnerabilities. The person assessing the vulnerability scanner reports needs to know how to interpret them and should compare them to the organization's business requirements. In addition, certain vulnerability scanners are able to make corrections and fix certain discovered vulnerabilities automatically.

40.3.9.2 Reviewing Patch Logs.

Log files keep track of the history of a system. Patch logs can assist the PVG and system administrators with tracking and verifying installed patches. Using patch logs to monitor an organization's systems can help to achieve consistency and compliance with the remediation plan. Patch logs can provide these capabilities:

• Identify which patches are installed on a system, allowing easy confirmation that the appropriate set of patches has been applied.
• Ensure that patches are applied in a consistent manner across the organization, through a comparison of log files.
• Verify that each patch has been installed properly.
• Determine whether the patch, or a subsequent update, improperly removed or damaged a previous patch.

40.3.9.3 Checking Patch Levels.

An organization might wish to verify the patch levels of hosts before allowing them to join its networks. This can be done through the use of separate virtual local area networks (VLANs) for unverified hosts. In most deployments, each host runs an agent that monitors various characteristics, such as OS and application patches and antivirus updates. When the host attempts to connect to the network, a network device such as a router requests information from the host's agent. If the host does not respond to the request, or the response indicates that the host is not fully patched, the network device causes the host to be placed onto a separate VLAN. This allows the organization to update the unpatched hosts while severely restricting what they can do. Once a host on the VLAN has been fully updated, it is moved automatically from the VLAN to the organization's regular network. The VLAN strategy can be particularly helpful for ensuring that mobile hosts are fully patched.

40.3.10 Vulnerability Remediation Training.

Although the PVG will monitor for new patches and vulnerabilities found within the software listed in the organizational software inventory, local administrators may use software not listed in the inventory. This situation may result from a management decision that the PVG only has resources to focus on the more popular software packages. In this situation, it is essential that local administrators have some knowledge of how to identify new patches and vulnerabilities. By providing them with such knowledge, a second line of defense is created in the patching process. Local administrators should be trained by the PVG on the various vulnerability and patching resources described in Section 40.4. Organizations may choose to train their administrators with only a few tools that are known to be comprehensive.

In addition, all end users who will be expected to implement recommended remediations on their own systems should be educated about the organization's vulnerability management process. These end users should also be provided with instructions on installing patches and performing other types of remedial actions. This education must be applied to the organization's remote workers.

40.4 PATCH AND VULNERABILITY MANAGEMENT ISSUES.

This section provides an overview of common issues in patch and vulnerability management. First, it explains the major types of patch management tools and discusses the advantages and disadvantages of each type. It also examines the security risks inherent in the use of the tools and provides guidance on effective deployment strategies for patch management tools. Next, this section explains how including patch management considerations in the purchase of IT products can reduce the need to patch the acquired products in the future. Finally, this section endorses the use of standardized configurations for IT resources and briefly provides recommendations for patching after a security compromise has occurred.

40.4.1 Enterprise Patching Solutions.

All moderate- to large-size organizations should be using enterprise patch management tools for the majority of their computers. Even small organizations should be migrating to some form of automated patching tool. Widespread manual patching of computers is becoming ineffective, as the number of patches that need to be installed grows and as attackers continue to develop exploit code more rapidly. Only uniquely configured computers, and other computers that cannot be updated effectively through automated means, such as many appliance-based devices, should continue to be patched manually.

40.4.1.1 Types of Patching Solutions.

There are two primary categories of enterprise patch management tools: those that use agents and those that do not. Some products support both approaches and allow the administrator to choose the approach that is most efficient for the environment. With both approaches, there usually is a central computer that holds all of the patches that should be, or could be, installed on computers participating in the patching solution. The central computer often contains a console that allows the patching administrator to control which computers get which patches. Some implementations use multiple central computers to provide redundancy and to divide the patching load across multiple devices and networks.

Both approaches utilize a centralized model with a single computer (or cluster of computers) controlling the patching process for all computers participating in the patching solution. This is in contrast to the standard Microsoft Windows Update service, which uses a completely decentralized model in which each computer (or the administrator of that computer) decides which patches to install and when to install them. Some products have features that combine the centralized and decentralized models. Such solutions usually follow the centralized model but give the end user some control over the process, such as the ability to choose not to install a patch.

Although the two primary categories of enterprise patch management tools have similarities, they also have important differences that should be considered when purchasing a particular solution.

40.4.1.1.1 Nonagent Patching Solutions.

Nonagent patching solutions are similar to network-based vulnerability scanners. There is usually a single computer that scans computers through the network. However, unlike many vulnerability scanners, the nonagent patching solution is usually given administrator access to the computers participating in the automated patching program. This gives the patching program access to much more information than is available through simple network scanning. It also gives the patching program the ability to install patches on participating computers. Given the similarity between nonagent patching solutions and vulnerability scanners, it is not surprising that some commercial nonagent patching solutions also detect vulnerabilities, and can do so with greater accuracy than a vulnerability scanning program that does not have administrator access to the computer.

Since nonagent patching solutions rely on network scanning, they may consume a large amount of network bandwidth. Most products resolve this problem by enabling the patch administrator to throttle the amount of network bandwidth that is used by the product. However, limiting the network bandwidth that can be used by the product may increase the total amount of time needed to complete the network scan. In large networks, it may not be possible to scan all computers as quickly as needed, and agent-based solutions may be preferable. Additionally, computers for employees that telecommute might not be included in the scan. Another problem with nonagent patching solutions is that personal firewalls on computers will typically block the scanning activity, unless they are specifically configured to permit it. Since the prevalence of personal firewalls is increasing, this is becoming a more significant problem.

40.4.1.1.2 Agent-Based Patching Solutions.

Agent-based patching solutions, as mentioned previously, usually use a centralized computer, or a cluster of computers, that manage the patching process for all participating computers. However, with this model a software program (agent) is installed on each participating computer. As discussed in Section 40.3.8, many appliance-based devices do not permit direct administrator access to the operating system, which typically prevents the installation of patching agents. Although each product works differently, the overall agent patching process generally works in this way:

1. The agent communicates with the central computer to learn about new patches. Depending on the implementation, the agent may poll the central computer periodically, or it may be contacted directly by the central computer, which is more efficient.
2. The agent has administrator or root access to the computer, and it uses that privilege to determine which patches are missing. This status is usually transmitted to the central computer, so that the overall patching administrator (e.g., the PVG representative) can view the status of all participating computers. This also enables the central administrator to produce patching reports regarding the patch security level for each system.
3. The agent receives instructions from the central computer on which patches to install and how to install them. In cases where a reboot is required, the central computer may instruct the agent to patch and automatically reboot the computer. Alternatively, the central computer may instruct the agent to patch and then notify the user that the computer needs rebooting (with the option of an automated reboot within a specified time frame).

The architecture of the agent-based solution eliminates the excessive network bandwidth usage that may occur with the nonagent-based solution. The primary drawback is that the agents must be installed on each computer and must run with administrator or root privileges. Second, computers already taxed by running with high processing or memory loads may suffer further performance degradation due to the agent process. Another possible drawback is that agents may not be available for all platforms, but platform support can also be an issue with nonagent approaches.

40.4.1.1.3 Advantages and Disadvantages.

Each approach has advantages and disadvantages that should be considered.

Nonagent Solution Advantages

• Does not need to install software agents on all participating computers.

Nonagent Solution Disadvantages

• Utilizes a significant amount of network bandwidth while scanning computers. (Both nonagent and agent solutions are likely to use approximately the same bandwidth for delivering patches to computers.)
• May require the use of ports and services that would otherwise be turned off as part of locking down the system (e.g., Remote Procedure Call [RPC] for UNIX, NetBIOS for Windows).
• May take a long time to scan large networks.
• May not produce accurate results for hosts that use personal firewalls.
• May require that the central computer be given administrator access to participating computers. (Managing the credentials that the central computer needs to log on to the individual hosts can be very challenging if there are many individual accounts, particularly if passwords are changed very frequently—e.g., monthly.)

Agent-Based Solution Advantages

• Can scan large networks quickly.
• Minimizes use of network bandwidth while scanning computers.

Agent-Based Solution Disadvantages

• Requires that software agents be installed, running, and managed on all participating computers. If an agent is not running due to failure or misconfiguration, the computer will not be patched.
• Must run agents with administrator or root privileges, which creates the possibility of remote attacks against such agents.

Deploying enterprise patch management tools within an enterprise can create additional security risks for an organization; however, organizations that do not effectively patch their systems face a much greater risk. Patching tools usually increase security far more than they decrease security, especially when the tools contain built-in security measures to protect against security risks and threats. Some risks with using these tools are listed next.

• A software vendor might distribute a patch to the enterprise patch management vendor that was corrupted with malicious code.
• The enterprise patch management vendor may provide a patch that has been maliciously altered by an employee or attacker.
• An attacker could break into the central patch computer and use the enterprise patch management tool as an efficient distribution tool for malicious code, potentially providing remote access to every participating computer.
• An attacker could break into the central patch computer on nonagent systems and steal the administrator passwords for all computers participating in the patch management program.
• An attacker could discover a locally exploitable vulnerability with the patch management agent software. This could enable the attacker to elevate access to a participating computer from user-level access to administrator access. This assumes that the attacker has already broken into the computer and gained access.
• An attacker could discover a remotely exploitable vulnerability with the patch management agent software. This could enable an attacker to remotely penetrate a participating computer and gain administrator access. It could also enable an attacker to launch a denial-of-service attack on the participating computer.
• An attacker could sniff enterprise patch management tool network communications to determine which patches have not been installed on particular computers.

These risks can be partially mitigated through the application of standard security techniques that should be used when deploying any enterprise-wide application. Examples of countermeasures include:

• Encrypting network connections
• Performing IP address authentication for network communications
• Disabling unneeded ports and services on the central patch management server
• Testing patches before deployment
• Performing timely application of patches
• Conducting timely mitigation of vulnerabilities for which there are no patches
• Using firewalls properly

40.4.1.2 Integrated Software Inventory Capabilities.

Enterprise patch management tools require administrator access to each participating computer and must inventory the software packages on each computer to determine which patches are needed. Therefore, it is natural for such programs to make this information available to the administrators and to incorporate a software inventory management capability. An increasing number of products provide this capability, and it appears that this is the natural way for the market to move. Such inventory products can be purchased separately but often require a separate agent to be installed on each computer. Since it is costly from an IT management point of view to install and manage multiple agents on each computer, it would be ideal if both functions (patching and inventorying) could be performed by the same product.

Some patch management systems can only recognize software, or versions of software, that have known vulnerabilities. This would preclude the use of such a patch management system as the sole source of software inventory information for an organization.

40.4.1.3 Integrated Vulnerability Scanning Capabilities.

Enterprise patch management tools are also beginning to incorporate vulnerability scanning functionality. This enables the administrator not just to see which patches are missing, but also to understand what vulnerabilities are associated with those patches and thus understand what real risks exist to the unpatched computers. This capability also allows administrators to see vulnerabilities within computers before the patches are even available. This is very important, given the speed at which attack tools are developed whenever a new vulnerability is announced.

Not only do some of these tools have the capability to scan for vulnerabilities, but they may also be able to scan for vulnerabilities with greater accuracy than network-based vulnerability scanners. Many network-based vulnerability scanners do not have administrator access to the computers that they scan, so they are forced to identify vulnerabilities by relying on imprecise guesses based on how different network ports respond to different inputs. Enterprise patch management tools do not have any such advantage over host-based vulnerability scanners. However, as with inventory management tools, it would be better to have patch management and vulnerability scanning capabilities integrated within one agent, instead of having to install and manage two separate agents on each computer. Readers should note the contrast between network-based vulnerability scanners that use banner grabbing to identify vulnerabilities versus software that actually has root access on a box. The industry has evolved, and most network-based scanners currently do have the ability to log in remotely as root to do scanning.

40.4.1.4 Deployment Strategies.

Although all moderate- to large-size organizations should be using enterprise patch management tools, deploying those tools universally within an organization can be difficult. Organizations should deploy enterprise patch management tools using a phased approach. This allows process and user communication issues to be addressed with a small group before deploying the patch application universally.

Most organizations deploy patch management tools first to standardized desktop systems and single-platform server farms of similarly configured servers. Once this has been accomplished, organizations should address the more difficult issue of integrating multiplatform environments, nonstandard desktop systems, legacy computers, and computers with unusual configurations. Manual methods may need to be used for operating systems and applications not supported by automated patching tools as well as for computers with unusual configurations; examples include embedded systems, industrial control systems, medical devices, and experimental systems. For such computers, there should be a written and implemented procedure for manual patching.

Nonstandard systems and legacy computers can hamper widespread deployment, but personnel issues can be an even greater challenge. System owners (and computer users) may have some initial qualms about giving administrator access to their computers to another group and having that group regularly install and update software. Their concerns include these issues:

• The agent software may decrease computer performance or stability.
• The patches being installed may cause unexpected problems with existing software.
• A user may lose data when the enterprise patching application reboots the computer to install a patch.
• The enterprise patching application itself may present a new security risk.
• A mobile user may become frustrated and confused when the enterprise patching application attempts to install a large set of patches as soon as the mobile user connects to the network.

These concerns should be discussed with system owners and computer users. All of them can be addressed by good communication, a careful phased rollout, and the selection of a robust and secure enterprise patch management tool.

40.4.2 Reducing the Need to Patch through Smart Purchasing.

Some software products have more vulnerabilities than other products with equivalent purpose and functionality. By considering several factors during the purchasing process, organizations may be able to reduce the number of future vulnerabilities experienced and thus reduce the need to patch the software. The future likelihood of vulnerabilities should not be the only factor in purchasing a product, but it should be an element in the decision-making process. A list of considerations in choosing products that are less likely to experience vulnerabilities in the future follows.

• Consider a product for which there is a detailed checklist specifying how to secure the product. NIST manages the Security Configuration Checklists Program for IT Products (http://checklists.nist.gov/), which collects reviewed checklists for a variety of operating systems and applications.
• Search a vulnerability database (such as the National Vulnerability Database at http://nvd.nist.gov/) for known vulnerabilities of products under consideration. Examine the type, severity, and quantity of vulnerabilities in the product under consideration. This is not foolproof because it often takes longer for vulnerabilities to be discovered (and patches released) for less popular software products.
• Consider a more mature product. Recently released products usually have more unknown vulnerabilities that will require future patches and possibly lead to increased exposure to risk.
• Consider less complicated products. More code, features, and services can mean more bugs, vulnerabilities, and patches. Consider not purchasing a product that has more features than needed. To the extent possible, delay implementing recently released major operating systems or applications until the experiences of others can be included in the decision-making process.
• Purchase products that conform to appropriate national or international security design standards (e.g., NIST FIPS PUB 140-2 for encryption modules).
• Consider software validated by independent testing. For the greatest assurance, the software's source code should be evaluated. (However, despite the benefit, software source code evaluations are generally not performed due to the cost of such analyses.)
• Use only versions of software that are currently supported. Obsolete software beyond its life cycle often has flaws that are addressed only in the newer, supported versions.
• Evaluate the speed with which the vendor responds to new vulnerabilities with a patch.

40.4.3 Using Standardized Configurations.

Using standardized configurations for IT resources reduces the labor involved in identifying, testing, and applying patches, and ensures a higher level of consistency, which leads to improved security. Organizations that use standard configurations for their IT resources will find it much easier and less costly to implement a patch and vulnerability management program. Comprehensive patch and vulnerability management is almost impossible (or at least very costly) within large organizations that do not deploy standard configurations.

A standard configuration should be defined for each major group of IT resources (e.g., routers, user workstations, file servers). Organizations should focus standardization efforts on types of IT resources that make up a significant portion of their entire IT inventory. Likely candidates for standardization include end user workstations, file servers, and network infrastructure components (e.g., routers and switches). The standard configuration description will likely include these items:

• Hardware type and model
• Operating system version and patch level
• Major installed applications, version, and patch level
• Security settings for the operating system and applications

In many cases, these standardized configurations can be maintained centrally, and changes can be propagated to all participating IT resources. An organization that relies on a hardware supplier to place a standard configuration on new computers should coordinate closely with that supplier to ensure that changes, including new patches, are implemented quickly. NIST SP 800-70, Security Configuration Checklists Program for IT Products—Guidance for Checklists Users and Developers, provides guidance on creating and using security configuration checklists, which are helpful tools for documenting standard security settings.⁶

40.4.4 Patching after a Security Compromise.

Patching after a security compromise is significantly more complicated than merely applying the appropriate patch. Although applying a patch after a security compromise will generally correct the vulnerability that was exploited, it will not eliminate rootkits, backdoors (secret avenues of access placed on a compromised computer system by an attack that allows future unauthorized access), or most other changes that might have been introduced by the intruder. For example, the Code Red II worm placed backdoors on compromised systems, and later the Nimda worm exploited those backdoors. In most cases a compromised system should be reformatted and reinstalled or restored from a known safe and trusted backup. If that is not possible, significant expertise will be required to manage the possible dangers inherent in compromised systems. Organizations may find it helpful to maintain fully patched images of their standard configurations. A current, known good image can replace a compromised system, and then data can safely be restored from backups. NIST SP 800-61, Computer Security Incident Handling Guide, is an extensive resource for handling security incidents and recovering compromised computers.⁷

40.5 CONCLUSION AND SUMMARY OF MAJOR RECOMMENDATIONS.

When designing a process for handling patches, consider the principles that make up the PVG patching concept. The core concepts should be found within the chosen patching methodology, although other patching variations may be acceptable. These concepts include using organizational inventories, patch and vulnerability monitoring, patch prioritization techniques, organizational patch databases, patch testing, patch distribution, patch application verification, patch training, automated patch deployment, and automatic updating of applications.

Except for the smallest organizations and selected areas of large organizations, the move to automated patching methods should be swift. Implementation of automated patch methods should parallel organizational plans to centralize services and to standardize desktop configurations. For this reason, computer security personnel must be actively involved in designing centralized services and standardized desktop models.

Although patching and vulnerability monitoring can often appear to be overwhelming tasks, consistent mitigation of organizational vulnerabilities can be achieved through a tested and integrated patching process. Having a mature patch and vulnerability management program will make the organization more proactive than reactive with regard to maintaining appropriate levels of security. Efficient patch automation, combined with effective preventive maintenance, should result in spending less time, resources, and money on incident response.

This chapter contains a variety of recommendations to assist organizations in implementing an effective patch and vulnerability management program. The primary recommendations are:

• 1. Create a patch and vulnerability group.
• 2. Continuously monitor for vulnerabilities, remediations, and threats.
• 3. Prioritize patch application and use phased deployments as appropriate.
• 4. Test patches prior to deployment.
• 5. Deploy enterprise-wide automated patching solutions.
• 6. Use automatically updating applications as appropriate.
• 7. Create an inventory of all information technology assets.
• 8. Use standardized configurations for IT resources as much as possible.
• 9. Verify that vulnerabilities have been remediated.
• 10. Consistently measure the effectiveness of the organization's patch and vulnerability management program, and apply corrective actions as necessary.
• 11. Train applicable staff on vulnerability monitoring and remediation techniques.
• 12. Periodically test the effectiveness of the organization's patch and vulnerability management program.

40.6 FURTHER READING

Nicastro, F. M. Curing the Patch Management Headache. Boca Raton, FL: Auerbach Publications, 2005.

CRC IIA Change and Patch Management Controls: Critical for Organizational Success. (Global Technology Audit Guide 2). Institute of Internal Auditors, 2005.

Jang, M. Linux^® Patch Management: Keeping Linux^® Systems Up To Date. Bruce Perens' Open Source Series. Prentice Hall PTR, 2006.

40.7 NOTES