Industry Best Practices
Security is no different from any other industry. Steps and techniques are expected as a baseline best practice. This section looks at some of them.
Use a Change Control Process
As mentioned earlier in this chapter, a good change control procedure has an identified owner, a path for customer input, an audit trail for any changes, a clear announcement and review period, testing procedures, and a well-understood back-out plan. Change control manages the process from start to finish. We should also mention that changes are typically applied only during nonwork hours. If your current procedure lacks any of these, reconsider carefully before using it for deployment of updates.
Read All Related Materials
Before applying any service pack, hotfix, or security patch, it is imperative you read all relevant documentation and have it peer reviewed. The peer review process is critical because it mitigates the risk of a single person missing critical and relevant points when evaluating the update. There is no worse feeling than pushing out an update and watching as your mail server reboots for the last time because you didn’t do due diligence.
Reading all associated documentation is the first step in assessing whether
1. The update is relevant and will resolve an existing issue.
2. Its adoption won’t cause other issues resulting in a compromise of the production system.
3. There are dependencies relating to the update. (That is, certain features being enabled or disabled for the update to be effective.)
Potential issues will arise from the sequencing of the update because specific instructions might state or recommend a sequence of events or updates to occur before the service pack, hotfix, or security patch is applied.
Documentation released with the updates is usually in the form of web pages, attached Word documents, and README.TXT files. These should be printed and attached to change control procedures as supporting documentation.
You can find specific write-ups on the Microsoft KB Articles; for instance, KB Article 892843 (http://support.microsoft.com/kb/892843) for a security update; Microsoft Outlook has a detailed description in the write-up.
Apply Updates as Needed
Apply updates only on an as-needed basis. One of the common misconceptions about any updates, be they from Microsoft, Apple, or Adobe, is that they are mandatory and urgent.
All updates, regardless of their type (whether they are service packs, hotfixes, or security patches), are to be applied on an as-needed basis. They need to be individually evaluated and treated as important optional updates.
Especially with security patches, the expectation is that it must be an urgent issue and must be quickly deployed. Without trying to detract from the urgency, security patches are very much a relative update; for example, customers using solely Windows XP (SP3) can ignore a patch for a security vulnerability in Windows 2007. However, if the issue is relevant and does plug a security hole, it should be urgently evaluated.
Only when it addresses or fixes an issue being experienced by the customer should it be considered. Of course, it still needs to be evaluated before being installed. Don’t read this section and think you can just use a Windows NT 4 machine on every desktop and that mitigates the risk and therefore you do not need to worry about security updates. No! There is a reason Microsoft, Linux, HP-UX, Apple, and so on evolved throughout the years. To be the most secure you can be, you need to update to the most current version of the most current operating system of your choosing. Stagnating at a lower version doesn’t help your security posture...it negates your security posture.
Testing
The prior points assist in giving you a feel (before installing) for the potential impact; however, testing allows for the “test driving” and eventual signing off of the update.
Service packs and hotfixes must be tested on a representative nonproduction environment prior to being deployed to production. This will help to gauge the impact of such changes.
Uninstall
Where possible, service packs, hotfixes, and security patches must be installed such that they can be uninstalled, if required.
Historically, service packs have enabled uninstalling, so verify there is enough free hard disk space to create the uninstall folder.
Consistency
Service packs, hotfixes, and security patch levels must be consistent on all servers and workstations throughout your computing environment. Inconsistent update levels across your company can lead to synchronization and replication-related problems. Anyone who has maintained a Microsoft domain knows it is extremely difficult to trap errors caused by domain controllers (DC) being out of sync, so it’s critical that you maintain consistency.
Backup and Scheduled Downtime
Server outages should be scheduled and a complete set of backup tapes and emergency repair disks should be available, in case a restoration is required.
Make sure that you have a working backup of your system. The only supported method of restoring your server to a previous working installation is from a backup. For more information on backup and recovery procedures, refer to your operating system forums for best practices for backing up.
Have a Back-Out Plan
A back-out plan enables the system and enterprise to return to their original state prior to the failed implementation. These procedures must be clear, and contingency management must test them because in the worst case a faulty implementation can make it necessary to activate contingency options.
Enterprises might need to exercise their back-out plan if the update does not have an uninstall process or the uninstall process fails. The back-out plan can be as simple as restoring from tape, or may involve many lengthy manual procedures.
Forewarn Helpdesk and Key User Groups
You need to notify helpdesk staff and support agencies of the pending changes so that they are ready for arising issues or outages.
To minimize the user impact, it is also a good idea to prepare key user groups of proposed updates; this can assist in managing user expectations.
Don’t Get More Than Two Service Packs Behind
Schedule periodic service pack upgrades as part of your operations maintenance, and try never to be more than two service packs behind. As mentioned before, service packs are composed of large security updates and hotfixes bundled together. If you do not keep up with the service packs as they are issued, you leave your computing environment vulnerable.
Target Noncritical Servers/Users First
If all tests in the lab environment are successful, start deploying on noncritical servers first, if possible, and then move to the primary servers after the service pack has been in production for 10–14 days. There is nothing worse than upgrading the CEO’s system first only to find out the upgrade crashed his system, and he not only lost the time it takes for you to have him back up and running, but also the potential loss of critical business plans and his recipe for white chicken chili.
Service Pack Best Practices
Great Microsoft TechNet articles reference service pack best practices. All you need to know can be found in the following documents:
• Steps to Take Before Installing Windows XP - SP3 (http://support.microsoft.com/kb/950717).
• How to Install the Latest Service Pack or Update Rollup for Exchange 2010 (http://www.microsoft.com/downloads/en/details.aspx?FamilyID=50b32685-4356-49cc-8b37-d9c9d4ea3f5b&displaylang=en).
• Learn How to Install Windows 7 SP1 (http://windows.microsoft.com/en-US/windows7/learn-how-to-install-windows-7-service-pack-1-sp1).
Apple products don’t have service packs per se; however, they do have annual updates that typically include new functionality and features to the system.
Hotfix Best Practices
The following sections outline some best practices for hotfixes.
Service Pack Level Consistency
Now take a moment to reflect on one of the major bullets listed under the general guidelines: consistency. There is a reason you should have the same service pack deployed to all machines running the same operating system. Don’t deploy a hotfix until you have all current service packs installed. A hotfix is related to a service pack and should be deployed with this in mind. If a hotfix is for post–Windows 2000 SP2, for example, you need to ensure that the server has SP2 installed. And if you are constant throughout your computing environment, you won’t need to do extra work in bringing a system up to level to patch it.
Latest Service Pack Versus Multiple Hotfixes
This last one is common sense. I don’t know about you, but my time at the office has me spread thin. If I don’t need to spend three hours uploading 200 hotfixes to a system, I won’t. My time can be better used elsewhere. If multiple hotfixes need to be applied to a system, and these are in the latest released service pack, apply the latest service pack instead of applying several hofixes unless issues relating to the latest service pack might cause the server to break.
Security Update Best Practices
The following sections outline some best practices for security updates.
Apply Admin Patches to Install Build Areas
It is crucial that not only systems deployed to the desk are retrospectively updated with security patches, but also the client built areas are updated for any new clients. For example, I do not have a “client build” area in my organization—unless you count my office—however, I have a procedure established that I have a fast patch disk and a fast secure disk. The fast patch has all the latest updates on it post SP3 (for my Windows XP builds). So after I install the base operating system and Service Pack 3 (SP3), I run the latest fast patch disk. When this is done, I run the fast secure disk on the system. This then enables me to safely put the newly built system on the network or domain to go out to Microsoft’s update services and get the updates listed from the end of the fast patch disk to present.
Apply Only on Exact Match
Apply fixes only if you encounter exactly the issue the fix solves or if the circumstances relate to your environment.
Subscribe to Email Notification
Subscribe to the notification alias to receive proactive emails on the latest security patches, or use newsgroups or other forums to help you stay apprised of the ever-changing risks. Security is not passive. You must be aggressive and continually read, upgrade, and plan for the worst.
Following are links to various online resources to assist you in maintaining a Microsoft, Novell, and Cisco environment:
• Microsoft: http://www.update.microsoft.com/microsoftupdate/v6/default.aspx?ln=en-us
• Novell: http://support.novell.com/patches.html