The primary tool used to ensure countermeasures are implemented is the POAM. The POAM is created with the risk assessment, but it is a living document because managers update it regularly. As the risk assessment transforms into a risk mitigation plan, the POAM document expands.

The POAM includes all the approved countermeasures and their timelines. A simple countermeasure may have only one or two milestones, whereas a complex countermeasure could have multiple milestones. If the milestones are met as expected, chances are better of ensuring the schedule is met. In other words, focusing on the final date is not as important as focusing on the milestone dates.

If a single milestone is missed, the entire project may be delayed. As mentioned previously, the critical path chart is invaluable in determining which milestones must be met to ensure the project stays on schedule.

When project management software is used, a quick glance at the display will often show whether the implementation of a countermeasure is on schedule. Project management software uses color-coded status symbols. For example, a green circle could indicate the project is on schedule; a yellow circle, the project is slightly delayed; and a red circle, a severe delay. Many project managers use lingo in reports indicating that a project is “in the green” or “in the red.” “In the green” indicates it is on schedule, and “in the red” indicates it is severely delayed, which often alerts senior management.

Countermeasures must be checked to be sure they are working as expected. Because the purpose of a countermeasure is to mitigate a risk, it should either reduce the impact of a threat or reduce a vulnerability.

The same is true of any product; it should work as expected. Watching an infomercial may convince people that an advertised product will help them lose weight, make them rich, or improve their health. However, when they receive it and it doesn’t perform up to their expectations, they realize they wasted their money.

Similarly, not all countermeasures perform as expected. The only way their performance will be discovered is to test and evaluate them, and some countermeasures are easier to evaluate than others. For example, a vulnerability scanner has detected a vulnerability, and the risk assessment recommends a countermeasure to eliminate the vulnerability. After the countermeasure has been implemented, the same vulnerability scan is run. If the scan doesn’t detect the risk, then clearly the countermeasure has closed the security gap. However, if the scan still detects the risk, then clearly the security gap remains open, and additional steps will need to be taken. The risk assessment doesn’t necessarily have to be redone from the beginning, but the gap should be addressed.

In the web farm and failover cluster example, the goal was to eliminate outages and increase availability. An outage will show that the solution isn’t working. However, an outage will cost money in lost revenue. Instead of waiting for an outage, testing can be performed to measure the countermeasure’s performance. Some tests and measurements that can be used are:

• Measuring the load on the web farm—During normal operation, the load should be balanced among servers and can be measured using load-balancing software. The resources on each individual server can also be measured. If the servers have similar hardware, the processor, memory, network card, and disk usage of each server should be about equal.
• Removing a server from the web farm—Removing a server simulates the failure of a server in the web farm and should not affect the entire farm. In other words, if a server is removed from the web farm, the other servers should pick up the load. Additionally, new clients should not be referred to the nonexistent server.
• Adding a server to the web farm—If the website experiences more growth, it should be able to handle another web server being added to the web farm, which allows the web farm to scale out. The network load balancing software should then balance the load with the new server. Therefore, additional clients can be added to the web service without changing the actual service.
• Transferring nodes on the failover cluster logically—Failover clusters include software that can logically swap nodes. For example, it can be used to logically switch node 2 from inactive to active and node 1 from active to inactive. If this switch fails during testing, it doesn’t cause an outage, but it does indicate the cluster will not prevent an outage during an actual failure.
• Shutting down the active node on the failover cluster—If logical transfers work, an actual failure of the active node can be simulated. Failover clusters usually identify a specific procedure used to simulate a failure. The inactive node should sense the failure and take over as the active node. If the test fails, the system can quickly be switched back with minimal impact on the operations. This type of failover testing can affect the service provided by the failover cluster. Therefore, testing should be done during a slow period and with much forethought and planning. One of the primary considerations is the ability to return the system to normal if it fails.

The goal of any testing and evaluation is to ensure that the countermeasure has acceptably closed the security gap. But, if the security gap hasn’t been closed, managers need to be informed of the remaining, or residual, risk, and they may decide that the gap has been closed satisfactorily. Managers may also decide that an additional countermeasure needs to be identified to further mitigate the risk.