Security troubleshooting aims at recovering from problems related to the countermeasures. Problems will occur with the defense mechanisms. Downtime of a security control is as critical as downtime of a core business process or asset. Troubleshooting failures of security controls is an important part of network security management. As with most concerns, prevention is always preferable to repair or response. By working to ensure that failures do not occur, or at least do not occur as often, maintaining effective security will be easier and less costly in terms of both budget and manpower.

Network security troubleshooting is often about triage—deciding which issues or problems are of the most imminent concern. The more a security component affects a mission-critical process, the more important rapid response and repair become. When security is down, the previously protected assets are put at greater risk for compromise. Minimizing the length of time consumed by the response is important to minimizing long-term losses.

One of the most effective preventative techniques in network security troubleshooting is installing patches and updates. As with patches and updates to production systems, always test the new code thoroughly before deployment. Once tested and approved for application, apply updates when downtime would cause the least number of problems. Always be prepared with a redundant option if the updating process causes further security control problems.

Possible complications from the application of patches and updates include resetting to factory defaults, loss of some—but not all—configuration settings, and bricking (i.e, making the device nonfunctional) the control. If the security control resets back to factory defaults, then it will need to be fully reconfigured. If a recent configuration backup is available, restoration might be a swift repair. Otherwise, manual resetting will be necessary. To facilitate this process, always have complete documentation of all settings of all security controls. Screenshots and images often make the process faster, as the desired state in the image can be compared with the information on the monitor in the same font with the same appearance for quick comparison.

If you lose some but not all configuration settings, then the security control is unlikely to operate as you expect. Restore or reconfigure the settings promptly. The update may have added or modified settings that need testing and verification for function and compatibility. If a new feature interferes with a business task, you might find it necessary to disable the feature or roll back the update.

If the update process causes a complete failure of the security control, the real possibility exists that the product is useless. This is known as bricking—turning a useful device into a worthless brick. In some cases, a hard reset can revive a seemingly bricked device, while there may be more esoteric repair and recovery options for other devices. If the vendor does not provide an unbricking solution, search the Internet for user groups or discussion forums for a home-grown solution.

Configuration errors might be the cause of a security control malfunction. Configuration errors may be caused by human error, oversight, or ignorance, as well as by updates, power fluctuations, and physical damage. When a security control is improperly configured, it does not provide the expected security. Troubleshooting this situation can be as simple as reverting to a previously saved configuration or manually reapplying the settings.

A more serious concern arises when configuration errors are not easily fixed. A recent patch or update may have rendered the product unstable, or a patch or update might be needed to stabilize it. If the configuration error reappears after every power cycle, then the device might be defective, it might need additional memory (or need defective memory replaced), or it might need to be attached to an uninterruptible power supply (UPS) to reduce the number of unplanned power fluctuations. Physical damage should be repaired expeditiously. If unrepairable, you may need to replace the device. Install preventative measures that will prevent the reoccurrence of the physical damage.

Power faults can take place for many reasons. If the building’s power grid is not reliable, if sags and spikes occur on a regular basis, the focus of the long-term repair should be on improving the building’s power distribution systems. Short-term responses can include surge protectors, UPSs, and generators.

If a device encounters a power fluctuation due to overheating, investigate whether the room where the device is operating has adequate HVAC service. The average temperature for a room dedicated to housing computer equipment should be at or below 70 degrees Fahrenheit with moderate relative humidity to avoid generating static electricity. In rooms where people work with computers, the temperature should remain below 80 degrees Fahrenheit. Also, check the device to see if it has sufficient internal airflow to maintain appropriate internal operating temperatures. Do not forget to check filters and vents for accumulation of dirt or dust.

Power faults can include power supply or power grid variances. In this situation, the only responses are surge protectors, UPSs, and generators, as customers are unable to affect the quality of the power company’s electricity by themselves. If switching power suppliers is an option, investigate this to determine the reliability of other providers, as well as the expense and hassle of switching. Sometimes actively and publicly “shopping” power suppliers can result in improved service from your present supplier.

In some situations, the problems with security are not with the security components, but with the overall infrastructure design. Perform a reassessment of the design on a periodic basis, such as once a year, to judge whether the current infrastructure continues to meet the security needs of your organization. Because security changes over time, the security design might need to evolve to meet the demands of new risks, threats, and concerns.

Static electricity is also a concern. Electrostatic discharge (ESD) or static-electric discharge (SED) can easily damage equipment, including security equipment. The amount of electricity discharged between someone’s finger and a doorknob when you can see the spark jump is more than enough voltage to destroy most computer chips. Take precautions to prevent static damage.

Physical damage can be a concern requiring troubleshooting. Damage might be caused through intentional destruction, accidents, or Mother Nature. If the damage is superficial or cosmetic, the device can be returned to service. However, if the damage prevents the security device from operating properly, then in most cases you need to replace it. Address the cause of the physical damage to reduce the likelihood of a reoccurrence.

Network security troubleshooting focuses on whether an intruder can bypass a security defense or restriction. A security control that can be bypassed is worthless. Once you discover that a security control can be bypassed, you need to investigate the method or mechanism of the bypass. Then, if possible, remove or block the method of bypass. If the flaw is a design or infrastructure concern, consider revising the design to remove the loophole.

From time to time, hackers find flaws in the programming of a security control. Once an exploit is written, the security control can be rendered useless through taking advantage of the flawed code. Vigilance in reviewing vulnerability databases and research, paying attention to vendor information, and watching the log files of the organization’s network should alert you when an exploit succeeds.

Defending against an exploitation focusing on a security control is the same as when one focuses on any other aspect of a network. If possible, reconfigure the control to minimize the effectiveness of the exploitation. Consider removing the component until you can implement a new defense. Apply a patch or update from the vendor as soon as it becomes available.

A final concern in relation to network security troubleshooting is hardware failure. Although it is not a common occurrence, hardware can fail. Over the life of an organization, you are almost guaranteed downtime caused by hardware failure. When that hardware is a security control, the downtime is more severe in that it places the rest of the environment at greater risk.

Manage hardware failure by having replacements on hand, or be able to obtain replacements quickly. Reduce downtime through redundant infrastructure design. Monitoring ongoing performance metrics might enable the detection of a future hardware failure as performance degrades. However, abrupt hardware failures are difficult to predict. Thus, being prepared with alternatives and replacement parts is often the best troubleshooting solution.