Once the outcome of a penetration test has been reported and delivered to the client, and the risks fully presented and discussed, there follows a ‘remediation phase’. This is the most crucial element of the penetration test lifecycle where identified vulnerabilities are addressed in order to reduce or negate the impact they have on the business.
One of the primary reasons this phase is often not given enough attention is the perception or mindset of organisations that consider a penetration test programme as a box-ticking exercise; where the initial delivered report can be used as supporting evidence for various purposes such as compliance, a tender or an audit process, without understanding the adverse impact certain vulnerabilities will have on the business if not fully perceived and acted on appropriately.
Furthermore, an indication of a vulnerability on a certain asset may originate from an erroneous chain of operational procedures that are built on insecure or weak foundations and which – if not identified through a proper risk assessment – will introduce the same risk again when remediation is attempted. As an example, if a server contains a number of weaknesses that have been remediated, but resurfaced in other systems, a root-cause analysis should also be undertaken for other operational functions such as technical training, with a revision of the build configuration standard for such server deployments. This is required because these weaknesses will be inherited in future server enrolments as a result of staff unawareness or from an ineffective, standard automated process.
It is always a challenge to correctly assess broadly the interdependent components and processes affected by a vulnerability and its associated risk. It is important in the remediation phase for an organisation to create both a long-term and short-term remediation road map that will include:
• correct development of fixes;
• enforcement and deployment of fixes;
• testing of the applied fixes;
• required expertise and resources;
• associated costs;
• timelines and milestones.
To effectively address vulnerabilities, a successful vulnerability remediation programme must be able to plan the required steps effectively and efficiently, including assigning responsibilities to personnel, allocating the right resources and managing associated costs.
When interpreting penetration testing results, a cool, collective approach should be taken that emphasises the prioritisation of remediating critical and high-risk vulnerabilities. It is advised that low-risk vulnerabilities are addressed at a later stage, especially when time is limited. This can be challenging due to the way that low-risk vulnerabilities are perceived as simpler or quicker to fix.
In situations where organisations have sufficient IT personnel, remediation strategies can be altered. In such events, based on the complexity of a vulnerability, suitable skilled personnel can be allocated consecutively to multiple vulnerabilities. It is important to note that even when a vulnerability has been rated as low risk, it should not be considered negligible from a security perspective.
A low-risk vulnerability is generally only rated as such because the likelihood and impact of exploitation is minimal at the time of assessment. This of course varies depending on newly available exploits or if it can be used in combination with other vulnerabilities that have the potential to lead to an organisational compromise. This combination of vulnerabilities is generally referred to as a ‘chained attack path’.
By their very nature, vulnerabilities can be coupled, and the complexity of such associations can rapidly become incomprehensible. Penetration tests generally include attack paths; graphical representations that illustrate how one vulnerability can be chained to interconnect with multiple other vulnerabilities. The introduction of such visual aids for vulnerability identification have dramatically reduced the strain of keeping track of associated vulnerabilities.
Prioritising results
Not all findings are considered equal and some often pose a much larger risk to an organisation. The exploitation of high-risk vulnerabilities can often lead to the compromise of a company; as such the following factors should be considered in prioritising the remediation activities in order to formulate an effective remediation plan.
1. Importance of the affected asset
It is vitally important to understand the value, liabilities, costs and sensitivity of the affected asset to the business. This can be assessed by understanding its involvement and criticality in the business’s operations and its interdependencies – including the data itself. Given an example of a domain controller that can issue and hold user accounts – one organisation might assess only the server itself where people can authenticate in the network to do their job functions; while another might assess what information the authentication controls themselves serve to provide.
2. Impact on the business if the vulnerability is exploited
Every asset carries a different weight in regard to its involvement in the business operations, the data it holds, its interdependencies and the impact it will have if exploited. Such exploitation could lead to various different impacts as listed below, but not limited to:
• productivity loss;
• financial loss from productivity;
• fines due to legal and regulatory constraints;
• reputational damage;
• liability and competitive advantage.
3. How easy it is to exploit the vulnerability
Given the fact that a vulnerability has been identified, a notable factor to be considered is how pragmatic it is that it can be exploited and under what conditions (i.e. if there are available exploits, easy prerequisites or conditions to allow an attacker to exploit the vulnerability, time required for the attacker to successfully exploit the vulnerability, and so on).
4. What the threat landscape is, by considering the attack vectors
There could be multiple threats to an asset wherever a weakness has been identified. It is important to have a good understanding of these different threats, together usually called a ‘threat landscape’. Each threat within the landscape can impact either the confidentiality, integrity or the availability of an asset. It’s important to have an understanding of the threat landscape and the various paths that could lead to a compromise/breach.
5. The effort, resources and cost needed to fix the issue
Fixing a certain vulnerability requires proper planning, testing and implementation of a fix. This can be a very resource-intensive exercise depending on the scope of the assessment, the size of the organisation and IT personnel (in-house or outsourced) expertise. Often, when businesses outsource their IT infrastructure operations to a third party, the time required for them to fix vulnerabilities could be longer than expected (and depending on the management or contract, may come with additional costs).
Vulnerability risk assessment
A risk assessment (RA) is an important tool in the assessment of an organisation’s asset exposure. It aims to identify the risk that is facing the organisation and takes into account the identified vulnerabilities, including the current threats. There are many quantitative and qualitative (or a combination of these two) risk methodologies and frameworks that can be used to provide a simplified understanding of the overall risk.
Qualitative analysis is an approach in which the risk is calculated based on the probability of that risk being triggered with its associated impact. These types of assessment rely on people’s context or expertise with the knowledge of the asset or process being assessed. Such risks often have relative scales or values (high, medium or low or 1–10) to display the level of severity, as this has no monetary or mathematical dependency. Using only this type of analysis may lead to biased results regarding the probability and associated impact rating, which can lead to non-pragmatic prioritisation of effort taken to address a risk.
Quantitative analysis
Quantitative analysis on the other hand deals with measurable data that are useful to express probability and impact values including the monetary loss for each threat, weakness and impact. As this type of assessment relies on mathematical formula calculations, it can be used for the benefit of controlling issues.
The framework and approach taken in risk assessment must be able to identify and describe the assets, threats and controls – including the impact and potential loss elements of these.
Vulnerabilities reported by the penetration testing provider will contain a risk rating and a measure of how easily they can be exploited, as the findings need to reflect pragmatic realities. These risk ratings need to be reviewed based on the impact they can have on the business and the types of processes that your business uses. Carrying out a RA helps in making an informed decision about the remediation steps, such as accepting or rejecting a risk and prioritising the actions (including any associated costs and resources required).
As it is practically impossible to fix everything at once, the risk assessment provides a metric of the types of risk that the organisation is facing, based on the outcome of the penetration test. The purpose of a RA is to enable organisations to view the different potential impacts of risks with regard to their business objectives, operations and reputation, which will result in more effective and appropriate resolutions. Based on these factors, an organisation can prioritise remediation activities.
ESTABLISHING A STRUCTURED REMEDIATION PLAN
A structured remediation plan outlines the follow-up strategy required to remediate issues identified from the penetration testing process.
Remediation activities
Remediation activities are a comprehensive collection of solution and test processes that outline the way in which an issue will be fixed. The ultimate goal of the remediation phase is to ensure that the risks posed by vulnerabilities are either completely removed by fixes or reduced to acceptable levels. There are different types of fixes that can be applied to remediate a vulnerability. They include the following:
1. Applying a certain patch and hotfix
Organisations or communities behind operating systems and applications provide regular patches and hotfixes when a security vulnerability is discovered in their products. Most of the time, depending on the product, an implementation guideline is issued together with the patch or ‘hotfix’, which makes applying a patch or a hotfix a straightforward process.
The difference between a patch and a hotfix is that the patch is released in order to fix a known issue or weakness and the hotfix to immediately fix a very specific issue.
2. Upgrading an application or host
Sometimes a small hotfix or a patch might not be enough to fix a security flaw. Hence, a firmware upgrade or a version upgrade may be required to completely negate a certain vulnerability. Most leading IT vendors periodically upgrade their products to enhance various features, and to enhance the security. Major firmware or version upgrades come with changes (i.e. deprecation of certain features) that a company needs to be aware of (i.e. upgrading PHP5 to PHP7).
3. Reducing the attack surface
Services and protocols may have inherent weaknesses, especially in the way they are designed (e.g. FTP and Telnet), and these should not be used anymore due to their inherent risks. If there is no business need to keep the services up and running, they must be replaced with services with similar functionality, or a compensating control should be added to remove or reduce the impact of exploiting such a protocol.
When remediating vulnerabilities, security teams should attempt to identify and remove important entry and exit nodes that are interlinked with multiple other branches of the attack path. Once this link has been identified, administrative teams can start to act upon remediating vulnerabilities. This is referred to as ‘breaking the attack chain’, and when done correctly can lead to an organisation having a much smaller attack surface.
4. Fixing weaknesses in the code
Any in-house developed products and applications (e.g. company website or login portal) that usually serve only the organisation that develops and maintains them could be found insecure during a penetration test. In such scenarios, source code has to be revisited to address and fix the identified vulnerabilities.
5. Developing a new solution or redesigning the current solution
This is necessary where a flaw is identified in the core of an asset and which is inherited by other various process; for example, if a web application has been found vulnerable to multiple issues such as SQL injection, cross-site scripting attacks, missing proper access control functionality and so on. To fix issues like these, the entire application may require a rework or a redesign. Such a fix or remediation is usually the most resource-consuming option, where there is a need either to build a new solution or to redesign the current one.
6. Implementing a compensating control
Sometimes, a fix might not always be a solution. There could be many reasons for this, such as impracticality, costs, resource constraints or no timely available patch from the vendor, which makes it very challenging. In situations like these, efforts should be made to delay, detect or completely deter an attack that could exploit a vulnerability. A compensating control, also known as an alternative control, can be set in place to satisfy a control that may be deemed impractical or difficult to set in place. It must have a similar risk reduction to the actual fix to lower the risk to an acceptable level.
7. Completely removing the affected asset
Based on the identified vulnerability, a business decision can be made to remove the affected asset or service completely. As an example, this could be a server that is not in production and hence has been left unmaintained with security vulnerabilities but still resides on the company’s network.
8. Sometimes, just accepting the risk
A risk assessment may indicate that the overall risk for a certain asset and its associated dependencies is low. Taking into consideration the possible solutions in this list, the organisation may decide not to act on a vulnerability, because the resources, time and costs may be higher than the value of the asset itself. However, it is strongly recommended whenever a certain risk is accepted that it is well justified, documented and revisited.
Applying changes or fixes within an IT environment is certainly not as easy as filling a small hole or applying a bandage to a wound. It is sometimes highly complicated: more akin to complex surgery than a bandage. Hence, the fixes need to be tested first in isolated mirrored environments (where possible) or on staging environments before being applied on the live production systems in an effective change control process. Fixes that have not been tested can result in severe outages, causing financial and reputational losses to the business.
Remediation strategy
A mature remediation strategy could ensure that similar vulnerabilities do not reappear in the future, which would save money on future remediation activities. A comprehensive strategy would lead to more secure baselines for solutions, which would allow further vulnerability risk assessments to focus on identifying more complex issues.
When defining a remediation strategy to fix a certain vulnerability, the following factors should be considered:
• Time, resources and costs required to plan the remediation. There will not be a single fix available for all the vulnerabilities that have been reported. Sometimes, a modification in the code of a system is required or a compensating control needs to be put in place. All interdependencies of the change should be captured in this phase in order to avoid any adverse disruption on other components. Moreover, the business needs to make sure that time constraints and resources required in planning, testing and applying the remedial activity have been considered.
• Time, resources and costs required to test the fix. Any changes that are planned have to be tested. During the testing of the fix, efforts should be made to make sure a certain fix does not open up a different attack surface or a security hole within the application or infrastructure.
• Time, resources and costs required to implement the fix. After successfully testing the fix, it should be scheduled to be rolled out to the production environment in a controlled manner. Efforts should be made to do this as soon as possible after a successful test through a controlled change management process.
An effective remediation strategy requires at least the following parameters for every item or vulnerability that has been identified:
• Timeline for all remediation phases (planning, deployment, testing).
• Associated costs of carrying out a fix. As an example, remediating a number of vulnerabilities in a custom-made application created by a supplier specifically for an organisation may require a substantial amount of work, and additional costs may be involved due to the way the contract agreement has been reached.
• Designation of individuals or teams that will be responsible in each of the remediation phases.
Allocating ownership
First and foremost, a dedicated team needs to be assigned the remediation tasks in order to carry out the fixes to the selected vulnerabilities in clear co-ordination with other relevant teams. The personnel overseeing the remediation phase should track the remediation progress at all stages, to provide a historical reference for the records:
• The testing and implementation progress of the fixes that are or have been applied on a particular asset including time frames.
• Other interdependencies that rely on the affected asset. These will come in very handy when there is an adverse impact or an incident reported, in case the fix has caused an unforeseen event.
• The outcome (successful or not).
• Personnel involved in the task.
Once all the remediation activities have been identified, they should be prioritised by the security team leads. After completing this step, it would be the ideal opportunity to present the action plan and feedback to the relevant teams. The teams should prioritise addressing their assigned activities to reduce the duration the associated vulnerability presents a risk to the organisation. Once teams have been debriefed, the team leads should communicate the expected time frame for remediation activities, based on feedback from the relevant teams. It should be noted that during the remediation process, risk could be re-evaluated or the remediation approach could change as the issue is explored.
The remediation plan should extend beyond only addressing vulnerabilities and should transition into a strategic lifecycle that focuses on creating strategies to reduce the likelihood that similar, organisation specific vulnerabilities are identified in the future. This transition should preferably only occur once the immediate risk – the affected technology issues – have been addressed.
It is important to measure how fast an organisation can classify and respond to vulnerabilities. Response time, as it is termed, has become increasingly important. It is the total time taken by an organisation to identify, classify and remediate a vulnerability. However, one has to measure the effectiveness of the remediation before confirming whether a vulnerability has indeed been remediated.
Due to the fact that penetration tests come in different flavours (i.e. have been executed from an external or internal perspective, been authenticated or unauthenticated and so on), it is strongly recommended for each organisation to set a framework or road map in place with a third party for such tests to commence at different time intervals, aiming at different components of the organisation – depending on the organisation’s requirements and risk appetite. These frameworks should also cover post-breach activity scenarios or ongoing system migration from one provider to another. This framework should be defined as part of the strategic organisational objectives to ensure that the cyber risk is kept as low as possible.
Re-testing is another crucial step during the remediation phase. After applying the remediation steps to an asset, the business should plan on executing a re-testing process to ensure that the fixes are effective in plugging the vulnerabilities. Re-testing is a simple and effective way of gauging the progress made against the remediation plan. Generally, a re-test should only be performed on all vulnerabilities where fixes have been attempted. Different re-tests can occur on different time frames, depending on the severity of the vulnerability. Efforts should be made to plan the re-testing phase into short-term and long-term assessments; that is, including re-testing of the critical and high-risk vulnerabilities before assessing other medium- or low-risk vulnerabilities. As mentioned previously, everything that is deemed unacceptable needs to be resolved in a timely manner, as waiting longer periods will mean that a complete reassessment is needed. Due to an ever-changing threat landscape and vulnerability disclosure, a delayed tailored re-test on the affected component may not be sufficient to uncover new weaknesses.
An acceptable time for remediation can be determined by the nature of the issue. This is expressed more specifically as:
Severity of the vulnerability × Likelihood and impact of the vulnerability being exploited × Asset value
Vulnerabilities that are ‘critical’ or ‘high’ risk should be remediated immediately, but realistically it has been industry-accepted for these to be resolved within a month. Other medium-to-low risks – those that do not have any direct or indirect impact on the confidentiality, integrity or availability of assets – can be fixed within three months. There are of course vulnerabilities that require more time to be completely resolved (i.e. migration to a new platform or re-writing core functionality of an application that has many interdependencies or an upgrade); here it is suggested to put mitigating controls in place to lower the risk to an acceptable level until a final solution is set in place.
Penetration testing offers the opportunity to validate an environment’s current security posture and to protect a business. By selecting the right scope and the right type of test, one can identify and remediate the identified security vulnerabilities. Far from being a stand-alone procedure, penetration tests need to be an integral part of the overall risk management programme. Always remember that true security is a holistic, overall approach that goes far beyond technical measures. Good security should be a culture within an organisation, based on a cycle of continuous improvement.
Acting effectively on penetration test results is of vital importance. This involves careful preparation of a road map for the required development and deployment of fixes, including the human resources and associated costs that will need to be involved. Equally important is the reassessment and verification that any applied security fix has the intended outcome, otherwise such processes must be repeated. Communication between all relevant parties and stakeholders in an organisation is essential and this lifecycle is an integral part of a continuous deployment programme to address business risks and act upon them.