CHAPTER 9: PREPARING THE REPORT
The ultimate goal of penetration testing is to help an organization to secure their networks, systems and databases. The pen tester is given the responsibility of providing meaningful information to the client organization, which will enable them to put forward a plan for mitigating any vulnerabilities and developing enhanced security strategies.
The penetration test report will often include several sections, including a brief summary, a listing of all work done, a detailed list of vulnerabilities found, a ranking of the risk level for each vulnerability, a list of recommendations, and a summary of the tester’s opinion on the overall health of the organization’s security framework. It is important, however, to remember to separate opinion from facts and findings, and not mix opinion with fact.
Overview
The first part of the report is an overview of the penetration testing project. It includes a description of the scope of the project (what was included in the test and what was determined to be out of scope). It is important to include this, so that the organization does not have the mistaken impression that the test covered areas that were not included in the test program and can schedule more tests to cover those areas not in the original scope. It can also be that the scope of the project changes during the performance of the tests. If a serious vulnerability is found during the test in an area not included in the original scope, then it may have been necessary to re-evaluate the test and make a determination of whether to expand or adjust the scope.
The overview should also detail the authority for the test – who approved it and what regulations, standards or organizational policies applied to the area being tested. This information is especially important when the test has been carried out to satisfy a compliance or audit requirement.
Executive summary
In most cases, the report should include an executive summary that presents a high-level overview of the test, the findings and the recommendations. This summary is prepared for upper management and regulatory bodies, and allows them to gain an overall picture of the test without disclosing or boring them with detailed descriptions of tests, findings and recommendations about configurations, etc. The executive summary of the report must not contain detailed technical descriptions that would only serve to confuse a senior manager and cause them to ignore the recommendations.
Detailed report
The detailed report contains the in-depth description of the test, vulnerabilities and recommendations. It may also contain more opinions from the tester on what the priorities should be, what the associated level of risk is for each vulnerability and a list of various options or potential solutions to resolve the identified security problems. The tester may also recommend where more testing may be needed.
This is where many reports fail to serve the needs of the client organization. The tester and report writers must remember that the primary purpose of the organization is to stay in business. That means that the systems and networks of the organization must serve to support some form of business function. When evaluating the vulnerabilities and providing recommendations on how to address or mitigate those vulnerabilities, the pen tester must not offer solutions that would prevent the system from providing its primary function – to support the business.
Case study: Utility company
The pen testing team had done excellent work and, with great pride, they began the exit interview meeting with the client organization. By following all the best practices of a good pen testing team, they had uncovered some potentially serious issues and were confident of the value they were providing to the client. As each member of the client organization entered the boardroom, they were presented a booklet outlining the scope, results and recommendations of the report. Printed in full color with graphics and notes, it was impressive and authoritative.
Forty-five minutes into the presentation – by which time the lead partner of the testing organization had already outlined the tests and findings, and was well into the list of recommendations – the Vice President of IT for the client organization interrupted with a question. “What business are we in?” he demanded. The leader of the testing team replied, “You are a service organization – a utility.”
The Vice President then said, “Yes, but what is the purpose of our network and our systems?”
Discussion ensued, in which it was agreed that the website that was tested was to allow for customers to interact with the client organization and order products, check order statuses, update their profiles, and check their payment histories.
“So, now that we know the business context of this system, let’s look at the recommendations you are giving us,” suggested the Vice President. “If I was to adopt these last few recommendations you have just suggested, would I still be able to support our business?”
The answer was, predictably, “No.” The pen testing team had just provided an excellent report that, in effect, would put the organization out of business. Such reports have little or no value to the client and result in the credibility of the pen testing team being damaged. A report that has good and poor recommendations is likely to be discarded. In fact, this report had several good recommendations that were then ignored, since the testing team had forgotten the most important directive of all: “Do not put the client out of business.”
Working papers and evidence
A pen testing report should be authoritative and factual. It must not be based on opinion or a generic standard that is not aligned to the business of the organization. Far too many pen tests are done clinically – without regard to the culture or operational environment of the client – and far too many tests are done haphazardly and with a lack of a clear, thorough methodology, which would ensure that everything is done according to the defined scope of the project.
Just as a scientist would do for a scientific hypothesis, the pen tester must first define the tests according to what needs to be proved, what a pass or fail standard would be, how the test will be conducted and what information the tests must provide. Once the test has been completed, there should be tangible evidence of what tests were conducted and what the results were. This leads to the analysis and interpretation of the results and then the determination of the recommendations for what needs to be corrected. Each recommendation in the report must be backed up by factual work papers and evidence that ensures the recommendation is valid and credible.
Determining risk levels
As seen earlier, the risk level is the likelihood of an adverse event affecting the ability of the business to meet its mission and objectives. However, risk can also be measured in terms of the amount of impact it could have. Would the entire organization be affected, or only one department? Would sensitive data potentially be disclosed, or only low-value information? Would the damage be tangible (quantitative – money, for example) or intangible (qualitative: reputation or morale)? The pen tester must be able to determine the level of risk associated with any finding in the report, so that the client organization will know how to prioritize and schedule their mitigation strategy. No organization will ever have enough money or resources (time and skilled manpower) to mitigate all risk immediately; setting out priorities is important, so that the most serious issues are addressed before the less serious, and so that the minor issues are not ignored for so long that they become major problems.
Risk response
The primary options for addressing the risk an organization is facing are quite straightforward. You can reduce (mitigate), accept, avoid or transfer the risk. The organization has to balance cost against benefit to know what the best strategy to take is.
Risk acceptance
Risk acceptance is knowingly and consciously agreeing that the organization is comfortable with the level of risk being faced. A person that loans a book or tool to a friend knows that there is a risk that the item may not be returned. They may attempt to mitigate the risk of loss by putting their name on it, or cautioning the friend to be sure to return it, but, regardless of the precautions they take, there is still the chance that the item (the asset) will be lost. The lender accepts this risk; they may be reluctant, but, in the end, they have a risk threshold or risk appetite that allows them to lend the item knowing that a loss may occur. If they were to not loan the item – being quite convinced that it would never be returned – they are practicing risk avoidance.
Organizations realize that not all risk can be mitigated or avoided, especially as the cost of mitigating the risk for a given incident may be higher than the impact of the incident itself. The level of risk that the organization is willing to accept is known as their “risk appetite” or “acceptable risk level.” A pen tester should never recommend a solution for a vulnerability that would be more costly to the organization than the cost of the impact if the problem were to occur.
Risk avoidance
Risk avoidance is avoiding a situation that may present an unacceptable level of risk to the organization. Part of a feasibility study and strategic plan is determining which initiatives the organization is willing to pursue and which ones should be avoided due to the risk of losing money being too high. In considering whether to enhance their website by adding in new functionality, an organization must practice risk avoidance and consider whether adding that functionality into their systems would pose a level of risk that is above the risk appetite of the organization. If the risk is too high, and the cost to mitigate it too severe, the organization may simply choose to avoid that risk by not adding in the new service or functionality.
Risk transference
Most organizations will purchase insurance as protection from catastrophic loss. The loss of a building or major asset may result in the financial collapse of the organization. Therefore, they purchase insurance to cover a portion of the costs associated with that loss in the event that it does occur. By purchasing insurance, they have transferred a portion of their risk to a third party – the insurance company.
Risk mitigation (reduction)
It could be stated that the primary purpose of a penetration test is to mitigate risk. The pen test finds vulnerabilities, determines the risk level associated with those vulnerabilities and provides the client organization with a set of recommendations for how they could reduce those vulnerabilities to an acceptable level.
Risk is reduced through the implementation of controls – technical, physical, administrative and operational controls. However, each control has a cost. The control may cost something to implement or to operate, and it may impact productivity, reduce functionality, or affect employee morale. Determining the correct response to a risk – which involves considering what is an acceptable level of risk and what is the cost versus the benefit of the control method – is the science of risk mitigation.
An example of deploying risk mitigation is the protection of a building. The owner wants to protect the building from break-ins and will deploy a series of controls to deter, prevent, detect and respond to break-ins. However, some buildings require more protection than others. It may not make sense to deploy expensive and complicated controls to protect a building that does not contain valuable assets, but it would be inappropriate not to deploy expensive controls to protect a building containing high value items – such as a bank or military installation. A pen tester must be careful not to recommend controls that would cost more to implement and maintain than the building is worth!
In some cases, the pen tester may identify a critical vulnerability during the test that requires immediate mitigation. When the pen tester finds a critical issue, they should notify management immediately, so that the problem can be addressed promptly. In this case, the issue should still be included in the penetration test report, but noted as having been fixed. This ensures that there is a record of the vulnerability and later tests can ensure that the vulnerability continues to be mitigated.
Residual risk
Residual risk is the risk that remains after the controls have been implemented. Risk cannot be totally eliminated, but it should be reduced to a level that is acceptable to the organization. Residual risk must be equal to or less than the acceptable risk. In the case where an organization is willing to self-insure for all risks that would cost less than one million dollars, the cost of many incidents or risks is going to be less than that risk acceptance threshold of one million dollars. Therefore, the residual risk or actual risk is less than an acceptable risk level.
Risk in the penetration test report
The pen tester should be able to provide the client organization with an assessment of their current level of risk and the risk levels associated with each vulnerability identified in the report. Since risk is always evolving as new technologies, vulnerabilities and threats emerge, and as the organization changes its operations or processes, risk management is a continuous process. A pen test is a picture of risk at a point in time, and should be repeated on a regular basis.
Audit risk
No penetration test is perfect, and there will always be a risk that a serious vulnerability in the client organization will not have been identified or included in the report. This is known as the audit risk. The pen tester should take precautions to avoid missing vulnerabilities by following methodologies, standards and best practices. However, every report should have a caveat that protects the pen tester from being held accountable for errors or omissions, and makes sure that they are not put in a position of being culpable if a subsequent attack exploits a vulnerability that had been missed.
Report confidentiality
A penetration testing report is an extremely confidential document and the pen testing organization must observe appropriate precautions in the preparation, delivery and storage of it.
Only personnel that have a need to know and have signed non-disclosure agreements should be able to access the report or any files or test results associated with the test. It is important to consider this in the printing of the report. There have been cases where a pen test for an organization has been released by employees of a company contracted to print the report. There have also been instances where copies of the report were found in the trash or recycling bins of the pen testing or printing company. This is especially likely to happen when draft copies of the report or the first print copy is discarded.
The report should be password protected and all extra copies should be shredded. The pen testing company may need to keep a copy of the report and test results (in case of future litigation or contests of the report), but it must be kept in a secure manner and a digital signature for the report must be generated. If the pen testing company is required to destroy all copies of the report upon delivery to the client, then they must have a sign-off and release from the client organization upon delivery.
Delivering the report
The penetration testing report should be delivered to the authorized personnel of the client organization. It is best to do this via a face-to-face meeting. A meeting should be arranged with appropriate personnel, so that the report can be presented to the client and the client has the opportunity to question or discuss the content of the report with the testing team.
The penetration tester should offer a clear and concise summary of the contents of the report, highlighting the important points and providing an overview of the recommendations and general state of the security of the client organization. Having graphs, tables and charts in the presentation and final report may help the client (especially senior management) understand the content of the report.
Prior to delivering the report, it may be advisable to have a meeting with the client’s IT department and other departments to gain a consensus on the recommendations contained in the report and ensure that the statements in the report regarding the client organization are accurate and acceptable.
Key learning points
• The report must be accurate, thorough, complete and credible.
• The report should be free from technical jargon and the use of terminology that may be easily misunderstood.
• Opinions must be separated from fact in the report.
• The report should have a standard layout, with an executive summary, detailed report and supporting evidence.
• The report should list the vulnerabilities, the relative risk of the vulnerabilities, and the recommendations to remediate the vulnerabilities.
• The report should be delivered in a face-to-face meeting with the client organization.
• The confidentiality of the report must be maintained at all times.
Questions
1. What is the purpose of the executive summary of the penetration testing report?
a) To provide detailed data on vulnerabilities and recommendations for the client organization
b) To define the schedule to fix any serious vulnerabilities
c) To provide a high-level overview of the penetration test
d) To gain consent from senior management to implement the recommendations of the report.
Answer: C
2. One of the primary elements of the pen testing report should be:
a) Work papers
b) Statement of scope
c) Test results
Answer: B
3. All personnel accessing pen testing reports should be:
a) Bound by non-compete agreements
b) Subject to non-disclosure requirements
c) Employees of the client organization
d) Managers or IT staff only.
Answer: B
4. Quantitative risk analysis is based on:
a) Financial considerations
b) Scenarios
c) Cost-benefit analysis
d) Risk avoidance.
Answer: A
5. Risk mitigation options may include:
a) Technical controls
b) Risk assessment
c) Qualitative risk review
d) Risk acceptance levels.
Answer: A