Penetration testing has been a standard security control for more than a decade, and has become, to an extent, a ‘routine’ requirement for organisations that require evidence of testing to comply with their regulatory or industry sector standards (the payment card industry PCI DSS standard for example). In response to this, a wide range of penetration test service providers has emerged within the information security marketplace. Some of these focus very closely on vulnerability and security testing, while others include test capability as one of many offerings within their portfolio.

AN OVERVIEW OF THE PENETRATION TESTING SERVICE PROVIDER MARKET

The security testing marketplace includes a spectrum of capability in terms of test quality and depth. At the simplest level, some organisations specialise in providing standard commoditised test suites, based on tools and ‘scanning’ technology. These provide a relatively low-cost and low-overhead (in terms of business support) test scope.

Standardised testing can establish that systems are secure against common and widely encountered security threats. This level of testing can be fully adequate for a small- or medium-sized organisation with relatively simple or small-scale IT systems. This is the level of testing typically commissioned in support of accreditation to a standard security level, such as the Cyber Essentials Plus Scheme.¹ It is even possible for some organisations to undertake standard tests of this type themselves using licensed tools, either in-house or from a cloud-based testing service. That said, where sensitive or high-value assets are to be assured a more informed and in-depth test approach would always be recommended.

More complex testing capability can include provision of a wider range of tools, particularly where a more high-end IT application set is in use. Organisations that have internet-facing applications with a wide range of user interaction, or in particular, supporting financial transactions, would normally opt to go beyond the level of ‘standard’ tests and incorporate more individual tester expertise. This requires more capability from the testing provider, and usually more input from the commissioning organisation on the nature of their applications, how they operate and what functions exhibit the most risk.

Going further, some organisations – particularly those in financial or market-sensitive sectors – may want a very deep and thorough assessment of their security risk profile. This can extend beyond simple penetration testing and into ‘Red Team’-style testing, where testers use expertise, guile and techniques such as profiling of their target and social engineering to develop a sophisticated threat ‘package’ which is used in an attempt to circumvent installed security controls.

TEST PROVIDER CAPABILITIES

At one time, security testing was as simple as running through a set of standard tests and checks against a system – looking for user accounts that do not require a password for access, looking for open access folders or web servers, looking for services that respond to queries or commands without requiring credentials, and so forth. In a way, penetration testing still follows a similar cycle; but the scope of the vulnerabilities and bugs to be assessed has grown exponentially with the combinations of system types, applications, services and interfaces that information systems have.

Identifying an appropriate testing partner to help you to understand security risks requires some awareness of the scale of this problem and an ability to converse with and assess the offerings and capabilities of potential testers. The ideal test provider will have an awareness of information technologies that is both wide and deep. They will also have some experience and familiarity with standards and applications used in your business sector.

They should be able to reference and use a range of testing tools and techniques to interrogate your systems, as well as adopt recognised methodologies for testing and know how to apply these. But, of course, they must also be able to express their findings from the use of these tools and techniques at a level that is clear and useful to your organisation.

A starting point for selecting test providers is their accreditation. In the UK, and increasingly internationally, the recognised source for establishing the credentials of cyber-security advisors is CREST.² In assessing providers for accreditation CREST reviews the skill and competency, methodology, quality controls and trust credentials (for example, through staff background checks) of cyber-security suppliers and service providers. The CREST system initially complemented, and has now largely succeeded, other accreditation initiatives such as the UK CESG CHECK scheme that applied similar criteria to assess the capability of security testing providers.

However, there are other accreditations to look for from reputable providers, and these may be useful in differentiating between potential offerings. Relevant assurance can be gained from certification to recognised security standards. In the UK the Cyber Essentials Plus scheme is gaining increased relevance, as is the ISO27001 standard internationally. Another area that can be assessed is the level of qualification held by key staff of a testing provider and their membership of appropriate professional bodies.

A range of qualifications are aimed directly at evidencing penetration testing capability. The most widely recognised are probably the GPEN qualification issued by GIAC or the OSCP from Offensive Security but there are also certifications issued by CREST, the CEH (Certified Ethical Hacker) or CPT (Certified Penetration Tester) qualifications. Broader information security professional qualifications, for example, the ISC2 CISSP³ or ISACA CISM⁴ and CISA qualifications, evidence a good level of competence to advise and investigate security capability.

An additional factor that can be used to indicate tester capability is the profile and experience of their team. Most security testing companies will provide prospective clients with an overview of their management and technical team covering qualifications, experience and expertise. Look for a wide range of experience and mix of competence. Good security testing requires contemporary and practised technical competence, but also a wider background in problem solving, planning, and even psychology and social engineering. A capable test provider will be able to offer you a good mix of experience, depth and technical currency.

WORKING RELATIONSHIPS WITH TESTERS

Any arrangement to perform security testing must be founded on both a sound formal contractual basis, and on a clear level of understanding between the client and the testers about the objectives and scope of the testing. Negotiating a contract for penetration tests is to some extent similar to other types of contract negotiation in business and IT. Many contracts are based on standard terms and conditions which set out the parameters within which both the testing team and the organisation being tested will operate.

Confidentiality is a prime consideration. In order to test effectively, testing providers require access to information about their client’s systems, networks and business structure. They may also need to share information with their client in order to make tests practical, such as network (IP) addresses or test system credentials. This creates a need for a mutual understanding and agreement on confidentiality around test information that must be observed by all stakeholders and participants. At the simplest level, this might be a non-disclosure agreement but there may also be a need to segregate access to information in order to make tests meaningful, particularly if these are more complex and multi-environment tests.

One of the biggest challenges for penetration testers is where systems are co-hosted in multi-tenanted environments, and where tests against one hosted application might have an impact on a separate system or system owner. There may also be some requirements around disclosure of test results; for example, where a system hosted by one supplier is being tested independently by another. Testing agreements should be clear and unambiguous about what party owns and is responsible for the dissemination of any test results and how these can be used.

Testing may involve access to personal data, in which case both the test client and testers must handle that data in accordance with privacy law. An example where this can come into play is where testing teams are located outside of the geographic area of the data hosting source, for example outside the EU, or they use testing tools hosted outside the EU. In that case, contracts must ensure that security and privacy of any personal data accessed during a test will be preserved in accordance with the General Data Protection Regulation (GDPR).

Where applicable, handling of personal data should be addressed in the contract with penetration testers. In light of GDPR this is particularly important where international boundaries are a factor in testing. While it is not within the scope of this chapter to cover GDPR in detail, testing providers will be aware of their privacy obligations and should recognise the need to clarify these contractually. The data privacy regulators (the ICO in the UK) provide guidance on contractual provisions regarding data protection and these should be adopted for testing that may impact individual privacy.⁵

REVIEW AND ‘ROTATION’ OF TEST PROVIDERS

A control that is often recommended for penetration testing is that organisations ‘rotate’ their testing provision between different providers over a rolling period. This can range from an arrangement in which several potential test providers are retained and offered the chance to ‘bid’ for specific test contracts as they arise (which can clearly offer commercial benefits), through to a periodic re-tender and re-contract for a longer lasting ‘exclusive’ test contract.

There are benefits in both building and maintaining a strong relationship with a trusted test service provider and in having a range of options. A test provider that has tested a client’s systems several times can build up a good understanding of their technical and organisational environment. This can lead to an improvement in the quality of the testing over time. It may also be easier to secure consent for tests by a known and trusted provider that has worked with your IT services or peer group companies previously.

Conversely, test strategies should take into account the dynamic nature of cyber threats. The risk environment changes frequently, and test techniques and capabilities evolve in line with this. A good test service provider should be aware of and respond to this, proactively reviewing their approach with you on a regular basis. If not, you should consider seeking alternative proposals for testing periodically to evaluate them against incumbent service providers. It can also be useful to consider specialist organisations with particular skills where you require a test of a new or ‘niche’ technology that might fall outside the skillset of previous test providers.

TEST CONSENTS

Consent for testing is rarely a bilateral issue between penetration tester and the client. It is not unusual for one or two separate layers of client service providers to be party to testing, and their awareness and consent to tests can be essential. For example, in testing a web-hosted application it is possible that one service provider is responsible primarily for providing the application and its functionality while another (possibly a cloud service provider) is responsible for the hosting.

Separate stakeholders will be subject to service liabilities, security controls of their own and contractual requirements, surrounding security tests. It is standard practice to seek consent from such service providers as a prerequisite of commissioning a penetration test. It can also be useful to engage with them to ensure that testing activity might not be misinterpreted and ‘reacted to’ inappropriately. For example, many hosting providers adopt overarching security controls which interpret a security scan as hostile activity and might block or redirect network traffic generated by the test process.

This is increasingly the case for cloud-hosted services where integral security controls are provided inside the cloud platform. Cloud service contracts generally require that clients request permission from the provider before conducting penetration testing of any part of their environment. Note that in some cases, standard security controls may have to be ‘relaxed’ to permit testing to take place at all. Because of this, most hosting providers will now have a standard approach to evaluation of and consenting to independent security tests. Engagement with this process should be a standard component of any penetration test plan.

Organisations that make extensive use of cloud services should study the assurance issued by their hosting provider and assess how penetration testing activity might complement this. It can be useful to review some of the developing standards and checklists for cloud security, for example those issued by the National Cyber Security Centre (NCSC)⁶ or the Cloud Security Alliance.⁷ You should also allow for some detailed discussions with your testing provider around the nature and scope of tests directed at cloud-hosted services.

COMMERCIAL AND TECHNICAL RELATIONSHIPS

Fees for performing penetration testing are obviously central to the contracting process. The actual cost of a test can vary significantly among test providers, business sectors and technical environments. It is always worth conducting initial surveys of potential suppliers through a ‘request for proposals’ (RFP) process prior to entering into a full negotiation, to gain an understanding of the potential costs.

Like all service markets, the security consultancy and testing business is subject to the rules of supply and demand. It is well known that good security skills are in short supply, and test service providers can sometimes charge a premium. Flexibility in scheduling can have a significant impact. If you set out to engage a large and complex penetration test programme to a short deadline with fixed delivery dates, you may find it hard to secure resource at a reasonable cost. Alternatively, if you engage a service provider well in advance, and plan for a series of test engagements with a flexible timetable over a long period, you could negotiate a very favourable deal overall for the cost of your test programme.

Be aware of the difference between skilled and in-depth security testing and standard technical or security consultancy. While technical support is normally charged at an hourly rate, elapsed time consumed by a penetration test does not necessarily bear much relationship to the total effort or cost required for it to be completed.

Penetration testing may involve long periods of automated assessment and scanning of networks and applications, interspersed with short, but intensive, periods of manual analysis to interpret results. In fact, the most valuable element of any testing engagement is the analysis and interpretation phase in which a range of findings, vulnerabilities and test logs are translated into a coherent picture of the risk profile of the test target. This may require more than one technical discipline – and hence imply collaboration between one or more testing specialists – adding to the complexity and cost of test analysis.

Test results have to be delivered in a usable format. Software tool-driven tests can produce copious amounts of technical information and test logs that could be unintelligible to a non-expert reviewer. The time taken to collate this into a presentable report is part of the cost of testing, as also is some additional time for the material to be reviewed and quality assessed for completeness. Good test providers tend to combine communication and documentation skills with their wider technical skill base, in order to provide high-quality reporting.

UNDERSTANDING AND USING TEST RESULTS

An important element of the relationship between a test provider and their client is the understanding that the penetration testers have of their client’s requirements and expectations from the test. Clients usually have some specific outputs they would like to see from test results, ranging from a ‘clean bill of health’ for their systems, through to a risk-based prioritisation of areas in which they should focus their defensive security investment and effort.

It is rare that penetration tests are conducted purely as a discovery exercise in which a client wants a full disclosure of all risks that can be identified, from the most critical down to the trivial. Testing engagements should seek clarity as to the level of reporting that is required by the client and what their appetite is for detail and risk level to be reported. When engaging penetration test providers, always ask to review sample reports (anonymised and desensitised, as appropriate) to gain an understanding up front of the type of material that will be issued at the end of the test. Ideally testing providers should be prepared to talk their clients through the results of these examples to clarify their meaning.

Penetration test service provider companies use standard vulnerability classification schemes that have a direct meaning to their clients in terms of risk levels. The most ‘common currency’ of vulnerability tends to be the categories used by Microsoft for classifying their software security updates. These range from low through moderate and important to critical levels of risk. In some environments, however, more sophisticated scoring and assessment models can be required (particularly IT infrastructure and service provider networks). A widely used example that some testing providers adopt is the Software Engineering Institute’s OCTAVE⁸ methodology, which measures both the impact of a particular vulnerability and the likelihood that it might be exploited in a particular organisational context.

Experienced testing providers will also be able to articulate their findings in relation to other models and methodologies. A commonly used reference is the OWASP Application Security Verification Standard⁹ project which sets out to analyse and categorise common risks to web applications. Another is the Cloud Security Alliance (2012)¹⁰ guidance on web application security assessments.

Ideally penetration test results should provide more than a simple list of risks and vulnerabilities. A capable test provider may be able to outline not just the vulnerability, but also what the outcome of it being exploited might mean for a given organisation. This could be in terms of service disruption, data loss or theft, or financial impact.

A good test report may also provide advice regarding remediation of vulnerabilities and provide references to sources of data, software or hardware updates or vendor recommendations for configuration to address test findings.

SUMMARY

This chapter has considered the market for penetration testing service provision and drawn attention to the varied services that are available. It has looked at a number of different types of testing and suggested approaches to selecting them in line with business requirements.

We have looked at some of the contractual and commercial factors that arise in penetration testing engagements. This has also included consideration of a range of certification and qualification regimes that can be used to gain assurance about potential test providers.

We have also looked at some of the processes that testing engagements will typically follow, including eliciting proposals, setting up agreements and contracting for provision of results. Finally we looked at the interpretation of test results and ways of gaining the optimal value from them.

REFERENCES

Cloud Security Alliance (2012) SecaaS Implementation Guidance: Category 5 // Security Assessments. Available at: https://downloads.cloudsecurityalliance.org/initiatives/secaas/SecaaS_Cat_5_Security_Assessments_Implementation_Guidance.pdf

Creasey, J. and Glover, I. (2017) A Guide for Running an Effective Penetration Testing Programme. CREST. Available at: https://crest-approved.org/wp-content/uploads/CREST-Penetration-Testing-Guide.pdf