CHOOSING A SUITABLE SUPPLIER
Organisations can carry out penetration testing themselves, sometimes very successfully. More often they will decide to employ the services of one or more specialist third party penetration testing providers.
Findings from the research project indicated that the main reasons why organisations hire external suppliers are because these suppliers can:
- Provide more experienced, dedicated technical staff who understand how to carry out penetration tests effectively, using a structured process and plan;
- Perform an independent assessment of their security arrangements;
- Carry out a full range of testing (eg black, white or grey box; internal or external infrastructure or web application; source code review; and social engineering);
- Conduct short term engagements, eliminating the need to employ your own specialised (and often expensive) technical staff.
‘We suspected that we had already been hacked and wanted to find out more about the threats to our systems, to help reduce the risk of another successful attack.’
When appointing an external provider of penetration services, it is important that you choose a supplier who can provide a reliable, effective and proven penetration testing service – but at the right price. To do this, it can be useful to:
- review requirements;
- define supplier criteria;
- appoint appropriate supplier.
A. Review requirements
The first step is to make sure that whoever chooses the supplier (preferably not just a procurement specialist) fully understands your organisation’s requirements, and is aware of any necessary management, planning and preparation activities.
There are many benefits in procuring penetration testing services from a trusted, certified external company who employ professional, ethical and highly technically competent individuals. CREST member companies are certified penetration testing organisations that fully meet this requirement, have been awarded the gold standard in penetration testing and build trusted relationships with their clients.
‘Our supplier is a trusted organisation who employs competent people – and this combination is important’
B. Define supplier selection criteria
To ensure that your chosen supplier will meet your requirements it can be helpful to define a set of supplier criteria, most of which your chosen supplier should be able to meet – or exceed. The six main criteria identified during the research project are outlined on the following pages together with examples of the types of questions you may wish to consider as part of the selection process.
‘A good supplier helps to assure the process for a proper security test without creating misunderstandings, misconceptions, or false expectations.’
Selection criteria 1 – Solid reputation, history and ethics
Typical questions to ask a potential supplier |
Comments |
a. Can you provide evidence of a solid reputation, history and ethics (eg. a full trading history; good feedback from both clients and suppliers; a reliable financial record; and a strong history of performance)? |
Two of the most important criteria for a buyer of penetration testing services to consider are the reputation (and history) of the supplier; and the ethical conduct they both adopt and enforce. |
b. Do you take part in specialised industry events (such as those run by CREST or OWASP chapters)? |
|
c. Are you able to demonstrate exploits or vulnerabilities you have found in other similar environments? |
|
d. Can you provide independent feedback on the quality of work performed and conduct of staff involved? |
They will also have processes in place for agreeing scope and obtaining permissions for the type of work to be conducted, where it will take place and what information and systems will be accessed. |
e. Do you adhere to a formal code of conduct overseen by an independent industry body? |
Selection criteria 2 – High quality, value-for-money penetration testing services
Typical questions to ask a potential supplier |
Comments |
a. Can you show that you provide high quality services, including the methodologies, tools, techniques and sources of information you will use as part of the testing process? |
Some suppliers will hit you with a volley of ‘vendor hype’ that can be difficult to penetrate. It can therefore be a real challenge to find the right quality of service at the right price. |
b. How do you perform rigorous and effective penetration tests to ensure that a wide range of system attacks are simulated? |
Suppliers should be able to produce insightful, practical and easy to read reports, engaging with senior management in business terms, resolving issues with IT service providers, and addressing global risk management issues. |
c. Can you describe your proven testing methodology that is tailored for particular types of environment (eg. infrastructure, web applications, mobile computing)? |
|
d. Can you demonstrate your organisations’ penetration testing capabilities (eg. by make a presentation; showing examples of similar (sanitised) projects they have undertaken) and providing a sample report? |
|
e. Do you have independently reviewed quality assurance processes that apply to each test being undertaken, to help make sure client requirements are being met in a secure, productive manner? |
‘If you have been compelled to conduct a penetration test, then our penetration testing services may not be for you, but if you want to conduct a proper test, give us a call.’
Selection criteria 3 – Research and development capability
Typical questions to ask a potential supplier |
Comments |
a. Do you have an active, continuous and relevant research and development capability? b. Have you produced research papers, published vulnerabilities or won awards in the industry? c. Do you perform sufficient research and development to be able to identify all significant vulnerabilities? d. How do you carry out specially tailored, manual tests to help detect unknown vulnerabilities, rather than just using a standard set of tools? |
One of the biggest selling points for some suppliers is the quality and depth of their technical research and development (R&D) capability. Some suppliers will constantly develop specific methodologies to address different environments, such as infrastructure, web application, wireless, mobile etc. A good technically competent supplier is likely to carry out about 70% manual testing (simulated hacking!), as opposed to 30% using automated tools. |
Selection criteria 4 – Highly competent, technical testers
Typical questions to ask a potential supplier |
Comments |
a. What qualifications do your testing staff hold in the various areas in which tests may be required (such as web application testing)? b. How do your testers identify ‘root cause’ findings, strategically analyse findings in business terms, help develop security improvement strategies and recommend countermeasures to both address vulnerabilities and prevent them recurring? c. Can you specify: named individuals who will be responsible for managing and conducting the test, their experience of the environment within the scope, their qualifications and the exact role each individual will perform? |
The penetration testers used by your supplier should have deep, technical capabilities in the specific areas that are relevant to your target environment (eg web application, infrastructure, mobile or vendor-specific). CREST provides accreditation in different technical areas, such as CREST web application testers and CREST infrastructure testers. There are also specific examinations in areas such as wireless testing. |
‘Put the right people from the right organisation on the right job at the right time’
Selection criteria 5 – Security and risk management
Typical questions to ask a potential supplier |
Comments |
a. Do you apply independently validated security and risk management controls over the testing process, all relevant people involved, key aspects of target systems and any client data affected? b. Can you provide written assurances that the security and risks associated with our critical systems and confidential information (together with any other business risks) will be adequately addressed – and compliance requirements met? c. How do you ensure that results of tests are generated, reported, stored, communicated and destroyed in a manner that does not put the organisation at risk? |
It is important that the supplier themself is secure – and has a positive approach to both security and risk. Your supplier should be able to provide assurances – preferably in writing – that the security and risks associated with your critical systems and confidential information (together with any other business risks) are being adequately addressed. During any security assessment it is likely that the test team will encounter sensitive or business critical data. You will need to be comfortable that you can trust both the supplier – and every individual tester they provide. |
Selection criteria 6 – Strong professional accreditation and complaint process
Typical questions to ask a potential supplier |
Comments |
a. Does your organisation hold strong professional accreditation? b. Can you outline the problem reporting and escalation processes that you adopt should there be a problem with the testing? c. Are you supported by a constructive, expert complaint process, with sufficient independence and authority to resolve issues? |
Penetration testing organisations who have been professionally accredited will provide you with confidence that major vulnerabilities have been identified and properly addressed. They will also bring with them a wealth of experience drawn from client work across a range of companies and sectors, allowing lessons learnt from one to be transferred to others. The CREST scheme requires organisations to demonstrate that they have appropriate procedures and controls in place to protect client information and systems. |
There can be a big difference between a cheap penetration testing service and one that provides real value for money. For example, many low cost services may not provide certified, professional staff that can uncover and address significant vulnerabilities or act in an ethical manner according to a defined code of conduct. Furthermore, there is typically little recourse in the event of a dispute (eg. no independent adjudication and sometimes not even any indemnity insurance).
‘CREST provides demonstrable assurance of the processes and procedures of member organisations and validates the competence of information security testers.’
C. Appoint appropriate supplier
It can often be difficult to produce a short list of potential suppliers, not least because there are so many to choose from. For example, penetration testing suppliers can include:
- Organisations specialising in penetration testing (often small boutique firms) – who may have specialist research and testing capabilities;
- Information security consultancies and integrators, with penetration testing teams – who may have wider links to information security strategy and integration with security management standards;
- Systems integrators and outsourcing service providers with penetration testing teams – who may have detailed understanding of your technical environment and knowledge of attacks to similar outsourced organisations;
- Regulated professional services firms, including the ‘Big 4’ accountancy firms, with penetration testing teams – who may be more heavily regulated, with links to wider audit and compliance requirements.
Although value can be obtained by appointing either certified testers or certified organisations, it is the combination of these that will provide you with the greatest assurance that the most effective tests will be conducted – and in the most professional manner.
By procuring penetration testing services from certified testers who work for certified organisations (as CREST require), you can rest assured that an expert and independent body – with real authority – is on hand to investigate any complaint thoroughly and ensure that a satisfactory conclusion is reached.
After carefully considering all the relevant supplier selection criteria – and evaluating potential suppliers – you will then need to formally appoint one or more suppliers. The key consideration should still be to select a supplier who can help you meet your specific requirements – at the right price – not just one who can offer a variety of often impressive products and services, some of which may not necessarily be relevant.
Tests are often carried out on a regular (typically annual) basis. However, they are often more effective if carried out immediately before (or after) a major change – often saving money in the longer run, too.
‘It is important to ensure that the right systems are being tested by the right people for the right reasons at the right time’