The purpose of web application security testing is to find any security weaknesses or vulnerabilities within an application and its environment, to document the vulnerabilities, and to explain how to fix or remediate them. The business drivers behind the testing may be requirements of corporate policy, security requirements mandated by the corporate financial auditors or an internal audit department, compliance requirements for PCI or other industry standards, or compliance with regulatory standards such as Sarbanes-Oxley or HIPAA. An evidentiary type of audit report, which contains evidence to back up claims of vulnerabilities, is even better, as the report will stand the test of time, and, over the years, explanations and thoughts about how the vulnerabilities were found may fade from people’s memories.

There are several types of testing methodologies. These include web application security audits, vulnerability assessments, and penetration tests. These methodologies have different scopes and goals, each with strengths and weaknesses. For clarity, these methodologies are all different from one another, but vulnerability testing and penetration testing may also both be part of an overall audit. However, an audit may contain steps that are not related to either vulnerability testing or penetration testing, as described in the section “Web Application Audits.”

The testing methodologies, in turn, can be executed with different levels of automation. Some testing is done in a completely automated fashion and other testing is done with a high component of manual intervention. This chapter briefly describes the goals and differences of the various types of testing and audits but does not attempt to delve into details of audit methodologies or audit standards to which audits may adhere.

Once testing is complete, the next recommended step is to fix the vulnerabilities identified by the testing. Once remediation is complete, the step following that should be postremediation testing to ensure all the repairs were done successfully.

In Chapter 3, we will walk through the process of these steps and describe in detail common web application vulnerabilities that are found during the course of audits, vulnerability testing, and penetration testing.

Understanding the Testing Process

Web application security testing comes in all shapes and sizes and it is sometimes difficult to differentiate between them. To add to the confusion, the names of the tests are sometimes used interchangeably, which sets incorrect expectations of all the tests concerned.

In a nutshell, the different aspects of web application testing can be understood in terms of the questions they answer:

• A web application audit answers the question, Is an organization implementing its web application policies correctly?
• A vulnerability assessment answers the question, What security weaknesses or vulnerabilities exist within an application?
• A penetration test answers the question, Was the tester, in a given amount of time, able to compromise any of the vulnerabilities?
• Postremediation testing answers the question, Have the vulnerabilities found during testing been completely remediated?

To summarize the terms used here since they seem very similar, an audit has the greatest scope and includes vulnerability testing, and web app audits try to find vulnerabilities in a broader scope of subjects including policies and procedures. Vulnerability testing is usually passive and seeks to identify but not compromise the vulnerabilities it identifies. Penetration testing is the next possible step after identifying the vulnerabilities and attempts to compromise those vulnerabilities. Another way of saying the same thing is that whereas vulnerability testing just identifies technical vulnerabilities, penetration testing actually tries to exploit those vulnerabilities. Postremediation testing occurs after remediation and identifies the degree of remediation success.

The main reason penetration testing is done is to satisfy a specific governmental or very high security requirement. However, in some cases companies simply want proof that the systems can be compromised. I recommend using the funds that would otherwise be used for penetration testing for postremediation testing and for implementing ongoing, regular vulnerability testing.

Web Application Audits

The scope of an audit is generally a superset of a vulnerability assessment. The scope may include other software associated with the application, such as databases, access controls for the application environment, application documentation, security policies and procedures for managing the application and its environment, change management, revision management, backup and restore procedures, and so on.

The audit process starts with a specific, clearly defined scope of requirements. These requirements may include vulnerability testing for the application and its associated database, access controls, and security policies and procedures.

The first step of the audit involves a planning meeting to ensure all objectives will be met by the various planned audit activities. Activities include collecting data about the security posture of the environment through vulnerability and other technological-security testing, manual security testing, interviews with staff members, and a review of operational and security-related policy/procedures documentation. After the data-collection phase of the audit is completed, analysis is done on all the data collected in order to create the required deliverables of:

1. Any available evidence of the presence of vulnerabilities
2. A description of the vulnerabilities
3. Recommended remediation for each type of vulnerability
4. Each vulnerability’s levels of security risk and business risk.
5. An executive summary that translates all the technical jargon into business risks upon which financial decision makers can act.

Vulnerability Assessment

A vulnerability assessment is a subset of an audit and is focused on finding weaknesses or vulnerabilities within the web application. It involves real-time testing and exercises the application components such as all input fields. There are different vulnerability testing tools commercially available such as Nexpose, Nessus, and NMap.

Vulnerability assessments can be completely automated or have a manual component. A manual component is usually done by an expert tester who utilizes several testing tools over a predetermined scope of time to find vulnerabilities in a step-by-step manner. The steps may involve launching several tools, with the intention that a vulnerability missed by one tool will be identified by another.

The steps may also involve the tester diving deeper into any vulnerability that she thinks may lead to finding other vulnerabilities. For instance, if a tester finds weak encryption in one section of a transaction processing application, she may dive more deeply into that section to look for weaknesses relating to out-of-date security certificates.

There are upsides and downsides to both fully automated vulnerability testing and for manual testing.

Fully Automated Testing

Fully automated testing is done using tools that are designed to run autonomously once they are given target IP addresses and URLs to test. Prior to starting the automated testing, the tester first needs to make sure the targets have visibility. For instance, if the IP addresses and URLs that are in scope for testing reside behind a firewall, the security person responsible for these items needs to grant him secure-access.

High-quality automated testing tools should have access to back-end databases of both current vulnerabilities and current threats so that they can test comprehensively and then tune out false positives. The method for tuning out false positives is to compare the vulnerabilities against the list of threats and then eliminate reporting on vulnerabilities for which there are no corresponding threats.

The main benefits of automated testing include the following:

• 100 percent scope: These tests run very fast and the scope of testing is 100 percent of an application.
• Exact number of instances reported: Since the test scope is 100 percent of an application, the tool can enumerate the exact number of instances of each type of vulnerability.
• Cost effectiveness: These tests are less expensive to run than ones involving expert testers’ time. For instance, an automated test may take one person-day to implement and only minutes to run. A comparable manual test would take four person-days to execute. If the testing tool can be rented or used as part of an outsourced service, then the all-in costs of the automated testing tool may be significantly less than for manual testing.
• Timely actionable information: Since the tests are less expensive than manual ones, it is more affordable to run them often, such as monthly, to obtain timely information about newly evolving vulnerabilities.

The primary downside of automated tests is that they cannot find all of the types of vulnerabilities. For example, the testing algorithms cannot anticipate issues that arise with real-time data, such as work flow errors or weak password protection.

Manual Testing

If manual testing is done by an expert in web application security, then this methodology offers the greatest-possible depth of testing. Manual testing is a step-by-step process where the tester looks for vulnerabilities, and, when they are found, attempts to drill down further into the vulnerabilities to clarify their magnitude and just how risky they are.

A manual tester uses testing tools to conduct much of the testing but directs the course of the tools. Also, since every tool has its limitations, a skilled tester will use at least two tools in order to minimize the chances of missing a vulnerability.

Since manual testing always has time and cost limitations, it is done only on sample sections of a web application. The reports then identify where and what type of vulnerabilities were found. Recommendations for where remediation can be done in every instance of the security weakness.

The most significant benefit of manual testing is that it can be more granular than automated testing and cover a wider scope. Since human experts are conducting the testing, they can understand vulnerabilities that automated testing tools cannot parse or understand. Also, experienced testers can dive deeper in an iterative manner to explore suspicious circumstances.

However, there are several downsides to manual testing:

• Expense: Person power is expensive.
• Scope limited to sampling: Since every testing engagement has a finite amount of time allocated for expert testers’ time, the testing is often limited to a sample of the application.
• Number of instances not reported: The total number of instances of each vulnerability is not usually reported. Instead, the type of vulnerabilities is reported, and it is up to the web application owner to identify all the instances.

Combining Automated and Manual Testing

The most accurate determinant of vulnerabilities and risk is the use of automated testing in concert with manual testing. Automated testing can be done monthly to provide information on a regular, timely basis at a relatively low cost. Manual testing can be done periodically, such as on a quarterly or annual basis, to find the vulnerabilities not detectable by the automated tester at a relatively higher cost.

The optimization of lower-cost automated testing in conjunction with higher-cost manual testing provides the benefits of both worlds:

• 100 percent scope of the application is tested
• Regular, timely reports of the latest vulnerabilities
• Quarterly or annual deeper-dive testing to identify vulnerabilities not otherwise found

A valuable enhancement to identifying vulnerabilities is to proceed to map the vulnerabilities against a database of existing exploits and attacks “in the wild” and then allocate higher risks to those vulnerabilities for which there exist actual threats.

Penetration Testing

A penetration test is a deeper dive of a vulnerability test. Here, the expert tester attempts to compromise vulnerabilities he finds. The tester’s goal it to prove he can gain a high level of administrative access. Testing is often done by teams of one or more testers, called tiger teams.

The main benefit of a penetration test is the proof of the risk. A compromised vulnerability proves the degree of risk. On the other hand, the time it takes for penetration testing is expensive, and it does not reduce risk; it only verifies the risk.

I believe it is a better return on investment for most companies to spend their security funds on eliminating vulnerabilities and hardening their security infrastructure rather than testing vulnerabilities that they already know need to be mitigated.

Postremediation Testing

It is surprising that vulnerabilities are sometimes not remediated even after a comprehensive web application test. Yet this problem often occurs and for many reasons. Sometimes remediations are effectively implemented but then unwound by an additional development of the application. Sometimes technologists remediate some but not all the instances of every type of vulnerability. There are also instances where third-party operators inadvertently undo the benefits of remediation by operational changes they make.

Therefore, a postremedial audit is a very useful tool for ensuring the remediation plan was successfully executed. The postremedial audit is usually smaller in scope than the initial audit and its focus is on identifying whether or not the remediation recommendations in the initial audit have been done correctly. The remediation audit report is therefore comprised of yes-or-no responses for each vulnerability in regard to whether each vulnerability has been successfully remediated.

Important Report Deliverables for All Testing Reports

Reports are the last stage of an audit engagement and are done after the testing team has completed information gathering, done the requisite analysis, drawn conclusions, and made recommendations. If the report is not crystal clear, actionable, complete, and easy to read, and does not include a provision for the recipient to ask questions, then the report may have little value. I have seen reports that were filled with unanalyzed data, did not provide actionable remediations, did not differentiate the levels of risk of the vulnerabilities, did not transparently identify evidence of vulnerabilities, did not explain what tools and methodologies were used, did not organize the results data in a format that is clear and able to be easily referenced, and did not have provisions for any subsequent questions to be answered.

Readers of audit reports want to see crystal clear observations, specific remedial recommendations, brevity that does not impune accuracy, and a linkage between vulnerabilities and their related business risks. That is what a good audit report provides.

Testing reports are most useful when they:

• provide only actionable data. This means filtering out false positives.
• provide up-to-date and accurate analysis based on extensive and constantly updated databases of vulnerabilities, malware, and attacks that exist in the wild.
• report the technical information by correlating each vulnerability with its:
  • associated threat
  • risk of compromise
  • business risk
  • remediation
  • evidence of existence
  • number of occurrences by vulnerability type wherever possible
• report estimated time for remediation for each type of vulnerability.
• identify the tools and methodology used for testing.
• precisely identify the scope of the audit, including IP address ranges, URLs, number of employees interviewed, number of pages of documentation read, the dates during which testing was done, and so forth.
• publish a Q & A session for recipients postreport with full transparency and disclosure for all types of questions.

This is a general list that will vary for reports looking at specific types of testing, such as penetration testing, where the vulnerabilities may already be known and the focus is on whether or not they can be compromised by a testing team of a specific size and testing for a specific period of time. The report for a postremedial audit will be severely truncated and can be as simple as a column of yes-or-no observations added to the vulnerability list in the initial audit portion.

Summary

There is a wide range of types and methodologies of web application security testing. It is important for those with expectations of the results of testing to understand the differences and overlap between different types of tests and how they are performed. This is important in order to ensure that expectations of results are clearly understood before funds are spent on the actual testing.

There is a different return on investment for each type of testing. Some testing is more drill down in depth, such as penetration testing, but may not have any return on investment at all. Other testing, such as automated regular-vulnerability testing, will be relatively inexpensive but may have a huge return on investment and may also meet all the business requirements imposed upon those responsible for web application security. In between these extremes exist various degrees and subsets of web application audits, whose return on investments will vary with the business requirements that drive the underlying testing needs.

The testing process (defining the pieces) for web application security audits, vulnerability assessments, and penetration testing can vary and is generally divided between automated and manual testing. Testing can have various degrees of automation and manual testing. Generally, automated testing is faster and less expensive than manual testing. There is a variety of testing tools available for web application security testing. It is useful to test with at least two tools to improve the chances that any vulnerability that is not found by one tool will be identified by another. Manual testing can find vulnerabilities that fully automated testing simply cannot.

After testing is completed, remediation should be done to fix vulnerabilities found during the testing phase. Postremediation testing should be done to make sure remediation has been done successfully. It is very important that false positives are tuned out during the analysis phase of all testing. This ensures that the reports are as meaningful and as actionable as possible.

Test reports must be clear, complete, actionable, and accompanied by an opportunity for the recipient to ask questions about the information provided.