Static Analysis

The first type of automation you should write, and possibly the most common is static analysis.

Static analyzers are scripts that look at source code, and evaluate the code for syntax errors and common mistakes.

Static analysis can take place locally during development (aka a linter) as well as on demand against a source code repository or on each commit/push to the master branch.

Many robust and powerful static analysis tools exist, for example:

1. Checkmarx (Most Major Languages—Paid)

2. PMD (Java—Free)

3. Bandit (Python—Free)

4. Brakeman (Ruby—Free)

Each of these tools can be configured to analyze the syntax of a document containing text and representing a file of code.

None of these tools actually execute code, as that would move them into the next category called “dynamic analysis” or sometimes “runtime analysis”.

Static analysis tools should be configured to look for common OWASP 10 vulnerabilities.

Many of these tools exist for major languages, in both free and paid form. Static analysis tools can also be written from scratch—but in-house built tools often do not perform well on code-bases at scale.

For example:

General XSS—Look for DOM manipulation with innerHTML. Reflected XSS—Look for variables pulled from a URL param. DOM XSS—Look for specific DOM sinks like setInterval().

SQL Injection—Look for user provided strings being used in queries.

CSRF—Look for state changing GET requests.

DoS—Look for improperly written regular expressions.

Further configuration of static analysis tooling can also help you enforce best secure coding practices.

For example, your static analysis tools should reject API endpoints that do not have the proper authorization functions imported—or functions consuming user input that do not draw from a single source of truth validations library.

Static analysis is powerful as a form of general purpose vulnerability discovery, but it may also be a source of frustration as they will report many false positives.

Additionally, static analysis suffers when dealing with dynamic languages (like JavaScript).

Statically types languages like Java or C# are much easier to perform static analysis on, as the tooling understands the expected data type and that data cannot change type as it traverses through functions and classes.

Dynamically typed languages on the other hand are much more difficult to perform accurate static analysis on. JavaScript is a fine example of this—because JavaScript variables (including functions, classes, etc.) are mutable objects—they can change at any point in time. Furthermore, with no typecasting it is difficult to understand the state of a JavaScript application at anytime without evaluating it at run-time.

To conclude, static analysis tooling is great for finding common vulnerabilities and misconfigurations—particularly in regards to statically typed programming languages.

Static analysis tooling is not effective at finding advanced vulnerabilities involving deep application knowledge, chaining of vulnerabilities, or vulnerabilities in dynamically typed languages.

Dynamic Analysis

Static analysis looks at code, typically prior to execution. On the other hand, dynamic analysis looks at code post execution.

Because dynamic analysis requires code execution—it is much more costly and significantly slower.

In a large application, dynamic analysis requires a production-like environment (servers, licenses, etc) prior to having any utility.

Dynamic analysis is fantastic at picking up actual vulnerabilities, whereas static analysis picks up many potential vulnerabilities but has limited ways of confirming them.

Dynamic analysis executes code prior to analyzing the outputs and comparing said outputs against a model which describes vulnerabilities and misconfigurations.

This makes it great for testing dynamic languages, as it can see the output of the code rather than the (vague) inputs and flow.

It is also great for finding vulnerabilities that occur as a side effect of proper application operation—for example, sensitive data improperly stored in memory or side-channel attacks.

Dynamic analysis tools also exist for many languages and frameworks. Some examples of these are:

1. IBM AppScan (Paid)

2. Veracode (Paid)

3. Iroh (Free)

Due to the increase complexity of functioning in a production-like environment, the better tools are often paid or require significant upfront configuration.

Simple applications can build their own dynamic analysis tools, but for complete automation at the CI/CD level they will require a significant effort and a bit of upfront cost.

Unlike static analysis tools, dynamic analysis tooling that is properly configured should have less false positives and give deeper introspection in regards to your application.

The trade-off is in maintenance, cost and performance when compared to static analysis tooling.

Vulnerability Regression Testing

Figure 20-1. Jest—you don’t need a special framework for security vulnerability tests. Any testing framework capable of reproducing the vulnerability should do. Pictured here is Jest—a fast, clean and powerful testing library for JavaScript applications. Jest can be easily modified to test for security regressions.

The final form of automation that is essential for a secure web application, is that of vulnerability regression testing nets.

Static analysis and dynamic analysis tools are cool, but compared to regression tests they are difficult to setup, configure and maintain.

A vulnerability regression testing suite is simple—it works similarly to a functional or performance testing suite, but tests previously found vulnerabilities to ensure they do not get released into the codebase once against as a result of a rollback or overwrite.

Imagine the following vulnerability:

“Steve introduced a new API endpoint in our application that allows a user to upgrade or downgrade their membership on demand from a UI component in their dashboard.”

const currentUser = require('../currentUser');
const modifySubscription = require('../../modifySubscription');

const tiers = ['individual', 'business', 'corporation'];

/*
 * Takes an HTTP GET on behalf of the currently authenticated user.
 *
 * Takes a param `newTier` and attempts to update the authenticated
 * user's subscription to that tier.
 */
app.get('/changeSubscriptionTier', function(req, res) {
 if (!currentUser.isAuthenticated) { return res.sendStatus(401); }
 if (!req.params.newTier) { return res.sendStatus(400); }
 if (!tiers.includes(req.params.newTier)) { return res.sendStatus(400); }

 modifySubscription(currentUser, req.params.newTier)
 .then(() => {
   return res.sendStatus(200);
 })
 .catch(() => {
   return res.sendStatus(400);
 });
});

Steve’s old friend Jed, who is constantly critiquing Steve’s code realizes that he can make a request like GET /api/changeSubscriptionTier with any tier as the newTier param and send it via hyperlink to Steve.

When Steve clicks this link, a request is made on his accounts behalf—changing the state of his subscription in his company’s application portal.

Jed has discovered a CSRF vulnerability in the application.

Luckily, although Steve is annoyed by Jed’s constant critiquing, Steve realizes the danger of this exploit and reports it back to his organization for triaging.

Once it is triaged, the solution is determined to be switching the request from a HTTP GET to a HTTP POST instead.

Not wanting to look bad in front of his friend Jed again, Steve writes a vulnerability regression test:

const tester = require('tester');
const requester = require('requester');

/*
 * Checks the HTTP Options of the `changeSubscriptionTier` endpoint.
 *
 * Fails if more than one verb is accepted, or the verb is not equal
 * to 'POST'.
 * Fails on timeout or unsuccessful options request.
 */
const testTierChange = function() {
 requester.options('http://app.com/api/changeSubscriptionTier')
  .on('response', function(res) {
   if (!res.headers) {
    return tester.fail();
   } else {
     const verbs = res.headers['Allow'].split(',');
     if (verbs.length > 1) { return tester.fail(); }
     if (verbs[0] !== 'POST') { return tester.fail(); }
   }
  })
  .on('error', function(err) {
    console.error(err);
    return tester.fail();
  })
};

This regression test looks similar to a functional test, and it is!

The difference between a functional test and a vulnerability test is not the framework but instead the purpose for which the test was written.

In this case, the resolution to the CSRF bug was determined to be that the endpoint should only accept HTTP POST requests.

The regression test written ensures that the endpoint changeSubscriptionTier only takes a single HTTP verb, and that verb is equal to POST.

If a change in the future introduces a non-POST version of that endpoint, or the fix is over-written—then this test will fail indicated the vulnerability has regressed.

Vulnerability regression tests are simple—sometimes they are so simple in fact that they can be written prior to a vulnerability being introduced.

This can be useful in the case of code where minor insignificant looking changes could have big impact.

Ultimately, vulnerability regression testing is a simple and effective way of preventing vulnerabilities that have already been closed from re-entering your codebase.

The tests themselves should be run on commit or push hooks when possible (reject the commit or push if the tests fail). Regular scheduled runs (aka daily) are the second best choice for more complex version control environments.