This chapter covers

• How immutability solves security problems
• How fail-fast contracts secure your design
• Types of validation and the order in which to do them

As developers, we’re constantly reminded about priorities and deadlines. Cutting corners and dirty hacks are sometimes part of reality that we must accept—or are they? The truth is, at the end of the day, you decide what syntax to use, what algorithms to apply, and how to steer the flow of execution. If you truly understand why certain code constructs are better than others, then using them becomes second nature and no more time-consuming than writing bad code. The same applies to security. Attackers don’t care about deadlines or priorities—a weak system is exploitable, regardless of why or under what circumstances it was built.

We all share the responsibility of designing secure software. In this chapter, you’ll learn why that doesn’t take any longer than building weak, exploitable software. To this end, we’ve organized this chapter into three sections, each discussing different strategies to solve security problems you might encounter in your daily work (table 4.1).

This way, we hope to empower you with a new set of tools, mindset, and best practices to use in your daily work. You’ll also learn how to spot weaknesses in legacy code and how to address them. As a result, you’ll see why security bugs are just bugs and how good design prevents them. We’ll start by dealing with change using immutability and provide an example for this.

4.1 Immutability

When designing an object, you need to decide whether it should be mutable or immutable. Mutability allows state to change; immutability prevents it. This might seem of minor importance, but from a security perspective, it makes a big difference. Immutable objects are safe to share between threads and open up high data availability—an important aspect when protecting a system against denial of service attacks. Mutable objects, on the other hand, are designed for change, which can lead to illegal updates and modifications. Often, mutability is introduced because frameworks require it or because it seems easier. But choosing mutability over immutability can be as expensive as it is dangerous. To illustrate, we’ll walk through an example where the mutable design of a webshop causes security problems that are easily solved using immutability.

4.1.1 An ordinary webshop

Picture an ordinary webshop where customers log in and add items to a shopping cart. Each customer has an associated credit score based on purchase history and membership points. A high credit score allows paying by invoice or credit card, whereas a low credit score allows only credit card payments. The credit score computation is quite expensive and is done continuously to even out the overall system load.

All in all, the system worked fine—until recently. When the last marketing campaign ran, lots of traffic hit the webshop. Then the system didn’t handle the load well, and customers complained about orders timing out, long waits, and inconsistent payment alternatives. The latter seemed a minor problem, but when the Finance Department reported that lots of customers with low credit scores had outstanding invoices, a full-blown security investigation started—the system must have been compromised! Obviously, the credit score computation was the primary suspect, but to everyone’s surprise, the root cause turned out to be a much bigger problem: the design of the Customer object.

The design of the Customer object

The Customer object in listing 4.1 shows two interesting details. First, all fields are initialized through setter methods, which implies that internal state is allowed to change after object creation. This is problematic because you never know when the object is properly initialized. The other observation is that each method is marked as synchronized to prevent concurrent field modification, which in turn can lead to thread contention (when threads are forced to wait for another thread to release one or more locks before executing).

Listing 4.1 Mutable Customer class

public class Customer {
   private static final int MIN_INVOICE_SCORE = 500;  ①  
   private Id id;    ②  
   private Name name;    ③  
   private Order order;    ④  
   private CreditScore creditScore;    ⑤  

   public synchronized Id getId() {
      return id;
   }

   public synchronized void setId(final Id id) {
      this.id = id;
   }

   public synchronized Name getName() {
      return name;
   }

   public synchronized void setName(Name name) {
      this.name = name;
   }

   public synchronized Order getOrder() {
      this.order = OrderService.fetchLatestOrder(id);
      return order;
   }

   public synchronized void setOrder(Order order) {
      this.order = order;
   }

   public synchronized CreditScore getCreditScore() {
      return creditScore;
   }

   public synchronized void setCreditScore(CreditScore creditScore){
      this.creditScore = creditScore;    ⑥  
   }

   public synchronized boolean isAcceptedForInvoicePayment() {
      return creditScore.compute() >
                             MIN_INVOICE_SCORE;    ⑦  
   }
   ...
}

How these design choices relate to security isn’t obvious, but when categorizing the webshop problems as data integrity or data availability issues, the correlation becomes clear. Let’s discuss those issues next.

Categorizing problems as integrity or availability issues

Data integrity involves the consistency of data during its entire life cycle; data availability ensures data is obtainable at the expected level of performance in a system.¹  Both concepts are essential to understanding the root cause of the webshop problems. For example, failure to retrieve data is an availability problem that often boils down to code that prevents parallel or concurrent access. Similarly, identifying code that allows modification is the place to start when analyzing integrity issues. Table 4.2 shows the webshop problems categorized as availability and integrity issues.

These categories give you an idea of what to look for in the Customer class. Let’s start by dealing with how implicit blocking can reduce availability.

Implicit blocking yields reduced availability

Choosing to allow or disallow concurrent and parallel access is often a balance between performance and consistency. If state always needs to be consistent and updates interleaved with read operations, then using a locking mechanism makes sense. But if access is mostly reads, then locking can result in unnecessary thread contention. For some reason, contention caused by concurrent access is often easier to reason about in code than contention caused by parallel access. For example, if a method is synchronized, as in listing 4.1, only one thread at a time is allowed to access the method, because the intrinsic lock of the method’s object must be acquired.²  All other threads that try to access the method concurrently must then wait until the lock is released—and this could cause thread contention.

Using synchronized on a method level can also yield thread contention during parallel access of two or more methods. As it turns out, the intrinsic lock acquired for a synchronized method is the same for all synchronized methods in an object. This means that threads accessing synchronized methods in parallel implicitly block each other, and this kind of contention can be hard to recognize.

If we go back to the webshop and analyze the ratio between read and write operations, it turns out that reading customer data is far more common than updating it. This is because data is primarily changed by the credit score algorithm, and reads are made by numerous client requests, including Finance’s reporting system. This gives you a hint that most of the time parallel and concurrent reads are safe, so why didn’t we remove the locking mechanism (synchronized) altogether?

Although it’s likely that parallel and concurrent reads are safe, you can’t ignore writes and remove the locking mechanism to minimize contention. Instead, other solutions must be considered. One is to use advanced locking mechanisms, such as a ReadWriteLock, which respects the read dominance.³  But locking mechanisms add complexity and cognitive load, which is something we’d prefer to avoid.

A simpler and better strategy is to use a design that favors parallel and concurrent access (for example, immutability). In listing 4.2, you see an immutable version of the Customer class that doesn’t allow state to change. This means it’s safe to share Customer objects between threads without using locks, and this yields high availability with low contention. In other words, no locking, no blocking.

Listing 4.2 Immutable Customer class

import static org.apache.commons.lang3.Validate.notNull;

public final class Customer {
   private final Id id;    ①  
   private final Name name;    ②  
   private final CreditScore creditScore;    ③  

   public Customer(final Id id, final Name name,
                   final CreditScore creditScore) {
      this.id = notNull(id);
      this.name = notNull(name);
      this.creditScore = notNull(creditScore);
   }

   public Id id() {
      return id;
   }

   public Name name() {
      return name;
   }

   public Order order() {
      return OrderService.fetchLatestOrder(id);
   }

   public boolean isAcceptedForInvoicePayment() {
      return creditScore.isAcceptedForInvoicePayment();
   }
}

But you still need to be able to change customer data. How do you do this if Customer is immutable? As it turns out, you don’t need mutable data structures to support change. What you need is to separate reads from writes and perform updates through channels other than those used when reading. This might seem overly complex, but if your system has an imbalance between reads and writes, it can be worth it. How to achieve this in practice is covered in chapter 7, where we discuss the Entity snapshot pattern in depth.

You’ve learned how immutability prevents availability issues by design, but what about the integrity issue in the webshop? Does immutability solve that too? Perhaps. Let’s see how the mutable design of Customer and CreditScore opens up the integrity issue.

Changing credit score, an integrity issue

Before diving into the analysis, let’s quickly recap the credit score problem. Each customer has an associated credit score, where a high value allows paying by invoice. During the last campaign, the system failed, and the Finance Department reported that lots of customers with low credit scores had outstanding invoices—a data integrity issue where credit scores changed to favor invoice payment. But how was this possible? Looking at the credit score logic of the mutable Customer object in listing 4.3, we see that

• The way creditScore is initialized in Customer opens up the possibility of changing the credit score at any moment.
• The creditScore reference is accidentally escaped in the getCreditScore method, which allows modification outside of Customer.
• The setCreditScore method doesn’t make a copy of the argument. This makes it possible to inject a shared creditScore reference.

Listing 4.3 Credit score logic in the mutable Customer object

public class Customer {
  private static final int MIN_INVOICE_SCORE = 500;
  private CreditScore creditScore;
  ...
  public synchronized void setCreditScore(
                         CreditScore creditScore) {  ①  
     this.creditScore = creditScore;    ②  
  }

  public synchronized CreditScore getCreditScore() {
     return creditScore;    ③  
  }

  public synchronized boolean isAcceptedForInvoicePayment() {
     return creditScore.compute() > MIN_INVOICE_SCORE;
  }
  ...
}

Let’s look at each one of these observations to see how they can cause a data integrity issue.

The first way to cause this sort of data integrity issue involves explicitly changing the credit score through initialization. The creditScore field in Customer is initialized through the setCreditScore method. Although done by design, this way of initializing the field allows changing a customer’s credit score at any moment, because the method doesn’t guarantee an invoke at-most-once behavior. This can seem acceptable because a client is only expected to read data, but the mutable design of Customer makes it impossible to prevent anyone from accidentally using the mutable API. This means you can’t guarantee the integrity of a Customer object.

The second way involves changing the credit score outside of Customer. If you look at the getCreditScore method in Customer, you’ll see that the internal creditScore field is accidentally escaped. This makes it possible to modify the credit score outside of the Customer object without acquiring the lock. This is extremely dangerous because Customer is a shared mutable object and updating it without synchronization is a disaster waiting to happen. (We’ll talk more about this in chapter 6.) But changing the credit score without a lock is one thing.

Due to the mutable design of CreditScore, it’s also possible to explicitly change the associated customer ID by invoking the setCustomerId method, as shown in listing 4.4. This implies that a Customer object can have one ID and the CreditScore object another—a dissociation that could yield an incorrect credit score value in the compute method!

Listing 4.4 CreditScore class

public class CreditScore {
   private Id id;

   public synchronized void setCustomerId(Id id) { 
      this.id = id;  ①  
   }

   public synchronized int compute() {
      List<Record> history =
          BillingService.fetchBillingHistory(id);    ②  
      Membership membership =
          MembershipService.fetchMembership(id);    ③  
      return CreditScoreEngine.compute(id, history,    ④  
                                       membership);
   }
   ...
}

To address this, the mutable design of CreditScore needs to be changed. In listing 4.5, you see an immutable version of the same object. What’s interesting is that the synchronized keyword and the dependency to the customer ID have been removed. This is because there’s no longer a need to acquire a lock when checking the credit score value, as it can’t change after being assigned in the constructor. This, in turn, means that the dependency to a specific customer is redundant, and the design can be simplified by moving the credit score computation outside of the object. As a result, this allows the credit score to be shared between threads without the risk of illegal updates, blocking, or locking.

Listing 4.5 Immutable CreditScore class

import static org.apache.commons.lang3.Validate.isTrue;

public class CreditScore {
   private static final int MIN_INVOICE_SCORE = 500;
   private final int score;

   public CreditScore(final int computedCreditScore) {
      isTrue(computedCreditScore > -1, "Credit score must be > -1");
      this.score = computedCreditScore;    ①  
   }

   public boolean isAcceptedForInvoicePayment() {
      return score > MIN_INVOICE_SCORE;
   }
   ...
}

The third way to change creditScore isn’t as obvious as the other two—it involves changing a shared credit score reference. If you look at the setCreditScore method in the mutable Customer object, you’ll see that the internal field is assigned the external mutable creditScore reference. This is fine as long as the external reference isn’t reused in another Customer object. But if it is, the computed credit score value will be the same for all customers sharing that reference—a major integrity issue that could explain the inconsistent payment alternatives in the webshop.

Identifying the root cause

All the scenarios we’ve explored are plausible explanations of the data integrity problem seen in the webshop, but which one caused it? Well, it doesn’t matter. The key point is that the decision to make Customer and CreditScore mutable made the code less secure in various ways. But when choosing a design that favors immutability, the need for locks and protection against accidental change disappears. Such a design yields implicit security benefits.

You’ve now learned how immutability solves data integrity and availability issues. You might have noticed how on some occasions we aggressively blocked bad data before it had a chance to establish itself in the object. Doing so is also an effective trick to uphold security, so let’s move on to how to fail fast using contracts.

4.2 Failing fast using contracts

As mentioned in chapter 1, a guiding principle for security is security in depth. Even if one security mechanism at the border fails, there are more mechanisms in place that can stop a breach from continuing or spreading. With physical security, the access card a visitor uses to gain entry to a building might also be a badge with a photo that they’re required to wear at all times. We’ve found the software design practices of preconditions and Design by Contract to be effective in making software more secure in a similar way. In this section, we show the pragmatic coding practices we’ve come to use, which you’ll see plenty of in the examples throughout the book.

Talking about preconditions was part of the theoretical studies of computer science in the late 1960s, especially by Sir C. A. R. Hoare (who also takes the blame for inventing the null-pointer exception).⁴  The term Design by Contract was coined in the late 1980s by Bertrand Meyer, who used it as a foundation for object orientation.⁵  Apart from being theoretically interesting, these ideas have direct practical benefits for security.

Thinking about design in terms of preconditions and contracts helps you clarify which part of a design takes on which responsibility. Many security problems arise because one part of the system assumes another part takes responsibility for something when, in fact, that part assumes the opposite. In this section, we’ll walk you through how to use contracts, preconditions, and failing fast to avoid those situations in practice.

How does Design by Contract work, and what do we mean by a contract? Let’s start with a nonsoftware example. Imagine you contract a plumber to fix the broken sink in your bathroom. The plumber might require that the door be unlocked and that the water shutoff valve be closed. These are the preconditions for the work. If they aren’t fulfilled, things might not go well. On the other hand, the plumber promises that after the work is finished, the bathroom sink will function properly. This is the postcondition of the contract.

Design contracts for objects work the same way. A contract specifies the preconditions that are required for the method to work as intended, and it specifies the postconditions for how the object will have changed after the method is completed. In listing 4.6, you see a class that’s part of the support system for a cat breeder. This particular class, CatNameList, helps keep track of cat names that can be given to new kittens. When the cat breeders come up with a new idea for a cat name, they queue it up using queueCatName. When they need a good cat name, they look up which is next in turn using nextCatName, and if they decide to use it, they remove the name with dequeueCatName. The method size tells them how many names are in the list.

Listing 4.6 CatNameList that keeps track of good names for cats

public class CatNameList {
    private final List<String> catNames = new ArrayList<String>();

    public void queueCatName(String name) {    ①  
        catNames.add(name);
    }

    public String nextCatName() {    ②  
        return catNames.get(0);
    }

    public void dequeueCatName() {    ③  
        catNames.remove(0);
    }

    public int size() {    ④  
        return catNames.size();
    }
} 

In the contract for this class, there are some pre- and postconditions, as listed in table 4.3.

The method queueCatName has some peculiar preconditions. It makes sense that a cat name can’t be null; it must be something, otherwise, this design doesn’t work properly. According to Swedish folklore, a good cat name must have an s sound in it (that makes the cat listen to it), so that’s also a precondition. Finally, cat breeders don’t want the same name to appear twice in the list, so this contract requires as a precondition that the new name isn’t already in the list.

The contract could have been written in another way. For example, the cat name list could have taken on the burden of avoiding duplicates. In that case, the pre- and postconditions would have been stated differently, as shown in table 4.4.

Note how this contract is about the entire class, not just one method at a time. The contract requires that the name sent to queueCatName contains an s and, at the same time, promises that the name returned from nextCatName indeed contains an s. This contract is about the responsibility of the class as a whole. The important aspect is that the contract points out the intended design. The task of avoiding duplicates has to fall on either CatNameList or the caller. Stating the contract makes it clear on whom the responsibility falls.

Many security problems arise from situations when one part of the system assumes that another part takes responsibility for something when, in actuality, it doesn’t—it assumes the callers would do that. Explicitly thinking in terms of Design by Contract avoids lots of those situations that give rise to vulnerabilities.

What should you do if preconditions aren’t fulfilled? Let’s look back to the situation of the plumber who comes to fix the broken sink, and who requires a few preconditions (door open, water turned off). If the door is locked, the plumber won’t be able to gain access, and if the water is still on, it’s not a good idea to start working. If work begins on the pipes with the shutoff valve still open, things might go seriously wrong. The better option is to fail fast —to terminate the job as soon as it becomes clear that the preconditions aren’t met.⁶  This is where the real security benefit comes in.

To make contracts powerful, the rules have to be enforced in code. The programming language Eiffel, designed by Bertrand Meyer, has support for this built into the language itself. As a programmer, you state the preconditions, postconditions, and invariants, and the runtime platform ensures they are checked. As you probably use other programming languages, you’ll need to build these checks into the code yourself. Let’s dig into some of those code patterns to help you create more secure software.

4.2.1 Checking preconditions for method arguments

In the contract for CatNameList, there are some restrictions on the names queued up by queueCatName: it’s not allowed to send in null, and the name must contain an s. If the caller doesn’t adhere to the contract, things will probably break sooner or later. If you do nothing in the implementation to uphold the contract, then names without an s could be queued, and much later, you’d end up with cats named Fido, Doggy, or Bonnie. To promote security, we advise to fail fast in a controlled manner instead of letting things break later, uncontrolled.

To fail fast, you check the preconditions at the beginning of the method before you do anything else, and fail hard if they aren’t met. If the preconditions aren’t met, then the program isn’t using its classes in a way they were designed to be used. The program has lost control of what’s happening, and the safest thing to do is to stop as fast as possible. This is nothing complicated; you can implement it yourself with an if statement that throws an exception along the lines of the following listing.

Listing 4.7 Enforcing fail fast of contract preconditions

public void queueCatName(String name) {
    if (name == null)    ①  
        throw new NullPointerException();
    if (!name.matches(".*s.*"))    ②  
        throw new IllegalArgumentException("Must contain s");
    if (catNames.contains(name))    ③  
        throw new IllegalArgumentException("Already queued");
    catNames.add(name);
}

This code is arguably a little verbose, but it does the trick. It stops if the cat names are missed completely (null) or bad cat names and prevents already existing cat names from entering the list. And it does so using an aggressive fast-fail approach.

In Java, if you encounter a null when there shouldn’t be one, you throw a NullPointerException.⁷  In the other cases, you throw an IllegalArgumentException to mark that the method call itself was valid but a specific argument couldn’t be accepted.

For brevity, we’ve found it useful to use the Validate utility class from Apache’s Commons Lang framework.⁸  It contains several useful helper methods that do exactly what we want: check a condition and throw an appropriate exception if the condition is false. Using the Validate.notNull, Validate.matchesPattern, and Validate.isTrue methods gives you the code in the following listing.

Listing 4.8 Using Validate to enforce preconditions

import org.apache.commons.lang3.Validate.*;
...
public void queueCatName(String name) {
    notNull(name);    ①  
    matchesPattern(name,".*s.*",
                   "Cat name must contain s");    ②  
    isTrue(!catNames.contains(name),
           "Cat name already queued");    ③  
    catNames.add(name);
}

Validate contains a lot of other helpful methods, like exclusiveBetween and inclusiveBetween to check if a value is within a range of valid values. Look through the documentation to see what’s there; it makes it much easier to check preconditions in your code. For brevity, we’ll use the framework in the examples in the rest of the book.

Checking preconditions is pretty easy, but thinking through contracts takes more work. When is it worth the effort? In our experience, formulating contracts and checking preconditions for public methods is definitely worth it. This goes for simple checks like checking for null, that numbers are within the expected range, and similar. For package internal methods, it’s a judgement call. If the package is large and the class has a lot of usage, it’s probably a good idea, but for a helper class with a few uses, we’d skip it. As for private methods, it’s not worth the effort. If the class needs internal contracts, it’s probably too big, does too many things, and should be split instead.

Now that you’ve looked at checking arguments for methods, let’s proceed to the similar subject of arguments for constructors. We’ll also talk about one more thing that Meyer mentioned in his theory, apart from preconditions and postconditions: invariants, which are things that always should be true for an object. Invariants must initially be upheld by the constructor on object creation.

4.2.2 Upholding invariants in constructors

Arguments to constructors add a bit of complexity compared to arguments for methods. The constructor arguments aren’t only primarily to process or to modify the state of the object but to create the object in its initial state. Because an object state can be created in one part of an application and used in a completely different part, an invalid state can produce hard-to-trace bugs that can be a breeding ground for security vulnerabilities. Better to fail fast than let that happen.

Enforcing the contract for a constructor could be as simple as saying one of the fields is mandatory and then checking that the value isn’t null. Here, you can interpret the word mandatory as meaning the contract of this object contains an invariant that states that this field must always be set to an object, never null. By adding a non-null check, you’re enforcing that invariant. Invariants can be a lot more complicated, but we’ll cover that in more depth in chapter 6 when we discuss securing mutable state.

In the next listing, you see the constructor of the class Cat. The contract states that as invariants, neither name nor sex is unspecified, so you must check that both fields are non-null. We also included the now familiar check that cat names must contain an s.

Listing 4.9 Enforcing contract for constructor

import org.apache.commons.lang3.Validate.*;

enum Sex {MALE, FEMALE;}

public class Cat {

    private String name;
    private final Sex sex;

    public Cat(String name, Sex sex) {
        notNull(name);    ①  
        matchesPattern(name,".*s.*",
                       "Cat name must contain s");    ②  
        notNull(sex);    ③  
        this.name = name;
        this.sex = sex;
    }
    ...
}

Due to a feature in Validate, this constructor can be compressed. The method notNull not only validates but also returns the validated object. Using this feature, your Cat constructor becomes even briefer, as seen in the following listing.

Listing 4.10 Enforcing contract for constructor, condensed version

public Cat(String name, Sex sex) {
    this.name = notNull(name);    ①  
    this.sex = notNull(sex);    ①  
    matchesPattern(name,".*s.*",
                   "Cat name must contain s");    ②  
}

Although most Validate methods return the validated value, there are a few that don’t. The pattern validator matchesPattern is one of these and, therefore, can’t use the condensed form but needs to stand on a line of its own. You’ll still have the desired behavior, where null results in a NullPointerException, whereas a malformed cat name yields an IllegalArgumentException.

You’ve probably noted that there are now two places in the code that do the same format check for cat names. This is a flagrant violation of the DRY (Don’t Repeat Yourself) principle, which states that the same idea should be represented only once.⁹  In chapter 5, we’ll show our preferred way of solving these kinds of issues by introducing a domain primitive that, in this case, takes the form of a class CatName.

Now that you’ve learned how to ensure that methods get appropriate arguments and that objects are created with constructors that uphold the invariants, it’s time to move on to the last kind of contract check we advise: preconditions checking that an object is in an appropriate state.

4.2.3 Failing for bad state

Finally, you want to ensure that preconditions that require an object to be in a certain state for an operation to be valid are met. For example, if the list of cat names is empty, it doesn’t make sense to look at the next name in the queue. It’s an operation that doesn’t fit the state of the CatNameList. If you remember from table 4.3, for nextCatName and dequeueCatName, the contract requires that CatNameList must contain something.

The obvious way to check this is to use Validate to ensure that the list in catNames isn’t empty. But the helper method Validate.isTrue doesn’t do the job well in this case. On failure, that method throws an IllegalArgumentException, which would be confusing for someone who had called the no-arg method nextCatName. Fortunately, for this situation, there’s Validate.validState, which you see used in the code for nextCatName in the following listing.

Listing 4.11 Enforcing state when looking at next name

public String nextCatName() {
    validState(!catNames.isEmpty());    ①  
    return catNames.get(0);
}

The next listing shows the CatNameList class in its final form, with preconditions that protect the class from bad usage and fast failure if data is received that would, intentionally or by mistake, invalidate the contract.

Listing 4.12 CatNameList with contract enforced through fast fails

import org.apache.commons.lang3.Validate.*;

public class CatNameList {
    private final List<String> catNames = new ArrayList<String>();

    public void queueCatName(String name) {    ①  
        notNull(name);    ②  
        matchesPattern(name,".*s.*",
                       "Cat name must contain s");    ③  
        isTrue(!catNames.contains(name),
               "Cat name already queued");    ④  
        catNames.add(name);
    }

    public String nextCatName() {
        validState(!catNames.isEmpty());    ⑤  
        return catNames.get(0);
    }

    public void dequeueCatName() {
        validState(!catNames.isEmpty());    ⑤  
        catNames.remove(0);
    }

    public int size() {    ①  
        return catNames.size();
    }
}

You can see that the class has grown with five lines of code, and it has become much more secure. Invalid argument data is stopped early, and the object can never be in an invalid state. The main effort goes into thinking through the design and phrasing it as a contract, but that’s work that needs to be done anyway.

Failing fast is a small code construct that promotes security. It comes in handy in the larger scheme of validating that data is sound and safe to use. In the next section, we’ll dig deeper into the different levels of validation.

4.3 Validation

To keep a system secure, it’s important to validate data. OWASP stresses input validation (validating data when it enters the system).¹⁰  This seems like an obvious and sound thing to do, but unfortunately it’s not that simple. There’s the challenge of structuring the code to ensure input is validated everywhere. There’s also the challenge of what constitutes valid data. What is valid or not varies from situation to situation.

Asking whether a particular value is valid or not is meaningless. Is 42 valid input? Is -1 or <script>install(keylogger)</script> valid? That depends on the situation. When ordering books, 42 is probably a valid input as a quantity, but -1 isn’t. On the other hand, when reporting temperatures, -1 is certainly sensible. The script string is probably not what you want in most circumstances, but on a site where you report security bugs, it definitely fits.

Validation is also a word that tends to mean lots of different things. One person might claim that AV56734T is a valid order number because it follows the format for order numbers the system uses. Someone else might claim that AV56734T isn’t a valid order number because there’s no order in the system with that number. Yet someone else might claim that it’s not valid even though there’s an order with that number because they tried to clear the order for shipping and, at that specific time, it wasn’t possible to do so. Obviously, there are many different kinds of validation.

You have probably come across the security advice “validate your input.” But with the confusion about validation in mind, this piece of advice is as helpful as “when driving a car, avoid crashing.” It’s certainly well meant, but it’s not helpful.

To clear up this confusion, we’ll walk through a framework that tries to separate the different kinds of validation. The list presented here also suggests a good order in which to do the different kinds of validation. Cheap operations like checking the length of data come early in the list, and more expensive operations that require calling the database come later. In this way, the early controls shield the more expensive and advanced controls.¹¹  You’ll want to perform the following types of validation, preferably in this order:

• Origin —Is the data from a legitimate sender?
• Size —Is it reasonably big?
• Lexical content —Does it contain the right characters and encoding?
• Syntax —Is the format right?
• Semantics —Does the data make sense?

As mentioned, checking for origin or checking the size can be done cheaply and quickly. Checking the lexical content requires a scan, which takes a bit more time and resources. Checking syntax might require parsing the data, something that consumes CPU resources and occupies a thread for quite some time. And checking if the data makes sense probably involves a heavy round-trip to the database. In this way, the earlier checks can prevent the more expensive later checks from being performed unnecessarily. We’ll take a look at these different types of validation one-by-one and see how they fit into a secure design. We’ll start with checking where data comes from.

4.3.1 Checking the origin of data

It certainly makes sense to check where data comes from before spending any effort on handling it. The reason for this is that many attacks are asymmetric in favor of the attacker. Asymmetric means that the effort required of the attacker to send malicious data is far less than the effort required of your system to handle that data. This is the basic logic of a denial of service (DoS) attack: the system is sent so much trash data that it becomes preoccupied with handling it, so no resources are left for the system to do its proper work.

A popular version is the distributed DoS (DDoS) attack, where lots of geographically distributed malicious clients send messages to the system at the same time. The clients of DDoS attacks are often botnets made up of computers that someone has infected with a remote-control function that’s silent until it receives a message from its master. Such botnets can be bought or rented on the less respectable street corners of the web.¹² 

Unfortunately checking the origin of data doesn’t help in all cases, but checking data’s origin is a first simple step to change the asymmetry in your favor. If the data comes from a legitimate origin, then continue. If not, discard it. Roughly, there are two mechanisms you can use: checking which IP address the data comes from or requiring an API access key. The simplest thing to do is check the IP address, so we’ll start there.

Checking the IP address

The most obvious way to check the origin of data is to check the IP address it’s sent from. The approach is somewhat losing its applicability as things become more and more connected, but it still has some merits.

If you have a microservice architecture, then some of the services will sit on the edge, accepting outside traffic, but others sit in the inside and only expect calls from other services. The services in the inside can be restricted to only accept traffic from a specific IP range (of the other services). For the edge services, this is probably not doable, as they possibly need to be accessible from any client.

In practice, this kind of check is nothing you do from inside your application. Instead, access is restricted by network configuration. In a physical hall, this is set up in the physical routers. On a cloud service, such as Amazon Web Services (AWS), you can set up a security group that only allows incoming traffic from a specific range or list of IP addresses. If your system is used inside a company, such as a point-of-sale system, it might also be possible to do origin checks through IP ranges. For example, if servers should only be contacted by the point-of-sale terminals or the office desktops inside the company network, you can lock down access to those IP ranges. Unfortunately, in this more-and-more connected world, such situations are becoming increasingly uncommon.

Also, be aware that IP filters, MAC address filters, and similar provide no guarantee. MAC addresses can be changed from the operating system, IP addresses can be spoofed (or real machines taken over). But at least they provide some sort of first line-of-defense.

Using an access key

If your system is open for requests from many places, you can’t check by IP origin. For example, if your clients could be anywhere on the internet, then all IP addresses are legitimate senders. This is the case for almost all public services facing customers. Sorting out attackers based on IP origin is hard or impossible in these cases. Thankfully, there’s one more way to restrict who’s allowed to contact your system: requiring an access key. Such a key can be given to all legitimate clients. If the client is another system, it can be given to that system on deploy. If the clients are applications, the key can be built into the application. If the clients access an API, the key can be given to them when they sign some kind of end-user agreement. The point is that the holder of an access key proves it’s allowed to send data to your system.

Taking AWS as an example, it has a REST API where you can manage your AWS resources, such as the S3 cloud data storage.¹³  When sending an HTTP request to this API, you need to provide your access key as part of the HTTP Authorization header, as shown in listing 4.13. If there’s no such header, the request is denied. It isn’t enough to just send the access key, however. A malicious attacker might be able to spoof your access key and could, thereafter, use it to impersonate you. To stop this, the Authorization header also contains a signature where the message content has been signed using your secret key, which is a secret shared by you and AWS alone. AWS can determine which secret key to use for checking the signature by looking at the access key.

Listing 4.13 Calling the REST API of AWS using access key and signature¹⁴ 

GET /photos/puppy.jpg HTTP/1.1
Host: johnsmith.s3.amazonaws.com
Date: Mon, 26 Mar 2007 19:37:58 +0000

Authorization: AWS AKIAIOSFODNN7EXAMPLE:frJIUN8DYpKDtOLCwo//yllqDzg=

In this case, you need to invest in some computation on the server side, as you need to check that the signature of the message matches. It’s not a particularly expensive operation, but you’re opening yourself up a little to attacks where someone can send you lots of data and fool you into processing it. The good part is that an attacker needs a real access key, and if they use that maliciously, you can blacklist them for a limited time. The drawback of using access keys is that it’s extra work. Fortunately, it’s something that typically only needs to be done once and might be provided if you use an API gateway or similar products. If you do the work yourself, remember to do appropriate validation of the access key as well (length, charset, format) so that it doesn’t become an attack vector.

Having checked the origin of the data, you can now safely spend some effort processing it. The first step is to check that the data is of a reasonable size.

4.3.2 Checking the size of data

Does it make sense to accept an order number that’s 1 GB of data? Probably not. If you’ve checked that the data comes from a legitimate origin, is the data of a reasonable size? Ideally, this check should be done as early as possible.

What is a reasonable size? That depends fully on the context. For example, an upload to a video site could easily be in the range of 100 MB to a few GB without anything being strange. On the other hand, a business application based on simple JSON or XML messages might not have messages bigger than some KB. What is a reasonable size for your data?

If the data comes over HTTP, you can check the Content-Length header and the size of the data.¹⁵  If you run a batch process, you can check the file size.

Apart from checking the size of the data as a whole, it’s also useful to check the size of each part when processing it. An attacker might hide a 1 GB order number inside a request. You might forget to check the size of each particular request, or the total size might be within reasonable limits, but if you also check the size of order numbers, there’ll be a defense mechanism every time you create an order number.

Hopefully this seems like sound design to you. If that’s the case, then you get a lot of security for free. If you do this trick consistently for order number, phone number, street name, zip code, and so forth, then it’s a lot harder for an attacker to find a place to insert big data. You have added protection that no part of the data can have a bizarre size.

For example, if you run an online bookstore, you might get batch files or HTTP requests regarding orders of books. These books are most probably identified using an International Standard Book Number (ISBN), which is a nine-digit number with an additional check digit. Checking that the entire order file or HTTP request is of reasonable size is good, but when you single out a part that should be an ISBN, you also want to check that part specifically. The following listing shows a class that represents an ISBN and validates its length to stop DoS attacks.

Listing 4.14 ISBN class containing a check that the number is short enough

import org.apache.commons.lang3.Validate.*;

public class ISBN {
    private final String isbn;

    public ISBN(final String isbn) {
        notNull(isbn);    ①  
        inclusiveBetween(10, 10,isbn.length());    ②  
        this.isbn = isbn;    ③  
    }
}

Checking the length of the string might seem redundant in some cases. In later steps, we often check content and structure using a regular expression (often called a regexp). Regexps might look like any of the following:

• [a-z]—a single character between a and z
• [A-Z]{4}—four letters, each between A and Z
• [1-9]*—digits between 1 and 9, repeated any number of times

If the next step is a format control matching the regexp [0-9]{20} (any digit between 0 and 9, repeated exactly 20 times), why should you check the length separately? A 25-char string won’t match the string anyway, so why check twice? The point here is that the length check protects the regexp engine. What happens if the string isn’t 25 chars but 1 billion? Most probably the regexp engine will load and start processing that humongous string, not realizing it’s too big. An early length check protects the latter stages.

Once you’ve checked that the input data is of reasonable size, it’s time to look inside the data. The first check is that the data contains the expected type of content, such as the expected characters and encoding, which is called the lexical content.

4.3.3 Checking lexical content of data

Knowing that the data comes from a trustworthy source and has a reasonable size, you can dare to start looking at the content. Most probably, sooner or later you’ll need to parse the data to extract the interesting parts (for example, if you receive the data in JSON or XML format). But parsing is expensive, both in terms of CPU and memory, so before you start parsing, you should do some further checks.

When checking the lexical content of data, we look at the content but not the structure. We scan through the data to see that it contains the expected characters and the expected encoding. If we encounter something suspicious, we discard the data instead of starting a parsing process that could bring our servers to their knees.

For example, if we expect data that only contains digits, we scan the stream to see if there’s anything else. If we find anything else, we draw the conclusion that the data is either broken by mistake or has been maliciously crafted to fool our system. In either case, the safe bet is to discard the data. Similarly, if we expect plain text that’s HTML-encoded, we scan the data to see that it only contains the expected content. In this case, each < should be encoded as <, so we shouldn’t encounter any brackets at all. If we come across a bracket, we get wary. Might it be someone trying to sneak in a JavaScript snippet? To be on the safe side, we discard the data.

Lexical content is how the data is supposed to look when you look at it up close, focusing on the details and not paying attention to larger structures. Simple regexps are a great way of checking lexical content in many simple situations. For example, the ISBN-10 formats for books can only contain digits and the letter X.¹⁶  An ISBN can also contain hyphens and spaces for readability, but for simplicity, we’ll ignore them. The following listing shows the ISBN class from earlier, now refined to check that an ISBN only contains the expected characters.

Listing 4.15 ISBN class with a regexp to control lexical content

import org.apache.commons.lang3.Validate.*;

public class ISBN {
    private final String isbn;

    public ISBN(final String isbn) {
        notNull(isbn);
        inclusiveBetween(10, 10,isbn.length());    ①  
        isTrue(isbn.matches("[0-9X]*"));    ②  
        this.isbn = isbn;
    }
}

If your input data is more complicated, for example XML, you might want to use a more powerful lexer. A lexer (or tokenizer) splits a sequence of characters into parts called lexemes or tokens. These can be defined as the smallest part with a meaning or the sequence of characters that forms a syntactic unit. In written English, words are considered to be the tokens. In XML, tokens are the tags and the content within them. The following two listings show an XML document and its tokens.

Listing 4.16 XML document with book information

<book>
   <title>Secure by Design</title>
   <authors>
      <author>Dan Bergh Johnsson</author>
      <author>Daniel Deogun</author>
      <author>Daniel Sawano</author>
   </authors>
</book>

Listing 4.17 Tokens found in book XML document, one token per row

<book>    ①  
<title>    ②  
S    ③  
e
c
u
r
e
...

In some situations, you might not want the full power of XML and might want to restrict it. In chapter 1, we presented an example of how dangerous it can be to allow XML entities in input. The reasonably short XML file you see in the following listing is less than 1,000 characters long. Still, it expands to a billion lol strings, something that will most probably break the poor XML parser.

Listing 4.18 XML expanding to a billion lols

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<!DOCTYPE lolz [
<!ELEMENT lolz (#PCDATA)>
<!ENTITY lol "lol">
<!ENTITY lol1 "&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;&lol;">
<!ENTITY lol2 "&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;&lol1;">
<!ENTITY lol3 "&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;&lol2;">
<!ENTITY lol4 "&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;&lol3;">
<!ENTITY lol5 "&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;&lol4;">
<!ENTITY lol6 "&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;&lol5;">
<!ENTITY lol7 "&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;&lol6;">
<!ENTITY lol8 "&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;&lol7;">
<!ENTITY lol9 "&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;&lol8;">
]>
<lolz>&lol9;</lolz>

Needless to say, a billion lol strings will make your parser fall off a cliff and die, so you can’t allow this to pass. One way of avoiding this is to disallow the use of XML entity definitions by having a lexer that scans the data, not recognizing <!ENTITY as a legitimate lexeme. Leaf back to chapter 1, section 1.5, if you want to review the details.

In practice, when working with XML documents, you’ll most probably parse them to pick out the interesting parts. You don’t want to do two parsings, one for checking content and one for parsing the structure. One way of avoiding this is to interleave the lexical check inside the parsing. The way to do this is to run the lexical check as part of stream processing, which was exactly what we did in chapter 1. Once you know that your data comes from a legitimate sender, is of reasonable size, and contains the right types of tokens, you can invest in digging deeper into the data, parsing it, and seeing if it has the right format and structure.

4.3.4 Checking the data syntax

It isn’t uncommon for data to arrive as XML or JSON, and to understand it, you need to parse it. In the previous section, we described how we look at the lexical content of data: how it looks up close. When checking the syntax, we zoom out so we can see the larger picture. In XML, is there a closing tag for each opening tag? Are attributes inside the tags well formed?

Sometimes checking syntax is as simple as using a regexp. But to be truthful, regexps aren’t always simple; we’ve seen numerous examples of people creating complex and sometimes incomprehensible things using regexps. We recommend using regexps in a simple way, for simple things. If a regexp makes your head hurt when you look at it, consider writing the logic as code instead.

One example where syntax checking is simple is with ISBNs. As mentioned earlier, an ISBN consists of nine digits and a control digit. The control digit might be 10, which is denoted by the letter X. The syntax of an ISBN is something that can be checked with a regexp. To do that, we check if the data matches [0-9]{9}[0-9X]. If it does, it has the right format, and not matching this regexp means we discard the data.

More complex data structures, like XML, require a parser. Parsing complex and large data structures is something that requires lots of computational resources. This is why the parser is a desirable target for someone who wants to launch a DoS attack on your system. Many of the preceding steps of checking origin, checking data size, and checking lexical content are there to shield the parser.

We’ve mentioned that checking syntax consists of checking that data comes in the proper form. We’ll look at one more common mechanism, a checksum, to see if the format is correct. An ISBN is a good example of this, as the tenth and last digit in the ISBN is a control digit. Without getting into the details of how the checksum is computed, checking this is also part of checking syntax. The next listing shows the ISBN class again, now with a syntax check of the format as well as a control of the check digit.

Listing 4.19 Checking an ISBN’s lexical content and syntax

import org.apache.commons.lang3.Validate.*;

public class ISBN {
    private final String isbn;

    public ISBN(final String isbn) {
        notNull(isbn);
        inclusiveBetween(10, 10,isbn.length());
        isTrue(isbn.matches("[0-9X]*"));    ①  
        isTrue(isbn.matches("[0-9]{9}[0-9X]"));    ②  
        isTrue(checksumValid(isbn));    ③  
        this.isbn = isbn;
    }

    private boolean checksumValid(String isbn) { /.../ }
}

When the syntax structure is simple enough, it looks ridiculous to first have a lexical check with one regexp and then a syntax check with a similar regexp. In these cases, the lexical check and syntax check are often collapsed into one step, as in the next listing.

Listing 4.20 Checking an ISBN’s lexical content and syntax in one step

import org.apache.commons.lang3.Validate.*;

...

    public ISBN(final String isbn) {
        notNull(isbn);
        inclusiveBetween(10, 10,isbn.length());
        isTrue(isbn.matches("[0-9]{9}[0-9X]"));    ①  
        isTrue(checksumValid(isbn));    ②  
        this.isbn = isbn;
    }

It might look like we’re sidestepping the validation order here. What happens in this case is that the lexical analysis and the syntax analysis are so similar that they are done by the same mechanism. You can look at it as if they’re interleaved or have collapsed. The essential idea of the validation order is still there. Now that you know that you have well-formatted data that contains the expected content, has a reasonable size, and comes from a legitimate sender, you can see if this data fits in with the data you have—if it makes sense.

4.3.5 Checking the data semantics

By checking origin, size, content, and structure, you’ll have gathered so much confidence in the input data that you’re ready to act on it. Up to this point, you might have performed checks that are CPU-intensive or require lots of memory. For example, you might have processed a large XML message that instructs you to add a lot of items to an order in an order system. But, thus far, you’ve only worked on the data by itself, not on the data in connection with the rest of the system. Chances are that you’ve probably not hit the database once during this time.

In the example with a large XML message for extending an order, you might have found a lot of order lines to add, but you haven’t checked that the specified product numbers exist, and certainly not their stock status. The specified order might not even exist or is perhaps already shipped and not open for extension. You’ve only checked the syntax.

Now it’s time for the semantic check: is this data sound and consistent with the view of the world that resides in the rest of the system? In the semantic check, you check things like whether this product number exists in your product catalog or if the order described by this order number can be amended with another item. We think the most intuitive place to put these constraints is in the domain model.

A search in your product catalog domain service could be asking does this product number exist? If there’s no match, an exception is thrown, and the flow is interrupted. In the same way, if someone tries to add an item to an order that’s closed (perhaps paid and shipped), it’ll show up as an IllegalStateException from the Order class. In fact, we think the domain model is such a natural place to put these kinds of checks that we don’t think of them as validations—to us, they’re part of the domain model.

Yet we agree that semantic validation is rightly part of the list, as the verb validate doesn’t make sense on its own. You always need to validate with respect to something. You validate that the data adheres to some rules and constraints, and those rules and constraints are what make up the domain model. The domain model states how you’ve chosen to look at the world. You have modeled order numbers to have a specific format, and you’ve modeled orders to be closed for amendments when they ship. After passing all these stages, you know that the data is considered sound with respect to the model—it’s been validated.

You have now reached the point where you know the data is from a legitimate sender, has a reasonable size, contains the expected content with the expected structure, and makes sense with respect to the rest of the data. You can now safely act on that data by adding a new book to the cart, accepting the payment, or sending the order for shipping.

To secure the design, it’s powerful to let the code reflect if data is validated and to what degree. The ISBN class you saw earlier is an example of this. It’s a small, domain-focused building block that we know contains an ISBN of the valid format. There’s no need to validate that number again. In our experience, designing such building blocks does wonders for the security of your system. It’s now time to dive into how to craft such blocks, which we call domain primitives.

Summary

• Data integrity is about ensuring the consistency and accuracy of data during its entire life cycle.
• Data availability is about ensuring data is obtainable and accessible at the expected level of performance in a system.
• Immutable values are safe to share between threads without locks: no locking, no blocking.
• Immutability solves data availability issues by allowing scalability and no locking between threads.
• Immutability solves data integrity issues by preventing change.
• Contracts are an effective way to clarify the responsibilities of objects and methods.
• It’s better to fail fast in a controlled manner than to risk uncontrolled failures later. Fail fast by checking preconditions early in each method.
• Validation can be broken down into checking origin, data size, lexical content, syntactic format, and semantics.
• Origin checks can be done by checking the origin IP or requiring an access key to counteract DDoS attacks.
• Data size checks can be done both at the system border and at object creation.
• Lexical content checks can be done with a simple regular expression (regexp).
• Syntax format checks might require a parser, which is more expensive in terms of CPU and memory.
• Semantic checks often require looking at the data in the database, such as searching for an entity with a specific ID.
• Earlier steps in the validation order are more economical to perform and protect the later, more expensive steps. If early checks fail, later steps can be skipped.