What would such a store look like? The best way to explain some of the basic design patterns is with an example. Table 15-1 shows a traditional database table in a traditional store.

This is the kind of information needed by the folks who stock the shelves and make decisions about marketing. They need to know what is being sold and, so they think, to whom.

People have different instinctual reactions to the existence of these databases. Some welcome them and see the efficiency as a net gain for society (Brin 1998). Others fret about how the information might be used for other purposes. We have laws protecting the movie rental records of people because the data became a focal point during the Senate confirmation hearings of Robert Bork (Etzioni 1999). Times and mores change, and no one knows what will be made of this information in the future. Cigarette smoking was once widespread and an accepted way to push the pause button on life. Now, companies fire people for smoking to save medical costs. What will the society of the future do with your current purchase records?

A store selling clothes might try to placate customer worries by announcing a privacy policy and pledging to keep it locked up, a strategy that often works until some breach happens. This may be intentional (Wald 2004, Anon. 2003, Shenon 2003) or unintentional (Wald 2007, Zeller 2006, Pfanner 2007, and many more), but it’s impossible to prevent when you use the fortress model of security.

The translucent model can solve the problem by scrambling some of the columns with a one-way function such as SHA256, a well-known hash function that’s designed to be practically impossible to invert. That is, if you start with the string x, you can reliably create SHA256(x), but if you get your hands on SHA256(x), it’s infeasible to figure out x.^[106] If the customers hash their names, the plain-text name can be replaced by the 256 bits produced by the computation BASE64(SHA256(name)). The result might look like Table 15-2.^[107]

The stock department can still track how many pairs of khakis were sold. The marketing department can still try to look for patterns, like the fact that the same customer, ick93LKJ0dkK, bought pants and a shirt at the same time. The names, though, are obscured.

The value of a one-way function is that it combines the two seemingly incompatible goals of hiding information while making it reliably persistent. That is, James Morrison can give the store a hash of his name in the form ick93LKJ0dkK and the store can be sure that his records in the database belong to him, but they can’t find his real name. This can be useful with warranties, frequent shopper programs, and automated suggestion tools. Websites that compute suggestions for customers based on their past purchases can still compute them for someone with the digital pseudonym ick93LKJ0dkK.

When James Morrison returns, he can provide his name and look up these values. He doesn’t need to remember ick93LKJ0dkK, because the software on his computer can reconstruct it by calculating BASE64(SHA256(name)). After that, ick93LKJ0dkK acts just like a regular name.

The raw data in the scrambled table isn’t much use. A drug fiend on a crime spree couldn’t use a translucent database from a pharmacy to find houses with narcotic prescriptions^[108] (Meier 2007, Harris 2004, Butterfield 2001).

There are still limitations. If James Morrison’s neighbor works at the store, that neighbor can still find out what James Morrison bought by just typing in the right name with the right spelling. This weakness can be reduced by forcing James Morrison to remember a secret password and computing a pseudonym from SHA256(name+password). This is a good solution until James Morrison forgets the password, effectively destroying all of the records.^[109] This is why this level of security may be best for more casual applications, such as online stores. It might not make sense for highly valuable records, such as certificates of deposit at banks.

There are a number of ways in which translucent databases can save system administrators the time and energy of worrying about the data under their care. Imagine a database that holds the schedule of babysitters so that parents can search for the babysitter who’s available on a particular evening, a service that can save parents and babysitters many annoying rounds of phone tag. I built a version of this system to explore the value of the idea; it required distributing a solid hash function to the client. The central server didn’t require any code change because the hashed identities were strings that could replace the names.

This information falls into a troubling limbo in terms of security motivation. The potential for malice is large and the downside is catastrophic, but there’s no great economic support for the system. The babysitters—unlike the U.S. government, for instance—don’t have the money to support the kind of system that protects the nuclear launch codes.

Translucent databases reduce the danger of aggregating so much information in one place protected by one root password. Instead of storing the name of each babysitter, the schedules can be stored under pseudonyms, as shown in Table 15-3.

How do parents get access to the schedules? The babysitters choose who can have access by computing SHA256(parent's name+parent's password). Then, they put an entry into an access table that looks like Table 15-4.

The parents log in by typing in their name and password to generate their pseudonym. They can then search the access table to find the pseudonyms of the babysitters’ names. This allows them to search the schedules and find a babysitter with an open schedule. If the babysitter keeps the parents’ pseudonym out of the second table, the parents are locked out. The babysitter is in control.

How do the parents locate the babysitter? The third column contains the babysitter’s name encrypted with the parents’ name and password. Thus, the babysitter’s real name in each row can be decrypted only by the parent indicated in that row. In this case, the second column with the pseudonym might be constructed from SHA256(parent's name+parent's password), as already described, while the key might be computed a bit differently with SHA256(parent's password+parent's name). Thus, parents need just one set of information (name and password) to calculate their pseudonyms and keys, but no one can guess the secret key from the public pseudonym.

The great benefit of my system is in protecting data kept by a store as a convenience. Many stores, both brick-and-mortar and online, keep the customers’ shipping addresses, credit cards, and other personalized information on file to avoid the trouble of retyping these for every purchase.

One solution is to combine traditional encryption with the one-way function and use the one-way function to compute the key from a name and a password. That is, encrypt the addresses and credit card number with a key defined by SHA256(password+name+padding), where padding is some random value added to ensure that the value is different from the digital pseudonym, SHA256(name+password), used to replace the name.

A store needs to keep track of a customer’s shipping address until the package leaves the building. Then they can delete the data and keep only the encrypted version until the customer comes back again.

Does it make sense to turn your database into an inscrutable pile of bits? Here are the advantages as I see them:

There are, however, prices to be paid:

These considerations suggest that this approach is best for protecting mildly sensitive data that can be lost without real consequences. A customer who loses access to the purchase history at a store is not seriously inconvenienced. A patient who can’t access medical records, however, may be pretty upset.

One of the best uses for this approach may be with multitiered databases. A hospital could keep its patient records intact in one centralized vault, but distribute depersonalized, translucent solutions to researchers conducting statistical studies. These researchers could still work with real data without the worry of protecting personal data (Schneier 2007).

One-way functions are just the beginning of the possibilities for working with protected data. Other traditional mathematical tools developed by cryptologists can be used with applications in the same way to preserve privacy. Steganographic applications, for instance, can mix a hidden layer of data in with the real layer, effectively creating a two-tier database. Only the users with the right keys for unscrambling the steganographic layer can extract the second tier. Public-key solutions can create functions that are one-way except for people who possess the secret key for unlocking the data. All of these solutions are explored in greater detail in Translucent Databases (see References).

The digital fortress is the traditional approach to storing sensitive information. The best database companies are filled with talented individuals who wrap adamantine layers around the data with access levels, security models, and plenty of different roles that allow the right queries while blocking the suspicious ones. These “unbreakable” systems have their place, but they are complicated, expensive, and fragile. A better solution is often keeping no real data at all and letting the user control the fate of his own information.

Anonymous. “Betraying One’s Passengers,” New York Times. September 23, 2003.

Brin, David. The Transparent Society. New York: Basic Books, 1998.

Butterfield, Fox. “Theft of Painkiller Reflects Its Popularity on the Street,” New York Times. July 7, 2001.

Etzioni, Amitai. “Privacy Ain’t Dead Yet,” New York Times. April 6, 1999.

Harris, Gardiner. “Tiny Antennas to Keep Tabs on U.S. Drugs,” New York Times. November 15, 2004.

Meier, Barry. “3 Executives Spared Prison in OxyContin Case,” New York Times. July 21, 2007.

Pfanner, Eric. “Data Leak in Britain Affects 25 Million,” New York Times. November 22, 2007.

Shenon, Philip with John Schwartz. “JetBlue Target Of Inquiries By 2 Agencies,” New York Times. September 23, 2003.

Schneier, Bruce. “Why ‘Anonymous’ Data Sometimes Isn’t,” Wired. December 13, 2007. http://www.wired.com/politics/security/commentary/securitymatters/2007/12/securitymatters_1213.

Wald, Matthew L. “U.S. Calls Release of JetBlue Data Improper,” New York Times. February 21, 2004.

Wald, Matthew L. “Randi A.J. v. Long Is. Surgi-Center, No. 2005-04976.” N.Y. App. Div, September 25, 2007.

Wayner, Peter. Translucent Databases. Flyzone, 2003. http://www.wayner.org/books/td/.

Zeller, Tom Jr. “U.S. Settles With Company on Leak of Consumers’ Data,” New York Times. January 27, 2006.