The problem with the Web and some other current protocols is that the content presented to a user by a single multimedia site must be obtained from various sources (including wholly different domains) and then combined. Web pages have their text and formatting information separate from actual images and other sizable goodies (a practice truly to be praised by those who have a limited bandwidth and just want to get to the point).

This situation makes it necessary for clients to make several requests in order to render a web page. The most naive way to achieve this is by requesting each piece, one by one, in sequence, but this is not the best practice in the real world because it leads to bottlenecks: Why wait for a page to load simply because the banner server is running slowly? Hence, to improve the speed of content retrieval, the browser issues numerous requests at once.

And herein lies the first shortcoming of HTTP: it offers no native ability to serve simultaneous requests. Instead, requests must be issued sequentially.

The sequential (also called serial) fetch model results in a considerable performance penalty if one of the web page elements needs to be downloaded from a slow server or over a spotty link or if it takes a while for the server to prepare and deliver a particular element. If sequential fetching were the only option, any such slow request would prevent subsequent requests from being issued and served until it (the slow request) is filled.

Because newer versions of HTTP have not improved this situation, most client software implements a kludge: the web browser simply opens a number of simultaneous, separate TCP/IP sessions to a server or a set of servers and attempts to issue many requests at once. This solution is actually quite sane when the page is requesting resources from several separate machines. However, it’s not a good fix when the requested resources are on a single system, where all requests could be made in a single session and reasonably managed by the server. Here’s why:

Alas, good or bad, this architecture is with us for now, and it is still better than serial fetch. We should acknowledge its presence and learn to take advantage of it.

How can this very property help us to identify the software that the client is using? Quite simply. The significance of parallel file fetching for the purpose of browser fingerprinting should be fairly obvious: no two concurrent fetch algorithms are exactly the same, and there are good ways to measure this.

But before we turn our attention to parallel fetching, we need to take a look at two other important pieces of the security and privacy equation for the Web: caches and identity management. Although seemingly unrelated, they make a logical whole in the end. Thus, a brief intermission.

Keeping local caches of documents received from the server is one of the more important features of the Web during its rapid expansion in recent years.^[30] Without it, the cost of running this business would have been considerably higher.

The problem with the increasing weight and complexity of a typical website is that it requires more and more bandwidth (which for businesses remains generally quite expensive), as well as better servers to serve the data at a reasonable speed.

If performance is not impacted by bandwidth bottlenecks, solutions such as concurrent sessions (as described earlier) put additional strain on service providers instead. The reason might be fairly surprising: if a person on a fairly slow link (such as a modem) opens four subsequent sessions to fetch even a fairly simple page, four connections and four processes or threads need to be kept alive on the server, taking away those resources from those with faster connections.

Finally, to make things worse, heavier and more complex websites don’t always mesh with user expectations. Relatively long web page load times that were once considered fairly decent now seem annoying and drive users away. In fact, research suggests that the average web user won’t wait more than 10 seconds for a page to download before they move on.^[101] The result is that corporations and service providers need more resources and better links to handle the incoming traffic. In fact, had things been left the way they were initially designed, the demand for serverside resources would have likely exceeded our capacity to fulfill the demand some time ago.

Of some help is that the contents served to web surfers is static or changes seldom, at least when compared with the rate at which a resource is retrieved by users. (This is especially true for large files, such as graphics, video, documents, executables, and so on.) By caching data closer to the end user—be it on the ISP level or even on the endpoint browser itself—we can dramatically decrease the bandwidth used for subsequent visits from users who share a common caching engine and make it easier on the servers handling the traffic. The ISP benefits from a lowered bandwidth consumption, as well, being able to serve more customers without having to invest in new equipment and connections. What HTTP needs, however, is a mechanism to keep the cache accurate and up-to-date. The author of a page (either human or machine) needs to be able to tell the cache engine when to fetch a newer version of a document.

To implement document caching, HTTP provides two built-in features:

This functionality is in practice achieved fairly simply: The server returns all cacheable documents with the regular HTTP session, but with an additional protocol-level header, Last-Modified. To no surprise, this header represents the server’s idea of the time this document was last modified. Documents that cannot be cached are, on the other hand, marked by the server with the header Pragma: no-cache (Cache-Control: no-cache in HTTP/1.1).

The client browser (or an intermediate cache engine run by the ISP) is supposed to cache a copy of every cacheable page based on the presence of an appropriate header, along with the last modification information. It should keep the cached page for as long as possible, either until the user-configured cache limit is exceeded or the user manually purges the cache, unless specifically instructed to discard it after a specific date with an Expires header.

Later, when the site is visited again, the client concludes that they have a previous instance of the page cached on the disk and follows a slightly different procedure when accessing it. As long as a document lives in the cache, the client attempts to fetch the file every time the user revisits a site, but specifies the If-Modified-Since header with every request, using the value previously seen in the Last-Modified header for <Since>. The server is expected to compare the Modified-Since value with its knowledge of the last modification time for a given resource. If the resource has not been changed since that time, the HTTP error message “304 Not Modified” is returned instead of the requested data. As a result, the actual file transfer is suppressed, thus preserving bandwidth (with only a couple of hundred bytes exchanged during this communication). The client (or intermediate cache engine) is expected to use a previously cached copy of the resource instead of downloading it again.

Another important and seemingly unrelated requirement for HTTP was that it be able to differentiate between sessions and track them across connections, store session settings and identity information. For example, some websites greatly benefit from the ability to adapt to one’s personal preferences and to restore the look and feel chosen by the user each time they visit the site. Naturally, a user’s identity can be established by prompting for a login and password every time a page is viewed, at which point the user’s personal settings can be loaded, but this bit of extra effort dramatically reduces the number of people who would be willing to do this to access the page.

A transparent and persistent way to store and retrieve certain information from the client’s machine was needed to ensure seamless and personalized access to web forums, bulletin boards, chats, and many other features that define the browsing experience for so many people. On the other hand, the ability for web server administrators to recognize and identify returning visitors by assigning them a unique tag and retrieving it later meant the surrender of anonymity in exchange for a little convenience. Such a mechanism would give companies with second-grade ethics a great tool to track and profile users, record their shopping and browsing preferences, determine their interests, and so forth. Search engines could easily correlate requests from the same user, and content providers that serve resources such as ad banners could use this information to track people even without their permission or the knowledge of site operators.^[31] Regardless of the concerns, however, there seemed to be no better, sufficiently universal alternative for this mechanism. And so web cookies were born.

Cookies, as specified in RFC2109,^[102] are small portions of text that are issued by a server when the client connects to it. The server specifies a Set-Cookie header in the response to the visitor. This portion of text is, by its additional parameters, limited in scope to a specific domain, server, or resource and has a limited lifespan. Cookies are stored by cookie-enabled client software in a special container file or folder (often referred to as a cookie jar) and are automatically sent back to the server using a Cookie header whenever a connection to a specific resource is established again.

Servers can choose to store (or push out) user settings in Set-Cookie headers and just read them back on subsequent visits; and here is where cookie functionality would end in a perfect world. Unfortunately, computers have no way of telling what is stored in a cookie. A server can choose to assign a unique identifier to a client using the Set-Cookie header and then read it back to link current user activity to previous actions in the system.

The mechanism is wildly regarded as having serious privacy implications. Some activists downright hate cookies, but the opposition to this technology is getting less and less vocal nowadays. Browsing the Web with cookies disabled gets increasingly more difficult—with some sites even refusing traffic from clients that do not pass a cookie check. Thankfully, many browsers offer extensive cookie acceptance, restriction, or rejection settings and can even prompt for every single cookie before accepting it (although the latter is not particularly practical). This makes it possible to mount a reasonable defense of your privacy, if only by defining who the “good guys” are and who to trust.

But is our privacy in our hands then?

The privacy of web browsing has long been considered a hot issue, and not without reason. Many people do not want others to snoop on their preferences and interests, even if their whereabouts are not particularly questionable. Why? Sometimes, you simply do not want a shoddy advertising company to know that you are reading about a specific medical condition and then be able to link this information to an account you have on a professional bulletin board, particularly because there is no way of knowing where this information will end up.

Cookie control makes our browsing experience reasonably comfortable, while keeping bad guys at bay. But even turning cookies off does not prevent information from being stored on one’s system to be later sent back to a server. The functionality needed to store and retrieve data on a victim’s machine has long been present in all browsers, regardless of cookie policy settings. The two necessary technologies work in a similar manner and differ only in terms of their intended use: cookies and file caching.

Somewhere back in 2000, Martin Pool posted a fairly short but insightful message^[103] to the Bugtraq mailing list, sharing an interesting observation and supporting it with some actual code. He concluded that there is no significant difference between the Set-Cookie and Cookie functionality versus Last-Modified and If-Modified-Since, at least for systems that do not use centralized proxy caches and that store copies of already fetched documents locally on disk (as is the case with most of us mere mortals). A malicious website administrator can store just about any message in the Last-Modified header returned for a page their victim visits (or, if this header is sanity-checked, it might simply use a unique, arbitrary date to uniquely identify this visitor). The client would then send If-Modified-Since with an exact copy of the unique identifier stored by a rogue operator on their computer whenever a page is revisited. A “304 Not Modified” response ensures that this “cookie” is not discarded.

Using your browser to slightly tweak Last-Modified data in response might seem like a neat way to prevent this type of exposure (while introducing some cache inaccuracy), but this is not the case. Another variant of this attack is to rely on storing data in cached documents, as opposed to using tags directly: a malicious operator can prepare a special page for the victim when a website is visited for the first time. The page contains a reference to a unique file name listed as an embedded resource (for example, an image). When a client revisits this page, the server notices the If-Modified-Since header and replies with the 304 error message, prompting the old copy of the page to be used. The old page contains a unique file reference that is then requested from the server, making it possible to map the client’s IP to a previous session in which that file name had been returned. Oops.

Naturally, the lifetime of cache-based “cookies” is limited by cache size and expiration settings for cached documents configured by the user. However, these values are generally quite generous, and information stored within metadata for a resource that is revisited once every couple of weeks can last for years, until the cache is manually purged. For companies that serve common components included on hundreds or thousands of sites (again, banners are a good example), this is a nonissue.

The main difference with these cache cookies, compared with cookies proper, is not a matter of the functionality they offer, but rather the ease of controlling the aforementioned exposure. (Cache data must also serve other purposes and cannot be easily restricted without a major performance impact associated with disabling caching partly or completely.)

In this bizarre twist, you can see how two aspects of the Web collide, effectively nullifying security safeguards built around one of them. Practice shows that intentions are not always enough, because rogues are not always willing to play by the rules and use the technology the way we want them to. Perhaps turning your cookies off does not make that much of a difference after all?

But then it is about time to go back to the main subject of our discussion.