Architecture

HTTP was created to support the World Wide Web, and its development over time has reflected the Web’s needs and its scalability challenges.

HTTP’s architecture is based on a client and server model. The definitions of client and server only define the particular roles that the programs or services fulfill for a particular connection, though. This means the same party can be a server in one connection and a client in others.

HTTP is stateless. This means that the communication protocol treats each request independently from previous ones and that the server does not retain session or status information regarding the parties involved in a certain communication.

The message exchange pattern used by HTTP is request/response. Messages are exchanged over a reliable transport or session layer connection.

The client in a HTTP communication is the party that establishes a connection to a server for sending one or more HTTP requests. Other terms that complete the HTTP protocol glossary are user agent, origin server, sender, and recipient.

A user agent is a software client originating an HTTP request. This can be a browser, a spider (robot program) crawling the Web, or any generic application. An origin server is a program that can originate authoritative responses for a certain resource. This is the server where the specified resource resides.

Sender and recipient are the terms used to indicate any party that sends and receives messages, respectively.

It is important to note that HTTP has been designed to be implemented in a wide range of situations. Not all user agents are general-purpose browsers accessing large public websites served by origin servers.

A user agent can be any sort of application, from mobile apps, to household appliances, to environmental sensors, to personal devices like fitness activity trackers.

The same applies when considering origin servers: these can be anything from city traffic cameras to networking components.

Most HTTP communication consists of a request for some resource indicated by a Uniform Resource Identifier (URI). The URI is a standard for identifying local and remote resources and describing the relationships between resources.

HTTP considers URIs as formatted strings identifying specific resources, composed from a limited set of characters and some graphic symbols and consisting of a hierarchical sequence of components: the scheme identifying the protocol, host name and port number, absolute path, query, and fragment. The format is (with optional components indicated by square brackets):

"http:""//"host[":"port][abs_path["?"query]["#"fragment]]

HTTP also allows the use of intermediaries between the client and server (as illustrated in Figure A-1), to handle requests through a chain of connections. Generally speaking, it is possible to identify three common HTTP intermediaries:

Proxy

A proxy’s function is to forward messages. The client selects the proxy via some local configuration rule to receive requests for all or some absolute URIs and redirects these requests through the HTTP interface toward some origin servers, in the attempt to satisfy them.

Gateway

A gateway, or reverse proxy, acts as an origin server translating received requests and forwarding these inbound to other servers. A gateway may communicate with inbound servers using different protocols, but if it wants to interoperate with third-party HTTP servers it will have to respect the user agent requirements for the inbound connections.

Tunnel

A tunnel is a relay between two connections that doesn’t change the messages exchanged in any way. Once it has been activated, a tunnel is not considered a party in the HTTP communication itself, and it is terminated when both ends of the connections are closed. An example of where tunnels are used is when Transport Layer Security (TLS) is used to establish a confidential communication.

This categorization is actually very versatile, as a single intermediary might act as an origin server, proxy, gateway, or tunnel in different communications, while at the same time handling many requests simultaneously.

HTTP moreover distinguishes between upstream and downstream communications, to describe directional requirements in relation to the message flow (messages always originate upstream and flow downstream). The terms inbound and outbound are instead used to describe directions regarding the request route, where inbound means toward the origin server and outbound means toward the user agent.

Figure A-1. A simple HTTP communication, initiated from a user client toward an origin server, can pass through a series of intermediaries

Parameters

Within HTTP request and response messages, colon-separated name/value pairs are transmitted in clear-text string format and terminated by a carriage return (CR) and line feed (LF) character sequence. These constitute the header fields of the HTTP message and define the operating parameters of an HTTP transaction. The end of the header fields is marked by two consecutive CRLF pairs.

If you were to visit http://www.oreilly.com, your browser would send a number of HTTP requests; if you captured one of these, the “headers” part of the request would look more or less like this:

GET / HTTP/1.1 Host: www.oreilly.com
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,es;q=0.6,it;q=0.4,en-GB;q=0.2,ca;
q=0.2
Cookie: <Some garbled cookie data>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 
Safari/537.36
X-Client-Data: <Some garbled client data>
HTTP/1.1 200 OK Accept-Ranges: bytes
Cache-Control: max-age=14400
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 16813
Content-Type: text/html; charset=utf-8
Date: Sat, 12 Jul 2014 16:59:01 GMT
Expires: Sat, 12 Jul 2014 20:59:01 GMT
Last-Modified: Sat, 12 Jul 2014 08:28:40 GMT
Server: Apache
Vary: Accept-Encoding

The Internet Engineering Task Force (IETF) has standardized a core set of field names that must be implemented by all HTTP-compliant protocol implementations. In addition to the core set, each application can also define their own field names and permissible values.

The Internet Assigned Numbers Authority (IANA) maintains the permanent registry of headers and a repository of provisional registrations. Up to June 2012, nonstandard header fields were conventionally marked by prefixing the field name with “X-”; however it was decided to deprecate this convention to avoid issues when nonstandard fields became part of the standard set.

Among the message headers, an HTTP-Version field is also specified to indicate the version of HTTP in use. This allows the sender to specify the format of a message and its capacity to understand further HTTP communications.

Messages

HTTP messages are either requests from client to server or responses from server to client.

Both requests and responses consist of a start line, zero or more header fields, an empty line (i.e., a line containing just a CRLF) that indicates the end of the header fields, and possibly a message body with the payload of the message. The format is therefore:

start-line 
*(message headers CRLF) 
CRLF 
[ message-body ]

The header fields include general headers, request headers, response headers, and entity header fields.

The message body of an HTTP message (if there is one) carries the entity body associated with the request or response. The entity body might be encoded as defined by the Transfer-Encoding field. In this case it will differ from the message body:

message-body = entity-body 
               | <entity-body as encoded per Transfer-Encoding>

Although request and response messages use a generic format for transferring the message payload, some syntax differences apply between the two.

Method = "OPTIONS" ;
| "GET" ;
| "HEAD" ;
| "POST" ;
| "PUT" ;
| "DELETE" ;
| "TRACE" ;
| "CONNECT" ;
| extension-method
| extension-method = token

Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF

entity-body := Content-Encoding( Content-Type( data ) )

Connections

HTTP supports the possibility of using a single TCP connection to send and receive multiple HTTP request/response messages. This method is called persistent connection, keep-alive, or connection reuse. Under HTTPv1.0, if the client supports the Keep-Alive header field, it will add it to the request:

Connection: Keep-Alive

Then, when attempting to handle this request, the server will add the same header field to the response. For following messages, the connection is kept open. When the client sends another request it will therefore use the same TCP connection. The connection will be interrupted only when either the client or the server drops it, signaling that the communication is over.

In HTTPv1.1 all connections are considered persistent unless otherwise specified. A short connection timeout is also implemented to allow origin servers to deliver multiple components of a web page rapidly enough, while not consuming excessive resources.

Clients supporting persistent connections may also wish to “pipeline” their requests, i.e., sending multiple requests without waiting for each response. The origin server will serve the responses in the same order that the requests were received.

Status Code Definitions

Status codes are three-digit codes that define the response of the server to a certain request. The first digit corresponds to the class of response. HTTP defines five classes:

1xx: Informational - Request received, continuing process
2xx: Success - The action was successfully received, 
  understood, and accepted
3xx: Redirection - Further action must be taken in order to 
  complete the request
4xx: Client Error - The request contains bad syntax or cannot 
  be fulfilled
5xx: Server Error - The server failed to fulfill an apparently 
  valid request

HTTPv1.1 also defines individual values of the status codes, with corresponding reason phrases that are intended to be used as recommendations for individual implementations (these can be replaced without affecting the protocol). Codes that you will probably use most frequently are:

Status-Code = 100 : Continue
| 200: OK
| 201: Created
| 202: Accepted
| 204: No Content
| 301: Moved Permanently
| 302: Found
| 303: See Other
| 304: Not Modified
| 400: Bad Request
| 401: Unauthorized
| 403: Forbidden
| 404: Not Found
| 500: Internal Server Error
| 501: Not Implemented
| 502: Bad Gateway
| 503: Service Unavailable

Access Authentication

HTTP provides different optional challenge-response authentication mechanisms, which can be used either by the server to challenge a client request, or by a client to provide authentication information.

These are defined separately from the protocol specification in RFC 2617. Note, however, that the authentication scheme provided by HTTP is not considered to be a secure method of user authentication unless used together with other external secure methods, such as SSL.

Content Negotiation

Content negotiation is the process of selecting a representation for a given response when multiple representations are available, in terms of type, language, encoding, and so on. HTTP defines two kinds of content negotiation: server-driven and agent-driven. The two methods are orthogonal and therefore can be used either independently or in combination. Transparent negotiation is one type of combination, where the negotiation work is taken over by the cache.

Caching

HTTP was designed for distributed information systems, where caching is known to improve performance. Therefore, the protocol definition includes a number of methods and elements to make the use of response caches as efficient as possible.

The ultimate goal of caching in HTTP is to reduce the need to send requests or response message in as many cases as possible, therefore limiting the number of network round-trips and network bandwidth required for a given communication.

Caching is an important aspect of HTTP and REST. Within RESTful architectures, the most efficient network request is one that doesn’t use the network, but efficiently relies on caching.

When defining its caching capabilities, HTTP introduces the concept of semantic transparency. It is said that a caching method is semantically transparent regarding a certain response when its use affects neither the client nor the origin server, except to improve performance. In other words, if the caching method is semantically transparent, the client receives the response from the cache exactly as it would have received it if the origin server had handled the request.

HTTPv1.1 allows clients, servers, and caching methods to explicitly reduce semantic transparency when necessary to improve performance.

HTTP also defines cache correctness as the idea that the cache, or caching method, responds to a certain request with the most up-to-date response possible that is appropriate to the request.

If the response returned by the cache is neither firsthand nor considered “fresh enough,” a Warning header is attached to the response so that the client can handle it properly.

Client and server caching and cache control mechanisms were discussed in detail in Chapter 12, where we saw how to best integrate external resources into our own application.

Security Considerations

HTTP clients and servers often exchange large amounts of user-generated data, and this should be handled carefully to prevent leakage to external sources through the HTTP protocol.

Chapter 16 discussed a number of security challenges that developers face, but it’s worth repeating a few key points.

Particularly, a server is in the position to log personal data contained in users’ requests that might reveal more than intended about them, and can be used by a potential attacker in a variety of ways. Users often reveal information regarding their thoughts, actions, tastes, and preferences by sharing them either indirectly, through their request messages, or directly, by voluntarily publishing this information online.

It is also important to note that as users’ online interactions become more complex and as shared documents are enriched with different media content, it is often difficult for the users to understand and ponder what kind of information is actually being shared with a single post, or what they might consider just innocuous activity.

It is therefore the responsibility of the application developer to implement methods and practices to safeguard privacy and protect user data from unintended leakages and unauthorized accesses.