5.1.1 A short conversation with a web server

To access content on the Web, we are used to typing URLs into our browser or to simply clicking on links to get from one place to another, to check our mails, to read news, or to download files. Behind this program layer that is designed for user interaction there are several more layers—techniques, standards, and protocols—that make the whole thing work. Together they are called the Internet Protocol Suite (IPS). Two of the most prominent players of this Protocol Suite are TCP (Transmission Control Protocol) and IP (Internet Protocol). They represent the Internet layer (IP) and the transportation layer (TCP). The inner workings of these techniques are beyond the scope of this book, but fortunately there is no need to manually manipulate contents of either of these protocols to conduct successful web scraping. What is worth mentioning, however, is that TCP and IP take care of reliable data transfer between computers in the network.¹

On top of these transportation standards there are specialized message exchange protocols like HTTP (Hyper Text Transfer Protocol), FTP (File Transfer Protocol), Post Office Protocol (POP) for email retrieval, SMTP (Simple Mail Transfer Protocol) or IMAP (Internet Message Access Protocol) for email storage and retrieval. All of these protocols define standard vocabulary and procedures for clients and servers to talk about specific tasks—retrieving or storing documents, files, messages, and so forth. They are subsumed under the label application layer.

Other than the name suggests, HTTP is not only a standard for hypertext document retrieval. Although HTTP is quite simple, it is flexible enough to ask for nearly any kind of resource from a server and can also be used to send data to the server instead of retrieving it.

Figure 5.1 presents a stylized version of ordinary user–client interactions. Simply put, when we access a website like www.r-datacollection.com/index.html, our browser serves as the HTTP client. The client first asks a DNS server (Domain Name System) which IP address is associated with the domain part of the URL we typed in.² In our example, the domain part is www.r-datacollection.com.³ After the browser has received the IP address as response from the DNS server, it establishes a connection to the requested HTTP server via TCP/IP. Once the connection is established, client and server can exchange information—in our case by exchanging HTTP messages. The most basic HTTP conversation consists of one client request and one server response. For example, our browser asks for a specific HTML document, an image, or some other file, and the server responds by delivering the document or giving back an error code if something went wrong. In our example, the browser would ask for index.html and start parsing the content of the response to provide the representation of the website. If the received document contains further linked resources like images, the browser continues sending HTTP requests to the server until all necessary resources are transmitted. In the early days of the Internet, one could literally observe how the browser loaded webpages piece by piece. By now, it almost seems like webpages are received all at once due to the availability of higher bandwidths, keeping HTTP connections alive or posing numerous requests in parallel.

There are two important facts about HTTP that should be kept in mind. First, HTTP is not only a protocol to transport hypertext documents but is used for all kinds of resources. Second, HTTP is a stateless protocol. This means that without further effort each pair of request and response between client and server is handled by default as though the two were talking to each other for the first time no matter how often they previously exchanged information.

Let us take a look at one of these standardized messages. For the sake of the example we establish a connection to www.r-datacollection.com and ask the server to send us index.html. The HTTP client first translates the host URL into an IP address and then establishes a connection to the server on the default HTTP port (port 80). The port can be imagined as a door at the server's house where the HTTP client knocks. Consider the following summary of the client-side of the conversation:⁴

After having established the connection the server expects a request and our client sends the following HTTP request to the server:

Now it is the client's turn to expect a response from the server. The server responds with some general information followed by the content of our requested document.⁵ The HTTP response reads as follows:

After having received all the data, the connection is closed again by the client …

… and the transaction is completed.

5.1.2 URL syntax

The location of websites and other web content are identified by Uniform Resource Locators (URLs). They are not part of HTTP but make communication via HTTP and other protocols straightforward for users.⁶ The general URL format can be expressed as follows:

scheme://hostname:port/path?querystring#fragment

A corresponding real-life example would be

http://www.w3.org:80/People/Berners-Lee/#Bio

Each URL starts with a scheme that defines the protocol that is used to communicate between client/application and server. In the example, the scheme is http, separated by a colon. There are other schemes like ftp (File Transfer Protocol) or mailto, which corresponds to email addresses that rely on the SMTP (Simple Mail Transfer Protocol) standard. Most enable communication in networks, but you will also find the file scheme familiar, which addresses files on local or network drives.

The hostname provides the name of the server where the resource of interest is stored. It is a unique identifier of a server. The hostname along with the port component tell the client at which door it has to knock in order to get access to the requested resource. The information is provided in the example as www.w3.org:80. Port 80 is the default port in the Transmission Control Protocol (TCP). If the client is fine with using the default port, this part of the URL can be dropped. Hostnames are usually human readable, but every hostname also has a machine-readable IP address. In the example, www.w3.org belongs to the IP address 128.30.52.37, making the following an equivalent URL:⁷

http://128.30.52.37/People/Berners-Lee/#Bio

As we usually provide the human-friendly versions of URLs, the Domain Name System (DNS) translates hostnames into numerical IP addresses. Therefore, the DNS is frequently compared to a worldwide phone book that redirects users who provide hostnames to services or devices.

The path determines the location of the requested resource on the server. It works like paths on any conventional file system where files are nested in folders that may again be nested in folders and so on. Path segments are separated by slashes (/).

In some cases, URLs provide supplementary information in the path that helps the server to process the request correctly. The additional information is delivered in query strings that hold one or more name=value pairs. The query string is separated from the rest of the URL by a question mark. It encodes data using a ‘field = value’ format and uses the ampersand symbol (&) to separate multiple name–value pairs.

https://www.google.com/search?q=RCurl+filetype%3Apdf

A comparable URL is constructed when we search for “RCurl” documents on Google that are of type PDF. The name–value pair q=RCurl+filetype%3Apdf is the transformed actual request written in the search form as “RCurl filetype:pdf,” a compact syntax to search for PDF files that include the term “RCurl.” One could easily extend the request with further search parameters such as tbs=qdr:y. This would limit the results to hits that are younger than one year.⁸

Finally, fragments help point to a specific part of a document. This works well if the requested resource is HTML and the fragment identifier refers to a section, image, or similar. In the example above, the fragment #Bio requests a direct jump to the biography section of the document. Note that fragments are handled by the browser, that is, on the client side. After the server has returned the whole document, the fragment is used to display the specified part.

There are some encoding rules for URLs. URLs are transmitted using the ASCII character set, which is limited to a set of 128 characters. All characters not included in this set and most special characters need to be escaped, that is, they are replaced by a standardized representation. Consider once again the example. The expression “RCurl filetype:pdf” is converted to q=RCurl+filetype%3Apdf. Both white space and the colon character seem to be “unsafe” and have been replaced with a + sign and the URL encoding%3A, respectively. URL encodings are also called percent-encoding because the percent character % initializes each of these encodings. Note that the plus character is a special case of a URL escape sequence that is only valid in the query part. In other parts, the valid URL encoding of space is %20. A complete list of URL encodings can be found at http://www.w3schools.com/tags/ref_urlencode.asp.

We can encode or decode characters in URLs with the base functions URLencode() and URLdecode() in R. The reserved argument in the former function ensures that non-alphanumeric characters are encoded with their percent-encoding representation:

R> t <-"I'm Eddie! How are you & you? 1 + 1 = 2"
R> (url <- URLencode(t, reserve = TRUE))
[1]"I'm%20Eddie!%20How%20are%20you%20%26%20you%3f%201%20+%201%20%3d%202"

R> URLdecode(url)
[1]"I'm Eddie! How are you & you? 1 + 1 = 2"

These functions can be useful when we want to construct URLs manually, for example, to specify a GET form (see below), without having to insert the percent-encodings by hand.

5.1.3 HTTP messages

HTTP messages, whether client requests or server response messages, consist of three parts: start line, headers, and body—see Figures 5.2 and 5.3. While start lines differ for request and response, the messages’ header and body sections are structured identically.

To separate start line from headers and headers from body, carriage return and line feed characters (CRLF) are used.⁹ Note that start line and headers are separated by one sequence of CRLF while the last header before the body is followed by two CRLF. In R, these characters are represented as escaped characters for carriage return and for new line feed.

The start line is the first and indispensable line of each HTTP message. In requests, the start line defines the method used for the request, followed by the path to the resource requested, followed by the highest HTTP version the client can handle. In our example we use the POST method requesting greetings.html and indicate that our client understands HTTP up to version 1.1.

The server response start line begins with a statement on the highest HTTP version the server can handle, followed by a status code, followed by a human-readable explanation of the status. Here, www.r-datacollection.com tells us that it understands HTTP up to version 1.1, that everything went fine by returning 200 as status code, and that this status code means something like OK.

The header section below the start line provides client and server with meta information about the other sides’ preferences or the content sent along with the message. Headers contain a set of header fields in the form of name–value pairs. Ordinarily, each header field is placed on a new line and header field name and value are separated by colon. If a header line becomes very long, it can be divided into several lines by beginning the additional line with an empty space character to indicate that they belong to the previous header line.

The body of an HTTP message contains the data. This might be plain text or binary data. Which type of data the body is composed of is specified in the content-type header, following the MIME type specification (Multipurpose Internet Mail Extensions). MIME types tell the client or server which type of data it should expect. They follow a scheme of main-type/sub-type. Main types are, for example, application, audio, image, text, and video with subtypes like application/pdf, audio/mpg, audio/ogg, image/gif, image/jpeg, image/png, text/plain, text/html, text/xml, video/mp4, video/quicktime, and many more.¹⁰

5.1.4 Request methods

When initiating HTTP client requests, we can choose among several request methods—see Table 5.1 for an overview. The two most important HTTP methods are GET and POST. Both methods request a resource from the server, but differ in the usage of the body. Whereas GET does not send anything in the body of the request, POST uses the body to send data. In practice, simple requests for HTML documents and other files are usually executed with the GET method. Conversely, POST is used to send data to the server, like a file or inputs from an HTML form.

If we are not interested in content from the server we can use the HEAD method. HEAD tells the server to only send the start line and the headers but not transfer the requested resource, which might be convenient to test if our requests are accepted. Two more handy methods for testing are OPTIONS, which asks the server to send back the methods it supports and TRACE, which requests the list of proxy servers (see Section 5.2.3) the request message has passed on its way to the server.

Last but not least there are two methods for uploading files to and deleting files from a server—PUT and DELETE—as well as CONNECT, a method for establishing an HTTP connection that might be used, for example, for SSL tunneling (see Section 5.3.1).

We will elaborate the methods GET and POST, the two most important methods for web scraping, when we discuss HTTP in action (see Section 5.4).

5.1.5 Status codes

When a server responds to a request, it will always send back a status code in the start line of the response. The most famous response that nearly everybody knows from browsing the Web is 404, stating that the server could not find the requested document. Status codes can range from 100 up to 599 and follow a specific scheme: the leading digit signifies the status category—1xx for informations, 2xx for success, 3xx for redirection, 4xx for client errors and 5xx for server errors—see Table 5.2 for a list of common status codes.

5.1.6 Header fields

Headers define the actions to take upon reception of a request or response. Headers can be general or belong to one specialized group: header fields for requests, header fields for responses, and header fields regarding the body of the message. For example, request header fields can inform the server about the type of resources the client accepts as response, like restricting the responses to plain HTML documents or give details on the technical specification of the client, like the software that was used to request the document. They can also describe the content of the message, which might be plain text or binary, an image or audio file and might also have gone through encoding steps like compression. Header fields always follow the same, simple syntax. The name comes first and is separated with a colon from the value. Some header fields contain multiple values that are separated by comma.

Let us go through a sample of common and important header fields to see what they can do and how they are used. The paragraphs in the following overview provide the name of the header in bold and the field type in parentheses, that is, whether the header is used for request, response, or body.

5.2.1 Identification

The communication between client and server via the HTTP protocol is an amnesic matter. Connections are established and closed for each session; the server does not keep track of earlier requests from the same user by default. It is sometimes desirable that server responses are built upon results from previous conversations. For example, users might prefer that sites are automatically displayed in their language or adapted to fit a specific device or operating system. Moreover, customers of an online shop want to place items into a virtual shopping cart and continue browsing other products, while the website keeps track of these operations. Apart from scenarios like these that enhance user experience, some basic knowledge about clients is interesting for web administrators who want to know, for example, from which other sites their pages are visited most frequently.

HTTP offers a set of procedures that are used for such purposes. We discuss the most popular and relevant ones in the context of web scraping—basic identification header fields and cookies.

R> getURL("http://httpbin.org/headers", referer ="http://www.r-datacollection.com/")

R> getURL("http://httpbin.org/headers", httpheader = c(From ="[email protected]"))

R> getURL("http://httpbin.org/headers", cookie ="id=12345;domain=httpbin.org")

5.2.2 Authentication

While techniques for client identification are useful to personalize web content and enable stateful communication, they are not suited to protect content that only the user should see. A set of authentication techniques exist that allow qualified access to confidential content. Some of these techniques are part of the HTTP protocol. Others, like OpenID or OAuth (see Section 9.1.11), have been developed more recently to extend authentication functionality on the Web.

The simplest form of authentication via the HTTP protocol is basic authentication (Franks et al. 1999). If a client requests a resource that is protected by basic authentication, the server sends back a response that includes the WWW-Authenticate header. The client has to repeat its request with a username and password in order to be granted access to the requested resource. Both are stored in the response's Authorization header. If the server can verify that the username/password combination is correct, it returns the requested resource in a HTTP 200 message. Technically, basic authentication looks as follows.

1. The client requests a protected resource:

   

2. The server asks the client for a user name and password:

   

3. The client/user provides the requested username and password in Base64 encoding:

   

4. The server returns the requested resource:

   

Note that in the third step, the username/password combination has been automatically “encrypted” into the string sequence “dXNlcm5hbWU6cGFzc3dvcmQ=.” This transformation is done via Base64 encoding. Base64 encoding is not actually an encrypting technique but follows a rather trivial and static scheme (see Gourley and Totty 2002, Appendix E). We can perform Base64 encoding and decoding with R; the necessary functions are implemented in the RCurl package:

The example reveals the insecurity of basic HTTP authentication: As long as it is done via standard HTTP, the sensitive information is sent practically unencrypted across the network. Therefore, basic authentication should only be used in combination with HTTPS (see Section 5.3.1).

A more sophisticated authentication technique is digest authentication (Franks et al. 1999). The idea behind digest authentication is that passwords are never sent across the Web in order to verify a user, but only a “digest” of it. The server attaches a little random string sequence to its response, called nonce. The browser transforms username, password, and the nonce into a hash code, following one of several algorithms that are known to both server and browser. This hash code is then sent back, compared to the hash calculations of the server, and if both match the server grants access to the client. The crucial point is that the hash alone does not suffice to learn anything about the password; it is just a “digest” of it. This makes digest authentication an improvement relative to basic authentication, as the encrypted client message is incomprehensible for an eavesdropper.

Steps 2 and 3 in the authentication procedure sketched above are slightly different. The server returns something like the following:¹⁵

• 2a. The server asks the client for a username and password and delivers a nonce, reports a “quality of protection” value (qop) and describes the realm as Protected area:

  

• 3a. The client provides the encrypted username and password in the response attribute, as well as the unencrypted username, the qop and nonce parameters and a client nonce (cnonce):

  

In Section 9.1.6 we give a short demonstration of HTTP authentication in practice with RCurl.

5.2.3 Proxies

Web proxy servers, or simply proxies, are servers that act as intermediaries between clients and other servers. HTTP requests from clients are first sent to a proxy, which evaluates the request and passes it to the desired server. The server response takes the way back via the proxy. In that sense, the proxy serves as a server to clients and as a client to other servers (see Figure 5.4).

Proxies are useful for several purposes. They are deployed for performance, economic, and security reasons. For users, proxies can help to

• speed up network use;
• stay anonymous on the Web;
• get access to sites that restrict access to IPs from certain locations;
• get access to sites that are normally blocked in the country from where the request is put; or
• keep on querying resources from a server that blocks requests from IPs we have used before.

Especially when proxies are used for any of the last three reasons, web scrapers might get into troubles with the law. Recent verdicts point in the direction that it is illegal to use proxy servers in order to get access to public websites that one has been disallowed to visit (see Kerr 2013). We therefore do not recommend the use of proxies for any of these purposes.

In order to establish connections via a proxy server, we have to know the proxy's IP address and port. Some proxies require authentication as well, that is, a username and a password. There are many services on the Web that provide large databases of open and free proxies, including their location and specification. Open proxies can be used by anyone who knows their IP address and port. Note that proxies vary in the degree to which they provide anonymity to the user. Transparent proxies specify a Via header field in their request to the server, filling it with their IP. Further, they offer an X-Forwarded-For header field with your IP. Simple anonymous proxies replace both the Via and the X-Forwarded-For header field with their IP. As both fields are delivered only when a proxy is used, the server knows that the requests comes from a server, but does not easily see the client's IP address behind it. Distorting proxies are similar but replace the value of the X-Forwarded-For header field with a random IP address. Finally, High anonymity proxies or elite proxies behave like normal clients, that is, they neither provide the Via nor the X-Forwarded-For header field but only their IP and are not immediately identifiable as proxy servers.

To send a request to a server detouring via a proxy with R, we can add the proxy argument to the request command. In the following, we choose a fictional proxy from Poland that has the IP address 109.205.54.112 and is on call on port 8080:

IP address and port of the proxy are specified in the proxy option. Further, we set the followlocation argument to TRUE to ensure that we are redirected to the desired resource.

5.3.1 HTTP Secure

Strictly speaking, the Hypertext Transfer Protocol Secure (HTTPS) is not a protocol of its own, but the result of a combination of HTTP with the SSL/TLS (Secure Sockets Layer/Transport Security Layer) protocol. HTTPS is indispensable when it comes to the transfer of sensitive data, as is the case in banking or online shopping. To transfer money or credit card information we need to ensure that the information is inaccessible to third parties. HTTPS encrypts all the client–server communication (see Figure 5.5). HTTPS URLs have the scheme https and use the port 443 by default.¹⁶

HTTPS serves two purposes: First, it helps the client to ensure that the server it talks to is trustworthy (server authentication). Second, it provides encryption of client–server communication so that users can be reasonably sure that nobody else reads what is exchanged during communication.

The SSL/TLS security layer runs as a sublayer of the application layer where HTTP operates. This means that HTTP messages are encrypted before they get transmitted. The SSL protocol was first defined in 1994 by Netscape (see Freier et al. 2011) and was updated as TLS 1.0 in 1999 (see Dierks and Allen 1999). When using the term “SSL” in the following, we refer to both SSL and TLS as their differences are of no importance to us.

A crucial feature of SSL that allows secure communication in an insecure network is public key, or asymmetric, cryptography. As the name already indicates, encryption keys are in fact not kept secret but publicly available to everyone. In order to encrypt a message for a specific receiver, the receiver's public key is used. In order to decrypt the message, both the public and a private key is needed, and the private key is only known to the receiver. The basic idea is that if a client wants to send a secret message to a server, it knows how to encrypt it because the server's public key is known. After encryption, however, nobody—not even the sender—is able to decipher the message except for the receiver, who possesses both the public and the private key.

We do not have to delve deeply into the details of public key encryption, how the secret codes (ciphers) work, and why it is so hard to crack them in order to understand HTTPS's purpose. For the details of cryptography behind SSL, we refer to the excellent introductions by Gourley and Totty (2002) and Garfinkel (2002). If you want to get a more profound understanding of digital cryptography, the books by Ferguson et al. (2010) and Paar and Pelzl (2011) are a good choice. What is worth knowing though is how secure channels between client and server are actually established and how we can achieve this from within R. A very simplified scheme of the “SSL handshake,” that is, the negotiation between client and server about the establishment of an HTTPS connection before actually exchanging encrypted HTTP messages, works as follows (see Gourley and Totty 2002, pp. 322–328).

1. The client establishes a TCP connection to the server via port 443 and sends information about the SSL version and cipher settings.
2. The server sends back information about the SSL and cipher settings. The server also proves his identity by sending a certificate. This certificate includes information about the authority that issued the certificate, for whom it was issued and its period of validity. As anybody can create his or her own certificates without much effort, the signature of a trusted certificate authority (CA) is of great importance. There are many commercial CAs, but some providers also issue certificates for free.
3. The client checks if it trusts the certificate. Browsers and operating systems are shipped with lists of certificate authorities that are automatically trusted. If one of these authorities has signed the server's certificate, the client trusts the server. If this is not the case, the browser asks the user whether she finds the server trustworthy and wants to continue, or if communication should be stopped.
4. By using the public key of the HTTPS server, the client generates a session key that only the server can read, and sends it to the server.
5. The server decrypts the session key.
6. Both client and server now possess a session key. Thus, knowledge about the key is not asymmetric anymore but symmetric. This reduces computational costs that are needed for encryption. Future data transfers from server to client and vice versa are encrypted and decrypted through this symmetric SSL tunnel.

It is important to note that what is protected is the content of communication. This includes HTTP headers, cookies, and the message body. What is not protected, however, are IP addresses, that is, websites a client communicates with.

We will address how connections via HTTPS are established in R and how much of the technical details are hidden deeply in the respective functions in Section 9.1.7—using HTTPS with R is not difficult at all.

5.3.2 FTP

The File Transfer Protocol (FTP) was developed to transfer files from client to server (upload), from server to client (download), and to manage directories. FTP was first specified in 1971 by Abhay Bhushan (1971); its current specification (see Postel and Reynolds 1985) is almost 30 years old. In principle, HTTP has several advantages over FTP. It allows persistent, keep-alive connections, that is, connections between client and server that are maintained for several transfers. This is not possible with FTP, where the connection has to be reestablished after each transfer. Further, FTP does not natively support proxies and pipelining, that is, several simultaneous requests before receiving an answer. On the upside, FTP may be faster under certain circumstances, as it does not come with a bunch of header fields like HTTP—just the binary or ASCII files are transferred.

FTP uses two ports on each side, one for data exchange (“data port,” the default is port 20) and one for command exchange (“control port,” the default is port 21). Just like HTTP, FTP comes with a set of commands that specify which files to transfer, what directories to create, and many other operations.¹⁷ FTP connections can be established in two different modes: the active mode and the passive mode. In active FTP, the client connects with the server's command port and then requests a data transfer to another port. The problem with this mode is that the actual data connection is established by the server. As the client's firewall has not been told that the client expects data to come in on a certain port, it usually blocks the server's attempt to deliver the data. This issue is tackled with the passive mode in which the client initiates both the command and the data connection. We are going to demonstrate accessing FTP servers with R in Section 9.1.2.

5.4.1 The libcurl library

Much of what we need to do with R on the web is dramatically facilitated by libcurl (Stenberg 2013). libcurl is an external library programmed in C. Development began in 1996 by Daniel Stenberg and the cURL project and has since been under continuous development. The purpose of libcurl is to provide an easy interface to various Internet protocols for programs on many platforms. Over time, the list of features has grown and now comprises a multitude of possible actions and options to configure, among others, HTTP communication. We can think of it as a tool that knows how to

• specify HTTP headers;
• interpret URL encoding;
• process incoming streams of data from web servers;
• establish SSL connections;
• connect with proxies;
• handle authentication;

and much more. In contrast, R’s url() and download.file() are precious little help when it comes to complex tasks like filling forms, authentication, or establishing a stateful conversation. Therefore, libcurl has been tapped to enable users to work with the libcurl library in their ordinary programming environment. In his manifest of RCurl and libcurl’s philosophy, Temple Lang points out the benefits of libcurl: Being the most widely used file transfer library, libcurl is extraordinarily well tested and flexible (Temple Lang 2012a). Further, being programmed in C makes it fast. To get a first impression about the flexibility of libcurl, you might want to start by taking a look at the available options of libcurl’s interface at http://curl.haxx.se/libcurl/c/curl_easy_setopt.html. Alternatively, you can type the following into R to get the comprehensive list of libcurl’s “easy” interface options that can be specified with RCurl:

R> names(getCurlOptionsConstants())

Currently, there are 174 available options. Among them are some that we have already relied on above, like useragent or proxy. We sometimes speak of curl options instead of libcurl options for reasons of convenience. curl is a command line tool also developed by the cURL software project. With R we draw on the libcurl library.¹⁸

5.4.2 Basic request methods

R> getURL("http://www.r-datacollection.com/materials/http/helloworld.html")
[1]"<html>
<head><title>Hello World</title></head>
<body><h3>Hello World</h3>
</body>
</html>"

R> pngfile <- getBinaryURL("http://www.r-datacollection.com/materials/http/sky.png")

R> writeBin(pngfile,"sky.png")

R> url <-"http://www.r-datacollection.com/materials/http/GETexample.php"
R> namepar <-"Eddie"
R> agepar <-"32"
R> url_get <- str_c(url,"?","name=", namepar,"&","age=", agepar)

R> cat(getURL(url_get))
Hello Eddie!
You are 32 years old.

R> url <-"http://www.r-datacollection.com/materials/http/GETexample.php"
R> cat(getForm(url, name ="Eddie", age = 32))
Hello Eddie!
You are 32 years old.

R> url <-"http://www.r-datacollection.com/materials/http/POSTexample.php"
R> cat(postForm(url, name ="Eddie", age = 32, style ="post"))
Hello Eddie!
You are 32 years old.

R> url <-"r-datacollection.com/materials/http/helloworld.html"
R> res <- getURL(url = url, customrequest ="HEAD", header = TRUE)
R> cat(str_split(res,"
")[[1]])
HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:07 GMT
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html

5.4.3 A low-level function of RCurl

RCurl builds on the powerful libcurl library, making it a mighty weapon in the hands of the initiated and an unmanageable beast in the hands of others. The low-level function curlPerform() is the workhorse of the package. The function gathers options specified in R on how to perform web requests—which protocol or methods to use, which headers to set—and patches them through to libcurl to execute the request. Everything in this function has to be specified explicitly so later on we will come back to more high-level functions. Nevertheless, it is useful to demonstrate how the high-level functions work under the hood.

We start with a call to curlPerform() to request an HTML document:

R> url <-"www.r-datacollection.com/materials/http/helloworld.html"
R> (pres <- curlPerform(url = url))
OK
0

Instead of getting the content of the URL we only get the information that everything seems to have worked as expected by the function. This is because we have to specify everything explicitly when using curlPerform(). The function did retrieve the document but did not know what to do with the content. We need to define a handler for the content. Let us create one ourselves. First, we create an object pres to store the document and a function that takes the content as argument and writes it into pres. As the list of options can get extensive we save them in a separate object performOptions and pass it to curlPerform():

R> pres <- NULL
R> performOptions <- curlOptions(url = url,
writefunc = function(con) pres <<- con )
R> curlPerform(.opts = performOptions)
OK
 0
R> pres
[1]"<html>
<head><title>Hello World</title></head>
<body><h3>Hello World</h3>
</body>
</html>"

That looks more like what we would have expected. In addition to the content handler, there are other handlers that can be supplied to curlPerform(): a debug handler via debugfunc, and a HTTP header handler via headerfunc. There are sophisticated functions in RCurl for each of these types that spare us the need to specify our own handler functions. For content and headers, basicTextGatherer() turns an object into a list of functions that handles updates, resets, and value retrieval. In the following example we make use of all three. Note that in order for debugfunc to work we need to set the verbose option to TRUE:

Using the value() function of content we can extract the content that was sent from the server.

R> str_sub(content$value(), 1, 100)
[1]"<html>
<head><title>Hello World</title></head>
<body><h3>Hello World</h3>
</body>
</html>"

header$value() contains the headers sent back from the server:

R> header$value()
[1]"HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:10 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 89
Content-Type: text/html

"

debug$value() stores various pieces of information on the HTTP request. See Section 5.4.6 for more information on this topic:

5.4.4 Maintaining connections across multiple requests

It is a common scenario to make multiple requests to a server, especially if we are interested in accessing a set of resources like multiple HTML pages. The default setting in HTTP/1.0 is to establish a new connection with each request, which is slow and inefficient. Connections in HTTP/1.1 are persistent by default, meaning that we can use the same connection for multiple requests. RCurl provides the functionality to reuse established connections, which we can exploit to create faster scrapers.

Reusing connections works with the so-called “curl handles.” They serve as containers for the connection itself and additional features/options. We establish a handle as follows:

R> handle <- getCurlHandle()

The handle in the handle object is of class CURLHandle and currently an empty container. We can add useful curl options from the list listCurlOptions(), for example:

In the example, we specify a User-Agent header field that contains the current R version and a From header field containing an email address, set the followlocation argument to TRUE, and activate cookie management (see Section 5.4.5). The curl handle can now be used for multiple requests using the curl argument. For instance, if we have a vector of URLs in the object url, we can retrieve them with getURL() fed with the settings in the curl handle from above.

R> lapply(urls, getURL, curl = handle)

Note that the curl handle container is not fixed across multiple requests, but can be modified. As soon as we specify new options in a request, these are added—or old ones overridden—in the curl handle, for example, with getURL(urls, curl = handle, httpheader = c(from ="[email protected]")). To retain the status of the handle but use a modified handle for another request, we can duplicate it and use the “cloned” version with dupCurlHandle():

Cloning handles can be especially useful if we want to reuse the settings specified in a handle in requests to different servers. Not all settings may be useful for every request (e.g., protocol settings or referrer information), and some of the information should probably be communicated only to one specific server, like authentication details.

When should we use curl handles? First, they are generally convenient for specifying and using curl options across an entire session with RCurl, simplifying our code and making it more reliable. Second, fetching a bunch of resources from the same server is faster when we reuse the same connection.

5.4.5 Options

We have seen that we can use curl handles to specify options in RCurl function calls. However, there are also other means. Generally, RCurl options can be divided into those that define the behavior of the underlying libcurl library and those that define how information is handled in R. The list of possible options is vast, so we selected the ones we frequently use and listed them in Table 5.3. Some of these options were already introduced above, the others will be explained below. Let us begin by showing the various ways to declare options.

We can declare options for single calls to the high-level functions (e.g., getURL, getURLContent, and getBinaryURL). In this case the options will only affect that single function call. In the following example we add header = TRUE in order to not only retrieve the content but also the response header:²⁰

R> url <-"www.r-datacollection.com/materials/http/helloworld.html"
R> res <- getURL(url = url, header = TRUE)
R> cat(str_split(res,"
")[[1]])
HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:11 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 89
Content-Type: text/html

<html>
<head><title>Hello World</title></head>
<body><h3>Hello World</h3>
</body>
</html>

Another, more persistent way of specifying options is to bind them to a curl handle as described in the previous section. Every function using this handle via the curl option will use the same options. If a function uses the handle and redefines some options or adds others, these changes will stick to the handle. In the following example we create a new handle and specify that the HTTP method HEAD should be used for the request:

R> handle <- getCurlHandle(customrequest ="HEAD")
R> res <- getURL(url = url, curl = handle)
R> cat(str_split(res,"
")[[1]])

The first function call using the handle results in an empty vector because HEAD provides no response body and the header option was not specified. In the second call we add the header argument to retrieve header information:

R> res <- getURL(url = url, curl = handle, header = TRUE)
R> cat(str_split(res,"
")[[1]])
HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:14 GMT
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html

The added header specification has become part of the handle. When we reuse it, we do not need to specify header = TRUE anymore:

R> res <- getURL(url = url, curl = handle)
R> cat(str_split(res,"
")[[1]])
HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:16 GMT
Server: Apache
Vary: Accept-Encoding
Content-Type: text/html

With dupCurlHandle() we can also copy the options set in one handle to another handle:

R> handle2 <- dupCurlHandle(handle)
R> res <- getURL(url = url, curl = handle2)

Two more global approaches exist. First, we can define a list of options, save it in an object, and pass it to .opts when initializing a handle or calling a function. The curlOptions() function helps to expand and match option names:

R> curl_options <- curlOptions(header = TRUE, customrequest ="HEAD")
R> res <- getURL(url = url, .opts = curl_options)

To specify further curl options when using getForm() and postForm(), we have to use the .opts argument. Otherwise the function cannot distinguish between form parameters and curl options. Further, instead of specifying the parameters of POST directly after the URL, they can also be processed in a list passed to the .params option:

Second, we can even use R’s global option system to specify standard values that will be part of each curl handle or function call unless specified otherwise:

R> options(RCurlOptions = list(header = TRUE, customrequest ="HEAD"))
R> res <- getURL(url = url)
R> options(RCurlOptions = list())

Now that we know how to set options, we should inspect two options a little closer because they can control HTTP methods and HTTP headers: customrequest and httpheader. The customrequest option was already used throughout the examples above and tells libcurl to use whatever method specified—for example, POST, HEAD, or PUT instead of the default GET. For instance, we can transform getURL() into a function that posts form information:

Individual HTTP headers can be added using the httpheaders option. We add them as a list where names of the list items identify the header name and their values correspond to header values. Let us specify some helpful standard headers and pass them to a call to getURL(). To check which headers are sent along the HTTP request, we send our request to a page that simply returns the HTTP request that was received. First we send a request without any further specifications:

R> url <-"r-datacollection.com/materials/http/ReturnHTTP.php"
R> res <- getURL(url = url)
R> cat(str_split(res,"
")[[1]])
GET /materials/http/ReturnHTTP.php HTTP/1.1
Authorization:
Host: r-datacollection.com
Accept: */*
Connection: close

The results from above show that only few headers are sent along our HTTP request. Now we want to add a from and user-agent header specification to the list:²¹

To conclude this section we provide an example of a list of default options. We recommend setting these options via options() directly at the start of a session after loading RCurl. This way it is transparent which options are set as default values for all functions and handles with the convenience of having to type the options only once. First, we include the from and user-agent options from above to always identify ourselves. Next we set followlocation to TRUE to tell libcurl to automatically follow redirections—maxredirs restricts these redirections to avoid infinite loops. Next, we specify a default connection timeout as well as a completion timeout (connecttimeout and timeout). The former tells libcurl to stop trying to connect to a server after 10 seconds while the latter timeout is for the maximum time we give libcurl to complete a request altogether. The standard libcurl connection timeout is 300 seconds. Setting the cookiefile option enables libcurl to receive, save, and send back cookies. The last option specifies the location for files that contain digital signatures for SSL certificate verification:

The list of default options can be emptied using

R> options(RCurlOptions = list())

5.4.6 Debugging

What happens in case of an error in an HTTP call? We have documented in Section 5.1.5 that many things can go wrong and the server communicates the presumed type of error. In the simplest of cases, we might have gotten the URL wrong:

R> getURL("http://www.stata-datacollection.com")
Error: Could not resolve host: www.stata-datacollection.com; Host not found

Some errors might be less obvious but still prevent us from receiving content from a server. In this section we will show some tools that help identify reasons why things do not work as expected. We know already that we can ask RCurl functions to capture the response headers in addition to the content by setting the header option to TRUE. Often, however, we want to have more information, for example the information that arrives at the server after we put a request to it.

A generally useful tool for HTTP debugging is the service at http://httpbin.org, which provides a set of endpoints for specific HTTP requests. To check whether a GET request is specified correctly and what information arrives at the server, we write

Moreover, RCurl provides its own way of checking HTTP calls by specifying a debug gatherer within the function call. The procedure is powerful and does not rely on external resources. It works as follows. First, we create an object that contains three functions (update(), value(), reset()) by calling the debugGatherer():

R> debugInfo <- debugGatherer()
R> names(debugInfo)
[1]"update""value""reset"
R> class(debugInfo[[1]])
[1]"function"

In a second step, we request a document using the ordinary getURL() function and use the debugfunction option. With this option we specify a function that gathers debug information as is supplied by libcurl—the update() function we stored in debugInfo. To make the necessary debugging information available, we have to set the verbose option to TRUE:

R> url <-"r-datacollection.com/materials/http/helloworld.html"
R> res <- getURL(url = url, debugfunction = debugInfo$update, verbose = T)

In a third and last step, we access the debugging information gathered during the execution of getURL() by calling the value() function stored in the debugInfo object. The value function provides seven items:

The first item of the resulting vector—text—captures information libcurl provides about the procedure:

R> cat(debugInfo$value()["text"])
About to connect() to r-datacollection.com port 80 (#0)
  Trying 173.236.186.125... connected
Connected to r-datacollection.com (173.236.186.125) port 80 (#0)
Connection #0 to host r-datacollection.com left intact
Closing connection #0

headerIn stores the HTTP response header:

R> cat(str_split(debugInfo$value()["headerIn"],"
")[[1]])
HTTP/1.1 200 OK
Date: Wed, 26 Mar 2014 00:20:25 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 89
Content-Type: text/html

The HTTP request header is stored in headerOut:

R> cat(str_split(debugInfo$value()["headerOut"],"
")[[1]])
GET /materials/http/helloworld.html HTTP/1.1
Host: r-datacollection.com
Accept: */*

The body of the response is contained in dataIn:

R> cat(str_split(debugInfo$value()["dataIn"],"
")[[1]])
<html>
<head><title>Hello World</title></head>
<body><h3>Hello World</h3>
</body>
</html>

The body of the sent data—for example, if we use the POST method—is found in dataOut:

R> cat(str_split(debugInfo$value()["dataOut"],"
")[[1]])

In this example it is empty as we used the GET method, which, by definition, does not send any data along with the request body.

The remaining two items sslDataIn and sslDataOut are analogous to dataIn and dataOut but for encrypted connections. They are also empty in our request:

R> cat(str_split(debugInfo$value()["sslDataIn"],"
")[[1]])
R> cat(str_split(debugInfo$value()["sslDataOut"],"
")[[1]])

Another source of valuable information might be the getCurlInfo() function, which provides additional information on the present state of a curl handle. To get the information we first specify a handle, use it in a function call, and then apply getCurlInfo() to the handle:

R> handle <- getCurlHandle()
R> url <-"r-datacollection.com/materials/http/helloworld.html"
R> res <- getURL(url = url, curl = handle)
R> handleInfo <- getCurlInfo(handle)

The information provided is manifold:

A useful operation might be to consider the total time it took to complete the request and the time it took to do all the things necessary to start the transfer—that is, resolve the host name, establish the connection to the host, and send the request—to get an idea where possible bottlenecks occur:

R> handleInfo[c("total.time","pretransfer.time")]
$total.time
[1] 0.219

$pretransfer.time
[1] 0.11

If the time before the actual download takes up a substantial part of the overall time it takes to complete a request, we should—for multiple requests to the same server—ensure that connections are reused. Let us gather the ratio of pre-transfer time to total time ten times in succession with and without reusing the same handle:

The gathered times show quite nicely how connection times can accumulate when establishing connections for each request and how this can be prevented by reusing curl handles that establish a connection once and reuse this connection to send multiple requests:

R> preTransTimeNoReuse
  [1] 0.110 0.094 0.109 0.109 0.094 0.109 0.110 0.109 0.125 0.110
R> preTransTimeReuse
  [1] 0.125 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

5.4.7 Error handling

After discussing tools for discovering why things might not work, this section presents an RCurl specific way to handle errors. We can experience various types of errors—those that we generated ourselves (e.g., by specifying a wrong host name or asking for a nonexisting resource), those that are due to a broken connection, and those generated on the server side. The set of functions provided by RCurl to retrieve content know a lot of different error types. We get a list by calling the getCurlErrorClassNames() function. We selected a couple of the most common ones:

Using tryCatch()²² we can specify individual actions to react to different types of errors. As an example we choose a user-generated error. We have a set of two URLs. We begin by trying to collect the first one. This operation fails because the host does not exist. The second URL serves as a replacement in case the first URL produces an error of class COULDNT_RESOLVE_HOST:

R> url1 <-"wwww.r-datacollection.com/materials/http/helloworld.html"
R> res <- getURL(url1)
Error: Could not resolve host: wwww.r-datacollection.com; Host not found

The call produces an error and res is not created. This can cause further errors in the program if we try to process the res object.

Now let us try to react to the error. In the following snippet, the object res stores the results from a call to tryCatch(). Within the function we first state the default—retrieving the URL. The next two statements are of the form errorType = errorFunction. For each error tryCatch checks whether the class of the error matches one of the error names provided—in our case COULDNT_RESOLVE_HOST and error—and executes the matching function. In the example the default statement produces a resolve host error and the second URL will be retrieved. Any other error would have produced an NA that would have been assigned to res and the default error message would have been printed.

5.4.8 RCurl or httr—what to use?

RCurl is a quite powerful package that helps to make the most sophisticated requests and to receive and process the incoming response. At times, it is a bit bulky, however. Fortunately, there is a package that offers a more slender interface: the httr package (Wickham 2012). It builds upon RCurl by wrapping the functions we have discussed so far.

In Table 5.4 we contrast functions of both packages to perform selected HTTP and authentication tasks. Some of the functions have quite a different syntax and do not always provide the same functionality. Although we do not want to go into more details of the httr package at this point, there is no reason to be dogmatic and work with only one of the two packages. In fact, httr offers several features that considerably ease some data collection tasks, for example, authentication via OAuth (see Section 9.1.11).