The first step for securing access to any APIs is the creation of the developer account. We will do the same to get access credentials for our news access APIs. We will use The Guardian's APIs to get the required news data for our analysis.

The step-by-step process for creating a The Guardian developer account is illustrated as follows:

Once the registration process is complete, you will get your API key through an e-mail to the e-mail address that you gave on the registration page:

E-mail with API key

This key will be required for each of your API calls. Please keep the key secret, as it is used for accounting purposes by the data provider.

The data access procedure in our earlier excursions was relatively simple when compared to the access process required for extracting the news data. The procedure involves a two-step process (we will detail both of them later):

The Guardian's APIs have an R wrapper around them, but it doesn't seem to work properly. So, we will borrow the API access mechanism that we developed in Chapter 4, Foursquare—Are You Checked in Yet?

To jog your memory about the steps involved in the process, here is a recap of them:

The Guardian's API has a very friendly resource here, which will allow us to skip the process of tediously going through the documentation to arrive at the required URL. Go to the address http://open-platform.theguardian.com/explore/. You will be greeted with the following page:

Exploring The Guardian's API

This useful web page allows you to construct your access URL by inputting the various filters available. The available filters can be accessed by selecting them from the Add filters... option box. Adding a filter adds a corresponding text box to the page, which can be used to specify the value for the filter.

For an example, suppose we want to search for articles mentioning brexit in the Opinion section of The Guardian between 01-06-2016 and 25-06-2016 (please note the British date format of dd-mm-yyyy as this is a UK website). The required API end point URL can be constructed from the preceding page by selecting the filters and then specifying the values. Take a look at the following completed web page to get the hang of the process:

Interactively creating API URL

We can directly use the URL generated to make our API call. This interface makes the URL creation a very simple process. The next step is to query it and then parse the JSON response to extract the required information.

Once we have generated the URL, we make the API call and then use the JSON returned by the call to extract the data. Let's use the same example as previously to extract the required data:

# The uri extracted from the online page

URI = "http://content.guardianapis.com/search?from-date=1016-06-01&to-date=2016-06-25&section=commentisfree&q=brexit&api-key=xxxxxxxxxxxxx"

# Making the API call

json_res <- getURL(URI, .mapUnicode=TRUE)

Once we have the response from the API call, we again have to parse the JSON result to get the data. To parse a JSON, it is very important to know the structure of the object. Here we are in luck, as the same web page that helped us in creating the URL will also give a sample of results that we can use to find out the structure of the JSON response. The image shown here gives the structure of the API call we have just executed:

JSON structure for parsing

The structure of result is a very simple one, which is good news for us as it simplifies our data extraction logic. Please revisit the parsing process explained in an earlier chapter to find out how this JSON can be parsed.

To parse this JSON we need to enter the results object, gather all its elements as an array, and then extract all the fields as string fields. The code that will do this is quite straightforward and can be easily written based on our earlier excursions. But there is an extra complexity here. Typically, for an article search, you will get multiple results and not all of those results will be returned in a single API call. We would need to solve this problem if we want to extract all our news articles' information.

To do this we need to pay attention to the other fields in the JSON response; we have two parameters called pagesize and pages. The pagesize parameter tells us the number of responses in a page and pages parameter gives us the total number of pages. We can combine these two pieces of information and write code so that we iterate the required number of pages, making multiple calls, and combine all that information in to a resultant DataFrame, which will be our final output. Here is the code that will complete this process:

# Find out the number of pages

num_pages = json_res %>% 
   enter_object("response") %>% 
   spread_values(num_pages = jnumber("pages"))
num_pages = num_pages$num_pages

# Initialize an empty data frame

out_df = data.frame()
for (num in 1:num_pages){
    uriTemplate <- "http://content.guardianapis.com/search?from-date=1016-06-01&to-date=2016-06-25&section=commentisfree&q=brexit&api-key=xxxxxxxxxxxxxxxxxxxxxxxxx&page=%s"
    apiurl<- sprintf(uriTemplate,num)
    json_res <- getURL(apiurl, .mapUnicode=TRUE)
    urls <- as.data.frame(json_res %>% 
enter_object("response") %>% 
enter_object("results")  %>%
            gather_array() %>% 
                spread_values(url = jstring("webUrl"), 
                type =   jstring("type"), 
                sectionName = jstring("sectionName"), 
                webTitle = jstring("webTitle"),
                sectionId = jstring("sectionId")
                              ))
    urls$document.id <- NULL
    urls$array.index <- NULL
    out_df = rbind(out_df, urls)
    Sys.sleep(10)
}

An important part of the preceding code is the Sys.sleep(10); this code instruction makes the execution of our code pause for specified time period, here 10 seconds, which is important when we are querying a data provider repeatedly and want to avoid hitting their ceiling API call rate.

The preceding code will compile a DataFrame; a part of which is shown here:

DataFrame with the links data

HTML scraping from the links – the bigger monster

Even after the elaborate first step, we are still not close to extracting the textual data. The procedure for extracting text is easy to explain: iterate through the DataFrame, visit each URL, and extract the text data. Those who are familiar with the complexities of web page scraping will understand the sarcasm of the previous lines. We will be using the rvest, tidyjson, magrittr, and RCurl libraries for our scraping process, which we assume are installed on your system.

The general steps involved in any web scraping task are as follows:

1. Extract the HTML source of the web page.
2. Analyze the HTML source of the web page.
3. Find tags of interest that contain the information.
4. Extract the data for those tags programmatically and repeat the whole process.

The first step is extracting the HTML source of the page. If you look at the DataFrame of results that we extracted in the last step, it contains a column that will give us the URL of the news story. After that, we can read the read_html function to extract the HTML tree of the URL.

# Extracting the HTML data

b <- read_html(curl(url, handle = curl::new_handle("useragent" = "Mozilla/5.0"))) 

It is important to add the handle argument, as it identifies our system as a legitimate client and we avoid being timed out by the server.

The next step is the toughest part of the process. Every web page is unique and yet similar in some sense. As we are looking at web pages only from a single source, the web pages will be similar, but even with that information we will have to analyze the source to find out how we can extract the textual data. Take a look at the HTML source of one of the URLs that we extracted:

Web source of a sample URL

You can observe that the HTML source of our URL is a scary sight, and it runs for some 2,000 lines. How can we ever get the required information from this huge garbled text?

This is the part of scraping where you go through the source looking for some clues, which can be used for the parsing. In our case, the clue is the text of the article. We also have access to the web page that is generated by this source. The web page will give us the text that we are looking for in our source. See the next image in which we have searched for some of the text snippets on the web page; it will reveal a very repeatable structure that we can use:

Web source of a sample URL (continued)

If you look closely at the image, you will observe that all the textual data is contained within <p> (paragraph tag of HTML) tags. This is interesting information as this will make our parsing process a breeze.

Note

Please keep in mind that the HTML structures of web pages are not only unique but they are very dynamic in nature. A simple change to the structure can break our nicely written scraping routine. To fix any such changes, the process is same as before. You go fishing in the source and find out information that can be used for parsing.

Now that we know the tags of interest, we can easily find them out using the html_nodes function from the library:

# Find the nodes for paragraph tags

paragraph_nodes = html_nodes(b, xpath = ".//p")

The preceding line of code will search for all paragraph nodes in the tree structure of the HTML source and extract them for processing. Once you have the nodes extracted, the next step is super simple:

# Tidy up by removing whitespaces, newlines and collapsing all nodes into a single node

nodes<-trimws(html_text(paragraph_nodes))
nodes<-gsub("
", "", nodes)
nodes<-gsub("  ", "", nodes)
content = paste(nodes, collapse = " ")

Now we have the textual content of the web page extracted. The following image gives the extracted textual content:

Extracted textual content

We can iterate the whole process for all of the DataFrame to end up with the content for each of our URLs.