Data scientists and data engineers get data from a variety of different sources. Often, data might come as CSV files or database dumps. Sometimes, we have to obtain the data through a web API.
An individual or organization sets up a web API to distribute data to programs over the Internet (or an internal network). Unlike websites, where the data is intended to be consumed by a web browser and shown to the user, the data provided by a web API is agnostic to the type of program querying it. Web servers serving HTML and web servers backing an API are queried in essentially the same way: through HTTP requests.
We have already seen an example of a web API in Chapter 4, Parallel Collections and Futures, where we queried the "Markit on demand" API for current stock prices. In this chapter, we will explore how to interact with web APIs in more detail; specifically, how to convert the data returned by the API to Scala objects and how to add additional information to the request through HTTP headers (for authentication, for instance).
The "Markit on demand" API returned the data formatted as an XML object, but increasingly, new web APIs return data formatted as JSON. We will therefore focus on JSON in this chapter, but the concepts will port easily to XML.
JSON is a language for formatting structured data. Many readers will have come across JSON in the past, but if not, there is a brief introduction to the syntax and concepts later on in this chapter. You will find it quite straightforward.
In this chapter, we will poll the GitHub API. GitHub has, over the last few years, become the de facto tool for collaborating on open source software. It provides a powerful, feature-rich API that gives programmatic access to nearly all the data available through the website.
Let's get a taste of what we can do. Type api.github.com/users/odersky in your web browser address bar. This will return the data offered by the API on a particular user (Martin Odersky, in this case):
{
"login": "odersky",
"id": 795990,
...
"public_repos": 8,
"public_gists": 3,
"followers": 707,
"following": 0,
"created_at": "2011-05-18T14:51:21Z",
"updated_at": "2015-09-15T15:14:33Z"
}The data is returned as a JSON object. This chapter is devoted to learning how to access and parse this data programmatically. In Chapter 13, Web APIs with Play, you will learn how to build your own web API.
Tip
The GitHub API is extensive and very well-documented. We will explore some of the features of the API in this chapter. To see the full extent of the API, visit the documentation (https://developer.github.com/v3/).
JSON is a format for transferring structured data. It is flexible, easy for computers to generate and parse, and relatively readable for humans. It has become very common as a means of persisting program data structures and transferring data between programs.
JSON has four basic types: Numbers, Strings, Booleans, and null, and two compound types: Arrays and Objects. Objects are unordered collections of key-value pairs, where the key is always a string and the value can be any simple or compound type. We have already seen a JSON object: the data returned by the API call api.github.com/users/odersky.
Arrays are ordered lists of simple or compound types. For instance, type api.github.com/users/odersky/repos in your browser to get an array of objects, each representing a GitHub repository:
[
{
"id": 17335228,
"name": "dotty",
"full_name": "odersky/dotty",
...
},
{
"id": 15053153,
"name": "frontend",
"full_name": "odersky/frontend",
...
},
...
]We can construct complex structures by nesting objects within other objects or arrays. Nevertheless, most web APIs return JSON structures with no more than one or two levels of nesting. If you are not familiar with JSON, I encourage you to explore the GitHub API through your web browser.