When you need to download more detailed information/data from sites, most often the website may have an API—it is a means to access data in a controlled manner. An API allows the user to specify simple URLs suffixed with queries to retrieve only the necessary fields/information.

Most APIs require users to sign up for an account and register a token. The token is generally application specific and each application can get its own token that will enable more granular control such as number of requests and throttling. Having an application token also prevents your applications from getting restricted as the site may not know your identity and, if there are a sufficiently high number of requests, block access to the resources.

Examples of APIs can be found at the following link: http://ropengov.github.io/projects/.

Examples of open data can be found at the following links: https://dev.socrata.com/docs/endpoints.html and https://cran.r-project.org/web/packages/RSocrata/index.html:

# Example of a Web API Call

install.packages("RSocrata")

library(RSocrata)

datasets <- ls.socrata("https://soda.demo.socrata.com")

names(datasets)

# View the datasets

data.frame(datasets$title, datasets$identifier)

# Simple download

population <- read.socrata("https://soda.demo.socrata.com/api/views/irft-er6i")

# Using App Tokens

# ** This needs to be set up, create an account at https://opendata.socrata.com/login **

token <- YOUR_TOKEN

population <- read.socrata("https://soda.demo.socrata.com/api/views/irft-er6i", app_token = token)

head(population)

# Using ID/Password

# ** This needs to be set up, create an account at https://opendata.socrata.com/login **

email <- Sys.getenv("SOCRATA_EMAIL", "Your_Registered_Email_Address")

password <- Sys.getenv("SOCRATA_PASSWORD", "Your_Password")

privateURL <- "https://soda.demo.socrata.com/resource/a9g2-feh2.csv" # dataset

# Uncomment the following line once you have set up your id and password

read.socrata(url = privateURL, email = email, password = password)

The following section highlights a very simple use case of an API along with R. The topic is vast as there are thousands, if not millions of APIs on the web.