Most social networks/platforms provide certain predefined ways of accessing their data. Across previous chapters, we have seen and utilized Application Programming Interfaces (APIs) in general. Unlike other platforms we have covered so far, StackExchange provides a slightly different way to access its data. Since the data on the platform is under the Creative Commons license, the StackExchange team takes it seriously and has opened up most of its data for the public. The following are popular methods of getting data related to any StackExchange website:

As discussed in the previous section, StackExchange provides multiple ways of getting data from the platform. Depending upon the use case, one method may be preferred over the other. We as users can be sure of the fact that the same data is available across the methods. For the use cases in this chapter, we will be utilizing the data dumps from the platform.

StackExchange's data dumps are fairly extensive in information they contain and granular enough to work with. These also free us of the query rate limitations imposed upon APIs or row count limitations set on the data explorer methods. This method may be limited from the perspective that the data is not real-time. This should not be much of a barrier given the fact that the data is not too old (usually lags by a month or so) and we can work on most behaviors with a little data lag unless we have a real-time analytics use case to work with. Before we dive into the use cases, let us get familiarized with the data dumps themselves.

Quick overview of the data in data dumps

StackExchange exposes quite an extensive set of data from the platform which is neatly segregated based on sites and entity levels. To get a complete picture of the data landscape, let us visually understand how the different files are associated:

StackExchange sample site wise entity relationships (diagram only shows key fields)

Note

This is not an ER diagram per se, it is a visual representation of relationships between different entities available as part of a StackExchange data dump. We have taken the liberty of using ER concepts and digressing wherever required for clarity and brevity purposes.

The preceding visual representation not just helps us understand the relationships between different data elements available, it also helps us during our analysis of the data to think critically and try and find potential spots for deriving insights. This will be clearer when we dive into the use cases.

As is evident from the preceding diagram, posts and users are the most important and central entities to the whole landscape. This shouldn't be surprising at all since posts (Q&A) are the site's bread and butter and it is the users who drive the whole platform.

Let us have a brief look at post and user files for a better understanding.

Posts

We discussed in the previous section about Posts.xml in brief. Now let us have a look at the contents of this file to see what we have at hand and prepare a strategy to utilize the contents to uncover insights. The Posts.xml file looks like the following snapshot:

Sample Posts.xml

The file consists of multiple <row> tags, one for each post (where a post can be a question or answer). Each row element consists of various attributes. Each post is uniquely identified by its Id field. It has attributes such as:

• PostTypeId: This signifies if a post is a question or an answer
• CreationDate: This creates a date and timestamp of the post
• Title: This is the question header we get on the website
• Body: This is the post's actual text

There are many more attributes. Each row consists of about 20 such attributes. The attributes are a mix of numeric, string, and categorical data types. The following is a snapshot of the data in tabular form at:

Posts data in tabular format

Users

It is the users who drive any social network and StackExchange is no exception. The Users.xml file contains public information associated with each registered user. Users on the platform are characterized by the following attributes:

• CreationDate: The date and timestamp when the user registered on the platform
• DisplayName: The identifiable string used by the user while interacting with the platform
• UpVotes/DownVotes: The count of votes this user has contributed to

There are many more attributes associated with demographic information such as age, location, and so on. We can utilize this information to get user-centric insights from the data. The following is a snapshot of this data in tabular format:

Users data in tabular format

Now that we have a basic understanding of what is available as part of the StackExchange data dumps, let us get our hands dirty and work upon a few use cases.

As discussed in the previous sections, the data is available on a per site basis and each site's data is segregated into separate XML files. For the purpose of the use cases to be discussed in the coming sections, we will be loading these XML files as required into R for analysis (mostly in the form of R DataFrames).

The following is a quick utility function we will be utilizing to load the XML files into R as DataFrames:

# load XML data
loadXMLToDataFrame<- function(xmlFilePath){
  doc <- xmlParse(xmlFilePath)  
  xmlList<- xmlToList(doc)
  total<-length(xmlList)
  data<-data.frame()
  
  for(i in 1: total){
    data <- rbind.fill(data,as.data.frame(as.list( xmlList[[i]])))
  }
    return(data)
}

The functions xmlParse() and xmlToList() are available through the package XML. This package contains many such utility functions. The rest of the code is pretty straightforward and returns a dataframe object from the XML input.

Now that we have the utility to get data into R, let us load the Posts.xml into R. The following snippet will return a DataFrame consisting of rows from Posts.xml:

> PostsDF <- loadXMLToDataFrame(paste0(path,"Posts.xml"))

Similarly, we can load other XML files as required. Let us now get started with our use cases.