2.1.1.1 Categorical Variables

Categorical variables can be classified into Nominal, Dichotomous, and Ordinal. We explain each type in a little more detail.

• Nominal

  These are variables with two or more categories without any regard for ordering. For example, in polling data from a survey, the variable state, or candidate names. The number of states and candidates are definite and it doesn't matter what order we choose to present our data. In other words, the order of state or candidate name has no significance in its relative importance in explaining the data.

• Dichotomous

  A special case of nominal variables with exactly two categories such as gender, possible outcomes of a single coin toss, a survey questionnaire with a checkbox for telephone number as mobile or landline, or the outcome of election win or loss (assuming no tie ).

• Ordinal

  Just like nominal variables, we can have two or more categories in ordinal variables with an added condition that the categories are ordered. For example, a customer rating for a movie in Netflix or a product in Amazon. The variable rating has a relative importance on a scale of 1 to 5, 1 being the lowest rating and 5 the highest for a movie or product by a particular customer.

2.1.1.2 Continuous Variables

Continuous variables are subdivided into Interval and Ratio:

• Interval

  The basic distinction is that they can be measured along a continuous range and they have a numerical value. For example, the temperature in degrees Celsius or Fahrenheit is an interval variable. Note here that the temperature at 0^o C is not the absolute zero, which simply means 0^o C has certain degree of temperature measure than just saying the value means none or no measure.

• Ratio

  In contrast, ratio variables include distance, mass, and height. Ratio reflects the fact that you can use the ratio of measurements. So, for example, a distance of 10 meters is twice the distance of 5 meters. A value 0 for a ratio variable means a none or no measure.

2.1.2.1 Comma-Separated Values

CSV or TXT is one of the most widely used data exchange formats for storing tabular data containing many rows and columns. Depending on the data source, the rows and columns have a particular meaning associated with them. Typical information looks like the following example of employee data in a company. In R, the read.csv function is widely used to read such data. The argumentssep specifies the delimiting character and header takes TRUE or FALSE, depending on whether the dataset contains the column names or not.

read.csv("employees.csv", header =TRUE, sep =",")
    Code First.Name Last.Name Salary.US.Dollar.
 1 15421       John     Smith             10000
 2 15422      Peter      Wolf             20000
 3 15423       Mark   Simpson             30000
 4 15424      Peter    Buffet             40000
 5 15425     Martin    Luther             50000

2.1.2.2 Microsoft Excel

Microsoft Excel file format (.xls or .xlsx) has been undisputedly the most popular data file format in the business world. The primary purpose being the same as CSV files, but Excel files offer many rich mathematical computations and elegant data presentation capabilities. Excel features calculation, graphing tools, pivot tables, and a macro programming language called Visual Basic for Applications. The programming feature of Excel has been utilized by many industries to automate their data analysis and manual calculations. This wide traction of Excel has resulted in many data analysis software programs that provide an interface to read the Excel data. There are many ways to read an Excel file in R, but the most convenient and easy way is to use the package xlsx.

library(xlsx)
read.xlsx("employees.xlsx",sheetName ="Sheet1")
    Code First.Name Last.Name Salary.US.Dollar.
 1 15421       John     Smith             10000
 2 15422      Peter      Wolf             20000
 3 15423       Mark   Simpson             30000
 4 15424      Peter    Buffet             40000
 5 15425     Martin    Luther             50000                              

2.1.2.3 Extensible Markup Language : XML

Markup languages have a very rich history of evolution by their first usage by William W. Tunnicliffe in 1967 for presentation at a conference. Later Charles Goldfarb formalized the IBM Generalized Markup Language between the year 1969 and 1973. Goldfarb is more commonly regarded as the father of markup languages. Markup languages have seen many different forms, including TeX, HTML, XML, and XHTML and are constantly being improved to suit numerous applications. The basics for all these markup language is to provide a system for annotating a document with a specific syntactic structure. Adhering to a markup language while creating documents ensures that the syntax is not violated and any human or software reader knows exactly how to parse the data in a given document. This feature of markup languages has found a wide range of usage, starting from designing configuration files for setting up software in a machine to employing them in communications protocols.

Our focus here is the Extensible Markup Language widely known as XML. There are two basics constructs in any markup language, the first is markup and the second is the content. Generally, strings that create a markup either start with the symbol < and end with a >, or they start with the character & and end with a ;. Strings other than these characters are generally the content. There are three important markup types—tag, element, and attribute.

• Tags

  A tag is a markup construct that begins with < and ends with >. It has three types.

  • Start tags: <employee_id>

  • End tags: </employee_id>

  • Empty tags: </>

• Elements

  Elements are the components that either begin with a start tag and end with an end tag, both are matched while parsing, or contain only an empty element tag. An example of element:

  • <employee_id>John </employee_id>

  • <employee_name type=”permanent”/>

• Attribute

  Within start tag or empty element tag, an attribute is a markup construct consisting of a name/value pair. In the following example, the element designation has two attributes, emp_id and emp_name.

  • <designation emp_id="15421" emp_name="John"> Assistant Manager </designation>

  • <designation emp_id="15422" emp_name="Peter"> Manager </designation>

Consider the following example XML file storing the information on athletes in a marathon.

<marathon>
<athletes>
<name>Mike</name>
<age>25</age>
<awards>
Two times world champion. Currently, worlds No. 3
</awards>
<titles>6</titles>
</athletes>
<athletes>
<name>Usain</name>
<age>29</age>
<awards>
Five time world champion. Currently, worlds No. 1
</awards>
<titles>17</titles>
</athletes>
</marathon>

Using the package XML and plyr in R, you can convert this file into a data.frame as follows:

library(XML)
library(plyr)

xml:data <-xmlToList("marathon.xml")

#Excluding "description" from print                    
ldply(xml:data, function(x) { data.frame(x[!names(x)=="description"]) } )
        .id  name age
 1 athletes  Mike  25
 2 athletes Usain  29
                                                             awards titles
 1  
      Two times world champion. Currently, worlds No. 3
          6
 2 
      Five times world champion. Currently, worlds No. 1
         17                                                                                                                      

2.1.2.4 Hypertext Markup Language : HTML

Hypertext Markup Language, commonly known as HTML, is used to create web pages. A HTML page, when combined with Cascading Style Sheets (CSS), can produce beautiful static web pages. The web page can further be made dynamic and interactive by embedding a script written in language such as JavaScript. Today's modern web sites are a combination of HTML, CSS, JavaScript, and many more advanced technologies like Flash players. Depending on the purpose, a web site could be made rich with all these elements. Even when the modern web sites are getting more sophisticated, the core HTML design of web pages still stands against the test of times with newer features and advanced functionality. Although HTML is now filled with rich style and elegance, the content still remains very central.

The ever-exploding number of web sites has made it difficult for someone to find relevant content on a particular subject of interest and that's where companies like Google saw a need to crawl, scrape, and rank web pages for relevant content. This process generates a lot of data, which Google uses to help users with their search queries. These kind of process exploits the fact that HTML pages have a distinctive structure for storing content and reference links to external web pages if any.

There are five key elements in a HTML file that are scanned by the majority of web crawlers and scrappers:

• Headers

  A simple header might look like

  <head>
  <title>Machine Learning with R </title>
  </head>

• Headings

  There are six heading tags, h1 to h6, with decreasing font sizes. They looks like

  <h1>Header </h1>
  <h2>Headings </h2>
  <h3>Tables </h3>
  <h4>Anchors </h4>
  <h5>Links </h5>
  <h6>Links </h6>

• Paragraphs

  A paragraph could contain more than few words or sentences in a single block.

  <p>Paragraph 1</p>
  <p>Paragraph 2</p>

• Tables

  A tabular data of rows and columns could be embedded into HTML tables with following tags

  <table>
  <tbody>
  <thead>
  <tr>Define a row </tr>
  </thead>
  </tbody>
  </table>
  <table> Tag declares the main table structure.
  <tbody> Tag specifies the body of the table.
  <thead> Tag defines the table header.
  <tr> Tag defines each row of data in the table

• Anchor

  Designers can use anchors to anchor a URL to some text on a web page. When users view the web page in a browser, they can click the text to activate the link and visit the page. Here’s an example

  <a href="https://www.apress.com/">__ Welcome to Machine Learning usingR!</a>

With all these elements , a sample HTML file will look like the following snippet.

<!DOCTYPE html>
<html>
<body>
<h1>Machine Learning usingR</h1>
<p>Hope you having fun time reading this book !!</p>
<h1>Chapter 2</h1>
<h2>Data Exploration and Preparation</h2>
<a href="https://www.apress.com/">Apress Website Link</a>
</body>
</html>

Python is one of most powerful scripting languages for building web scraping tools. Although R also provides many packages to do the same job, it’s not very robust. Like Google, you can try to build a web-scrapping tool, which extracts all the links from a HTML web page like the one shown previously. We will show a very basic example of reading a local HTML file in R named html_example.html with the previous information. It’s easy to extend this to any web page.

library(XML)
url <- "html_example.html"
doc <-htmlParse(url)
xpathSApply(doc, "//a/@href")
                      href
 "https://www.apress.com/"

2.1.2.5 JSON

JSON is a widely used data interchange format in many application programming interfaces like Facebook Graph API, Google Maps, and Twitter API.

An example of a JSON file is when you get data from the Facebook API. It might also be used to contain profile information that can be easily shared across your system components using the simple JSON format.

{
"data":[
      {
"id": "A1_B1",
"from":{
"name": "Jerry", "id": "G1"
         },
"message": "Hey! Hope you like the book so far",
"actions":[
            {
"name": "Comment",
"link": "http://www.facebook.com/A1/posts/B1"
            },
            {
"name": "Like",
"link": "http://www.facebook.com/A1/posts/B1"
            }
         ],
"type": "status",
"created_time": "2016-08-02T00:24:41+0000",
"updated_time": "2016-08-02T00:24:41+0000"
      },
      {
"id": "A2_B2",
"from":{
"name": "Tom", "id": "G2"
         },
"message": "Yes. Easy to understand book",
"actions":[
            {
"name": "Comment",
"link": "http://www.facebook.com/A2/posts/B2"
            },
            {
"name": "Like",
"link": "http://www.facebook.com/A2/posts/B2"
            }
         ],
"type": "status",
"created_time": "2016-08-03T21:27:44+0000",
"updated_time": "2016-08-03T21:27:44+0000"
      }
   ]
}

Using the library rjson, you can read such JSON files into R and convert the data into a data.frame. The following R code displays the first three columns of the data.frame.

library(rjson)
url <- "json_fb.json"
document <-fromJSON(file=url, method='C')
as.data.frame(document)[,1:3]
   data.id data.from.name data.from.id
 1   A1_B1          Jerry           G1

2.1.2.6 Other Formats

Apart from all these widely used data formats , there are many more formats supported by R. It’s possible to read data directly from databases using ODBC connections. R also supports data formats from other data analytics software like SPSS, SAS, Stata, and MATLAB.

2.1.3.1 Structured

Structured data is everywhere and it’s always the easiest to understand, represent, store, query, and process such data. So, if you dream of an ideal world, all data will have rows and columns stored in a tabular manner. The widespread development around the various business applications, database technology, business intelligent system, and spreadsheet tools has given rise to enormous amount of clean and good looking data. Every row and column is well defined within a connected schematic tables in a database. The data coming from CSV and Excel files generally has this structure built into it. We will show many examples of such data throughout the book to explain the relevant concepts.

2.1.3.2 Semi-Structured

Although structured data gives us plenty of scope to experiment and ease of usage, it’s not always possible to represent information in rows and column. The kind of data generated from Twitter and Facebook has significantly moved away from the traditional Relational Database Management System (RDBMS) paradigms , where everything has a predefined schema, to a world of NoSQL (Chapter 9 covers some of the NoSQL systems), where data is semi-structured. Both Twitter and Facebook rely heavily on JSON or BSON (Binary JSON) . Databases like MongoDB and Cassandra store this kind of NoSQL data.

2.1.3.3 Unstructured

The biggest challenge in data engineering has been dealing with unstructured data like images, videos, web logs, and click stream. The challenge is pretty much in handling the volume and velocity of this data generation process on top of not finding any patterns. Despite having many sophisticated software systems for handling such data, there are no defined processes for using this data in modeling or insight generation. Unlike semi-structured data, where we have many APIs and tools to process the data into a required format, here, a huge effort is spent in processing this data to a structured form. Big data technologies like Hadoop and Spark are often deployed for such purposes. There has been significant work on unstructured textual data generated from human interactions. For instance, Twitter Sentiment Analysis on tweets (covered in Chapter 6).

2.2.1.1 Function str()

The str() function in R comes in very handy when you first look at the data. Referring back to the employee data we used earlier, the str() output will look something like what’s shown in the following R code snippet. The output shows four useful tidbits about the data.

• The number of rows and columns in the data

• Variable name or column header in the data

• Datatype of each variable

• Sample values for each variable

Depending on how many variables are contained in the data, spending a few minutes or an hour on this output will provide significant understanding of the entire dataset.

emp <-read.csv("employees.csv", header =TRUE, sep =",")
str(emp)
 'data.frame':    5 obs. of  4 variables:
  $ Code             : int  15421 15422 15423 15424 15425
  $ First.Name       : Factor w/ 4 levels "John","Mark",..: 1 4 2 4 3
  $ Last.Name        : Factor w/ 5 levels "Buffet","Luther",..: 4 5 3 1 2
  $ Salary.US.Dollar.: int  10000 20000 30000 40000 50000

2.2.1.2 Naming Convention : make.names()

In order to be consistent with the variable names throughout the course of the analysis and preparation phase, it’s important that the variable names in the dataset follow the standard R naming conventions. The step is critical for two important reasons,

• When merging the multiple datasets, it’s convenient if the common columns on which the merge happens have the same variable name.

• It’s also a good practice with any programming language to have clean names (no spaces or special characters) for the variables.

R has this function called make.names(). To demonstrate it, lets make our variable names dirty and then will use make.names to clean them up. Note that read.csv functions have a default behavior of cleaning the variable names before they loads the data into data.frame. But when we are doing many operations on data inside our program, it’s possible that the variable names will fall out of convention.

#Manually overriding the naming convention                    
names(emp) <-c('Code','First Name','Last Name', 'Salary(US Dollar)')

# Look at the variable name                    
emp
    Code First Name Last Name Salary(US Dollar)
 1 15421       John     Smith             10000
 2 15422      Peter      Wolf             20000
 3 15423       Mark   Simpson             30000
 4 15424      Peter    Buffet             40000
 5 15425     Martin    Luther             50000

# Now let’s clean it up using make.names                    
names(emp) <-make.names(names(emp))

# Look at the variable name after cleaning                    
emp
    Code First.Name Last.Name Salary.US.Dollar.
 1 15421       John     Smith             10000
 2 15422      Peter      Wolf             20000
 3 15423       Mark   Simpson             30000
 4 15424      Peter    Buffet             40000
 5 15425     Martin    Luther             50000

2.2.1.3 Table(): Pattern or Trend

Another reason it’s important to look at your data closely up-front is to look for some kind of anomaly or pattern in the data. Suppose we wanted to see if there were any duplicates in the employee data, or if we wanted to find a very common name among employees and reward them for a fun HR activity. These tasks are possible using the table() function. Its basic role is to show the frequency distribution in a one- or two-way tabular format.

#Find duplicates                    
table(emp$Code)

 15421 15422 15423 15424 15425
     1     1     1     1     1
#Find common names                    
table(emp$First.Name)

   John   Mark Martin  Peter
      1      1      1      2

This clearly shows no duplicates and the name Peter appearing twice. These kind of patterns might be very useful to judge if the data has any bias for some variables, which will tie back to the final story we would want to write from the analysis.

2.2.2.1 Merge and dplyr Joins

The most useful operation while preparing the data is the ability to join or merge two different datasets into a single entity. This idea is easy to relate to the various joins of SQL queries. A standard text on SQL queries will explain the different forms of joins elaborately, but we will focus on the function available in R. Let’s discuss the two functions in R, which help to join two datasets.

We will use another extended dataset of our employee example where we have the department and educational qualification and merge it with the already existing dataset of employees. Let’s see the four very common type of joins using merge and dplyr. Though the output is same, there are many differences between merge and dplyr implementations. dplyr is somewhat regarded as more efficient than merge, but the merge() function merges two data frames by common columns or row names, or does other versions of database join operations, whereas dplyr provides a flexible grammar of data manipulation focused on tools for working with data frames (hence the d in the name).

merge(emp, emp_qual, by ="Code")
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD

merge(emp, emp_qual, by ="Code", all.x =TRUE)
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15424      Peter    Buffet             40000    <NA>
 5 15425     Martin    Luther             50000    <NA>

merge(emp, emp_qual, by ="Code", all.y =TRUE)
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15426       <NA>      <NA>                NA     PhD
 5 15429       <NA>      <NA>                NA     Phd

merge(emp, emp_qual, by ="Code", all =TRUE)
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15424      Peter    Buffet             40000    <NA>
 5 15425     Martin    Luther             50000    <NA>
 6 15426       <NA>      <NA>                NA     PhD
 7 15429       <NA>      <NA>                NA     Phd

library(dplyr)

inner_join(emp, emp_qual, by ="Code")
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD

left_join(emp, emp_qual, by ="Code")
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15424      Peter    Buffet             40000    <NA>
 5 15425     Martin    Luther             50000    <NA>

right_join(emp, emp_qual, by ="Code")
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15426       <NA>      <NA>                NA     PhD
 5 15429       <NA>      <NA>                NA     Phd

full_join(emp, emp_qual, by ="Code")
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15424      Peter    Buffet             40000    <NA>
 5 15425     Martin    Luther             50000    <NA>
 6 15426       <NA>      <NA>                NA     PhD
 7 15429       <NA>      <NA>                NA     Phd

2.2.3.1 Correcting Factor Variables

Since R is a case-sensitive language, every categorical variable with definite set of values like the variable Qual in our employee dataset with PhD and Masters as two values needs to be checked for any inconsistencies. In R, such variables are called factors, with PhD and Masters as its two levels. So, a value like PhD and Phd are treated differently, even though they mean the same. A manual inspection using the table() function will reveal such patterns. A way to correct this would be:

employees_qual <-read.csv("employees_qual.csv")

#Inconsistent                    
employees_qual
    Code    Qual
 1 15421 Masters
 2 15422     PhD
 3 15423     PhD
 4 15426     PhD
 5 15429     Phd
employees_qual$Qual =as.character(employees_qual$Qual)
employees_qual$Qual <-ifelse(employees_qual$Qual %in%c("Phd","phd","PHd"), "PhD", employees_qual$Qual)

#Corrected                    
employees_qual
    Code    Qual
 1 15421 Masters
 2 15422     PhD                                  
 3 15423     PhD
 4 15426     PhD
 5 15429     PhD

2.2.3.2 Dealing with NAs

NAs (abbreviation for “Not Available”) are missing values and will always lead to wrong interpretation, exceptions in function output, and cause models to fail if we live with them until the end. The best way to handle NAs is either to remove/ignore if we are sitting in a big pool of data or if we couldn't afford to lose anything from the precious small dataset we have got, we impute, which is a process of filling the missing values.

The technique of imputation has attracted many researchers to devise novel ideas but nothing can beat the simplicity that comes from the complete understanding of the data. Let’s take up an example from the merge we did previously, in particular the output from the right join. It’s not possible to impute First and Last Name, but it might not be relevant for any aggregate analysis we might want to do on our data. Rather, the variable that’s important for us is Salary, where we don't want to see NA. So, here is how we impute a value in the Salary variable.

emp <-read.csv("employees.csv")
employees_qual <-read.csv("employees_qual.csv")

#Correcting the inconsistency                    
employees_qual$Qual =as.character(employees_qual$Qual)
employees_qual$Qual <-ifelse(employees_qual$Qual %in%c("Phd","phd","PHd"), "PhD", employees_qual$Qual)

#Store the output from right_join in the variables impute_salary                    
impute_salary <-right_join(emp, employees_qual, by ="Code")

#Calculate the average salary for each Qualification                    
ave_age <-ave(impute_salary$Salary.US.Dollar., impute_salary$Qual,
FUN = function(x) mean(x, na.rm =TRUE))

#Fill the NAs with the average values                    
impute_salary$Salary.US.Dollar. <-ifelse(is.na(impute_salary$Salary.US.Dollar.), ave_age, impute_salary$Salary.US.Dollar.)

impute_salary
    Code First.Name Last.Name Salary.US.Dollar.    Qual
 1 15421       John     Smith             10000 Masters
 2 15422      Peter      Wolf             20000     PhD
 3 15423       Mark   Simpson             30000     PhD
 4 15426       <NA>      <NA>             25000     PhD
 5 15429       <NA>      <NA>             25000     PhD

Here, the idea is that a particular qualification is eligible for paychecks of a similar value, but there certainly is some level of assumption we have taken, that the industry isn't biased on paychecks based on which institution the employee obtained the degree. However, if there is a significant bias, then a measure like average might not be a right method; instead something like median could be used. We will discuss these kinds of bias in greater detail in the descriptive analysis section.

2.2.3.3 Dealing with Dates and Times

In many models, date and time variables play a pivotal role. Date and time variables reveal a lot about the temporal behavior, for instance, sales data of a supermarket or online store could give us details like most important time of the day with sales volume at peak, sales trend of weekday versus weekend, and much more. Often, dealing with date variables is a painful task, primarily because of many available date formats, time zones, and daylight savings in few countries. These challenges makes any arithmetic calculation like difference between days and comparing two date values even more difficult.

The lubridate package is one of the most useful packages in R, and it helps in dealing with these challenges. The paper, “Dates and Times Made Easy with Lubridate,” published in the Journal of Statistical Software by Grolemund, describes the capabilities the lubridate package offers. To borrow from the paper, lubridate helps users:

• Identify and parse date-time data

• Extract and modify components of a date-time, such as years, months, days, hours, minutes, and seconds

• Perform accurate calculations with date-times and timespans

• Handle time zones and daylight savings time

The paper gives an elaborate description with many examples, so we will take up here only two uncommon date transformations like dealing with time zone and daylight savings.

library("lubridate")
date <-as.POSIXct("2016-03-13 09:51:48")
date
 [1] "2016-03-13 09:51:48 IST"
with_tz(date, "UTC")
 [1] "2016-03-13 04:21:48 UTC"

dst_time <-ymd_hms("2010-03-14 01:59:59")
dst_time <-force_tz(dst_time, "America/Chicago")
dst_time
 [1] "2010-03-14 01:59:59 CST"

dst_time +dseconds(1)
 [1] "2010-03-14 03:00:00 CDT"

2.2.4.1 Derived Variables

Deriving new variables requires lot of creativity. Sometimes it demands a purpose, situations where a derived variable helps to explain certain behavior. For example, while looking at the sales trend of any online store, we see a sudden surge in volume on a particular day, so on further investigation we found the reason to be a heavy discounting for end-of-season sales. So, if we include a new binary variable EOS_Sales assuming a value 1, if we had end of season sales or 0 otherwise, we may aid the model in understanding why a sudden surge is seen in the sales.

2.2.4.2 n-day Averages

Another useful technique for deriving such variables, especially in time series data from the stock market, is to derive variables like last_7_days, last_2_weeks, and last_1_month average stock prices. Such variables work to reduce the variability in data like stock prices, which can sometime seem like noise and can hamper the performance of the model to a great extent.

2.3.1.1 Quantile

If we divide our population of data into four equal groups, based on the distribution of values of a particular numerical variable, then each of the three values creating the four divides are called first, second, and third quartile. In other words, the more general term is quantile; q-Quantiles are values that partition a finite set of values into q subsets of equal sizes.

For instance, dividing in four equal groups would mean a 3-quantile. In terms of usage, percentile is more widely used terminology, which is a measure used in statistics indicating the value under which a given percentage of observations in a group of observations fall. If we divide something in 100 equal groups, we have 99-Quantiles, which leads us to define the first quartile as the 25th percentile and the third quartile as the 75th percentile. In simpler terms, the 25th percentile or first quartile is a value below which 25 percent of the observations are found. Similarly, 75th percentile or third quartile is a value below which 75 percent of the observations are found.

First Quartile:

quantile(marathon$Finish_Time, 0.25)
  25%
 2.65

Second Quartile or Median:

quantile(marathon$Finish_Time, 0.5)
 50%
 4.3
#Another function to calculate median                    

median(marathon$Finish_Time)
 [1] 4.3

Third Quartile:

quantile(marathon$Finish_Time, 0.75)
   75%
 6.455

The interquartile range is the difference between the 75th percentile and 25th percentile, would be the range that contains 50% of the data of any particular variable in the dataset. Interquartile range is a robust measure of statistical dispersion. We will discuss this further in the later part of the section.

quantile(marathon$Finish_Time, 0.75, names =FALSE) -quantile(marathon$Finish_Time, 0.25, names =FALSE)
 [1] 3.805                              

2.3.1.2 Mean

Though median is a robust measure of central tendency of any distribution of data, mean is a more traditional statistic for explaining the statistical property of the data distribution. The median is more robust because a single large observation can throw the mean off. We formally define this statistic in the next section.

mean(marathon$Finish_Time)
 [1] 4.6514

As you would expect, the summary (mean and median are often counted one as a measure of centrality ) listed by the summary function's output are in the increasing order of their values. The reason for this is obvious from the way they are defined. And if these statistical definitions were difficult for you to contemplate, don't worry, we will turn to visualization for explanation. Though we have a dedicated chapter on it, here we discuss some very basic plots that are inseparable from theories around any exploratory analysis.

2.3.1.3 Frequency Plot

A frequency plot is showing the number of runners in each type. Such simple plots explain the distribution of a categorical variable. As seen in the plot, we how many first-timers, frequents, and professional runners participated in the marathon.

plot(marathon$Type, xlab ="Marathoners Type", ylab ="Number of Marathoners")

Figure 2-1. Number of athletes in each type

2.3.1.4 Boxplot

A boxplot is the alternative to the summary statistics in visualization. Though looking at numbers is always useful, an equivalent representation of the same in a visually appealing plot could serve as an excellent tool for better understanding, insight generation, and ease of explaining the data.

In the summary, we saw the values for each variable but were not able to look how the finish time varies for each type of runner. In other words, how type and finish time are related. Figure 2-2 clearly helps to illustrate this relationship. As expected, the boxplot clearly shows that professionals have a much better finish time than frequents and first-timers.

Figure 2-2. Boxplot showing variation of finish times for each type of runner

boxplot(Finish_Time ∼Type,data=marathon, main="Marathon Data", xlab="Type of Marathoner", ylab="Finish Time")

2.3.2.1 Variance

Varianceis a measure of the spread for the given set of numbers. The smaller the variance, the closer the numbers are to the mean and the larger the variance, the farther away the numbers are from the mean. Variance is an important measure for understanding the distribution of the data, more formally it’s called probability distribution. In the next chapter, where various sampling techniques are discussed, we examine how a sample variance is considered to be an estimate of the full population variance, which forms the basis for a good sampling method. Depending on whether our variable is discrete or continuous, we can define the variance.

Mathematically, for a set of n equally likely numbers for a discrete random variable, variance can be represented as follows:

And more generally, if every number in our distribution occurs with a probability p_i, variance is given by:

As seen in the formula, for every data point, we are measuring how far the number is from the mean, which translates into a measure of spread. Equivalently, if we take the square root of variance, the resulting measure is called the standard deviation, generally written as a sigma. The standard deviation has the same dimension as the data, which makes it convenient to compare with the mean. Together, both mean and standard deviation, can be used to describe any distribution of data. Let’s take a look at the variance of the variable Finish_Time from our marathon data.

mean(marathon$Finish_Time)
 [1] 4.6514
var(marathon$Finish_Time)
 [1] 4.342155
sd(marathon$Finish_Time)
 [1] 2.083784

Looking at the values of mean and standard deviation, we could say, on average, that the marathoners have a finish time of 4.65 +/- 2.08 hours. Further, it’s easy to notice from the following code snippet that each type of runner has their own speed of running and hence a different finish time .

tapply(marathon$Finish_Time,marathon$Type, mean)
  First-Timer    Frequents Professional
     7.154118     4.213158     2.207143
tapply(marathon$Finish_Time,marathon$Type, sd)
  First-Timer    Frequents Professional
    0.8742358    0.5545774    0.3075068

2.3.2.2 Skewness

As variance is a measure of spread, skewness measures asymmetry about the mean of the probability distribution of a random variable. In general, as the standard definition says, we could observe two types of skewness:

• Negative skew: The left tail is longer; the mass of the distribution is concentrated on the right. The distribution is said to be left-skewed, left-tailed, or skewed to the left.

• Positive skew: The right tail is longer; the mass of the distribution is concentrated on the left. The distribution is said to be right-skewed, right-tailed, or skewed to the right.

Mathematicians discuss skewness in terms of the second and third moments around the mean. A more easily interpretable formula could be written using standard deviation.

This formula for skewness is referred to as the Fisher-Pearson coefficient of skewness. Many software programs actually compute the adjusted Fisher-Pearson coefficient of skewness, which could be thought of as a normalization to avoid too high or too low values of skewness:

Let’s use the histogram of beta distribution to demonstrate skewness.

library("moments")
 Warning: package 'moments' was built under R version 3.2.3
par(mfrow=c(1,3), mar=c(5.1,4.1,4.1,1))

# Negative skew                    
hist(rbeta(10000,2,6), main ="Negative Skew" )
skewness(rbeta(10000,2,6))
 [1] 0.7166848
# Positive skew                    
hist(rbeta(10000,6,2), main ="Positive Skew")
skewness(rbeta(10000,6,2))
 [1] -0.6375038
# Symmetrical                    
hist(rbeta(10000,6,6), main ="Symmetrical")

Figure 2-3. Distribution showing symmetrical versus negative and positive skewness

skewness(rbeta(10000,6,6))
 [1] -0.03952911

For our marathon data , the skewness is close to 0, indicating a symmetrical distribution.

hist(marathon$Finish_Time, main ="Marathon Finish Time")

Figure 2-4. Distribution of finish time of athletes in marathon data

skewness(marathon$Finish_Time)                                                        
 [1] 0.3169402

2.3.2.3 Kurtosis

Kurtosis is a measure of peakedness and tailedness of the probability distribution of a random variable. Similar to skewness, kurtosis is also used to describe the shape of the probability distribution function. In order words, kurtosis explains the variability due to a few data points having extreme differences from the mean, rather than lot of data points having smaller differences from the mean. Higher values indicate a higher and sharper peak and lower values indicate a lower and less distinct peak. Mathematically, kurtosis is discussed in terms of the fourth moment around the mean. It’s easy to find that the kurtosis for a standard normal distribution is 3, a distribution known for its symmetry, and since kurtosis like skewness measures any asymmetry in data, many people use the following definition of kurtosis:

Generally, there are three types of kurtosis:

• Mesokurtic: Distributions with a kurtosis value close to 3, which means in the previous formula, the term before 3 becomes 0, a standard normal distribution with mean 0 and standard deviation 1.

• Platykurtic: Distributions with a kurtosis value < 3. Comparatively, a lower peak and shorter tails than normal distribution.

• Leptokurtic: Distributions with a kurtosis value > 3. Comparatively, a higher peak and longer tails than normal distribution.

While the kurtosis statistic is often used by many to numerically describe a sample, it is said that, “there seems to be no universal agreement about the meaning and interpretation of kurtosis”. Tukey suggests that, like variance and skewness, kurtosis should be viewed as a “vague concept” that can be formalized in a variety of ways.

#leptokurtic                  
set.seed(2)
random_numbers <-rnorm(20000,0,0.5)
plot(density(random_numbers), col ="blue", main ="Kurtosis Plots", lwd=2.5, asp =4)
kurtosis(random_numbers)
 [1] 3.026302
#platykurtic                  
set.seed(900)
random_numbers <-rnorm(20000,0,0.6)
lines(density(random_numbers), col ="red", lwd=2.5)
kurtosis(random_numbers)
 [1] 2.951033
#mesokurtic                  
set.seed(3000)
random_numbers <-rnorm(20000,0,1)
lines(density(random_numbers), col ="green", lwd=2.5)
kurtosis(random_numbers)
 [1] 3.008717
legend(1,0.7, c("leptokurtic", "platykurtic","mesokurtic" ),
lty=c(1,1),
lwd=c(2.5,2.5),col=c("blue","red","green"))

Figure 2-5. Showing kurtosis plots with simulated data

Comparing these kurtosis plots to the marathon finish time, it’s platykurtic with a very low peak and short tail.

plot(density(as.numeric(marathon$Finish_Time)), col ="blue", main ="Kurtosis Plots", lwd=2.5, asp =4)

Figure 2-6. Showing kurtosis plot of finish time in marathon data

kurtosis(marathon$Finish_Time)
 [1] 1.927956