Get data from Twitter

There are several ways to get data from Twitter, but, in our opinion, the simplest and quickest way is by using the R package twitteR (Gentry, 2015).

Before using this package you need a developer account. To register go to https://dev.twitter.com/. Once you have it, you can obtain the API code creating a new application. In the bottom of the page, click “Manage Your Apps” under “Tools” and then on “Create New Application.” On the application creation page, you have to fill all the questions except the Callback. When you finish the creation step, you can check the details of our application. The generated consumer keys and secrets (API) would be under the tab “Keys and Access Token.”

This information is important to successfully connect to Twitter, and you need to run this procedure only once: with the same code you can connect forever:

install.packages(c("devtools", "rjson", "bit64", "httr"))
#RESTART R session!
library(devtools)
install_github("twitteR", username="geoffjentry")
#this is important to be sure to use the last version of the package

library(twitteR)
api_key <‐ "YOUR API KEY"
api_secret <‐ "YOUR API SECRET"
access_token <‐ "YOUR ACCESS TOKEN"
access_token_secret <‐ "YOUR ACCESS TOKEN SECRET"
setup_twitter_oauth(api_key,api_secret,access_token,access_token_secret)

Goal 1: Determine what people are talking about regarding “Expo 2015” and “Expo 2020”

In many cases, the main question that we want to answer from a twitter analysis is to know what people are talking about for a given topic. Usually the quickest way to know more about what people are talking about is by visualizing the most frequent words and terms contained in the tweets. We can do this by creating a plot—a bar plot with the most frequent terms—or we can use a more visually attractive way using a word cloud. A wordcloud is generally very helpful if we only want to take a quick and simple look at the data.

We download a maximum number of 1000 tweets in one week, containing Expo+2015+Milan and Expo+2020+Dubai:

expo2015 <‐ searchTwitter("Expo+2015+Milan",n=1000, lang="en")
expo2020 <‐ searchTwitter("Expo+2020+Dubai",n=1000, lang="en")

Before beginning our analysis, we load all the required packages for text mining (Feinerer, 2015) and for creating (Fellows, 2013) wordcloud:

library(tm)
library(wordcloud)

We extract the text from the tweets in two different vectors:

text_2015 <‐ sapply(expo2015, function(x) x$getText())
text_2020 <‐ sapply(expo2020, function(x) x$getText())

We then need to clean the data, but tolower doesn’t always behave as expected and returns error messages. This is why we apply a modified version that skips such errors:

newTolower = function(x)
{
   y = NA
   try_error = tryCatch(tolower(x), error=function(e) e)
   if (!inherits(try_error, "error"))
      y = tolower(x)
   return(y)
}

text_2015 <‐ sapply(text_2015, newTolower)
names(text_2015) = NULL
text_2015 <‐ text_2015[text_2015 != ""]
text_2020 <‐ sapply(text_2020, newTolower)
names(text_2020) <‐ NULL
text_2020 <‐ text_2020[text_2020 != ""]

clean.text = function(x)
{
   x = tolower(x)
   # remove rt
   x = gsub("rt", "", x)
   # remove at
   x = gsub("@\w+", "", x)
   # remove punctuation
   x = gsub("[[:punct:]]", "", x)
   # remove numbers
   x = gsub("[[:digit:]]", "", x)
   # remove links http
   x = gsub("http\w+", "", x)
   # remove tabs
   x = gsub("[ |	]{2,}", "", x)
   # remove blank spaces at the beginning
   x = gsub("^ ", "", x)
   # remove blank spaces at the end
   x = gsub(" $", "", x)
   return(x)
}

text_2015 = clean.text(text_2015)
text_2020 = clean.text(text_2020)

We then construct the lexical corpus. Furthermore, we consider some stop words and create the term document matrix:

corpus_2015 <‐ Corpus(VectorSource(text_2015))
corpus_2020 <‐ Corpus(VectorSource(text_2020))

skipwords <‐c("and","with","than","expo","the","for","2015","2020","from","will",
         "you","not","this","these","those","has","are","our","in","into",
         "your","which", "it","its")

corpus_2015 <‐ tm_map(corpus_2015,removeWords,skipwords)
corpus_2020 <‐ tm_map(corpus_2020,removeWords,skipwords)

tdm_2015 <‐ TermDocumentMatrix(corpus_2015)
tdm_2020 <‐ TermDocumentMatrix(corpus_2020)

Now we can obtain words and their frequencies:

milan = as.matrix(tdm_2015)
word_freqs_milan = sort(rowSums(milan), decreasing=TRUE) 
dmilan = data.frame(word=names(word_freqs_milan), freq=word_freqs_milan)
dubai = as.matrix(tdm_2020)
word_freqs_dubai = sort(rowSums(dubai), decreasing=TRUE) 
ddubai = data.frame(word=names(word_freqs_dubai), freq=word_freqs_dubai)

Finally, we create the Wordcloud plots (Figure 14.3). Wordclouds are graphical representations of word frequency that give greater prominence to words that appear more frequently in a source text. The larger the word in the visual, the more common the word was in the document. This type of visualization can assist evaluators with exploratory textual analysis by identifying frequent words. It can also be used for communicating the most salient points or themes in the reporting stage:

pdf("expo_2015.pdf")
wordcloud(dmilan$word,dmilan$freq,
          random.order=FALSE,colors=brewer.pal(8,"Dark2"), max.words=100)
dev.off()
pdf("expo2020.pdf")
wordcloud(ddubai$word,ddubai$freq,
          random.order=FALSE,colors=brewer.pal(8,"Dark2"), max.words=100)
dev.off()

Goal 2: Compare what people are talking about regarding “Expo 2015” and “Expo 2020”

When we work on a project comparing two text files, we can generate a unique wordcloud that can provide more insights. For this purpose, we have to make some changes in the script.

First, we join texts in a vector for each event:

m <‐ paste(text_2015,collapse="")
d <‐ paste(text_2020,collapse="")
all <‐ c(m, d)

We create the corpus, clean it, and create the term document matrix:

corpus <‐ Corpus(VectorSource(all))
corpus <‐ tm_map(corpus,removeWords,skipwords)
tdm <‐ TermDocumentMatrix(corpus)
tdm <‐ as.matrix(tdm)
colnames(tdm) <‐ c("Expo 2015", "Expo 2020")

For this goal, we can use the other type of graphic in the wordcloud package: the comparison and the commonality wordcloud before we plot a cloud, and we can compare the frequencies of words across documents:

pdf("comp_expo.pdf")
comparison.cloud(tdm,random.order=FALSE,colors=c("green","red"), max.words=100)
dev.off()

Let p_ij be the rate at which word i occurs in document j and p_j be the average across documents: p_j =  . The size of each word is mapped to its maximum deviation (max_i(p_ij − p_i)), and its angular position is determined by the document where that maximum occurs. The commonality cloud plot permits to plot a cloud of words shared across the two searches:

pdf("comm_expo.pdf")
commonality.cloud(tdm,random.order=FALSE,colors=c("green","red"), max.words=100)
dev.off()

The wordclouds in Figure 14.4 are only examples used to analyze Twitter data with R.

Wordclouds are an easy to use and inexpensive option for visualizing text data, but we have to remember that the right way to interpret wordclouds is that the display emphasizes frequency of words, not necessarily their importance. Wordclouds will not accurately reflect the content of text if slightly different words are used for the same idea. They also do not provide the context, so the meaning of individual words may be lost. Because of these limitations, wordclouds are best suited for exploratory qualitative analysis.

An interesting analysis about Expo 2015 using R was conducted by the Voices from the Blogs.² This application with ExpoBarometro³ presents an indicator that measures the daily social experience of Expo 2015, meaning the explicit reactions of the visitors of the event in Milan as they appear from posts published on Twitter, every day. ExpoBarometro presents an indicator of sentiment constructed as the ratio between the percentage of posts making a positive reference to the Expo experience and the sum of the positive and negative posts, excluding from the calculation those of neutral content.

All ExpoBarometro analyses are carried out through iSA® technology by Voices from the Blogs, a spin‐off of the University of Milan. In‐depth quality is achieved with supervised statistical techniques that initially predict a classification of a training set by means of human coders. This is the key to retrieve meaningful information from texts published on the Web, avoiding problems of linguistic expression that appear in analysis based on semantic engines or on other fully automated techniques.

The data feed analyzed daily by iSA relates to all tweets written in Italian containing one of the following expressions: “expo, Expo 2015, Expo 2015, expomilan, expomilano.” These tweets are obtained through Twitter’s API streaming, and the analysis does not contain duplications and excludes posts by official Expo accounts. Figure 14.5 shows a screenshot of the results obtained and published on the official website of Expo 2015 organization team.

Goal 1: Determine the performance of a sensory panel

For this goal, we consider the experts dataset. For each sensory attribute, it is possible to test the product effect, panelist effect, and session effect. Using R and in particular using the functions panelperf() and coltable() of SensoMineR, it is possible to get a useful plot that summarizes 12 different ANOVA models, one for each attribute, in which the cells are highlighted in yellow when p‐values are lower than 0.05:

## Goal 1 ###
Experts <‐ read.table(file="perfumes_qda_experts.csv",
          header=TRUE, sep=",", quote="/", dec=".")
experts$Session <‐ as.factor(experts$Session)
experts$Panelist <‐ as.factor(experts$Panelist)
res.panelperf1 <‐ panelperf(experts,formul="~Product+Panelist
          +Session+Product:Panelist+Product:Session+Panelist: Session",firstvar=5)
coltable(res.panelperf$p.value[order(res.panelperf$p.value[,1]),],
          col.lower="yellow")

Figure 14.7 shows that the panel discriminates between the products for all the sensory attributes, except Citrus. It also shows that the panelists have particularly well differentiated the products.

Goal 2: Represent the product space in a map

For this goal, we consider again the experts dataset. It is possible to define the sensory profile (adjusted mean) of the products using the decat() function in SensoMineR and then produce individual factor map and variable factor map using the PCA() function (see Figure 14.8):

## Goal 2 ###
res.decat<‐decat(experts,formul="~Product+Panelist",
           firstvar=5,lastvar=ncol(experts))
res.pca<‐PCA(res.decat$adjmean)

The main dimension of variability (i.e., the first component) contrasts products such as Pleasures with products such as Angel. In the second plot, for each dimension, only significant attributes are shown. Wrapping and Heady are most positively linked with the first dimension, whereas the sensory attributes that are the most negatively linked are Fruity and Floral.

A possible new question that arises looking at these plots can be: How would the positioning of the perfumes evolve if we would slightly change the composition of the panel? To answer this question, the developers of SensoMineR provide the panellipse() function:

res.panellipse <‐ panellipse(experts,col.p=4,col.j=1,firstvar=5,
               level.search.desc=1)

In Figure 14.9, the more one sees ellipse overlaps, the less distinctive (or the closer) the two products are. Hence it appears clearly that Aromatics Elixir and Shalimar are perceived as similar, whereas Shalimar and Pleasure are perceived as different.

Goal 3: Compare the sensory profile of a panel of experts and a panel of customers

For this goal, we consider the experts dataset and the consumers dataset. A multiple factor analysis can be applied using MFA() function of SensoMineR in order to produce maps including both consumer and expert sensory responses, which are based on different sets of attributes (see Figure 14.10):

experts.avg <‐ decat(experts,formul="~Product+Panelist+Session+Product:Panelist+
           Product:Session+Panelist:Session",firstvar=5,graph=FALSE)$adjmean
consumers <‐ read.table("perfumes_qda_consumers.csv",sep=",",header=TRUE,
           dec=".",quote=""")
consumers$consumer <‐ as.factor(consumers$consumer)
consumers.avg <‐ decat(consumers,formul="~product+consumer",firstvar=3,
           graph=FALSE)$adjmean
data.mfa2 <‐ cbind(experts.avg,consumers.avg[rownames(experts.avg),])
res.mfa2 <‐ MFA(data.mfa2,group=c(12,21),type=c("s","s"),
           name.group=c("Experts","Consumers"))

From this representation, consumers and experts agree with the perfumes’ similarity even if they use different sets of attributes. The correlation circle highlights also the correlation between the attributes evaluated by experts and the attributes evaluated by consumers.

Goal 4: What is the best product?

For this goal, we consider the liking dataset, that is, the hedonic score for the 12 perfumes given by consumers. In order to make the plot more readable, we consider a set of only two panelists, A and B.

The null hypothesis to be tested is that there are no differences between A and B. This can be tested using a t‐test. Moreover, a useful plot implemented in SensoMineR is the graphinter() function:

Small <‐ read.table(file="perfumes_linking_small.csv", header=TRUE,
         sep=",", quote="/", dec=".")
graphinter(small,col.p=2,col.j=1,firstvar=3,numr=1,numc=1)
graphinter(small,col.p=1,col.j=2,firstvar=3,numr=1,numc=1)

As shown in Figure 14.11 (left panel), although the consumers provided different scores to the products (consumer A scoring higher than consumer B), both seem to evaluate the product in a similar way, at least in terms of ranking. Indeed, both consumers appreciate J’adore EP and J’adore ET the most and Shalimar the least. This finally reveals a strong product effect. Figure 14.11 (right panel) shows some inversion ranking for some perfumes, J’adore EP and J’adore ET and Chanel N°5 and Aromatics Elixir. In their book, Lê and Worch (2014) consider the complete liking dataset with all the consumers; ANOVA is applied and Fisher’s LSD is used. For a multivariate approach, the so‐called internal preference mapping is considered.

Lê and Worch (2014) also present other advanced analysis methods combining different datasets. Other possible data structure is textual data, napping data, or just‐about‐right (JAR) data.

Textual data are textual comments on product given by consumers. Text mining techniques have to be applied in order to handle such unstructured data. In this case typical words can be considered as attributes and used to describe the product profile.

Napping data are issued from sessions where panelists arrange all the products on a blank paper sheet. Napping allows collecting directly in only one session the sensory distances panelists perceived between products. Each panelist discriminates products according to his/her own criteria. The closer the two products are, the more alike the panelist perceives them. The dataset has as many rows as there are products and twice the number of panelist columns, each pair of columns corresponding to the X and Y coordinates of the products for one panelist.

JAR scales are commonly used in consumer research to identify whether product attributes are perceived at levels that are too high, too low, or just about right for that product.

Many other goals can be identified, for example, how the relationship between each sensory attribute and the hedonic score can be evaluated at different levels, how to get homogeneous clusters of products or of consumers, etc. All these goals can be easily reached using R and his packages.

Goal 1: Based on data from the Eurostat database, what are European tourism patterns?

Using R, it is possible to import and manipulate the Eurostat data very easily. The eurostat package (Leo et al., 2015) provides tools to access the data directly from the Eurostat database in R with search and manipulation utilities.

We can search the available variables using search_eurostat() function. In order to efficiently represent the European tourism, we consider the number of overnight stays in tourist accommodation by NUTS 2 regions (table tour_occ_nin2, source Eurostat). This indicator represents both the length of stay and the number of visitors.

To download the data, we use the get_eurostat() function:

dt <‐ get_eurostat("tour_occ_nin2", time_format = "raw")

An effective way to visualize how a measurement varies across a geographic area is a choropleth map, that is, a thematic map in which areas are shaded in proportion to the value assumed by the variable of interest in that particular region.

R offers a wide variety of ways to create maps. We’ll use principally two packages: maptools and ggplot2. The maptools package (Bivand and Lewin‐Koh, 2015) provides tools for reading and manipulating geographic data. In particular we use it to read shapefiles of Europe published by Eurostat. We will use the shapefile in a 1 : 60 million scale from year 2010 and subset it at the level of NUTS2. The ggplot2 package (Wickham and Chang, 2015) provides powerful graphics language for creating complex plots. In order to prepare the data for the visualization, we will use the fortify() function in ggplot2 to convert the shapefile into a data frame, and we will join the data about the tourism and the spatial data. Finally, we will create the choropleth map using the geom_polygon() function. R is not the easiest way to generate maps, but it allows for full control of what the map looks like.

The map in Figure 14.12 represents the total number of nights spent in tourist accommodation in 2013 at regional level. Using this representation, it is easy to identify where tourism in the European Union is concentrated. The most attractive areas are the coastal regions, Alpine regions, and some major cities. Spain and Italy are the most common tourism destinations in the European Union.

Goal 2: Identify the determinants of overall tourists’ satisfaction

The tourism phenomenon raised several policy implications for the efficient supply of tourist services and amenities to the visitors from abroad. Among other issues, some works were devoted to the analysis of the foreigner tourists’ experience and their self‐reported satisfaction.

This example is taken from Cugnata and Perucca (2015). We use a large sample of data from a survey conducted every year since 1996 on behalf of the Bank of Italy. In every survey study, a sample of foreign tourists is asked to report (among other things) their satisfaction with various aspects of their journey in Italy. At the same time, we include some variables for the socio‐economic characteristics of the provinces visited by the respondent.

R offers efficient ways to integrate multiple data frames with different structures, for example, the merge function merges two data frames by common columns or row names or other versions of database join operations.

The goal of the application paper is to identify the determinants of overall tourists’ satisfaction, among individual characteristics and some features of the tourist services supplied.

We use a Bayesian network (BN) approach to the study of tourist satisfaction. As discussed by Salini and Kenett (2009), BN presents several advantages compared with other statistical techniques. First, they are able to outline the complex set of links and interdependencies among the different components of satisfaction. Second, they are extremely useful for policy‐design purposes, since they allow for the building of hypothetical scenarios, letting change the distribution of some parameters under the control of the policymakers and controlling the impact on the variables of interest.

Several R packages implement algorithms and models for constructing BN. We use the bnlearn package (Scutari, 2010, 2015). The hc() function is applied to learn the structure of the BN using the Hill‐climbing algorithm and the graphviz.plot() function to plot the graph associated with the estimated BN. Figure 14.13 shows the BN obtained with the Hill‐climbing algorithm with score functions such as AIC.

As expected, the three groups of variables are (within each category of data) joined together. However, both daily expenditure and the dimensions of satisfaction (in particular with the accommodation) are influenced by the individual characteristics and the socio‐economic indicators.

One of the main benefits of BNs is that they provide an opportunity to conduct “what‐if” sensitivity scenarios. Such scenarios allow us to set up goals and improvement targets. Comparing R to other BN software, an advantage of R is that it is a flexible tool.

Considering the dimensions of self‐reported satisfaction, overall satisfaction is directly influenced by safety and food satisfaction and (indirectly) from prices and accommodation. If we fix the dimension of satisfaction to a certain level, the proportion of (overall) very satisfied respondents (9–10) changes. Figure 14.14 shows the distribution of the overall satisfaction for each level of each variable.