The following are the key concepts/terms in context of sentiment analysis.

In the analytics space, particularly machine learning and/or text analytics, a feature is one of the central entities upon which an algorithm works. A feature is simply a measureable entity being observed for the current problem at hand.

For example, while performing stock price prediction measures, like the current stock price, time, rate of change of stock price, and so on, are features utilized by different algorithms for generating outputs.

In most use cases, it is common to start with a basic set of features, analyze them and then gradually move towards feature generation and feature extraction/selection. While feature generation is a process of deriving new or additional features from data and existing feature set, feature extraction/selection in general refers to the process of fine tuning and/or reducing the feature set to the best possible subset which can describe a dataset.

The following are a few features commonly utilized in the context of text analytics:

The previous mentioned list of features are just the tip of the iceberg. As data scientists, we need to get creative and derive features based on the dataset at hand, given we have explored the standard plethora of existing features.

Now that we have a sound understanding of sentiment analysis and its key concepts, let us get our hands dirty with an actual use case.

Twitter is a recent favorite amongst celebrities, politicians and leaders to get in touch with their respective audience. Making the best possible use of its follower relationship graph, Twitter enables them to get in touch and stay updated. Our next use case revolves around such Twitter handles itself. With this use case we intend to understand the personality behind one such Twitter handle and analyze:

For this use case, let us tap into the timeline of the most powerful and watched man on the planet, The President of United States of America. The official Twitter handle for POTUS (acronym for President of the United States) is @POTUS.

We will follow the usual analytical workflow again and try to answer the aforementioned questions regarding tweets from @POTUS.

We assume the required libraries such as twitteR, ggplot2, tm and so on are loaded and we have an active connection to our Twitter app using R. For this particular use case, we will load one additional package called the syuzhet.

In the following snippet we connect to Twitter and extract 1500 tweets for the handle @POTUS. We have written a simple utility called extractTimelineTweets(), which extracts tweets, converts the text to UTF-8 format and returns as DataFrame:

# connect to twitter app
setup_twitter_oauth(consumer_key = CONSUMER_KEY,
                    consumer_secret = CONSUMER_SECRET)

tweetsDF <- extractTimelineTweets("POTUS",1500)

First let us begin with the topics Mr. President is talking about. A word cloud is usually a quick way to visualize such an analysis. Before we build one, we need to perform some text processing to clean up and transform the tweets. In our previous use case we were mostly concerned about handles generating the tweet for building the word cloud. In this case we are interested in the actual tweet text and hence the additional care.

We'll make use of the package tm's tools like Corpus and VectorSource to vectorize our tweets corpus and then use tm_map to apply transformations like stopword removal (a list of words like a, an, the and so on, which occur frequently in text but do not carry much factual information), whitespace removal and so on.

The following snippet prepares our data to be used for wordcloud generation:

nohandles <- str_replace_all(tweetsDF$text, "@\w+", "")
wordCorpus <- Corpus(VectorSource(nohandles))
wordCorpus <- tm_map(wordCorpus, removePunctuation)
wordCorpus <- tm_map(wordCorpus, content_transformer(tolower))
wordCorpus <- tm_map(wordCorpus, removeWords, stopwords("english"))
wordCorpus <- tm_map(wordCorpus, removeWords, c("amp"))
wordCorpus <- tm_map(wordCorpus, stripWhitespace)

# prepare wordcloudwordcloud(words = wordCorpus,
          scale=c(5,0.1),
          max.words=1000, 
          random.order=FALSE, 
          rot.per=0.35, 
          use.r.layout=FALSE, 
          colors=pal)

The following wordcloud marks out the common topics which interest the @POTUS, for example climate, world, health, women, gun, congress and so on. Of course, the word American comes out as the most used word in the center of the wordcloud as shown in the following figure:

Tweet text wordcloud from @POTUS

Next, let us now try to understand the sentiments expressed by this user. World leaders play an important role in the dynamics of the world and understanding their sentiments can help us understand them better.

In this case, we will perform sentiment analysis using polarity analysis. Polarity analysis is the process of scoring and aggregating sentiment of the corpus for deriving insights. Such an analysis utilizes a pre-compiled list of words which denotes positive, negative and neutral sentiments. We then parse each tweet/document against this list of polar words and assign a sentiment score based on a certain agreed upon equation.

For example, a simple way of assigning a score to each tweet could be:

Sentiment Score = Sum (positive words) – Sum (negative words)

We can build a sophisticated formula/equation based on our requirement and use case. For example, we may try to include neutral words as well, or we may normalize the whole score to get the final value in a pre-defined range.

We will make use of the syuzhet package to score our tweets. The function get_sentiment() works upon a corpus of text and returns a vector of scores for each text. The scores are on a real scale with negative values referring to negative polarity and vice-versa.

The following snippet performs sentiment scoring and then plots these scores on a histogram for better understanding:

tweetSentiments <- get_sentiment (tweetsDF$text,method = "syuzhet")
tweets <- cbind(tweetsDF, tweetSentiments)
tweets$sentiment <- sapply(tweets$tweetSentiments,encodeSentiment)

qplot(tweets$tweetSentiments) + 
      theme(legend.position="none")+
      xlab("Sentiment Score") +
      ylab("Number of tweets") + 
      ggtitle("Tweets by Sentiment Score")

Polarity analysis of tweets from @POTUS

The plot is slightly skewed towards the right of 0, thus showing that most of the tweets are positive, but there are some tweets with negative sentiments. Users may subset and explore the occasions when @POTUS wasn't all that cheery.

It can be a little difficult to gauge different sentiment aspects from the previous plot. We can view the same from a different perspective by binning sentiments into categories such as positive, very positive, negative, very negative, and neutral. These categories could be based on a range of polarity scores, say from 0 to 0.5 could be marked positive and greater than 0.5 as very positive and similarly for negative polarity.

Let us see which class most sentiments belong to:

tweetSentiments <- get_sentiment (tweetsDF$text, method = "syuzhet")
tweets <- cbind(tweetsDF, tweetSentiments)
tweets$sentiment <- sapply(tweets$tweetSentiments,encodeSentiment)

ggplot(tweets, aes(sentiment)) +
       geom_bar(fill = "aquamarine4") + 
       theme(legend.position="none",
       axis.title.x = element_blank()) +
       ylab("Number of tweets") + 
       ggtitle("Tweets by Sentiment")

The following is the plot generated by using ggplot:

Polarity categories of tweets from @POTUS

The preceding plot shows that a very high percentage of tweets come under the very positive category while the other categories are all comparatively low. #phew!

Note

Not just positive, negative, and neutral sentiments

As humans we express far more emotions than just positive, negative and neutral: we express anger, joy, fear, surprise, and so on. Significant research into the field of sentiment analysis has made it possible to generate lexicons and corpora which can help us better label sentiments. One such research by Dr. Saif Mohammad and Peter Turney is also exposed through the same syuzhet package. Readers are encouraged to explore this further. The previous polarity analysis using NRC lexicon (http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm) generates the following output: #amazing!

As discussed, polarity analysis utilizes a dictionary or list of polar words which are then used to score the text in our corpora. Yet, this is not the only way to perform sentiment analysis. If we look carefully, in the process of sentiment analysis we are basically trying to classify our tweets (or text in general) into positive, negative and neutral classes. In formal terms, this is what we term as a classical classification problem.

In machine learning, classification is a supervised learning problem. In this, we learn about the class labels from a training set which is usually labeled/scored manually or through some measurements.

For example, in the case of sentiment analysis, the training set will constitute a list of documents which would be manually labeled by a team as positive, negative or neutral based on certain pre-agreed rules. Once we have a labeled training set, an algorithm such as Support Vector Machines (SVM) would learn based on the training dataset and then utilize this information to assign labels to our new tweets or unlabeled documents. Such a process requires human supervision and is time-consuming. Yet it has the benefit of learning the underlying process of assigning sentiments to a certain document, which is generally not possible with the polarity based analysis we did previously. There are various datasets, such as WordNet, SentiNet, SentiWordNet, and so, on which contain labeled texts. These datasets are regularly updated and used for research purposes as well as for training supervised algorithms.

Before we finish this use case, let us also touch upon an unsupervised approach for analysis of textual data or tweets in this case. Using a word cloud, we have already identified the various topics @POTUS tweets about. Let us now try and see if we can derive some common themes out of these tweets.

Hierarchical clustering is one such unsupervised learning algorithm widely used to generate hierarchical grouping of textual information. Like any clustering algorithm, hierarchical clustering helps us group similar entities together for better analysis. Hierarchical clustering can be understood in the form of the following three steps:

The merge and compute steps are repeated until we obtain a single cluster containing all the elements of the dataset. As the name suggests, the output of this algorithm is a hierarchical structure which is similar to an inverted tree, also called a dendrogram. The root of the tree represents the most general theme and it goes granular as we go deeper into the tree.

Let us now apply the algorithm on our dataset and analyze the results. We will begin by creating a Term-Document Matrix from our tweets DataFrame using the tm package. We then compute a distance matrix using the dist() function. For this use case, we utilize the Euclidian distance (readers may experiment with other measures as well). Once computed, we use the hclust() function to perform the hierarchical clustering and generate a dendrogram for our dataset.

A Term-Document Matrix is simply a data structure in which all the words (terms in general) of our dataset represent the rows while the documents are represented as columns. If a term exists in a particular document, that row-column intersection is marked as 1 (0 otherwise). The Term-Document Matrix may also utilize frequency of a term instead of 0-1 encoding depending upon the use case.

The following snippet illustrates this point:

# computer term-document matrix
twtrTermDocMatrix <- TermDocumentMatrix(wordCorpus)

twtrTermDocMatrix2 <- removeSparseTerms(twtrTermDocMatrix,
                                        sparse = 0.97)

tweet_matrix <- as.matrix(twtrTermDocMatrix2)

# prepare distance matrix
distMatrix <- dist(scale(tweet_matrix))
# perform hierarchical clustering
fit <- hclust(distMatrix,method="single")
# plot the dendrogram
plot(fit)

The output of this process clearly groups similar terms and can help us derive insights from such tweets by analyzing the grouped tweets together rather than in isolation:

Cluster dendrogram of tweet corpus

With the help of this use case, we used real data from Twitter to analyze sentiments portrayed by tweeple (@POTUS in particular). We touched upon concepts of sentiment analysis and utilized powerful R packages like tm, stringr and syuzhet for the task at hand. Visualizations helped us better understand our data along with the ability to better express our results and insights. Sentiment analysis is utilized under various scenarios, and, most prominently, by brands and organizations. Techniques like sentiment analysis assist marketers to judge their audience/ customer perception/feedback of a product and update their strategies accordingly. These, along with many more scenarios, showcase how important a technique like sentiment analysis could be in this data driven world!