In this chapter, we will demonstrate how to use various NLP APIs to perform text classification. This is not to be confused with text clustering. Clustering is concerned with the identification of text without the use of predefined categories. Classification, in contrast, uses predefined categories. We will focus on text classification where tags are assigned to text to specify its type.
The general approach used to perform text classification starts with the training of a model. The model is validated and then used to classify documents. We will focus on the training and usage steps.
Documents can be classified according to any number of attributes such as its subject, document type, time of publication, author, language used, and reading level. Some classification approaches require humans to label sample data.
Sentiment analysis is a type of classification. It is concerned with determining what text is trying to convey to a reader, usually in the form of a positive and negative attitude. We will investigate several techniques to perform this type of analysis.
Classifying text is used for a number of purposes:
- Spam detection
- Authorship attribution
- Sentiment analysis
- Age and gender identification
- Determining the subject of a document
- Language identification
Spamming is an unfortunate reality for most e-mail users. If an e-mail can be classified as spam, then it can be moved to a spam folder. A text message can be analyzed and certain attributes can be used to designate the e-mail as spam. These attributes can include misspellings, lack of an appropriate e-mail address for recipients, and a non-standard URL.
Classification has been used to determine the authorship of documents. This has been performed for historical documents such as for The Federalist Papers and for the book Primary Colors where the authors have been identified.
Sentiment analysis is a technique that determines the attitude of text. Movie reviews have been a popular domain but it can be used for almost any product review. This helps companies better assess how their product is perceived. Often, a negative or positive attribute is assigned to text. Sentiment analysis is also called opinion extraction/mining and subjectivity analysis. Consumer confidence and the performance of a stock market can be predicted from Twitter feeds and other sources.
Classification can be used to determine the age and gender of a text's author and to provide more insight into its author. Frequently, the number of pronouns, determiners, and noun phrases are used to identify the gender of a writer. Females tend to use more pronouns and males tend to use more determiners.
Determining the subject of text is useful when we need to organize a large number of documents. Search engines are very much concerned with this activity but it has also been used simply to place documents in different categories such as used with tag clouds. A tag cloud is a group of words reflecting the relative frequency of occurrence of each word.
The following image is an example of a tag cloud generated by IBM Word Cloud Generator (http://www.softpedia.com/get/Office-tools/Other-Office-Tools/IBM-Word-Cloud-Generator.shtml) and can be found at http://upload.wikimedia.org/wikipedia/commons/9/9e/Foundation-l_word_cloud_without_headers_and_quotes.png:
The identification of the language used by a document is supported using classification techniques. This analysis is useful for many NLP problems where we need to apply specific language models to the problem.