TextBlob⁶ is the fundamental class for NLP with the textblob module. Let’s create a TextBlob containing two sentences:

In [1]: from textblob import TextBlob

In [2]: text = 'Today is a beautiful day. Tomorrow looks like bad weather.'

In [3]: blob = TextBlob(text)

In [4]: blob
Out[4]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

TextBlobs—and, as you’ll see shortly, Sentences and Words—support string methods and can be compared with strings. They also provide methods for various NLP tasks. Sentences, Words and TextBlobs inherit from BaseBlob, so they have many common methods and properties.

[Note: We use snippet [3]’s TextBlob in several of the following Self Checks and sub-sections, in which we continue the previous interactive session.]

Natural language processing often requires tokenizing text before performing other NLP tasks. TextBlob provides convenient properties for accessing the sentences and words in TextBlobs. Let’s use the sentence property to get a list of Sentence objects:

In [7]: blob.sentences
Out[7]:
[Sentence("Today is a beautiful day."),
 Sentence("Tomorrow looks like bad weather.")]

The words property returns a WordList object containing a list of Word objects, representing each word in the TextBlob with the punctuation removed:

In [8]: blob.words
Out[8]: WordList(['Today', 'is', 'a', 'beautiful', 'day', 'Tomorrow',
'looks', 'like', 'bad', 'weather'])

Parts-of-speech (POS) tagging is the process of evaluating words based on their context to determine each word’s part of speech. There are eight primary English parts of speech—nouns, pronouns, verbs, adjectives, adverbs, prepositions, conjunctions and interjections (words that express emotion and that are typically followed by punctuation, like “Yes!” or “Ha!”). Within each category there are many subcategories.

Some words have multiple meanings. For example, the words “set” and “run” have hundreds of meanings each! If you look at the dictionary.com definitions of the word “run,” you’ll see that it can be a verb, a noun, an adjective or a part of a verb phrase. An important use of POS tagging is determining a word’s meaning among its possibly many meanings. This is important for helping computers “understand” natural language.

The tags property returns a list of tuples, each containing a word and a string representing its part-of-speech tag:

In [12]: blob
Out[12]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

In [13]: blob.tags
Out[13]:
[('Today', 'NN'),
 ('is', 'VBZ'),
 ('a', 'DT'),
 ('beautiful', 'JJ'),
 ('day', 'NN'),
 ('Tomorrow', 'NNP'),
 ('looks', 'VBZ'),
 ('like', 'IN'),
 ('bad', 'JJ'),
 ('weather', 'NN')]

By default, TextBlob uses a PatternTagger to determine parts-of-speech. This class uses the parts-of-speech tagging capabilities of the pattern library:

https://www.clips.uantwerpen.be/pattern

You can view the library’s 63 parts-of-speech tags at

https://www.clips.uantwerpen.be/pages/MBSP-tags

In the preceding snippet’s output:

• Today, day and weather are tagged as NN—a singular noun or mass noun.

• is and looks are tagged as VBZ—a third person singular present verb.

• a is tagged as DT—a determiner.⁷

• beautiful and bad are tagged as JJ—an adjective.

• Tomorrow is tagged as NNP—a proper singular noun.

• like is tagged as IN—a subordinating conjunction or preposition.

Let’s say you’re preparing to purchase a water ski so you’re researching them online. You might search for “best water ski.” In this case, “water ski” is a noun phrase. If the search engine does not parse the noun phrase properly, you probably will not get the best search results. Go online and try searching for “best water,” “best ski” and “best water ski” and see what you get.

A TextBlob’s noun_phrases property returns a WordList object containing a list of Word objects—one for each noun phrase in the text:

In [15]: blob
Out[15]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

In [16]: blob.noun_phrases
Out[16]: WordList(['beautiful day', 'tomorrow', 'bad weather'])

Note that a Word representing a noun phrase can contain multiple words. A WordList is an extension of Python’s built-in list type. WordLists provide additional methods for stemming, lemmatizing, singularizing and pluralizing.

One of the most common and valuable NLP tasks is sentiment analysis, which determines whether text is positive, neutral or negative. For instance, companies might use this to determine whether people are speaking positively or negatively online about their products. Consider the positive word “good” and the negative word “bad.” Just because a sentence contains “good” or “bad” does not mean the sentence’s sentiment necessarily is positive or negative. For example, the sentence

The food is not good.

clearly has negative sentiment. Similarly, the sentence

The movie was not bad.

clearly has positive sentiment, though perhaps not as positive as something like

The movie was excellent!

Sentiment analysis is a complex machine-learning problem. However, libraries like TextBlob have pretrained machine learning models for performing sentiment analysis.

In [18]: blob
Out[18]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

In [19]: blob.sentiment
Out[19]: Sentiment(polarity=0.07500000000000007,
subjectivity=0.8333333333333333)

In [20]: %precision 3
Out[20]: '%.3f'

In [21]: blob.sentiment.polarity
Out[21]: 0.075

In [22]: blob.sentiment.subjectivity
Out[22]: 0.833

In [23]: for sentence in blob.sentences:
    ...:     print(sentence.sentiment)
    ...:
Sentiment(polarity=0.85, subjectivity=1.0)
Sentiment(polarity=-0.6999999999999998, subjectivity=0.6666666666666666)

By default, a TextBlob and the Sentences and Words you get from it determine sentiment using a PatternAnalyzer, which uses the same sentiment analysis techniques as in the Pattern library. The TextBlob library also comes with a NaiveBayesAnalyzer⁹ (module textblob.sentiments), which was trained on a database of movie reviews. Naive Bayes¹⁰ is a commonly used machine learning text-classification algorithm. The following uses the analyzer keyword argument to specify a TextBlob’s sentiment analyzer. Recall from earlier in this ongoing IPython session that text contains 'Today is a beautiful day. Tomorrow looks like bad weather.':

In [28]: from textblob.sentiments import NaiveBayesAnalyzer

In [29]: blob = TextBlob(text, analyzer=NaiveBayesAnalyzer())

In [30]: blob
Out[30]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

Let’s use the TextBlob’s sentiment property to display the text’s sentiment using the NaiveBayesAnalyzer:

In [31]: blob.sentiment
Out[31]: Sentiment(classification='neg', p_pos=0.47662917962091056,
p_neg=0.5233708203790892)

In this case, the overall sentiment is classified as negative (classification='neg'). The Sentiment object’s p_pos indicates that the TextBlob is 47.66% positive, and its p_neg indicates that the TextBlob is 52.34% negative. Since the overall sentiment is just slightly more negative we’d probably view this TextBlob’s sentiment as neutral overall.

Now, let’s get the sentiment of each Sentence:

In [32]: for sentence in blob.sentences:
    ...:     print(sentence.sentiment)
    ...:
Sentiment(classification='pos', p_pos=0.8117563121751951,
p_neg=0.18824368782480477)
Sentiment(classification='neg', p_pos=0.174363226578349,
p_neg=0.8256367734216521)

Notice that rather than polarity and subjectivity, the Sentiment objects we get from the NaiveBayesAnalyzer contain a classification—'pos' (positive) or 'neg' (negative)— and p_pos (percentage positive) and p_neg (percentage negative) values from 0.0 to 1.0. Once again, we see that the first sentence is positive and the second is negative.

Inter-language translation is a challenging problem in natural language processing and artificial intelligence. With advances in machine learning, artificial intelligence and natural language processing, services like Google Translate (100+ languages) and Microsoft Bing Translator (60+ languages) can translate between languages instantly.

Inter-language translation also is great for people traveling to foreign countries. They can use translation apps to translate menus, road signs and more. There are even efforts at live speech translation so that you’ll be able to converse in real time with people who do not know your natural language._¹¹,¹² Some smartphones, can now work together with in-ear headphones to provide near-live translation of many languages._¹³,¹⁴ _,¹⁵ In the “IBM Watson and Cognitive Computing” chapter, we develop a script that does near real-time inter-language translation among languages supported by Watson.

The TextBlob library uses Google Translate to detect a text’s language and translate TextBlobs, Sentences and Words into other languages.¹⁶ Let’s use detect_language method to detect the language of the text we’re manipulating ('en' is English):

In [36]: blob
Out[36]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

In [37]: blob.detect_language()
Out[37]: 'en'

Next, let’s use the translate method to translate the text to Spanish ('es') then detect the language on the result. The to keyword argument specifies the target language.

In [38]: spanish = blob.translate(to='es')

In [39]: spanish
Out[39]: TextBlob("Hoy es un hermoso dia. Mañana parece mal tiempo.")

In [40]: spanish.detect_language()
Out[40]: 'es'

Next, let’s translate our TextBlob to simplified Chinese (specified as 'zh' or 'zh-CN') then detect the language on the result:

In [41]: chinese = blob.translate(to='zh')

In [42]: chinese
Out[42]: TextBlob("")

In [43]: chinese.detect_language()
Out[43]: 'zh-CN'

Method detect_language’s output always shows simplified Chinese as 'zh-CN', even though the translate function can receive simplified Chinese as 'zh' or 'zh-CN'.

In each of the preceding cases, Google Translate automatically detects the source language. You can specify a source language explicitly by passing the from_lang keyword argument to the translate method, as in

chinese = blob.translate(from_lang='en', to='zh')

Google Translate uses iso-639-1¹⁷ language codes listed at

https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes

For the supported languages, you’d use these codes as the values of the from_lang and to keyword arguments. Google Translate’s list of supported languages is at:

https://cloud.google.com/translate/docs/languages

Calling translate without arguments translates from the detected source language to English:

In [44]: spanish.translate()
Out[44]: TextBlob("Today is a beautiful day. Tomorrow seems like bad
weather.")

In [45]: chinese.translate()
Out[45]: TextBlob("Today is a beautiful day. Tomorrow looks like bad
weather.")

Note the slight difference in the English results.

Inflections are different forms of the same words, such as singular and plural (like “person” and “people”) and different verb tenses (like “run” and “ran”). When you’re calculating word frequencies, you might first want to convert all inflected words to the same form for more accurate word frequencies. Words and WordLists each support converting words to their singular or plural forms. Let’s pluralize and singularize a couple of Word objects:

In [1]: from textblob import Word

In [2]: index = Word('index')

In [3]: index.pluralize()
Out[3]: 'indices'

In [4]: cacti = Word('cacti')

In [5]: cacti.singularize()
Out[5]: 'cactus'

Pluralizing and singularizing are sophisticated tasks which, as you can see above, are not as simple as adding or removing an “s” or “es” at the end of a word.

You can do the same with a WordList:

In [6]: from textblob import TextBlob

In [7]: animals = TextBlob('dog cat fish bird').words

In [8]: animals.pluralize()
Out[8]: WordList(['dogs', 'cats', 'fish', 'birds'])

Note that the word “fish” is the same in both its singular and plural forms.

For natural language processing tasks, it’s important that the text be free of spelling errors. Software packages for writing and editing text, like Microsoft Word, Google Docs and others automatically check your spelling as you type and typically display a red line under misspelled words. Other tools enable you to manually invoke a spelling checker.

You can check a Word’s spelling with its spellcheck method, which returns a list of tuples containing possible correct spellings and a confidence value. Let’s assume we meant to type the word “they” but we misspelled it as “theyr.” The spell checking results show two possible corrections with the word 'they' having the highest confidence value:

In [1]: from textblob import Word

In [2]: word = Word('theyr')

In [3]: %precision 2
Out[3]: '%.2f'

In [4]: word.spellcheck()
Out[4]: [('they', 0.57), ('their', 0.43)]

Note that the word with the highest confidence value might not be the correct word for the given context.

TextBlobs, Sentences and Words all have a correct method that you can call to correct spelling. Calling correct on a Word returns the correctly spelled word that has the highest confidence value (as returned by spellcheck):

In [5]: word.correct() # chooses word with the highest confidence value
Out[5]: 'they'

Calling correct on a TextBlob or Sentence checks the spelling of each word. For each incorrect word, correct replaces it with the correctly spelled one that has the highest confidence value:

In [6]: from textblob import Word

In [7]: sentence = TextBlob('Ths sentense has missplled wrds.')

In [8]: sentence.correct()
Out[8]: TextBlob("The sentence has misspelled words.")

Stemming removes a prefix or suffix from a word leaving only a stem, which may or may not be a real word. Lemmatization is similar, but factors in the word’s part of speech and meaning and results in a real word.

Stemming and lemmatization are normalization operations, in which you prepare words for analysis. For example, before calculating statistics on words in a body of text, you might convert all words to lowercase so that capitalized and lowercase words are not treated differently. Sometimes, you might want to use a word’s root to represent the word’s many forms. For example, in a given application, you might want to treat all of the following words as “program”: program, programs, programmer, programming and programmed (and perhaps U.K. English spellings, like programmes as well).

Words and WordLists each support stemming and lemmatization via the methods stem and lemmatize. Let’s use both on a Word:

In [1]: from textblob import Word

In [2]: word = Word('varieties')

In [3]: word.stem()
Out[3]: 'varieti'

In [4]: word.lemmatize()
Out[4]: 'variety'

Various techniques for detecting similarity between documents rely on word frequencies. As you’ll see here, TextBlob automatically counts word frequencies. First, let’s load the e-book for Shakespeare’s Romeo and Juliet into a TextBlob. To do so, we’ll use the Path class from the Python Standard Library’s pathlib module:

In [1]: from pathlib import Path

In [2]: from textblob import TextBlob

In [3]: blob = TextBlob(Path('RomeoAndJuliet.txt').read_text())

Use the file RomeoAndJuliet.txt18 that you downloaded earlier. We assume here that you started your IPython session from that folder. When you read a file with Path’s read_text method, it closes the file immediately after it finishes reading the file.

You can access the word frequencies through the TextBlob’s word_counts dictionary. Let’s get the counts of several words in the play:

In [4]: blob.word_counts['juliet']
Out[4]: 190

In [5]: blob.word_counts['romeo']
Out[5]: 315

In [6]: blob.word_counts['thou']
Out[6]: 278

If you already have tokenized a TextBlob into a WordList, you can count specific words in the list via the count method:

In [7]: blob.words.count('joy')
Out[7]: 14

In [8]: blob.noun_phrases.count('lady capulet')
Out[8]: 46

WordNet¹⁹ is a word database created by Princeton University. The TextBlob library uses the NLTK library’s WordNet interface, enabling you to look up word definitions, and get synonyms and antonyms. For more information, check out the NLTK WordNet interface documentation at:

https://www.nltk.org/api/nltk.corpus.reader.html#module-nltk.corpus.reader.wordnet

In [1]: from textblob import Word

In [2]: happy = Word('happy')

In [3]: happy.definitions
Out[3]:
['enjoying or showing or marked by joy or pleasure',
 'marked by good fortune',
 'eagerly disposed to act or to be of service',
 'well expressed and to the point']

In [4]: happy.synsets
Out[4]:
[Synset('happy.a.01'),
 Synset('felicitous.s.02'),
 Synset('glad.s.02'),
 Synset('happy.s.04')]

In [5]: synonyms = set()

In [6]: for synset in happy.synsets:
   ...:     for lemma in synset.lemmas():
   ...:         synonyms.add(lemma.name())
   ...:

In [7]: synonyms
Out[7]: {'felicitous', 'glad', 'happy', 'well-chosen'}

In [8]: lemmas = happy.synsets[0].lemmas()

In [9]: lemmas
Out[9]: [Lemma('happy.a.01.happy')]

In [10]: lemmas[0].antonyms()
Out[10]: [Lemma('unhappy.a.01.unhappy')]

Stop words are common words in text that are often removed from text before analyzing it because they typically do not provide useful information. The following table shows NLTK’s list of English stop words, which is returned by the NLTK stopwords module’s words function²⁰ (which we’ll use momentarily):

The NLTK library has lists of stop words for several other natural languages as well. Before using NLTK’s stop-words lists, you must download them, which you do with the nltk module’s download function:

In [1]: import nltk

In [2]: nltk.download('stopwords')
[nltk_data] Downloading package stopwords to
[nltk_data] C:UsersPaulDeitelAppDataRoaming
ltk_data...
[nltk_data] Unzipping corporastopwords.zip.
Out[2]: True

For this example, we’ll load the 'english' stop words list. First import stopwords from the nltk.corpus module, then use stopwords method words to load the 'english' stop words list:

In [3]: from nltk.corpus import stopwords

In [4]: stops = stopwords.words('english')

Next, let’s create a TextBlob from which we’ll remove stop words:

In [5]: from textblob import TextBlob

In [6]: blob = TextBlob('Today is a beautiful day.')

Finally, to remove the stop words, let’s use the TextBlob’s words in a list comprehension that adds each word to the resulting list only if the word is not in stops:

In [7]: [word for word in blob.words if word not in stops]
Out[7]: ['Today', 'beautiful', 'day']

An n-gram²¹ is a sequence of n text items, such as letters in words or words in a sentence. In natural language processing, n-grams can be used to identify letters or words that frequently appear adjacent to one another. For text-based user input, this can help predict the next letter or word a user will type—such as when completing items in IPython with tab-completion or when entering a message to a friend in your favorite smartphone messaging app. For speech-to-text, n-grams might be used to improve the quality of the transcription. N-grams are a form of co-occurrence in which words or letters appear near each other in a body of text.

TextBlob’s ngrams method produces a list of WordList n-grams of length three by default—known as trigrams. You can pass the keyword argument n to produce n-grams of any desired length. The output shows that the first trigram contains the first three words in the sentence ('Today', 'is' and 'a'). Then, ngrams creates a trigram starting with the second word ('is', 'a' and 'beautiful') and so on until it creates a trigram containing the last three words in the TextBlob:

In [1]: from textblob import TextBlob

In [2]: text = 'Today is a beautiful day. Tomorrow looks like bad weather.'

In [3]: blob = TextBlob(text)

In [4]: blob.ngrams()
Out[4]:
[WordList(['Today', 'is', 'a']),
 WordList(['is', 'a', 'beautiful']),
 WordList(['a', 'beautiful', 'day']),
 WordList(['beautiful', 'day', 'Tomorrow']),
 WordList(['day', 'Tomorrow', 'looks']),
 WordList(['Tomorrow', 'looks', 'like']),
 WordList(['looks', 'like', 'bad']),
 WordList(['like', 'bad', 'weather'])]

The following produces n-grams consisting of five words:

In [5]: blob.ngrams(n=5)
Out[5]:
[WordList(['Today', 'is', 'a', 'beautiful', 'day']),
 WordList(['is', 'a', 'beautiful', 'day', 'Tomorrow']),
 WordList(['a', 'beautiful', 'day', 'Tomorrow', 'looks']),
 WordList(['beautiful', 'day', 'Tomorrow', 'looks', 'like']),
 WordList(['day', 'Tomorrow', 'looks', 'like', 'bad']),
 WordList(['Tomorrow', 'looks', 'like', 'bad', 'weather'])]