Vector Space Model

For information retrieval, indexing, filtering, and relevancy ranking, the vector space model is an algebraic model used to represent any set of documents (text or images) as vectors of identifiers (rows and columns), where rows represent documents and columns represent document terms.

Let's illustrate vector space based on the following search queries from two documents:

• Doc_1: customer loves meat

• Doc_2: customer loves vegetables

Without doing any text filtering, the vector space model of Doc1 and Doc2 is represented in the table in Exhibit 3.16. It is a binary representation of the documents and their terms, where:

• 1 means the word (term) appears in the document
• 0 means the word (term) does NOT appear in the document
• Word frequency location filtering and normalization have been applied to this representation

Term

Term refers to words, punctuation, phrases, multiword terms, expressions, or simply put, a token in any given document.

Document

A document is a collection of terms. It could be a title, a phrase, a sentence, a paragraph, a query, or a file. For instance, a resume, a news article, or a blog post could be considered documents in text analytics.

Corpus

A corpus is a collection of text data that is used to describe a language. It could be a collection of documents, writings, speeches, and conversation.

Exhibit 3.17 showcases examples of term, document, and corpus.

Stop Word List

Stop word lists are a set of commonly used words in any language that are ignored or removed during the text parsing due to their lack of relevance or value in the text analysis. For instance, articles, conjunctions, and prepositions provide little information in the context of a sentence.

Stop words are important in text analytics because they enable focus on the more important words to find pattern, trend, and meaning in text data. For instance, if we were to perform a search: “How to become a data scientist?” The search engine would find web pages that contain the terms how, to, become, a, data, and scientist. All the pages found with the terms how, to, and a would flood the search, since they are more commonly used terms in the English language, compared to data and scientist. So, by disregarding the frequent terms, the search engine can focus on retrieving pages that contain the keywords become, data, and scientist and the results will be closer to what we want.

Stop words are useful before clustering is applied. Commonly used stop words include:

• Prepositions: There are about 150 prepositions in English, such as about, above, across, as, at, along, around, before, below, but, by, beyond, for, from, in, into, of, over, than, to, since, via, with, without, and the like.
• Determiners: a, an, any, another, other, the, this, that, these, those, and the like.
• Some adjectives: kind, nice, good, great, other, big, high, different, few, bad, same, able, and the like.

It is important to note that for some analyses, such as sentiment analysis, some stop words should be included in the text mining to be able to find patterns, trends, and sentiments. When performing sentiment analysis, some information retrieval tools (search engine) will simply avoid removing those stop words to optimize the search results.

You can define and build your own stop words based on the type of analytics, domain, or industries you envision to apply text analytics. You can also define a role for your stop words and decide to include or exclude them if they are used as verbs or nouns. A good example is the word show. It could be excluded if it were used as a verb (display) while it would be included when it is used as a noun (spectacle). There are publicly available stop word lists that many text analytics software and open source tools use.

Start Word List

These lists enable you to control which terms are used in a text mining analysis. A start word list is a dataset that contains a list of terms to include in the parsing results. If you use a start word list, then only terms that are included in that list appear in parsing results.

Stemming

Stemming refers to the process of finding the stem or root form of a term. Like the start and stop word lists, most software uses predefined dictionary-based stemming. You can also customize your stemming dictionary to include a specific stemming technique, as in using declar as the stem for declared, declares, or declaration.

Parts of Speech

Parts of speech (POS) are the basic types of words that any language has. Each part of speech explains how the word is used. Words that are assigned the same POS have similar behaviors in terms of syntax and have similar roles within the grammatical structure of sentences in terms of morphology. In English, there are nine parts of speech: articles, nouns, pronouns, adjectives, verbs, adverbs, conjunctions, prepositions, and interjections.

Entity

An entity is any of the several types of information that should be distinguished from general text during text mining. Most text analytics software can identify entities and also analyze them as a unit. Entities are also normalized. You can customize the list of entities based on the domain to which you are applying text mining. Most software programs have a dictionary of normalized entities, such as company name, that will include a list of companies in a given country with all the taxonomy related to the name. An example of entity company name is: International Business Machine: IBM.

Entities that are commonly used include: person, proper noun, product, location, address, phone, time, date, company, currency, measure, organization, percent, Social Security number, code, or vehicle. You can modify this list of entities to include or exclude others based on the analysis you envision.

Noun Groups

A noun group is a group of words, nouns or pronouns, in a document collection. Noun groups are identified based on linguistic relationships that exist within sentences. Noun groups identified in text mining act as single units. You can choose, therefore, to parse them as single terms.

It is important to note that stemming noun groups will treat the group as a single term; for example, case studies is parsed as case study.

Synonymy

A synonym refers to a word or phrase that shares the same meaning with another word or phrase in the same language. Words that are synonyms are said to be synonymous, and the state of being synonymous is called synonymy.

Synonymy is very useful in text analytics, as it helps to reduce the redundancies by treating words with the same meaning as equivalents. A synonym list enables you to specify different words that should be processed equivalently, for instance, vehicle and car.

Polysemy

Polysemy refers to the capacity of a word term to have multiple meanings or senses: for instance, the word show could refer to a spectacle (noun), but it could also mean to display (verb). Polysemy is very useful in NLP and text mining, such as sentiment analysis.

N-grams of texts refer to a set of co-occurring words from a given sequence or window. An n-gram of size 1 is referred to as a unigram, size 2 is a bigram, size 3 is a trigram. Larger sizes are sometimes referred to by the value of n, for example: four-gram, five-gram, and so on. When building the n-grams you will basically move one word forward. You can also move two or more n words forward. For example, consider the sentence “The weather was really bad yesterday.”

If n = 2 (known as bigrams), then the n-grams would be:

• the weather
• weather was
• was really
• really bad
• bad yesterday

N-grams are extensively used in text mining and NLP tasks such as type-ahead spelling, correction, text summarization, and tokenization.

Zipf's Law: Modeling the Distribution of Terms

Zipf's law refers to the power law that is used to showcase terms' distribution across documents. Zipf's law is mathematically simple; it states frequencies proportionally. The most frequent word (i = 1) has a frequency proportional to 1, the second most frequent word (i = 2) has a frequency proportional to 1/2, the third most frequent word has a frequency proportional to 1/3, and so on (1/i: of Cfi ∼ 1/i).

Latent Semantic Analysis (LSA)

Latent semantic analysis (LSA)⁵ is a technique in NLP, like distributional semantics, in which relationships between a set of documents and the terms they contain are analyzed by producing a set of concepts related to the documents and terms. LSA assumes that words that are close in meaning will occur in similar pieces of text. A matrix containing word counts per paragraph (rows represent unique words and columns represent each paragraph count) is constructed from a large piece of text. Then SVD is used to reduce the number of rows while preserving the similarity structure among the columns. Words are compared by taking the cosine of the angle between the two vectors or the dot product between the normalizations of the two vectors formed by any two rows. Values close to 1 represent very similar words while values close to 0 represent very dissimilar words. LSA is used for information retrieval from search queries.

Dot Product

The dot product is a measure of a distance or angle between two vectors in each space coordinate. This operation takes two equal-length vectors (or sequence of numbers) and returns a single number. Algebraically, it is the sum of the products of the corresponding entries of the two sequences of numbers. Geometrically, it is the product of the Euclidian magnitudes of the two vectors and the cosine of the angle between them. For instance, let vectors V1 = [1, 2, 3] and V2 = [4, 5, 6]; the dot product (V1, V2) = 1 × 4 + 2 × 5 + 3 × 6 = 32.

Dot products and cosines are used to measure the similarity between documents. Once a quantitative representation of document is created, the cosine is used to measure the similarity between vectors that represent each term document.