Measuring the similarity of documents

The representation of documents as word vectors assigns to each document a location in the vector space created by the vocabulary. Interpreting vector entries as Cartesian coordinates in this space, we can use the angle between two vectors to measure their similarity because vectors that point in the same direction contain the same terms with the same frequency weights.

The preceding diagram (the one on the right) illustrates—simplified in two dimensions—the calculation of the distance between a document represented by a vector d1 and a query vector (either a set of search terms or another document) represented by the vector q.

Cosine similarity equals the cosine of the angle between the two vectors. It translates the size of the angle into a number in the range [0, 1] since all vector entries are non-negative token weights. A value of 1 implies that both documents are identical concerning their token weighs, whereas a value of 0 implies that two documents only contain distinct tokens.

As shown in the diagram, the cosine of the angle is equal to the dot product of the vectors; that is, the sum product of their coordinates, divided by the product of the lengths, measured by the Euclidean norms of each vector.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.137.161.222