A key insight into how to present textual data (specifically, individual words, or lexemes) to an algorithm comes from linguistics. In word embedding, we pay attention to the neighborhood of each word and use it to establish its meaning and importance. Neighbourhood of a word is the set of words surrounds a particular word. The context of a word is determined by its neighborhood. 

Note that in BoW, a word loses its context, as its context is from the neighborhood that it is in.