In this chapter, we will be using Gensim module (https://github.com/RaRe-Technologies/gensim) to train our word2vec model. Gensim provides large-scale multicore processing support to many popular algorithms like Latent Dirichlet Allocation (LDA), Hierarchical Dirichlet Process (HDP) and word2vec. There are other approaches that we could take like using Tensorflow (https://github.com/tensorflow/models/blob/master/tutorials/embedding/word2vec_optimized.py) to define your own computation graph and build the model, and we will look into that later on.
Know the Code! Python dependencies are quite manageable. You can learn more here https://packaging.python.org/tutorials/managing-dependencies/.
"This tutorial walks you through the use of Pipenv to manage dependencies for an application. It will show you how to install and use the necessary tools and make strong recommendations on best practices. Keep in mind that Python is used for a great many different purposes, and precisely how you want to manage your dependencies may change based on how you decide to publish your software. The guidance presented here is most directly applicable to the development and deployment of network services (including web applications), but is also very well suited to managing development and testing environments for any kind of project."
"This tutorial walks you through the use of Pipenv to manage dependencies for an application. It will show you how to install and use the necessary tools and make strong recommendations on best practices. Keep in mind that Python is used for a great many different purposes, and precisely how you want to manage your dependencies may change based on how you decide to publish your software. The guidance presented here is most directly applicable to the development and deployment of network services (including web applications), but is also very well suited to managing development and testing environments for any kind of project."
We will be using the seaborn package to plot the word clusters, sklearn to implement the t-SNE algorithm and Tensorflow for building tensorboard plots.
import multiprocessing
import os , json , requests
import re
import nltk
import gensim.models.word2vec as w2v
import sklearn.manifold
import pandas as pd
import seaborn as sns
import tensorflow as tf
from tensorflow.contrib.tensorboard.plugins import projector