There are many public Datasets available for the consumption of the general public that can be used for education, research, and development purposes. The MovieLens website lets users rate and personalize movie recommendations. GroupLens Research published the rating Datasets from MovieLens. These datasets are available for download from their website, http://grouplens.org/datasets/movielens/. In this chapter, the MovieLens 100K Dataset is being used to demonstrate the usage of distributed data processing with Spark in conjunction with Python, NumPy, SciPy, and matplotlib.
Tip
On the GroupLens Research website for the dataset download, apart from the preceding dataset, there are more voluminous datasets such as MovieLens 1M dataset, MovieLens 10M dataset, MovieLens 20M dataset, and MovieLens latest datasets available for download. Once the reader is quite familiar with the programs and has achieved a sufficient level of comfort playing around with data, these additional datasets can be used by the reader to do their own analysis work to strengthen the knowledge acquired from this chapter.
The MovieLens 100K dataset has data in multiple files. The following are the ones that are going to be used in the data analysis use cases of this chapter:
u.user: The demographic information about the users who have rated movies. The structure of the dataset is given as follows, reproduced as it is from the README file accompanying the dataset:- User ID
- Age
- Gender
- Occupation
- Zip code
u.item: The information about the movies that are rated by the users. The structure of the dataset is given as follows, reproduced as it is from the README file accompanying the dataset:- Movie ID
- Movie title
- Release date
- Video release date
- IMDb URL
- Unknown
- Action
- Adventure
- Animation
- Children's
- Comedy
- Crime
- Documentary
- Drama
- Fantasy
- Film-Noir
- Horror
- Musical
- Mystery
- Romance
- Sci-Fi
- Thriller
- War
- Western