This validation technique is applicable only on CF methods and linear regression CBF since the predicted ratings are generated only by these algorithms. Given each rating rij in u_vals in the validation sets, the predicted rating is calculated using each method and the root mean square error is obtained:

RMSE =

Here, Nval is the number of ratings in the u_vals vectors. The presence of the square factor in this formula highly penalizes the large errors, so the methods with low RMSE (best values) are characterized by small errors spread over all the predicted ratings instead of large errors on few ratings, like the mean absolute error MAE= would prefer.

The code to calculate the RMSE for the memory-based CF user-based and item-based methods is as follows:

For each method, the SE function is called to compute the error for each fold and then the total RMSE of the folds is obtained.

Using 5 nearest-neighbors for item-based CF with slope one and 20 for user-based CF, the methods have the following errors:

All have similar RMSE values but the best method is item-based Collaborative Filtering.

For the model-based methods, instead of not hidden validation ratings, u_test are included in the utility matrix for training and then the RMSE is calculated using the following script:

The code calculates the RMSE only for CBF regression and SVD, and the reader can easily replicate the code to calculate the error for the other algorithms since most of the required code is just commented (SVD expect-maximization, SGD, ALS, and NMF). The results are shown in the following table (K dimension feature space):

As expected, the ALS and SGD are the worst methods but they are discussed because they are instructive from a didactic point of view (they are also slow because the implementation is not as optimized as the methods from sklearn library).

All the others have similar results. However, just note that the hybrid methods have slightly better results than the corresponding SVD and CF user-based algorithms. Note that the movies to predict are chosen randomly so the results may vary.

The rating error RMSE does not really indicate the quality of a method but is an academic measure that is not really used in a commercial environment. The goal of a website is to present content that is relevant to the user regardless of the exact rating the user gives. In order to evaluate the relevance of the recommended items, the precision, recall, and f1 (see Chapter 2, Unsupervised Machine Learning) measures are used where the correct predictions are the items with ratings greater than 3. These measures are calculated on the first 50 items returned by each algorithm (if the algorithm return a recommended list or the 50 items with the highest predicted ratings for the other methods). The function that calculates the measures is as follows:

Here, Boolean ratingsval indicates if the method returns ratings or recommended list. We use the function ClassificationMetrics in the same way we compute the RMSE for all the methods, so the actual code to evaluate the measures is not shown (you can write it as an exercise). The following table summarizes the results for all the methods (neighs is number of nearest-neighbors, K dimension feature space):

From the results you can see that the best method is association rules, and there is good precision also for the LLR, hybrid CBFCF user-based, and CF user-based methods. Note that the results may vary since the movies to predict have been randomly chosen.