5.2.1. Attention network mechanism in recommender systems

Although several deep learning models provide a satisfactory solution to recommendation systems, they are less effective when new items do not accumulate enough data. The attention mechanism (Vaswani et al. 2017) has attracted interest from researchers as a solution to this cold start problem in recommendation systems. The attention mechanism enables the model to impose different weights on inputs, depending on the context. Tay et al. (2018) propose a multi-pointer learning scheme that learns to combine multiple views of user-item interactions based on the pointer networks. They apply a co-attention mechanism to select the most informative review from the review bank of each user and item, respectively. Liu et al. (2021) have developed a sequential neural recommendation to model users’ long- and short-term preferences through aspect-level reviews. The model embeds user-related and item-related reviews into a continuous low-dimensional dense vector space using aspect-aware convolution and self-attention. Ying et al. (2018) introduce a memory attention-aware recommender system based on a memory component and an attentional mechanism to learn deep adaptive user representations.

Kang et al. (2018) propose a self-attention-based sequential model that allows long-term semantics to be captured, but makes its predictions based on relatively few actions. At each time step, the model seeks to identify which items are relevant from a user’s action history and use them to predict the next item. Zhou et al. (2018) developed a Deep Interest Network (DIN) by designing a local activation unit to adaptively learn the representation of user interests from historical behaviors concerning a particular ad. The algorithm learns to aggregate the sequence of users’ history to form a user embedding. User embedding is regarded as the weighted sum of item embeddings, and an attention network obtains the weight of each item embedding. In Yakhchi et al. (2020), the authors propose a Deep Attention-based Sequential (DAS) model. It deals with the representation of false dependencies between items when modeling a user’s long-term preferences by attention mechanism.

The recommender engine learns to aggregate the users’ history sequence to compute the attention weighted sum of item embeddings in previous work. By contrast, the proposed framework does not only learn the interaction between user interests and item embeddings, but it also adds a new component to learn the mutual influence generated by the contributions of items that carry collaborative signals on user decisions, helping to account for complex user-item interactions.

5.2.2. Stacked machine learning for optimization

Ensemble machine learning algorithms enhance the predictions of simple and understandable models. They require a more sophisticated technique to prepare the model and more computational resources to train the model. They are wrappers for a set of smaller, concrete pre-trained models. The wrapper then follows a well-defined strategy to aggregate their predictions into a final one. Several ensemble approaches have been proposed, such as voting classifiers, bagging and pasting, boosting and stacking (Pavlyshenko et al. 2018). Stacking is a rather simple idea that can leverage one of the most powerful ensemble models possible. Instead of using a trivial aggregation function such as “mode” and “mean”, stacking uses a meta-learner model that learns the mapping from lower predictions to the label. A stacked generalization ensemble can use the set of predictions as a context and conditionally decide to weigh the input predictions differently, potentially resulting in better performance (Wolpert et al. 1992).

Numerous approaches combining multiple classifiers based on meta-learning have been proposed. Reid and Grudic (2009) apply regularization to linear stack generalization at the combiner level to avoid the over-fitting problem and improve performance. Jurek et al. (2014) propose a meta-learning-based approach incorporating an unsupervised learning method at the meta-level. All of the base classifiers initially classify each instance from the validation set (Jurek et al. 2014). Outputs of the classification process are considered later as new attributes. The K-Means clustering technique divides all instances from the validation set into clusters according to the new attributes. The collection of clusters is considered as a final meta-model, where each cluster represented one class. Deng et al. (2012b) propose a kernel deep convex networks (K-DCN) architecture composed of a variable number of modules stacked together to form the deep architecture. The deep stacking network (DSN) enabling parallel training on very large-scale datasets is proposed in Deng et al. (2012a).

For recommendation, Otunba et al. (2019) stack an ensemble of a generalized matrix factorization (GMF) and multi-layer perceptron (MLP) that propagates the prediction from constituent models through other constituent models to final output. Bao et al. (2009) combine predictions from multiple recommendation engines to generate a single prediction. They first define each component recommendation engine as a level-1 predictor. Then they learn a level-2 predictor, using a meta-learning algorithm, with predictions of the component engines as meta-features. The level-2 predictor can be either a linear function or a nonlinear function based on the meta-learning algorithm employed. They define a new meta-feature which represents properties of the input users/items as additional meta-features. It allows the combination of component recommendation engines at runtime based on user/item characteristics. In Da et al. (2016), the authors develop three ensemble approaches based on multimodal interactions. Different types of users feedback processed individually by traditional recommendation algorithms are combined in order to optimize the modeling of users’ profiles. Unlike previous works with stacked recommenders, the stacking content-based recommender considers the content for recommendation to create a profile model for each user. Its main advantage is the ability of the embedding representation to integrate the side information in the hybrid architecture.

5.3.1. Notation

A robust way to formulate a recommendation task is to approach it as a prediction problem. Thus, the interactions between users and items are represented in the form of a utility matrix R. For each user u ∈ U, the recommender system attempts to predict all of the unspecified ratings ȓ_u. Let U = u₁, u₂, ..., u_n and I = i₁, i₂, ..., i_m be the sets of users and items, respectively, where n is the number of users, and m is the number of items. The matrix factorization algorithm decomposes a matrix M_(n×m) into two matrices P E IR^N×K and Q E IR^M×K. A user’s interaction on an item is modeled as the inner product (interaction function) of their latent vectors. Let ui be the ground truth rating assigned by the user u on the item i. The utility matrix is defined as:

[5.1]

where K denotes the dimension of the latent space. Latent factor models are a state-of-the-art methodology for model-based collaborative filtering. Matrix factorization techniques are a class of widely successful latent factor models that attempt to characterize items and users using vectors of factors inferred from item rating patterns. High correspondence between item and user factors leads to a recommendation. The matrix factorization performs remarkably well on dyadic data prediction tasks. However, it fails to capture the heterogeneous nature of objects and their interactions, as well as failing to model the context in which a rating is given. In fact, latent factor models are inherently linear, which limits their modeling capacity for recommendation. Recently, a growing body of work adding crafted nonlinear features into the linear latent factor models has been introduced, powered by neural networks. In this chapter, we use the GloVe embedding with 50 dimensions in order to represent the latent space.

A training set T is consisted of N tuples. Each tuple (u, i, ȓui) denotes a rating by user u for item i. For a given predicted rating matrix, the normalization is done on a user-basis. For each user ∈ U, the minimal rating min = min (r^s(ui)) and the maximal rating max = max (r^s (ui)) are extracted. Then, the min/max scaling function is applied to as follows:

[5.2]

The normalization step can be performed before, during or after the generation of the rating matrix. Moreover, one can easily adjust neural-based recommenders for this purpose. Sigmoid is used as the activation function for the output layer, ultimately skipping the use of the Min/Max scaler. In the remainder of this chapter, we use the notations given in Table 5.1.

5.3.2. The interactive attention network recommender

The first component of the proposed framework is an interactive attention network recommender. Its role is to discover latent features that exhibit mutual influence between users and items. The attention mechanism has proved to be effective in various machine learning tasks such as image/video captioning (Rital et al. 2002, 2005; Xu et al. 2015) and machine translation (Bahdanau et al. 2014). It allows different parts to contribute when compressing them to a single representation. The attention mechanism is a process that mimics the actions of the human brain to selectively concentrate on a few relevant things, while ignoring others in deep neural networks. The attention mechanism emerged naturally from problems that deal with time-varying data. The core idea is that we can look at all of the different words at the same time and learn to pay attention to the correct ones depending on the task at hand.

The attention mechanism, which is simply a notion of memory, gained from attending at multiple inputs through time, emerged from problems that deal with time-varying data.

The attention network is based on the encoder and decoder which are stacked RNN layers. The encoder processes the input and produces one compact representation, called z, from all of the input time steps. It can be regarded as a compressed format of the input. The attention ensures a direct connection with each timestamp, in which the context vector z should have access to all parts of the input sequence instead of just the last one (see Figure 5.2). Thus, applying a co-attention mechanism in the context of collaborative filtering recommendation allows the discrimination of the items that are interesting for users, even those with no previous interaction, through higher attention weights.

The attention network-CF approach is inspired by the paper by Chen et al. (2017). Previous works put more emphasis on only learning the complex relationship between the target users (or items) and their neighbors by attention network. Here, we aim to exploit the encoding ability of the interactive attention between the users and the items to learn the most relevant weights that represent the users’ mutual influence on the item. The underlying idea is that some correlation between users and items with particular characteristics can reveal the possibility that an item is interesting for similar users (see Figure 5.3).

First, the list of users U and the list of items I are fed into two different embedding layers: e_u and e_i, respectively. It makes it possible to capture some useful latent properties of users p_u and items q_i. Each of these embedding layers is chained with a long short-term memory (LSTM) layer. The LSTM (Sherstinsky et al. 2020) is a variety of RNNs that can learn long sequences with long time lags. The advantage of this architecture is that LSTM units are recurrent modules, which enable long-range learning. Each LSTM state has two inputs, the current feature vector and the output vector of the previous state, h_t-1, and one output vector h_t. The LSTM based representation learning can be denoted as follows:

[5.3]

[5.4]

The learned representation can be denoted as H and H, respectively. The respective dimensions of H and H are d × n and d × m (d-dimensional vectors of LSTM).

The attention mechanism is used to project the users and items embedding inputs into a common representation space. The proposed neural attention framework can model the high-order nonlinear relationship between users and items and mutual influence. Indeed, interactive attention on both the users and items is applied. The final rating prediction is based on all the interactive users and items features. This mechanism is explored as follows: joint user and item interactive attention maps are built and combined recursively to predict a distribution over the items. First, a matrix L ∈ IR^n×m as L = tanh is computed, where W_pq is a d × d matrix of learnable parameters. The feature interaction attention map is given by:

[5.5]

[5.6]

Therefore, the interactive attention models the mutual interactions between the user latent factors and item latent factors by applying a tangent function (tanh). Subsequently, an attention distribution is calculated as a probability distribution over the embedding space. The attention weights are generated through the softmax function:

[5.7]

[5.8]

The function f is a multi-layer neural network (MLP). Besides, the attention vectors of high-order interaction features can be generated by a weighted sum using the derived attention weights or a sigmoid function, given by β_p and β. These latent spaces of users and items are then concatenated as follows:

In order to form the predicted score R_ui, the concatenation spaces are fed into a dense layer with a sigmoid activation function as follows:

[5.9]

[5.10]

During the training phase, a grid-search method is used to learn the model parameters and set the cost function as defined below:

[5.11]

The model predicts interest probabilities for each possible combination of the prediction set ℘, as shown in Algorithm 1.

5.3.3. The stacked content-based filtering recommender

The content-based recommender system injects item attributes as a deciding factor for recommendations. The stacked recommender generates a user profile for each user in the form of a learned model. First, the recommender extracts “tags” from item descriptions or reviews and uses them to calculate each item’s embedding vector with the help of Stanford’s pre-trained GloVe vectors (Pennington et al. 2014). Afterward, the recommender employs the stacking ensemble learning technique to learn the profile for each user, as shown in Figure 5.4.

Let L = {L₁, L₂, ..., L_l} be different regression models, and x_train be the training dataset. U and emb are independent variables. The base regression models’ hyperparameters are ∀l ∈ L θ_l. The number and the type of regression algorithms are tunable.

Each l ∈ L is trained separately with the same training dataset. Each model provides predictions for the outcomes (R), which are then cast into a meta-learner (blender). In other words, the L predictions of each regressor become features for the blender. The latter can be any model, such as linear regression, SVR or Decision Tree,...etc., as expressed in [5.11].

[5.12]

where the weight vector w is learned by a meta-learner.

A blender model can then be defined and tuned with its hyperparameters 0_blende. It is then trained on the outputs of the stack L. It learns the mapping between the outcome of the stacked predictors and the final ground-truth ratings. The expression of the final prediction is as follows:

[5.13]

In order to train the two recommenders needed for the task at hand, one applies an aggregation function to merge their outputs into a single utility matrix. The IPRec framework uses the simple unweighted average aggregation function followed by encoder layers. The final predicted utility matrix _ui is as follows.

[5.14]

5.4.1. The datasets

MovieLens dataset: It presents real, timestamped 5-star ratings, as given by users of the MovieLens Website on different films. Furthermore, GroupLens supplied movies with their genomic tags to describe them in a machine-friendly manner. The dataset has several versions, depicted by the total number of ratings packaged in the files (Harper et al. 2015). They are:

• – MovieLens 25M: this huge dataset offers 25 million ratings, applied to 62,000 movies by 162,000 users. It was released in December 2019.
• – MovieLens 20M: the predecessor of the previous version offers 20 million ratings given by 138,000 users over 27,000 movies. It was first released in April 2015 and last updated in October 2016. Its support has been discontinued ever since.
• – MovieLens 10M: it is a rather old dataset, dated February 2009. It provides 10 million ratings applied to 10,000 movies by 72,000 users. An Interactive Attention Network 137
• – MovieLens 1M: this is the most popular MovieLens Dataset version. It is widely used for research and benchmarking recommender systems. It contains 1 million ratings from 6,000 users over 4,000 movies. Its exact specifications are in Table 5.2.
• – MovieLens Small: a small subset of the latest MovieLens Dataset version (currently MovieLens 25M), which usually contains 100,000 ratings applied to 9,000 movies over 600 users. It is generally used for educational purposes only, such as learning recommender systems.

5.4.2. Evaluation metrics

Two influential metrics are adopted to evaluate the predictive accuracy of the recommendation framework: the mean average precision and the normalized discounted cumulative gain.

The mean average precision (MAP) is a popular metric in measuring the accuracy of information retrieval, and object detection systems (Lasfar et al. 2000; Labatut et al. 2012). For a set of queries Q, MAP calculates the mean of the average precision scores for each query q∈ Q, The result is always a value between 0 and 1, where the higher the score, the better the accuracy.

[5.15]

[5.16]

where:

• – TP: True Positives: positive items that are detected by the system as positive;
• – FP: False Positives: negative items that are detected by the system as positive;
• – Pr: the precision value considering only q-first items.

In recommender systems, we usually trim the results to return the top-k elements, where 1 <k<=q. The number of elements k depends on use: a system may show the top three trending items or the best 10 items that match the taste of the current user, etc. Therefore, a more flexible variant of MAP, referred to by MAP @, is used. The latter performs the same calculation procedure but over the smaller set of top-k elements:

[5.17]

The rank in which recommendations are shown has an impact on the performance of a recommender system. The normalized discounted cumulative gain (NDCG) is a ranking quality evaluation metric that is also used in information retrieval systems (Demirkesen et al. 2008; Balakrishnan et al. 2012). NDCG measures the normalized usefulness of items based on their positions in the resulting list by calculating the ratio between discounted cumulative gain (DCG) of the recommended items over the DCG of their ideal ranking. The result ∈ [0 –1] indicates the gain of the recommender. Hence, the higher this value, the better.

[5.18]

[5.19]

[5.20]

where:

• – DCG: the discounted cumulative gain of the predicted item set ranked by the recommender system;
• – i DCG: the discounted cumulative gain of the ideal ranking of predicted items; An Interactive Attention Network 139
• – Pos(q): the position of q, as predicted by the recommender system;
• – iPos(q): the ideal position of q.

NDCG@k is used to evaluate the ranking quality of the top-k recommended items.

5.4.3. Baselines

The personalized recommender system is compared with the following recommender systems:

• – Interactive attention network recommender: it is a novel collaborative filtering recommender system. This state-of-the-art technique is based on the co-attention mechanism. It takes its power from discriminating the importance of different feature interactions between all of the entities (user-user, item-item and users-item).
• – The stacked content-based filtering recommender: this content-based recommender system is proposed in Drif et al. (2020). The stacked ensemble exploits the strengths of embedding representation and the stacking ensemble learning for modeling the content. Here, we adopt a Random forest (Chollet et al. 2021) model as a meta-learner because it improves the predictive accuracy by fitting several decision tree classifiers on various sub-samples of the dataset.
• – Neural Collaborative Filtering (NCF) (He et al. 2017): this recommender system applies the MLP to learn the user-item interaction function.
• – Variational Autoencoders for Collaborative Filtering (MultiVAE) (Liang et al. 2018): this technique models the collaborative information in a multinomial distribution to sample prediction for items on the long tail.
• – EnsVAE:Ensemble Variational Autoencoders for Recommendations (Drif et al. 2020): the EnsVAE-compliant recommender system provides simple guidelines to build hybrids for a plethora of use cases. It is adjusted to output interest probabilities by learning the distribution of each item’s ratings and attempts to provide various novel items pertinent to users.

5.5.1. Hyperparameter analysis

This section depicts the hyperparameters analysis step performed separately on each recommender. The proposed implementation method is based on Keras (Chollet et al. 2021). In addition, the latter runs over Tensorflow (Abadi et al. 2016), which ensures the ability to reproduce the same results by replicating the same developing and evaluating environment. The machine used throughout the whole development and evaluation phases is a MacBook Pro (16 GB 1600 MHz DDR3, GPU 2.2GHz 6-core Intel Core i7).

The stacked content-based recommender is composed of a set of regression models and a meta-learner. The idea behind stacking is to enhance weak learners with a strong learner. Therefore, a small group of weak learners that exhibit different errors would perform the same as a large set of powerful learners with similar errors (Géron et al. 2019). Consequently, we perform the analysis on stacks 1–5 and focus on the learning algorithms used by each learner, alongside the meta-learner’s hyperparameters.

The first step is to find the combination of regression models that compose the best stack. A grid search is performed over six algorithms: polynomial-kernel support vector machines (SVM_poly) (Singh et al. 2016), RBF-kernel support vector machines (SVM_rbf), decision trees (DT), automatic relevance detection regression (ARD), the simple linear regression (LR) and the Random Forest. Table 5.3 reports the results of the top 5 stacks for k= 10, 30 and 50.

Stack 4 exhibits the higher score compared to the other stacks. The structure of each stack is as follows:

• – Stack 1: DT – LR – SVM_rbf | ARD
• – Stack 2: LR – DT – ARD | SVM_poly
• – Stack 3: DT – LR – SVR_poly | ARD
• – Stack 4: LR – SVR_poly – ARD | Random Forest
• – Stack 5: LR – SVR_poly – Random Forest| ARD

The second step is to fine-tune the stack’s learners and the meta-learners hyperparameters. Here, we present the results of fine-tuning the Stack base Random Forest meta-learner. A grid search algorithm over Random Forest hyperparameters, namely: n_estimators (the number of trees in the forest), max_depth (the maximum number of levels in each decision tree), max_features (the maximum number of features considered for splitting a node), min sample Leaf (min number of data points allowed in a leaf node), min sample Split (the minimum number of data points placed in a node before the node is split), is performed. The evaluation metric used is the normalized discounted cumulative gain (NDCG) at k = 10. Figure 5.5 shows the changes in NDCG scores while tweaking these hyperparameters.

The final meta-learner hyperparameters for the stacked recommender are as follows: n_estimators = 300, max_depth = 40,

We developed an interactive attention network recommender that learns the interaction between the users and items and the mutual influence between the items and user interaction. The predicted utility matrix is generated based on the learned embeddings. Figure 5.6 illustrates the hyperparameters analysis.

For each hyperparameter, a range of values is explored to find the best set of hyperparameters. They are: the dimensions of the embedding a ∈ [30, 100], the number of dense layers after 0 ∈ [2, 20], the number of neurons per dense layer r ∈ [30, 150], the activation function used in the dense layers <Y ∈ { selu, elu, relu} and the optimizer Jc ∈ { sgd, adam, adagrad}. Results show that the best performance is achieved for the following settings: a = 50, 0 = 03, r = 100, <Y = elu, Jc = Adam.

5.5.2. Performance comparison with the baselines

To provide a better insight into the interactive attention network recommender results, we visualize a sample of the interactive co-attention weights for 10 items and four users. Figure 5.7 shows that this recommender constructs a user-item co-attention map for the final rating prediction. It identifies the most relevant weights that represent users’ mutual influence on the item. This is due to its ability to capture complementary information from each user contribution and combine them to predict the utility matrix. Indeed, the co-attention module emphasizes the dependencies between the different entities.

Table 5.4 presents the recommendation performance of all methods on MovieLens. The proposed framework outperforms all baselines according to the mean average precision. The most likely reason is that the IPRec can model the higher order feature interactions. Figure 5.8 represents the MAP@k performance versus k-top items. IPRec generates a personalized recommendation, as the MAP measure represents the fraction of relevant items in the top k recommendations averaged over all of the users.

To put it differently, IPRec can recall the relevant items for the user better than the other models. It acquires the user-item interaction and creates a user personalized task recommendation justifying its high recorded scores. In addition, both the interactive attention recommender and EnsVAE recommender produce competitive empirical results. For example, the IPRec, EnsVAE and Inter-active attention recommenders achieve MAP@10=0.89, MAP@10=0.87 and MAP@10=0.86, respectively, which are higher than the baselines. The MultiVAE models the collaborative information in the form of a multinomial distribution to sample the likelihood of presenting certain items to certain users. However, it scores weak MAP values because it does not model a rich semantic representation of data. The neural collaborative filtering (NCF) method gives a good score with k=10, MAP@10=0.74 due to the modeling of the interaction between user and item features.

As shown in Figure 5.9, we observe that the interactive attention recommender (Co-attention-CF) records a high NDCG score on the MovieLens dataset. In other words, the rank of an item in its top-k recommendation list is close to the one observed in the ground-truth list. Applying the attention mechanism enhances the modeling of user-item interaction. Indeed, the more positive the users toward an item, the more likely it is recommended by users with similar preferences. Note that the EnsVAE recommender and the stacked recommender still exhibit good NDCG scores. The stacked recommender is effective in obtaining the user’s overall interest built by the stack’s learners. It captures the side-information to create a profile model for each user, while optimizing the stack’s learners’ objective function.

The IPRec scores are lower compared to the above models. Due to the average aggregation of the two recommenders, it reduces the rating probabilities of some items. Given that NDCG is a ranking metric, this reduction may cause items to be misplaced on the query, reducing the NDCG score. Nevertheless, the high MAP values indicate that the top-k items are still relevant to the user, although ranked differently. In general, many opportunities exist for the hybridization of recommender systems. The ultimate goal is to use all of the knowledge available in different data sources and the algorithmic power of these various recommender systems to make robust inferences. The content-based recommender tends to integrate the various data sources more tightly (side information). In addition, the collaborative recommender is more effective when a lot of data is available. Here, we combine the scores of the two recommender systems into a single unified score by computing the weighted aggregates (average aggregation) of the scores from individual ensemble components. Using different combinations of features for building the entire recommendation system is challenging because we consider two aspects: tailoring the recommender system and improving performance. Therefore, it is a complex task to find a tradeoff between these two aspects. The main advantage of the proposed approach is that we simulate the different configurations in some recommender systems where user behavior is tightly related to the content information available in the items. Leveraging a combination of recommendation techniques boosts the recommendation system performance. Hence, the IPRec framework outperforms other baselines with a significant margin in terms of the mean average precision.