Problem

You want to explain sentiment analysis prediction using SHAP.

Solution

The solution takes into account the most common dataset available, which is the IMDB sentiment classification dataset from the SHAP library. It can be accessed using the SHAP dataset. We will be using the SHAP library to explain the predictions.

How It Works

Let’s take a look at the following example (see Figure 5-1 and Figure 5-2):

!pip install shap

import warnings

warnings.filterwarnings("ignore")

import sklearn

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.model_selection import train_test_split

import numpy as np

import shap

import pandas as pd

from keras.datasets import imdb

corpus,y = shap.datasets.imdb()

corpus_train, corpus_test, y_train, y_test = train_test_split(corpus, y, test_size=0.2, random_state=7)

vectorizer = TfidfVectorizer(min_df=10)

X_train = vectorizer.fit_transform(corpus_train).toarray() # sparse also works but Explanation slicing is not yet supported

X_test = vectorizer.transform(corpus_test).toarray()

corpus_train[20]

Well how was I suppose to know this was................................

y

array([False, False, False, ..., True, True, True])

model = sklearn.linear_model.LogisticRegression(penalty="l2", C=0.1)

model.fit(X_train, y_train)

explainer = shap.Explainer(model, X_train, feature_names=vectorizer.get_feature_names())

shap_values = explainer(X_test)

shap.summary_plot(shap_values, X_test)

A summary plot illustrates the S H A P value of the features. The feature values of bad and great extend from approximately negative 2.0 to 0.1, and negative 0.1 to 1.0 S H A P values, respectively.

A summary plot illustrates the S H A P value of the features. The feature value of the sum of 16407 other features extends between approximately negative 2 and 2 S H A P values.

Problem

You want to explain sentiment analysis prediction using ELI5.

Solution

The solution takes into account the most common dataset available, which is the IMDB sentiment classification. We will be using the ELI5 library to explain the predictions.

How It Works

Problem

You want to explain an individual review in the sentiment analysis prediction using ELI5.

Solution

The solution is takes into account the most common dataset available, which is the IMDB sentiment classification dataset. We will be using the ELI5 library to explain the predictions.

How It Works

this movie was so poorly written and directed i fell asleep 30 minutes through the movie……………….

The green patches show positive features for the target class positive, and the red parts are negative features that correspond to the negative class. The feature value and the weight value indicate the relative importance of words as features in classifying sentiments. It is observed that many stop words or unwanted words are present in the tokenization process; hence, they are appearing as features in the feature importance. The way to clean it up is to use preprocessing steps such as applying stemming, removing stop words, performing lemmatization, removing numbers, etc. Once the text cleanup is completed, then the previous recipes can be used again to create a better model to predict the sentiments.