To evaluate the predictive accuracy of our first classification tree, we will use our test set to generate predicted class probabilities, as follows:

y_score = classifier.predict_proba(X=X_test)[:, 1] # only keep probabilities for pos. class

The .predict_proba() method produces one probability for each class. In the binary class, these probabilities are complementary and sum to 1, so we only need the value for the positive class. To evaluate the generalization error, we will use the area under the curve based on the receiver-operating characteristic that we introduced in Chapter 6, The Machine Learning Process. The result indicates a significant improvement above and beyond the baseline value of 0.5 for a random prediction:

roc_auc_score(y_score=y_score, y_true=y_test)
0.5941