As we have seen, decision trees perform well in multi-classification problems. There are other approaches we could follow too. One of them is logistic regression, which gives six possible outcomes to a problem. Nevertheless, this approach has some limitations. For example, we are assuming that there is no order to the categories in the target variable. It means that different categories or classes in the target variable are nominal. In the case of ratings, this assumption is not necessarily true, because ratings assign a ranking. Moreover, differences between credit ratings are not the same, which means that the difference between AAA and AA+ ratings is not necessarily equal to the difference between BBB and BBB- ratings.
Thus, in this part of the book, we are going to implement an ordered logistic regression, which assumes an order in our target variable and a non-constant difference between ratings.
The model can be deployed using the polr function from the MASS package. This function only needs the formula of the model, the dataset and, in our case, the Hess=TRUE option. This option will allow us to calculate and visualize the standard errors of the variables in our model:
library(MASS)
ordered_logistic <- polr(RatingMayT1 ~ ., data = train_macro_trn[,c(variables)], Hess=TRUE)
A summary of the model is then printed:
summary(ordered_logistic)
## Call:
## polr(formula = RatingMayT1 ~ ., data = train_macro_trn[, c(variables)],
## Hess = TRUE)
##
## Coefficients:
## Value Std. Error t value
## CARA -0.3624 0.2520 -1.4381
## DCPI 0.1432 0.1807 0.7924
## DGDP -0.2225 0.2129 -1.0452
## ILMA 1.5587 0.2592 6.0126
## PSBR 0.6929 0.2209 3.1371
## PUDP -2.8039 0.3886 -7.2145
## TDRA 0.3070 0.2464 1.2461
## YPCA 2.6988 0.7100 3.8011
## MEANWGI 2.2565 0.4707 4.7937
## YearsFromLastDefault 0.8091 0.2191 3.6919
##
## Intercepts:
## Value Std. Error t value
## 1|2 -10.0770 1.1157 -9.0321
## 2|3 -5.6306 0.6134 -9.1789
## 3|4 -2.4390 0.4011 -6.0810
## 4|5 0.4135 0.3615 1.1439
## 5|6 4.8940 0.5963 8.2070
##
## Residual Deviance: 236.9271
## AIC: 266.9271
The preceding table provided us with the regression coefficient table. Moreover, it shows the estimates for the different intercepts, which are sometimes called cutpoints. The intercepts indicate where the predicted result should be cut, to make the different credit ratings observed in the data.
Moreover, the model provides us with the residual deviance and the AIC metrics, which are useful metrics to compare different models.
In the preceding results, we cannot see any p_values indicating whether a variable is significant, as it is not usually displayed in any regression. So, we need to calculate them.
The p_values can be approximately calculated by comparing the t value against the standard normal distribution. First, we will store our coefficients using the following code:
coefs <- coef(summary(ordered_logistic))
print(coefs)
## Value Std. Error t value
## CARA -0.3623788 0.2519888 -1.4380749
## DCPI 0.1432174 0.1807448 0.7923737
## DGDP -0.2225049 0.2128768 -1.0452282
## ILMA 1.5586713 0.2592360 6.0125575
## PSBR 0.6928689 0.2208629 3.1371002
## PUDP -2.8038553 0.3886409 -7.2145133
## TDRA 0.3069968 0.2463570 1.2461463
## YPCA 2.6988066 0.7100112 3.8010760
## MEANWGI 2.2564849 0.4707199 4.7936888
## YearsFromLastDefault 0.8090669 0.2191455 3.6919175
## 1|2 -10.0770197 1.1156894 -9.0321014
## 2|3 -5.6306456 0.6134365 -9.1788566
## 3|4 -2.4389936 0.4010815 -6.0810418
## 4|5 0.4134912 0.3614860 1.1438653
## 5|6 4.8940176 0.5963226 8.2069960
If we observe the sign of the coefficients, there are some negative values. Apparently, some variables display a non-intuitive or unexpected sign, but we don't need to be concerned about it in this example.
Coefficients of the model can be somewhat difficult to interpret, because they are scaled in terms of logs. Thus, it is common to convert previous raw coefficients into odds ratios.
Odds ratios will be obtained as follows:
exp(coef(ordered_logistic))
## CARA DCPI DGDP
## 0.69601870 1.15398065 0.80051110
## ILMA PSBR PUDP
## 4.75250240 1.99944358 0.06057607
## TDRA YPCA MEANWGI
## 1.35933662 14.86198455 9.54946258
## YearsFromLastDefault
## 2.24581149
Finally, the p_values of the different variables are calculated and merged with our obtained coefficients:
p_values <- pnorm(abs(coefs[, "t value"]), lower.tail = FALSE) * 2
coefs <- cbind(coefs, "p value" = p_values)
print(coefs)
## Value Std. Error t value p value
## CARA -0.3623788 0.2519888 -1.4380749 1.504128e-01
## DCPI 0.1432174 0.1807448 0.7923737 4.281428e-01
## DGDP -0.2225049 0.2128768 -1.0452282 2.959175e-01
## ILMA 1.5586713 0.2592360 6.0125575 1.826190e-09
## PSBR 0.6928689 0.2208629 3.1371002 1.706278e-03
## PUDP -2.8038553 0.3886409 -7.2145133 5.412723e-13
## TDRA 0.3069968 0.2463570 1.2461463 2.127107e-01
## YPCA 2.6988066 0.7100112 3.8010760 1.440691e-04
## MEANWGI 2.2564849 0.4707199 4.7936888 1.637422e-06
## YearsFromLastDefault 0.8090669 0.2191455 3.6919175 2.225697e-04
## 1|2 -10.0770197 1.1156894 -9.0321014 1.684062e-19
## 2|3 -5.6306456 0.6134365 -9.1788566 4.356928e-20
## 3|4 -2.4389936 0.4010815 -6.0810418 1.194042e-09
## 4|5 0.4134912 0.3614860 1.1438653 2.526795e-01
## 5|6 4.8940176 0.5963226 8.2069960 2.267912e-16
Once we develop our model, we are going to predict the outcome of our model:
Ord_log_pr_train <- cbind(train_macro_trn[,c("CountryName","Year","RatingMayT1")], predict(ordered_logistic, train_macro_trn, type = "probs"))
colnames(Ord_log_pr_train)<-c("Country","year","Observed","X1","X2","X3","X4","X5","X6")
head(Ord_log_pr_train,1)
##Country year Observed X1 X2 X3 X4
1 Austria 2010 6 5.468638e-06 4.608843e-04 0.010757249 0.15316033
## X5 X6
## 1 0.7811701 0.05444599
The model gives a different probability for each rating. The predicted rating is the one with the highest predicted probability. For example, for Austria in 2010, the model assigns the highest probability to the 5 (X5) rating, so the predicted rating is a 5.
With the following code, we assign the predicted rating to the highest probability:
for (j in 1:nrow(Ord_log_pr_train))
{
Ord_log_pr_train$maximaPD[j]<-max(Ord_log_pr_train$X1[j],Ord_log_pr_train$X2[j],Ord_log_pr_train$X3[j],Ord_log_pr_train$X4[j],Ord_log_pr_train$X5[j],Ord_log_pr_train$X6[j])
}
Ord_log_pr_train$Predicted<-ifelse(Ord_log_pr_train$X1==Ord_log_pr_train$maximaPD,1,ifelse(Ord_log_pr_train$X2==Ord_log_pr_train$maximaPD,2,ifelse(Ord_log_pr_train$X3==Ord_log_pr_train$maximaPD,3,ifelse(Ord_log_pr_train$X4==Ord_log_pr_train$maximaPD,4,ifelse(Ord_log_pr_train$X5==Ord_log_pr_train$maximaPD,5,6)))))
Let's see the accuracy of the model in the train sample:
model_assessment(Ord_log_pr_train,"Ordered_logistic")
## notche perc_Ordered_logistic cumulative
## 1 0 0.69047619 0.6904762
## 2 1 0.29761905 0.9880952
## 3 2 0.01190476 1.0000000
As we can see, the model is able to correctly predict 69.05% of credit ratings using the training sample. These results were better when using a decision tree.
Let's see the performance of the model in the test sample. The following code gives us the predicted probabilities of each country for each of the rating levels. The predicted rating is given by the category with the highest probability:
Ord_log_pr_test <- cbind(test_macro_trn[,c("CountryName","Year","RatingMayT1")], predict(ordered_logistic, test_macro_trn, type = "probs"))
colnames(Ord_log_pr_test)<-c("Country","year","Observed","X1","X2","X3","X4","X5","X6")
The following code finds the rating category with the highest probability and assigns this rating as the predicted rating:
for (j in 1:nrow(Ord_log_pr_test))
{
Ord_log_pr_test$maximaPD[j]<-max(Ord_log_pr_test$X1[j],Ord_log_pr_test$X2[j],Ord_log_pr_test$X3[j],Ord_log_pr_test$X4[j],Ord_log_pr_test$X5[j],Ord_log_pr_test$X6[j])
}
Ord_log_pr_test$Predicted<-ifelse(Ord_log_pr_test$X1==Ord_log_pr_test$maximaPD,1,ifelse(Ord_log_pr_test$X2==Ord_log_pr_test$maximaPD,2,ifelse(Ord_log_pr_test$X3==Ord_log_pr_test$maximaPD,3,ifelse(Ord_log_pr_test$X4==Ord_log_pr_test$maximaPD,4,ifelse(Ord_log_pr_test$X5==Ord_log_pr_test$maximaPD,5,6)))))
The accuracy of the model in the test sample is as follows:
model_assessment(Ord_log_pr_test,"Ordered_logistic")
## notche perc_Ordered_logistic cumulative
## 1 0 0.57142857 0.5714286
## 2 1 0.39285714 0.9642857
## 3 2 0.01785714 0.9821429
## 4 3 0.01785714 1.0000000
The results are also slightly worse than in the decision tree model.
You can now save the workspace before starting the next section:
save.image("Backup5.RData")
In the next section, we will use macroeconomic data to predict country ratings. All the variables we used are quantitative variables. In the following section, we will use country reports for the same purpose.