Performance metrics

Performance metrics are used to quantify the performance of the trained models. Based on this, let's define the following four metrics:

Metric Formula
Accuracy
Recall
Precision
F1 score

 

Accuracy is the proportion of correction classifications among all predictions. While calculating accuracy, we do not differentiate between TP and TN. Evaluating a model through accuracy is straightforward, but in certain situations, it will not work.

Let's look at the situations where we need more than accuracy to quantify the performance of a model. One of these situations is when we use a model to predict a rare event, such as in the following examples:

  • A model to predict the fraudulent transactions in a banks transactional database
  • A model to predict the likelihood of mechanical failure of an engine part of an aircraft

In both of these examples, we are trying to predict a rare event. Two additional measures become more important than accuracy in these situations—recall and precision. Let's look at them one by one:

  • Recall: This calculates the hit rate. In the first of the preceding examples, it is the proportion of fraudulent documents successfully flagged by the model out of all the fraudulent documents. If, in our testing dataset, we had 1 million transactions, out of which 100 were known to be fraudulent, the model was able to identify 78 of them. In this case, the recall value would be 78/100.
  • Precision: The precision measures how many of the transactions flagged by the model were actually bad. Instead of focusing on the bad transactions that the model failed to flag, we want to determine how precise the bad bins flagged by the model really is. 

Note that the F1 score brings both the recall and precision together. If a model has perfect scores for both precision and recall, then its F1 score will be perfect. A high F1 score means that we have trained a high-quality model that has high recall and precision.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.141.244.153