We built a model, but we do not know if it can be trusted. To estimate its performance, we can apply a cross-validation technique that was explained in Chapter 1, Applied Machine Learning Quick Start.

Weka offers an Evaluation class for implementing cross-validation. We pass the model, data, number of folds, and an initial random seed, as follows:

Classifier cl = new J48(); 
Evaluation eval_roc = new Evaluation(data); 
eval_roc.crossValidateModel(cl, data, 10, new Random(1), new Object[] {}); 
System.out.println(eval_roc.toSummaryString()); 

The evaluation results are stored in the Evaluation object.

A mix of the most common metrics can be invoked by calling the toString() method. Note that the output does not differentiate between regression and classification, so make sure to pay attention to the metrics that make sense, as follows:

    Correctly Classified Instances          93               92.0792 %
    Incorrectly Classified Instances         8                7.9208 %
    Kappa statistic                          0.8955
    Mean absolute error                      0.0225
    Root mean squared error                  0.14  
    Relative absolute error                 10.2478 %
    Root relative squared error             42.4398 %
    Coverage of cases (0.95 level)          96.0396 %
    Mean rel. region size (0.95 level)      15.4173 %
    Total Number of Instances              101  
  

In the classification, we are interested in the number of correctly/incorrectly classified instances.