As discussed in detail in the previous chapter, bootstrap averaging (bagging) improves the performance of an otherwise noisy classifier.

Stochastic gradient boosting uses sampling without replacement at each iteration to grow the next tree on a subset of the training samples. The benefit is both lower computational effort and often better accuracy, but subsampling should be combined with shrinkage.

As you can see, the number of hyperparameters keeps increasing, driving up the number of potential combinations, which in turn increases the risk of false positives when choosing the best model from a large number of parameter trials on a limited amount of training data. The best approach is to proceed sequentially and select parameter values individually or using combinations of subsets of low cardinality.