Information criteria is a collection of different and somehow related tools that are used to compare models in terms of how well they fit the data while taking into account their complexity through a penalization term. In others words, information criteria formalizes the intuition we developed at the beginning of this chapter. We need a proper way to balance how well a model explains the data on the one hand, and how complex the model is on the other hand.

The exact way these quantities are derived has to do with a field known as information theory, something that is beyond the scope of this book, so we are going to limit ourselves to understand them from a practical point of view.