A note on the reliability of WAIC and LOO computations

When computing WAIC or LOO, you may get a warning message, indicating that the result of either computation could be unreliable. This warning is raised based on a cut-off value that was determined empirically (see the Keep reading section for a  reference). While it is not necessarily problematic, it could be an indication of a problem with the computation of these measures. WAIC and LOO are relative newcomers and we probably still need to develop better ways to access their reliability. Anyway, if this happens to you, first, make sure that you have enough samples and that you have a well-mixed, reliable sample (see Chapter 8Inference Engines). If you still get those messages, the authors of the LOO method recommend using a more robust model, such as using a Student's t-distribution instead of a Gaussian one. If none of these recommendations work, then you may need to think about using another method, such as directly performing K-fold cross-validation.

On a more general note, WAIC and LOO can only help you choose among a given set of models, but they cannot help to decide if a model is really a good solution to our particular problem. For this reason, WAIC and LOO should be complemented with posterior predictive checks, along with any other information and tests that help us to put models and data in the light of the domain knowledge relevant to the particular problem we are trying to solve.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.137.216.175