Data-snooping and backtest-overfitting

The most prominent challenge to backtest validity, including to published results, relates to the discovery of spurious patterns due to multiple testing during the strategy-selection process. Selecting a strategy after testing different candidates on the same data will likely bias the choice because a positive outcome is more likely to be due to the stochastic nature of the performance measure itself. In other words, the strategy is overly tailored, or overfit, to the data at hand and produces deceptively positive results.

Hence, backtest performance is not informative unless the number of trials is reported to allow for an assessment of the risk of selection bias. This is rarely the case in practical or academic research, inviting doubts about the validity of many published claims.

The risk of overfitting a backtest to a particular dataset does not only arise from directly running numerous tests but includes strategies designed based on prior knowledge of what works and doesn't, that is, knowledge of different backtests run by others on the same data. As a result, backtest-overfitting is hard to avoid in practice.

Solutions include selecting tests to undertake based on investment or economic theory rather than broad data-mining efforts. It also implies testing in a variety of contexts and scenarios, including possibly on synthetic data.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.86.235.207