Mean-variance optimization is very sensitive to the estimates of expected returns and the covariance of these returns. The covariance matrix inversion also becomes more challenging and less accurate when returns are highly correlated, as is often the case in practice. The result has been called the Markowitz curse: when diversification is more important because investments are correlated, conventional portfolio optimizers will likely produce an unstable solution. The benefits of diversification can be more than offset by mistaken estimates. As discussed, even naive, equally-weighted portfolios can beat mean-variance and risk-based optimization out of sample.
More robust approaches have incorporated additional constraints (Clarke et al., 2002), Bayesian priors (Black and Litterman, 1992), or used shrinkage estimators to make the precision matrix more numerically stable (Ledoit and Wolf [2003], available in scikit-learn ( http://scikit-learn.org/stable/modules/generated/sklearn.covariance.LedoitWolf.html). Hierarchical risk parity (HRP), in contrast, leverages unsupervised machine learning to achieve superior out-of-sample portfolio allocations.
A recent innovation in portfolio optimization leverages graph theory and hierarchical clustering to construct a portfolio in three steps (Lopez de Prado, 2015):
- Define a distance metric so that correlated assets are close to each other, and apply single-linkage clustering to identify hierarchical relationships
- Use the hierarchical correlation structure to quasi-diagonalize the covariance matrix.
- Apply top-down inverse-variance weighting using a recursive bisectional search to treat clustered assets as complements rather than substitutes in portfolio construction and to reduce the number of degrees of freedom.
A related method to construct hierarchical clustering portfolios (HCP) was presented by Raffinot (2016). Conceptually, complex systems such as financial markets tend to have a structure and are often organized in a hierarchical way, while the interaction among elements in the hierarchy shapes the dynamics of the system. Correlation matrices also lack the notion of hierarchy, which allows weights to vary freely and in potentially unintended ways.
Both HRP and HCP have been tested by JPM on various equity universes. The HRP, in particular, produced equal or superior risk-adjusted returns and Sharpe ratios compared to naive diversification, the maximum-diversified portfolios, or GMV portfolios.
We will present the Python implementation in Chapter 12, Unsupervised Learning.