Problems with Mean Variance Optimization

Whenever there is a major bear market, investors begin to question everything. In particular, following the Financial Crisis of 2007–2008, many families decided that mean variance optimization (MVO), the main tool for designing investment portfolios for decades, simply didn't work.

Certainly MVO was overhyped by advisors, who should have been using MVO techniques simply as “what if” modeling exercises, not to design final, real-world portfolios. But now that the crisis is several years behind us and the markets have largely recovered, let's take a calmer look at the problems presented by MVO.

Computational Power

This process is known as mean variance optimization, or MVO, and so far, so good. But there are serious problems here. The first problem is that there are many, many possible asset combinations to be looked at. If, for example, we wish to consider including 10 asset classes in our portfolios, our optimizer will have to search through almost 33 million possible portfolios even before it begins to think about changing the percentages of each asset.⁶ Even as recently as the early 1970s it required a computer the size of a large room, two days of computational time, and tens of thousands of dollars to run one mean variance optimization. In other words, the delay in adopting Markowitz's ideas was not entirely due to intransigence on the part of real-world investors.

Oddly, however, the unavailability of cheap, massive computational power was in some ways a blessing for investors. Without access to simple optimization tools, investors relied on reviewing portfolio combinations that had been developed by large financial firms and academic institutions, then applied their own judgment to the application of those portfolios to their own circumstances. In other words, they had little choice but to be thoughtful about the underlying theory of mean variance optimization.

Today, however, investors face a very different and, probably, more intransigent problem. MVO calculations are now simplicity itself. Optimization programs are inexpensive and investors have access to massive and cheap computational power. These days any laptop computer carried around by a college freshman can run an optimization in a few seconds. But as a result, investors have come to rely on the machine, rather than to consider the underlying theory in a thoughtful way. Let's consider ways in which the machine might produce results that are dangerous to our wealth.

Garbage In, Garbage Out

I mentioned, earlier, the three inputs needed to conduct an MVO exercise: future risks, returns, and correlations for each asset to be included in the optimization. The trouble is, we not only don't know what these values are, we don't really have any idea what they are. This is partly because we can't foresee the future. If we could, we wouldn't be wasting our time designing our portfolios—we would simply pick the stock that is going to be the next Google and load up on it. But we can't.

The best work on asset class risks, returns, and correlations, for example, has been done by Ibbotson Associates, whose annual Ibbotson Stocks, Bonds, Bills, and Inflation Yearbook gives values for asset class data on a quarterly basis from 1925 to the present. Yet, when Ibbotson Associates attempted to issue projections for future asset class returns, their estimates were so bizarrely off the mark that the firm soon stopped issuing forward projections altogether.

But our inability to predict the future isn't the only problem—we also can't know the past. That may seem like an extraordinary remark, especially considering that I have just cited the excellent Ibbotson data. But the problem is one of definition—what past are we talking about? If, for example, we are talking about the returns on stocks since 1925, then we will get one answer. If we are talking about returns in the post-World War II period, we will get another answer. If we are talking about returns over the past 20 years, we will get still a third answer. Which of these (or many other) pasts is most likely to resemble the future? We can't know, of course.

But if we can't know the future and don't know the past—that is, if we are putting garbage data into the optimizer and are therefore certain to get garbage advice out of it—what good is mean variance optimization?⁷ The answer is that, if used thoughtfully and with a clear sense of its limitations, MVO can lead investors toward portfolios that are more efficient and more appropriate for their needs than any other approach known to the financial world. But for MVO to play this role, our financial advisors must work exactly at the cutting edge of knowledge in this area.

The Challenge of Developing Thoughtful Data Inputs

If we are working with competent financial advisors, they will not use purely historical returns in their MVO calculations. Instead, they will adjust past returns to meet the conditions existing in the capital markets today and over our effective time horizons as investors. Future returns are extremely sensitive to starting values, and therefore, in particular, advisors must give serious thought to the valuations of assets at the time the MVO exercise is being undertaken. Advisors, frankly, tend to be very bad at this exercise, for two reasons. The first is that it is an enormously difficult task—if we really knew which assets were undervalued and (especially!) when those undervaluations were going to be reversed, we would be far wealthier than we are. The second reason is that most advisors are sales-oriented, not advice-oriented, and hence they are far better at telling investors what they want to hear than they are at giving their honest opinions about important matters.

Consider MVO analyses performed in the late 1970s—clearly, they should have taken into account the very low valuations of most equity securities at that time. And consider MVO analyses performed in the late 1990s—clearly, they should have taken into account the very high valuations of equity securities at that time. Instead, most portfolios designed during those periods of time used long-term historical returns (or something close to them) and performed miserably. Investors in the 1970s found themselves seriously underexposed to stocks just as the greatest bull market in U.S. history was starting. Investors in the 1990s found themselves seriously overexposed to stocks just as one of the worst bear markets in history was starting.

In addition, in thinking about the investment environment going forward, family advisors must consider not some generic time period that is convenient to the advisory firm, but the family's own actual investment time horizons. Thus, for example, the advisor might believe that, over the next decade, U.S. large-cap stock returns will reflect essentially their long-term averages. But if the family's real investment time horizon is 5 years, not 10, the advisor's opinion is largely irrelevant. If a family hopes to experience any investment success, they will want to work with advisors who can tailor their MVO inputs to the family's actual needs, rather than employing a central-office approach where one size fits all.

Multivariate Modeling

Virtually all modeling now being performed by advisors to create client portfolios looks nothing like real-world capital markets. The key problem is that single-period optimizers assume that the expected risks, returns, and correlations that are programmed into the optimizer will not change, even over long periods of time. In fact, risk levels, returns, and correlations vary, sometimes quite dramatically, over time. Markets can be calm or volatile, correlations can change overnight as markets move from risk-off to risk-on, and returns vary hourly. Skewness and kurtosis vary over time.

What is needed, but what is generally not available, is a multivariate input model that represents multivariate processes with time-varying joint distributional properties, that applies to a wide variety of dependence structures, and that allows the representation of many time series. While a few firms are working on designing such systems, most advisors continue to use overly simplistic modeling that generates portfolios that will not be robust under real-world scenarios.

Taking Taxes into Account

Family investors can't spend gross returns. Like it or not, Uncle Sam is our investment partner, as are the governor and, very often, the mayor. Because that is the case, why do advisory firms persist in conducting MVO analyses using pretax returns? The answer is that that's the way it has always been done. The first investors to use (and pay for!) MVO analyses were pension plans and large endowed institutions. When advisory firms began to seek private clients they didn't bother to adjust their analyses to reflect the singular nature of family clients—the most singular feature being that private clients pay taxes and institutional clients don't.⁸

Even if the tax consequences of all investment assets were the same, advisors who use pretax MVO analyses with private investors would be seriously misleading those clients. Assuming nontaxable and taxable investors with identical risk tolerances, the returns achievable by the former will be substantially higher than those achievable by the latter. In other words, private investors will either have to accept lower returns than institutional investors owning the same portfolios, or they will have to own different and more risky portfolios to achieve the same returns.

But the tax consequences of owning different investment assets are in fact not the same for private investors. Consider this question: Would you rather own a well-performing hedge fund that returns 15 percent per annum or a poorly performing private equity partnership that returns 15 percent per annum? If you are an institutional investor, the answer is that you would prefer to own the hedge fund—it is less risky and has better liquidity than the private equity partnership, yet generates the same return.

But if you are a family investor, the answer is likely to be quite different: You would probably prefer to own the private equity partnership. Why? Because the hedge fund will generate its 15 percent return primarily through short-term capital gains, while the private equity fund will generate its 15 percent return primarily through long-term capital gains. Most wealthy investors will pay high taxes on the hedge fund gains, but much lower taxes on the private equity gains. Hence, the question can be rephrased as follows: Would you rather get 12 percent per year or 9 percent per year?⁹

We can conduct a similar analysis with essentially every asset we might wish to use in our portfolios. We would notice, for example, that “value” stocks will generate much of their return through dividend payments, while growth stocks will generate much of their return through long-term appreciation. We will note that real estate and oil and gas investments offer tax advantages to families. And so on.

The point of all this is simple—our advisors need to conduct their portfolio design studies using after-tax returns, not pretax returns. The result of such an approach will be that different assets will be used in different proportions than would be the case if the studies were performed using pretax data—and the portfolios thus designed will be efficient and appropriate for taxable investors, as opposed to nontaxable institutions.

Monte Carlo Simulations

In the process of designing our portfolios, financial advisors generate a substantial collection of modern portfolio theory statistics about the characteristics of various possible portfolios. These statistics are useful to advisors, but they are hardly useful to investors. What we need to see is the range of possible outcomes for various portfolios in dollars—and, preferably, in after-tax dollars. Such outcomes are developed using Monte Carlo¹⁰ or similar simulations. These simulations begin with a portfolio starting value, then run many iterations based on the risk, return, and correlation characteristics of the portfolio.

Monte Carlo simulations are not perfectly accurate for several reasons. A principal reason is that what the computer is actually doing is taking a portfolio, holding it for a one-year period, and seeing what the outcome looks like. It then runs the initial portfolio for another one-year period and looks at that outcome, and so on. In the real world, our portfolios change every year and it is the range of outcomes associated with those changed portfolios that matter. Note that in reviewing Monte Carlo simulations, we will want to observe the range of outcomes for short, intermediate, and longer time periods. Because negative outcomes rise much more slowly than positive outcomes over time, if we look only at longer term periods we might be tempted to select a portfolio that will, in all likelihood, prove too risky for us over shorter term periods. It is therefore important that we look at each portfolio over a range of time periods.

The Problem of Fat Tails

Many natural events occur in a similar pattern; namely, a range of outcomes clustered around a long-term average outcome: the height of trees, the weight of smallmouth bass, visits of birds to a sanctuary. If you live in a region that, on average, receives 40 inches of rain in a year, the actual pattern may show that, two-thirds of the time, annual rainfall lies between 30 inches and 50 inches. A statistician would tell you that the mean rainfall in your area is 40 inches with a standard deviation (S.D.) of 10. Rainfall results therefore distribute themselves along the familiar bell curve.

Knowing this, we can calculate that, 95 percent of the time, rainfall in the area will be between 20 inches and 60 inches—two S.D.s. It might be climatically possible for rainfall in the area to be as low as 5 inches or as high as 75 inches, but those outcomes would be statistically almost impossible. We would not expect to observe such outcomes even over many centuries, and hence it doesn't pay us to prepare for or anticipate them in any way. (The so-called 100-year flood is really just the most remote of the possible outcomes, the one that falls in the 95th percentile, or two S.D.s. above the average.)

The returns produced by the capital markets also distribute themselves in bell curve fashion. But there is a crucial difference between capital markets and natural phenomena. The example of rainfall is an example of natural behavior largely uncompromised by human interaction. But securities prices don't somehow magically change from time to time—they change only in response to the action of human beings in buying and selling them. And human behavior, as we all know, is at best complex and at worst truly bizarre. We can behave with calm resolution one moment and panic along with the rest of the crowd the next moment.

The result is that the bell curve distribution of capital markets performance is not shaped exactly like a bell curve. Instead, it's shifted slightly to the left and is characterized by so-called fat tails: Extreme outcomes, good and bad, are far more common than statistics would suggest.

Thus, if the mean return on U.S. large-capitalization stocks is 10 percent and the S.D. is 20 (both roughly right over very long periods), then we would almost never expect to see annual losses of more than about −30 percent; that is, more than two standard deviations below the mean return. Indeed, if we experienced such a loss even once in a lifetime of investing we could consider ourselves to be relatively unlucky.

In fact, however, extremely poor and extremely good returns are far, far more likely to occur than a pure statistical analysis would lead us to believe. When Eugene Fama did his doctoral thesis on price movements of the Dow Jones Industrial Average, he discovered that for each stock in the index there were many more days of extreme price movements than would occur in a normal distribution. Random distribution couldn't explain these outliers:

If the population of price changes is strictly normal, on the average for any stock … an observation more than five standard deviations from the mean should be observed about once every 7,000 years. In fact, such observations seem to occur about once every three to four years.”¹¹ (Emphasis supplied.)

Or consider the breathtaking market collapse that occurred over two days in October of 1987, an event that would have been statistically unlikely to occur “had the life of the universe been repeated one billion times.”¹²

Investors thus face a curious dilemma. On the one hand, we have carefully constructed our portfolios using the best modern portfolio theory techniques, employing the necessary¹³ assumption that market events will occur in a normally distributed fashion. On the other hand, we know from examining the actual data that some market events—namely, very bad portfolio performance¹⁴—will occur far more often than a normal distribution would lead us to expect. We have, in effect, succeeded in controlling a mathematical version of risk—normally distributed price volatility—but we have not adequately addressed the real risk that investors face: the risk that we will depart from our sound strategies in the face of unexpectedly poor results, thus incurring permanent losses of capital.

Price volatility matters, to be sure, and not just in the world of financial theory. Given our druthers, most of us would prefer to obtain a handsome long-term rate of return with little or no price volatility along the way. But we have to live in the real world, and in that world we must choose between, on the one hand, low price volatility and low returns and, on the other hand, high returns and higher price volatility. Given that reality, most investors can get used to price volatility so long as it stays within an expected, reasonable range. It is not price volatility itself that causes us to flee from sensible long-term strategies—it is unanticipated, apparently irrational price volatility.