CHAPTER EIGHT
Sorting Luck from Skill
Why Investors Excel at Buying High and Selling Low
THE BOSS BLEW HIS STACK. After the storied New York Yankees won only four of their first twelve games in 2005, George Steinbrenner, the baseball team’s owner, could not contain his frustration. “I am bitterly disappointed by the lack of performance of our team,” he seethed. “It is unbelievable to me that the highest-paid team in baseball would start the season in such a deep funk. They have the talent to win and they are not winning.” Even with 93 percent of the season to go, Joe Torre, the team’s manager, could only agree, “He’s not saying anything we certainly don’t know ourselves. When he does spend the money, he expects more than he’s obviously gotten so far.”1
The Yankees did come through, tying for first place in their division during the regular season, but not because of the Boss’s tonguelashing. But how much was because of skill, and how much was due to luck? Hard to say. We have difficulty sorting skill and luck in lots of fields, including business and investing. As a result, we make a host of predictable and natural mistakes, such as failing to appreciate the team’s or the individual’s inevitable reversion to the mean. This chapter will offer you a fresh perspective for interpreting your own team’s winning streaks and slumps—or, for that matter, the performance of employees, business units, stockbrokers, and other professionals as individuals and groups.
Sweet Peas in the 1800s: A Brief History
Francis Galton, cousin of Charles Darwin, was a Victorian polymath who liked to count things. Curious about a broad range of topics including evolution, psychology, and meteorology, he brought an empiricist’s discipline to testing his ideas. During his life, he gathered and analyzed a huge amount of data.
Through a process of inquiry and investigation, Galton discovered the phenomenon of reversion to the mean, a towering achievement in statistics.
The idea is that for many types of systems, an outcome that is not average will be followed by an outcome that has an expected value closer to the average. While most people recognize the idea of reversion to the mean, they often ignore or misunderstand the concept, leading to a slew of mistakes in their analysis.2
Galton’s interest in this topic started with the idea that genius was inherited. He noticed that geniuses—musicians, artists, scientists—were way above the average, and that while their children were above the average, they were closer to it. Genius, however, was hard to measure. So Galton turned to something he could measure: sweet peas. He separated sweet pea seeds by size and showed that while the offspring tended to resemble the parent seed, their average size was closer to the mean of the full population.3
While normal, or bell-shaped, distributions were well known at that time, thinkers of the day generally assumed that the distributions were the result of a large number of small errors around an average. For instance, numerous scientists might make an estimate of a planet’s position. Each estimate captures the position with some error, reflecting imperfect instruments or calculation. If those errors are as likely to be in one direction as another, they will cancel out, and the average of the estimates will be the planet’s true position.
But the theory of errors could not explain Galton’s findings. He recognized there had to be a different mechanism at work. Heredity clearly played an important role in determining the size of the peas; it wasn’t simply that errors were distributed around some sort of universal average.
So Galton rolled up his sleeves and embarked on a detailed study of stature. Galton gathered the heights of four hundred parents and more than nine hundred of their grown children. He combined the heights of the mothers and fathers into what he called “mid-parent stature” and found that they followed a normal distribution. He then calculated the height of their children and found that they reverted to the mean. Tall parents tend to have tall children. But the height of the children is closer to the average of all children. Short parents generally have short children. But those children are taller than their parents (see figure 8-1). This data allowed Galton to demonstrate and define reversion to the mean.4
FIGURE 8-1
Reversion to the mean in human heights
Source: Based on Francis Galton, “Regression towards Mediocrity in Hereditary Stature,” Journal of the Anthropological Institute 15 (1886): 246–263.
Galton’s significant insight was that, even as reversion to the mean occurs from one generation to the next, the overall distribution of heights remains stable over time. This combination sets a trap for people because reversion to the mean suggests things become more average over time, while a stable distribution implies things don’t change much. Fully grasping how change and stability go together is the key to understanding reversion to the mean.5
Skill, Luck, and Outcomes
In many human endeavors, the outcomes are a combination of skill and luck. In baseball, for example, a pitcher can hurl a great game and yet his team may lose due to chance events. Naturally, the amount of influence that skill and luck will have depends on the activity. There is no skill involved in playing a slot machine. But winning at chess requires a great deal of skill and only a small amount of luck. Yet even when a player’s skill doesn’t change, his luck will come and go.
For example, consider how a golfer may score on two rounds on different days. If the golfer scores well below his handicap for the first round, how would you expect him to do for the second one? The answer is not as well. The exceptional score on the first round resulted from his being skillful but also very lucky. Even if he is just as skillful while playing the second round, you would not expect the same good luck.6
Any system that combines skill and luck will revert to the mean over time. Daniel Kahneman neatly captured this idea when he was asked to offer a formula for the twenty-first century. He actually provided two. Here’s what he submitted:7
Success = Some talent + luck
Great success = Some talent + a lot of luck
Naturally, poor outcomes can reflect the combination of some skill and a lot of bad luck. That was the case with the first dozen games for the Yankees in 2005. However, over time, skill shines through as luck evens out, which helps explain why the Yankees finished in first place. Steinbrenner’s view of his team was too narrow. He saw that the Yankees had lost eight of twelve games. But he didn’t take into account that the Yankees were among the most skillful ballplayers in the nation (even though he was paying their sizable salaries). They started to win when their luck improved.
When you ignore the concept of reversion to the mean, you make three types of mistakes. The first mistake is thinking you’re special. I once met with a company’s senior management team and discussed my interpretation of reversion to the mean in corporate performance. The executives all nodded knowingly. Then the CEO chimed in, “Yes, we understand the idea of mean reversion well. But it doesn’t apply to us because we’ve figured out a better way to run our business.” If it were only so.
One example of ignoring reversion to the mean comes from the world of investing. Which investment manager would you rather hire: the one who recently beat the market or the one who lagged behind the index? Of course, there is no easy answer. Luck clearly plays a large but elusive role in how much money you’ll make from any investment, especially in the short term. But even though industry pros intellectually understand the importance of luck, they consistently fail to incorporate that knowledge into their decisions.
Amit Goyal, a finance professor at Emory University, and Sunil Wahal, a finance professor at Arizona State University, analyzed how thirty-four hundred retirement plans, endowments, and foundations (plan sponsors) hired and fired firms that manage investment funds over a ten-year period. The researchers found that plan sponsors tended to hire managers who had performed well in the recent past. And the number one reason to fire a manager was poor performance. Consistent with reversion to the mean, the researchers noted that in subsequent years, many of the managers who were fired went on to outperform the managers who were hired (see figure 8-2).8
Individual investors behave similarly. Individuals earn returns that are generally 50 percent to 75 percent of the S&P 500 Index precisely because they pour money into hot markets and yank it out after a drop. They buy high and sell low. Those people who ignore reversion to the mean forgo substantial investment returns on their hard-earned money.9
FIGURE 8-2
Hire them when they are hot and hold them when they are not
Source: Amit Goyal and Sunil Wahal, “The Selection and Termination of Investment Management Firms by Plan Sponsors,” Journal of Finance 63, no. 4 (2008): 1805–1847.
In my research, I found that analysts on Wall Street ignore the effects of reversion to the mean when they build their models of a company’s future financial results. Analysts regularly neglect the evidence for reversion to the mean in considering essential drivers like company sales growth rates and levels of economic profitability.10
Secrist’s Blunder
“Mediocrity tends to prevail in the conduct of competitive business,” wrote Horace Secrist, an economist at Northwestern University, in his 1933 book, The Triumph of Mediocrity in Business. With that stroke of the pen, Secrist became a lasting example of the second mistake associated with reversion to the mean—a misinterpretation of what the data says.11 Secrist’s book is truly impressive. Its four hundred-plus pages show mean-reversion in series after series in an apparent affirmation of the tendency toward mediocrity. My research gives an example of Secrist’s idea. Figure 8-3 shows how the spread between return on invested capital (ROIC) and cost of capital reverts to the mean for a sample of more than a thousand companies, broken into quintiles, over a decade (the figure tracks the median ROIC for each quintile). While contemporary, this picture would have fit comfortably inside Secrist’s text.12
Secrist’s book was warmly received for the most part, with the notable exception of a scathing review by Harold Hotelling, an economist and statistician at Columbia University. The problem, Hotelling pointed out, is “these diagrams really prove nothing more than the ratios in question have a tendency to wander about.”13 The best visual for understanding Hotelling’s criticism is figure 8-4. At the top is the distribution of ROICs for the sample in 1997. In the middle is the reversion to the mean from figure 8-3, and on the bottom is the distribution of ROICs for 2007. Note that the distribution on the top and bottom look very similar.
FIGURE 8-3
Corporate returns on invested capital mean revert (1997–2007)
FIGURE 8-4
Reversion to the mean does not imply the triumph of mediocrity
In contrast to Secrist’s suggestion, there is no tendency for all companies to migrate toward the average or for the variance to shrink. Indeed, a different but equally valid presentation of the data shows a “movement away from mediocrity and [toward] increasing variation.”14 A more accurate view of the data is that over time, luck reshuffles the same companies and places them in different spots on the distribution. Naturally, companies that had enjoyed extreme good or bad luck will likely revert to the mean, but the overall system looks very similar through time.
What if you ran the analysis of reversion to the mean from the present to the past instead of from the past to the present? Are the parents of tall children more or less likely to be taller than their children?
A counterintuitive implication of mean reversion is that you get the same result whether you run the data forward or backward. So the parents of tall children tend to be tall, but not as tall as their children. Companies with high returns today had high returns in the past, but not as high as the present. Figure 8-5 illustrates this point by reversing the arrow of time. The quintiles are based on 2007 ROICs—and are therefore different from the quintiles in figure 8-3—and go back to 1997. The similarity to figure 8-3 is clear.
FIGURE 8-5
Mean reversion also works from the present to the past (2007–1997)
Here’s how to think about it. Say results are part persistent skill and part transitory luck. Extreme results in any given period, reflecting really good or bad luck, will tend to be less extreme either before or after that period as the contribution of luck is less significant.
What Kind of Feedback Helps Performance?
More than forty years ago, Daniel Kahneman was asked to help flight instructors in the Israeli air force sharpen their training skills. After watching the instructors hurl obscenities at the trainees, Kahneman told the instructors about research with pigeons that demonstrated how positive feedback can motivate better than castigation. One instructor retorted, “With all due respect, sir, what you’re saying is for the birds.” The agitated instructor went on to explain that pilots almost always did worse on their next flight after praise and consistently did better after a tongue lashing.
Initially taken aback, Kahneman soon realized that the instructor was committing our third mistake. The instructor believed that his insults caused the pilots to fly better. In reality, their performance was simply reverting to the mean. If a pilot had an unusually great flight, the instructor would be more likely to pay him a compliment. Then, as the pilot’s next flight reverted to the mean, the instructor would see a more normal performance and conclude praise is bad for pilots. The instructors didn’t see that their feedback was less important for the performance on the next flight than reversion to the mean.15
Those who thought that Steinbrenner’s tantrum helped to put the Yankees in first place in 2005 made the same mistake. The main lesson is that feedback should focus on the part of the outcome a person can control. Call it the skill part, or the process. Feedback based only on outcomes is nearly useless if it fails to distinguish between skill and luck.
The Halo Effect
The halo effect, first described in the 1920s by Edward Thorndike, a psychologist at Columbia University, is closely related to reversion to the mean and illustrates a fatal flaw in much of the research for business managers. The halo effect is the human proclivity to make specific inferences based on general impressions. For example, Thorndike found that when superiors in the military rated their subordinate officers on specific qualities (e.g., intelligence, physique, leadership), the correlations among the qualities were impossibly high. If the officer liked his subordinate, he awarded generous grades across the board. If he didn’t like him, he gave poor marks. In effect, the overall impression the officer made on his superior obscured the details.16
In The Halo Effect, Phil Rosenzweig showed that this mistake pervades the business world. Rosenzweig pointed out that we tend to observe financially successful companies, attach attributes (e.g., great leadership, visionary strategy, tight financial controls) to that success, and recommend that others embrace the attributes to achieve their own success. Researchers who study management often follow this formula and rarely recognize the role of luck in business performance. And the substantial data the researchers use to support their claims is all for nothing if they fall into the trap of the halo effect.17
For example, Rosenzweig suggests that the press will praise a company that is doing well for having “a sound strategy, a visionary leader, motivated employees, an excellent customer orientation, a vibrant culture, and so on.”18 But if the company’s performance subsequently reverts to the mean, onlookers will conclude all of those features went wrong, when in reality nothing of the sort happened. In many cases, the same people are running the same business with the same strategy. Mean reversion shapes company performance, which in turn manipulates perception.
Rosenzweig offers ABB, the Swedish-Swiss industrial company, as a good example. In the mid-1990s, the Financial Times named ABB the “Most Respected Company” in Europe three years in a row, suggesting the company “rated exceptionally highly for business performance, corporate strategy, and maximizing employee potential.” ABB’s chief executive officer, Percy Barnevik, also collected accolades. The Korean Management Association named him “the world’s best honored top manager”—an award for getting the most awards.
In the late 1990s and early 2000s, ABB’s performance slipped. And the same factors that the press cited as the key to ABB’s success, such as agility from having decentralized management, were now the reason for the downfall, as “the far-flung units ended up causing conflicts.”19 But the press saved its largest swing in opinion for Barnevik, who went from being described as “charismatic, bold, and visionary” to “arrogant, imperial, and resistant to criticism.” Richard Tomlinson and Paola Hjelt, journalists at Fortune magazine, reviewed ABB’s highs and lows and concluded that, “Barnevik was never as good as the rave reviews he received in the 1990s, nor was he half as bad as the recent damning press coverage might suggest.”20
While Tomlinson and Hjelt got it right, the media often perpetuates the halo effect. Successful individuals and companies adorn magazine covers, along with glowing stories explaining the secrets of their success. The halo effect also works in reverse, as the press points out the shortcomings in poor-performing companies. The press’s tendency to focus on extreme performance is so predictable that it has become a reliable counter-indicator.
Tom Arnold, John Earl, and David North, finance professors at the University of Richmond, reviewed the cover stories that Business-Week, Forbes, and Fortune had published over a period of twenty years. They categorized the articles about companies from most bullish to most bearish. Their analysis revealed that in the two years before the cover stories were published, the stocks of the companies featured in the bullish articles had generated abnormal positive returns of more than 42 percentage points, while companies in the bearish articles underperformed by nearly 35 percentage points, consistent with what you would expect. But for the two years following the articles, the stocks of the companies that the magazines criticized outperformed the companies they praised by a margin of nearly three to one. Reversion to the mean. Sports fans have a variant of this called the Sports Illustrated jinx—teams or athletes tend to do worse right after they have appeared on the magazine’s cover.21
Rosenzweig shows in devastating fashion that most of the thinking from best-selling business books falls prey to the halo effect. These books are commercially successful, he suggests, because they tell managers a story they want to hear: any company can be successful by taking these steps. In fact, no simple formula ensures success in a rapidly changing business environment.
For instance, in his widely read book Good to Great, Jim Collins identifies eleven great companies and notes they are all so-called hedgehogs. They focus on what they do best. They direct their efforts to doing whatever results in economic growth. And they are passionate. So one lesson of the book is that your company can succeed too, if it adopts the habits of a hedgehog. However, the important question is not, “were all great companies hedgehogs?” but rather, “were all hedgehogs great?” If the answer to the latter question is no—and it assuredly is—then dwelling on the survivors creates a bias in the analysis, leading to faulty conclusions.
Now that you are alert to the one-two punch of reversion to the mean and the halo effect, you will see it everywhere. In the late 1990s, the Corporate Executive Board did some thought-provoking research on corporate growth. I found the analysis useful and immediately integrated it into my work. About a decade later, the firm published an updated version of its analysis. At first, I was excited to get my hands on the latest findings based on “exhaustive study.”
But I was soon deflated by the realization that the new work suffered from the halo effect. Unlike the prior work, the updated analysis defined a pattern of rising and falling company sales growth, mined decades of data to find corporate performance that matched the pattern, and then attached attributes (specific strategic, organizational, and external factors) to the companies that fit. While alluring and well packaged, the findings were based on flawed analysis.
How do you avoid mistakes associated with reversion to the mean? Here’s a checklist that may help you identify important issues:
1. Evaluate the mix of skill and luck in the system that you are analyzing. Discerning the contributions of skill and luck is rarely an easy task, even if analytical tools are available. 22 To make the thought more concrete, consider the continuum of games in table 8-1. On the left are complete-information games, where each player knows the positions, payoffs, and strategies available to his opponent. In these games, the outcomes are largely settled through skill. On the right are games based on luck, where skill plays no role. The middle games combine skill and luck.
Here’s a simple test of whether an activity involves skill: ask if you can lose on purpose.23 Think about casino games like roulette or slots. Winning or losing is purely a matter of luck. It doesn’t matter what you do. But if you can lose on purpose, then skill is involved. This simple test reveals the role of luck in investing. While most people recognize that it is hard to construct a portfolio that beats the S&P 500, most people don’t know how hard it is to build a portfolio that will do a lot worse than the benchmark.
TABLE 8-1
What determines the outcome—skill or luck?
| Skill | Skill and luck | Luck |
|---|---|---|
| Chess | Poker | Roulette |
| Checkers | Backgammon | Slot machines |
| Go | Monopoly | Chutes and Ladders |
You should therefore be careful when you draw conclusions about outcomes in activities that involve luck—especially conclusions about short-term results. We’re not very good at deciding how much weight to give to skill and to luck in any given situation. When something good happens, we tend to think it’s because of skill. When something bad happens, we write it off to chance. So forget about the outcome and concentrate instead on the process.
Recognize, too, there is no lack of commentary about systems that are strongly influenced by chance. As the story of George Steinbrenner made us aware, luck plays an important role in baseball, especially in the short term. Yet baseball announcers analyze the games play-by-play with little awareness that luck explains most of what’s going on. This same principle applies in business and markets.
2. Carefully consider the sample size. Daniel Kahneman and Amos Tversky established that people extrapolate unfounded conclusions from small sample sizes.24 But thinking clearly about sample size is essential for a few reasons.
The more that luck contributes to the outcomes you observe, the larger the sample you will need to distinguish between skill and luck. Baseball is a good example. Over a 162-game season, chances are good the best teams will rise to the surface. In the short term, however, almost anything can happen. In Moneyball, Michael Lewis, an author who frequently provides fresh views on issues, points out, “In a five-game series, the worst team in baseball will beat the best about 15 percent of the time.”25 You do not see this in chess or tennis matches, games in which the best player almost always beats the worst, regardless of time frame.
In addition, when a large number of people participate in an activity that is influenced by chance, some of them will succeed by sheer luck. So you have to scrutinize even long, successful track records in fields with lots of participants. Investment track records are a good example.
Fans often misunderstand hot hands and streaks in games and sports. The term hot hand refers to the belief that success breeds success. We tend to believe that if a basketball player has made one shot, he is more likely to make the next one.
Michael Bar-Eli, a professor of business at Ben-Gurion University, studies the psychological determinants of human performance, especially as they relate to sports. With some colleagues, Bar-Eli did a detailed review of hot-hand studies, concluding tepidly that “the empirical evidence for the existence of the hot hand is considerably limited.”26
This is not to say that players don’t have streaks of made or missed shots. Naturally, they do. The point is that these strings of successes and failures are consistent with the skill level of the player. For instance, a basketball player who makes 60 percent of her shots has about a 7.8 percent chance (0.6)5 of making five in a row. A player who makes 40 percent of his shots has only a 1 percent chance (0.4)5 of hitting five in a row. The best players have more streaks than the worst players, just as you would expect, given the statistics.
Streaks, continuous success in a particular activity, require large doses of skill and luck. In fact, a streak is one of the best indicators of skill in a field. Luck alone can’t carry a streak. My analysis of various sports streaks in basketball and baseball clearly suggests streak holders are among the most skilled in their fields.
Jerker Denrell, a professor of organizational behavior at Stanford Business School, has shown the link between the sample size and learning. In his paper, “Why Most People Disapprove of Me: Experience Sampling and Impression Formation,” Denrell argues that the first impression you have of a person or organization can determine your future degree of interaction. So if you run a business that deals with customers, it is especially important to make sure that you make a favorable first impression.27
Imagine trying a new restaurant with two possible outcomes. In the first case, the restaurant is at its best. You have a wonderful meal with attentive service at a reasonable price. Would you go back? In the second case, the restaurant has an off day. You have a so-so dinner with indifferent service at the high end of what you had hoped to pay. Would you go back?
Most people would go back in the first case but not in the second. Given reversion to the mean, what’s likely to happen the second time you go to the restaurant? Chances are the meal won’t be quite as good, or the service will slip a bit. But in this case you have gathered a more accurate view of the restaurant, even if it’s less flattering. On the other hand, if you never return to the restaurant because of a bad experience, you are assured you will gather no additional information, even if that information—as reversion to the mean suggests—would be more favorable. So people tend to have a better picture of people and things they like than what they don’t like because they have a fuller sample.
3. Watch for change within the system or of the system. Not all systems remain stable over time, so it’s important to consider how and why the system has changed. One obvious example is individual changes in skill level. An athlete’s age is a good example. In many professional sports, athletic skill improves through the late twenties, at which point it begins to steadily deteriorate. So above-average athletes revert to the mean over time as a consequence of diminished skill. Loss of skill naturally applies to other pursuits as well, including business and medicine.
Further, the system itself may change. Stephen Jay Gould analyzed why baseball has not seen a player sustain a .400 batting average for a complete season since Ted Williams in 1941. After entertaining some possible explanations—none persuasive—Gould showed that while the mean batting average in the major leagues has been fairly stable over the years, the standard deviation has shrunk from roughly 32 percent in 1941 to about 27 percent today. The bell of the bell-shaped distribution has a narrower width than it used to. That the right side of the distribution is closer to average may explain the lack of.400 hitters. Gould attributed the reduction in standard deviation to a greater and more consistent overall skill level in the major leagues.28
4. Watch out for the halo effect. A whole cottage industry, including business school professors and consultants, is working hard to offer businesspeople tidy solutions for their problems. Here’s how you grow sales. Here’s how you innovate. Here’s how you manage your people. But any time you see an approach offering secrets, formulas, rules, or attributes to achieve success, you can be sure that someone is selling you a nostrum. Still, spotting the halo effect requires discipline, because the purveyors are selling alluring stories and suggest substantial, albeit phony, rigor.
If you’re like me and want to find a cause for every effect, you should spend some time disentangling skill from luck. An appreciation of the relative contributions of skill and luck will allow you to think clearly about reversion to the mean. To me, the greatest lesson and opportunity from understanding reversion to the mean is to keep your cool. When outcomes are really good because of a dose of good luck, prepare for the times when they will be closer to the average. When outcomes are disappointing as the result of bad luck, recognize things will get better.