The Data Deluge

Yes, we could give these decision makers some benefit of the doubt. We could possible argue that, even if they wanted to, these managers might not have ready access to data to test and verify their hypotheses. Alternately, we could reason that the data are actually available to test management beliefs and their mental models but are not deployed effectively while making key strategic and tactical decisions. While until a few years ago, it might have been possible to make a case for the absence of good quality data, a number of recently published studies suggest that high-quality data are now widely available. Most organizations today live in an extremely data-rich environment. Recently reported statistics³ suggest that over 160 exabytes (1 exabyte = 10¹⁸ bytes) of digital data were generated worldwide in 2006. This is the equivalent to 36 billion digital movies, 43 trillion digital songs, or 1 million digital copies of every book in the Library of Congress. In 2006, these books would represent about 6 tons of books for every man, woman, and child on earth—approximately the weight of a large adult elephant. By the end of 2010, this number was expected to grow to about 1,000 exabytes at a whopping 57% compounded annual growth rate and was projected to outpace the capacity to store such data. Another way to think of this volume of data is that in 2006, the digital universe was the equivalent of 12 stacks of books extending from the earth to the sun, or one stack of books twice around the earth’s orbit. By the end of 2010, the stack of books could reach from the Sun to Pluto and back!⁴

While a lot of the information, such as digital entertainment, is for consumer consumption, it is also generated by various for-profit and nonprofit organizations through customer relationship management (CRM) systems, internal metrics and processes, as well as external organizational measures of sales and competitive activities. In 2006, about 25% of the worldwide digital data were generated in the workplace—approximately 40 exabytes. By the end of 2010, this volume is expected to rise to about 30%—about 300 exabytes. Wal-Mart’s gargantuan database, for instance, has grown from 110 terabytes in 2000 (1 terabyte = 10¹² bytes) to half a petabyte in 2004 (1 petabyte = 10²¹ bytes). These data are generated primarily to support internal decisions and provide information to other partners, such as suppliers in the value chain. The point is that an absence of adequate and readily available data no longer seems to be a justifiable reason for the pattern of intuition-based decision making that is rampant among various layers of management today. Instead, it is perhaps time for them to leverage the vast amounts of well-structured data at their disposal and to hone their skills to make more effective decisions.