Most Big Data Projects Will Fail, and That’s a Good Thing

Monetizing information is the number one challenge among those planning to invest in Big Data. This is an indication that many organizations are embarking on Big Data projects speculatively and without any strategy or defined business outcomes. Some Big Data projects are driven by IT without buy-in from the business, or have been initiated inside a department without any buy-in from the rest of the organization. In both of these cases, the ability to demonstrate value from Big Data will be at risk. This can lead to the project being abandoned as a pilot because crucial funding and support to get it into production will be missing.

Big Data projects are inherently experimental, and most experiments fail. Some projects will not deliver value and will justifiably be discarded. Others may last short term and be shelved after providing some insight. These outcomes should be expected, but having failure as the common case does not mean Big Data projects should lack a desired business outcome.

Recognizing the Organizational Roadblocks to Information Monetization

While the recent and ongoing diminishment of Big Data hype is a relief, fast followers and late adopters will struggle to achieve the kind of innovative and high-value successes that many early adopters had with Big Data. Even as technologies and techniques for accumulating, managing, and deploying Big Data have matured, most organizational cultures and business models remain garrisoned against information-based innovation.

It’s true that Big Data skills—especially real data scientists—are in short supply. This will continue for several years. However, most limiting is the continued inertia toward thinking about and using data merely for generating ever more reports and spreadsheets, rather than evolving to consider information a strategic asset and raw material for transforming businesses, products, and services. Digitalization and a shift toward the data-driven organization, of course, require a sophisticated application of information assets, as opposed to more aesthetic pie charts. Finding ways to make use of orders of magnitude, higher volume, velocity, and variety of data can require drastic business model changes for which many are simply unprepared.

Giving the Gift of Information to Your Competitors

Digital industry change will seldom be initiated by incumbents—instead it’s the technology providers and startups that will often take the lead. The master agreements of these technology providers often gives them the right to take ownership of their customers’ intellectual property, even (in some cases) that which may have been designated as confidential.

To date, as a result of digitalization, a small number of industries have been deeply disrupted and fundamentally redefined. Examples of these industries include music, books, newspapers, travel agencies, local advertising, and photographic film. However, developments over the last five years show that now every industry must be considered as susceptible to deep disruption.

With digital remastery of an industry or company, the intellectual property and knowledge that differentiates your company is virtual and IT-driven. Perhaps, because of this, we are increasingly seeing that the protagonists responsible for some of these industry changes and digital remastery have not originated from the industry incumbents, but rather have derived from technology-savvy and information-savvy companies outside the industry.

The Great Chief Data Officer Balancing Act

To derive value from information assets, organizations must focus first on data quality and data governance. This is the primary reason to hire a CDO. Once the data management aspects of risk, audit, and compliance are addressed, CDOs should shift focus to maximize the value of information assets consistent with the business’s strategies.

As CDOs mature in their role and improve their organization’s ability to better manage the risks associated with data (such as data quality, security, and data breaches), their roles will evolve and different priorities will emerge. While still responsible for the strategy and execution of data policies in support of compliance and risk mitigation, CDOs will adjust focus toward maximizing the business value of information assets. Specifically, in the age of Big Data, CDOs will be tasked with raising the analytic competency of their organizations. CDOs will show how information assets and analytics enhance the customer experience, increase market share, and deliver new forms of information innovation to improve brand recognition.

CDOs also will become tasked with identifying barter deals for information assets, participating in information exchanges with partners and suppliers, and driving other sources of competitive differentiation across their ecosystems. In this evolution from risk to value, CDOs will apply advanced analytics and information valuation techniques to measure and communicate the economic value of their data assets. And value-focused CDOs, will deploy information assets to generate supplemental and significant revenue streams.

Advanced Analytics, Data Science, and Artificial Intelligence

Not just a global trend, but also a technology trend, advanced analytics solutions are becoming increasingly popular in driving business innovation and experimentation—and creating competitive advantage by monetizing available information assets inside and outside the organization. Over the foreseeable future, enterprises will be seeking to adopt advanced analytics and adapt their business models, establish specialist data science teams, and rethink their overall strategies to keep pace with the competition.

Machine learning, represented by neural networks are supplanting most forms of advanced analytics due to their dynamic nature that adapts to changing and unknowable inputs. These types of algorithms can help to monetize almost any kind of information, be it granular IoT data or macro-level economic figures. Expect these kinds of algorithms to be a standard component in the vast majority of data scientist toolboxes and increasingly accepted by business leaders, despite their “black box” models. As a result, many data science tasks will become automated, increasing the productivity of data scientists and enabling a class of “citizen data scientists” to emerge. This will put signifi-cant pressure on information asset supply chains and information curation efforts, and engender a boom in information monetization ideas from all corners of the organization across all industries.

Increasingly, predictive analytics solutions that make copious use of available information assets within and external to the organization will evolve into artificial intelligence (AI) solutions capable of automating decision making and processes. Around the beginning of the next decade, some industries will attribute up to one-third of their net new revenue growth to information-fueled AI solutions. Still, humans will remain involved. Many machine learning implementations will require dedicated workers to help train, monitor, and guide them. Expect these roles to be mandated by various industry regulations. As well, industry regulations will require AI applications and their underlying information sources to be audited able, thereby presenting a new set of challenges.

Information Infrastructure

Information infrastructure is moving toward a complementary environment that encourages simultaneous deployment on premises and across multiple cloud environments. An increasing pressure to manage data in multiple deployment models, while also optimizing its access and retrieval, is enabling extended information ecosystems, while making an organization distinct information supply chain more obtuse.

Over the coming years, organizations with data virtualization capabilities will spend significantly less time on building and managing data integration processes for connecting distributed data assets. The ability to logically connect data sources, rather than physically integrate them (and do so with nominal performance degradation) will enable expanded information ecosystems, and reduce the security risks associated with information sprawl. As well, data transformation flows increasingly will make use of machine learning algorithms. I have long anticipated and referred to this as “self-organizing information.” Early attempts at this, however, will be plagued with erroneous interpretations of the data, both by the algorithms themselves and unsuspecting recipients (humans and applications).

Blockchain is worth mentioning as well, since it is a hot (if overhyped) topic at present. Into the next decade, expect most blockchain efforts to miss the mark, either due to performance challenges endemic to the technology or the misapplication of the technology by overzealous architects. That said, blockchain may very well prove useful for rationalizing metadata or master data long before being practicable for managing most high volume/velocity/variety information assets themselves.

Information Asset Privacy and Security

Other global trends with specific information-related implications are privacy and security. Information asset risk-related concerns increasingly will demand serious planning and robust execution in the areas of privacy and security. This will involve the vast majority of enterprises continuously monitoring for sensitive data incidents. First, however, this will require organizations to automatically inventory their information assets and scour their systems for instances of sensitive data such as PII. And these efforts will lead to the use of automated information masking or similar pseudonymization techniques, along with monitoring the information’s usage itself. To further buffer against missteps, most large organizations that process personal information will perform periodic impact assessments and more formal audits.

For information asset security implementations, many organizations will continue to attempt to implement manual data classification policies. But these will be fraught with limited deployments and insufficient tangible benefits. Formal (and eventually standardized) calculations for information asset liabilities will supplant these efforts to provide a degree of economic risk modeling and security prioritization. Within a few years, a small percentage of penetration tests will begin to be conducted by machine learning–based systems.

Information Monetization

In part I we looked at dozens of ways organizations of every size—and in every industry and geography—are generating measurable economic benefits from their own and external information assets. One can only imagine that what we’ve seen the past few years is information monetization in its infancy. Today, information monetization efforts are targeted, functionally specific, or experimental. Most are limited to just a few kinds of information and are deliberate, customized solutions. Tomorrow, expect a wider variety of information monetization approaches, and the emergence of standardized models, mechanisms, and technologies for doing so.

Moreover, I anticipate robust industry markets for data and content, automatic information trading, and an economic fluidity for information approaching or even surpassing that of cash. Remember, certain unique economic and behavioral characteristics of information render it more useful than other assets in some circumstances. This infonomics boom will lead to organizations finding unimaginably creative ways of harvesting or generating valuable information assets, and of marketing them.

Information Management

In part II we considered how current information management standards fall far short of the discipline with which other kinds of assets are managed. In examining the concepts of the supply chain and ecosystem, and traditional asset management approaches, we can envision how information assets may (or should) be managed with orders of magnitude improved discipline.

The emergence of the chief data officer (CDO) role in many organizations, and across all industries, indicates a growing recognition of information as a strategic business asset. Although today most CDOs are “chiefs” in title only, I expect IT organizations will begin to bifurcate into separate “I” and “T” organizations, thereby elevating the CDO to a level of prominence, influence, responsibility, and budget aligned with other CXOs within the organization. At this point, no longer will information be considered the domain of the IT organization, but a distinct business asset in its own right.

One of the greatest IAM challenges, particularly for CDOs, will be the continually dissolving walls between one organization’s information assets and those of another. Information assets increasingly will be shared in the cloud, not transmitted, among business partners and customers, creating true information ecosystems with established protocols, rewards, and penalties for its participants. In addition to pure IAM challenges, these information ecosystems will stress the notions of information rights, control, ownership, and sovereignty. Adopting and institutionalizing precepts such as the Generally Accepted Information Principles suggested in chapter 9 should help to lay a foundation for IAM within and among organizations.

Building on the popularity of cloud storage, combined with an age-old banking business model, I have long anticipated the emergence of “information services organizations.” Such companies will provide not just storage and access to information assets, but also a full range of banking-type services including: lending (licensing), interest (enrichment and cleansing), investment (new forms of monetization), credit (using information as a form of payment or collateral), portfolio analysis (measuring and reporting on information health), and advice.

Information Measurement

As we just covered in part III, the impetus to improve the way information is monetized and managed will lead to organizations measuring it in a variety of ways. Information quality, performance, relevance, impact, cost, and value measures deployed independently by organizations will become mainstream competencies, later giving rise to industry standards. At first, industry regulators and investors will require them. Later, insurers—realizing they could be offering new “information insurance” products—will solicit or generate information-related risk and value measurements. Once insurers acknowledge information assets as a form of insurable property, the accounting profession will be compelled to acknowledge information as a capitalizable asset. Although, this could happen in the reverse order—accountants followed by insurers. It’s uncertain which profession will blink first.

Eventually, the formalized measurement of information value and risk will lead to multi-billion dollar industry opportunities not only for the insurance industry, but also for accounting firms to offer formal information auditing services. In addition, accounting and valuation firms will start offering enhanced M&A services that rightly consider the value of a firm’s information assets.

Finally, I expect CDOs, enterprise architects, business leaders, and economists to begin thoroughly looking into how to apply classic economic models to information—not as some nouveau academic exercise, but as legitimate means for determining how to better understand and harness information’s unique characteristics and behavior. This is the promise of infonomics—information’s emergence over the past couple of decades from that of a business byproduct and business resource, to a performance fuel and marketable commodity, and ultimately to a true, recognized economic asset.