The Mr. McGuires of the world are no longer advising newly-minted graduates to get into plastics. But perhaps they should be recommending data. In today’s digital world data is the key, the ticket, and the Holy Grail all rolled into one.

I do not just mean it’s growing in importance as a profession, although it is a great field to get into, and I’m thrilled that my sons Jake and Josh are pursuing careers in data and technology. Data is where the dollars are when it comes to company budgets. Every few years there is another report showing that business intelligence (BI) is at or near the top of the chief information officer’s (CIO) list of priorities.

Enterprises today are driven by data, or, to be more precise, information that is gleaned from data. It sheds light on what is unknown, it reduces uncertainty, and it turns decision-making from an art to a science.

But whether it’s Big Data or just plain old data, it requires a lot of work before it is actually something useful. You would not want to eat a cup of flour, but baked into a cake with butter, eggs, and sugar for the right amount of time at the right temperature it is transformed into something delicious. Likewise, raw data is unpalatable to the business person who needs it to make decisions. It is inconsistent, incomplete, outdated, unformatted, and riddled with errors. Raw data needs integration, design, modeling, architecting, and other work before it can be transformed into consumable information.

This is where you need data integration to unify and massage the data, data warehousing to store and stage it, and BI to present it to decision-makers in an understandable way. It can be a long and complicated process, but there is a path; there are guidelines and best practices. As with many things that are hard to do, there are promised shortcuts and “silver bullets” that you need to learn to recognize before they trip you up.

It will take a lot more than just reading this book to make your project a success, but my hope is that it will help set you on the right path.

Enterprises need this information to understand their operations, customers, competitors, suppliers, partners, employees, and stockholders. They need to learn about what is happening in the business, analyze their operations, react to internal and external pressures, and make decisions that will help them manage costs, grow revenues, and increase sales and profits. Forrester Research sums it up perfectly: “Data is the raw material of everything firms do, but too many have been treating it like waste material—something to deal with, something to report on, something that grows like bacteria in a petri dish. No more! Some say that data is the new oil—but we think that comparing data to oil is too limiting. Data is the new sun: it’s limitless and touches everything firms do. Data must flow fast and rich for your organization to serve customers better than your competitors can. Firms must invest heavily in building a next-generation customer data management capability to grow revenue and profits in the age of the customer. Data is an asset that even CFOs will realize should have a line on the balance sheet right alongside property, plant, and equipment” [1].

It can be a problem, however, when there is more data than an enterprise can handle. They collect massive amounts of data every day internally and externally as they interact with customers, partners, and suppliers. They research and track information on their competitors and the marketplace. They put tracking codes on their websites so they can learn exactly how many visitors they get and where they came from. They store and track information required by government regulations and industry initiatives. Now there is the Internet of Things (IoT), with sensors embedded in physical objects such as pacemakers, thermostats, and dog collars where they collect data. It is a deluge of data (Figure 1.1).

This is a massive, potential headache for CIOs. According to the 2014 State of the CIO Survey [2], leveraging data and analytics is the most important technology initiative for 2014, with 72% of CIOs surveyed stating that it is a critical or high priority. Gartner concurs, with the prediction that BI and analytics will remain a top focus for CIOs through 2017, and that the benefits of fact-based decision-making are clear to business managers in a broad range of disciplines, including marketing, sales, supply chain management (SCM), manufacturing, engineering, risk management, finance, and HR [3].

Adding to the complexity, now many more people in an organization need the information that comes from all this data. Where in the past only a few managers received information to analyze, now business people at all levels are using analytics in their jobs. For example:

All of this information has had a tremendous impact on businesses’ ability to make informed decisions. According to a study conducted by The Economist [4], the flood of information and analytics has had these effects:

My son Jake’s first job out of college was at a firm that analyzed large volumes of data for resume parsing and job matching. His boss voiced a similar lament when he said “What’s the use of Big Data if it’s just a lot of data? We need information we can analyze, not just a big pile of data.”

So much is riding on your data. No one can afford to be sloppy or wasteful in their BI and data integration strategies. Mistakes are expensive and it is highly embarrassing when a customer finds errors before you do. Businesses, now more than ever, need to understand who their current and potential customers are as well as how much revenue and profit each product or service line generates. This demands clean, consistent, conformed, current, and comprehensive data.

Do not be swayed by vendor sales pitches for quick-hit solutions that promise short cuts. Prepackaged analytics and corporate dashboards offer the allure of off-the-shelf solutions that appear to take almost no work and instantly provide the business with the answers to all their questions. If the solution looks too good to be true, step back and ask some questions. Read the fine print, such as the qualifiers that everything works fine only if all your data is clean, all the data is included in the supported sources, and your business analytics match the prebuilt models. In short, the prebuilt solution works if you already have all your data in place. These quick and dirty solutions often have the unintended consequence of creating more data silos.

It would be great if all it took was buying the right tool to provide your business with comprehensive, cleansed, consistent, conformed, and current business information from source data. You do need to buy tools, but it does not happen in minutes, hours, or days. It takes time and hard work.

The best situation for an enterprise is when, in addition to using clean, consistent, conformed, current, and comprehensive data, they are routinely using analytic-driven processes to manage and grow the business. Advanced analytics tools such as data visualization, predictive analytics, and data discovery often play a large role in this. The reality is that most enterprises are not in this state of “analytical nirvana,” but it should always be their goal.

Their path should include:

See Figure 1.3 for a visual representation of how they interrelate.

Data integration—combining data from different sources and bringing it together to ultimately provide a unified view. Data integration and data shadow systems are often at opposite ends of the DW/BI spectrum. If an enterprise has inconsistent data, it is highly likely that it has a data integration problem. The components of data integration include the data sources; the processes to gather, consolidate, transform, cleanse, and aggregate data and metadata; standards; tools; and resources and skills.

Data warehousing—the process of storing and staging information, separate from an enterprise’s day-to-day transaction processing operations, and optimizing it for access and analysis in an enterprise. In this process, data flows from data producers to the data warehouse, where it is transformed into information for business consumers. It encompasses all the data transformations, cleansing, filtering, and aggregations necessary to provide an enterprise-wide view of the data.

As for reporting and decision-support systems, historically it was a centralized database. In the classic definition from Bill Inmon’s book Building the Data Warehouse it is:

BI—to present data to business people so they can use it to gain knowledge. BI enables access and delivery of information to business users. It is the visible portion of the corporate data systems, as opposed to data warehousing, which is in the “back room.” BI is what business people see via tools and dashboards. The data comes from relational data sources or enterprise applications such as enterprise resource planning (ERP), customer resource management (CRM), and SCM. The source of the data can be a “black box” from the business person’s perspective; they mainly care about what it is, not where it came from.

A few of the key terms are depicted in Figure 1.4. Some enterprises will use data warehousing as the umbrella term for everything depicted in that diagram while others will use BI instead. As long as it is designed, developed, tested, and deployed either term is fine. Table 1.1 presents some of the key terms and the discipline to which they apply.

Each building block includes a host of different technologies, many listed in Table 1.1. Although there is overlap, we have tried to group them by BI, DW, and data integration. This is not meant to be an all-inclusive list or definitions, but rather our perspective on some of the bigger terms. The rest of this book shows how these technologies are put into action and where they fit with one another. Becoming familiar with these terms now will help make the concepts easier to understand.