Why Is the Data Ecosystem Important?

The data ecosystem relates to the data management and technological infrastructure that needs to be in place to enable analytics across the entire organization. This includes implementing technology architectures, governance, and procedures to manage the company's full data needs. It has been proven that those companies that have a strong data-driven decision making corporate culture correlate with executive sponsorship and support. As a result, the appreciation and understanding of the importance of the data ecosystem in driving the benefits of analytics become the new cultural norm.

The primary purpose of data ecosystems is to capture data to produce useful insights. As consumers purchase products, especially via digital formats, they leave data imprints. Companies now can create data ecosystems that capture and analyze data imprints so product teams can determine what their users like and don't like, and how they respond to different marketing programs. Product teams can use insights to tweak sales promotions, features, displays, and more to improve consumer demand. Data ecosystems were originally designed to be relatively centralized and static. The birth of the Web and cloud services has changed the data ecosystem framework. Now data is captured and distributed across organizations with IT professionals having less central control. Hence, the term data ecosystem, indicating these are data environments designed to evolve. There is no one data ecosystem solution framework.

Every business can create its own data ecosystem, sometimes referred to as a technology stack. It starts with a collection of hardware and software to capture, store, process, analyze, and act upon data insights. The best data ecosystems are built around an analytics solution platform that is open source with specific point-design capabilities—such as demand forecasting and planning—which tie the whole ecosystem together. Analytics platforms help teams integrate multiple data sources, provide AI/machine learning tools to automate the process of capturing, harmonizing, and normalizing data, and then conducting analysis and tracking users' performance metrics.

Why Data and Analytics?

The reality is that data and analytics are complicated. However, the results can significantly improve an organization's productivity and performance. The results could be increased revenue, improved profit margins, reduction in inventory costs, or an entirely new business framework. Any successful initiative requires a cultural and mindset shift where data and analytics change from supportive and secondary to fundamental for digital transformation. Data and analytics become central to how organizations do business every day, and thus are the basis for all decisions.

Those companies who make data a part of everything ask the right questions:

• How will the insights uncovered from this data as a result of the analytics change the value proposition for our customers?
• How will it improve our ability to predict shifting consumer demand patterns?
• How will we be able to improve the consumer experience?
• How will this help us deliver value and profitability?
• How will the results drive new business processes, innovation, and consumer response as a result of these new insights?⁴

The ability to answer these questions based on data and analytics will not only add value but will expand data and analytics competencies by encouraging data literacy across the organization. Organizations need to become smarter at understanding what results can be improved and how that investment drives positive outcomes across the company. It might be an investment in access to new data, or the implementation of artificial intelligence, or the ability to easily visualize data, or other activities. A clear understanding that considers data quality, data governance, and data literacy is vital to the success of a company's data and analytics investment.

The challenge at many companies is that demand planners, as well as many businesspeople, don't understand the importance of how data and analytics supports their work. They feel that business knowledge, experience, and intuition play a more practical role. On the other hand, demand analysts and data scientists lack a clear understanding of the business acumen required to successfully model and analyze the business effectively. Data and analytics are increasingly valuable assets to organizations, which requires all employees to embrace, understand, and utilize the relevant analytics information that affects their areas of expertise, particularly during disruptions. In the digital economy, data is as important as the classic business drivers of “people, process, analytics, and technology,” making it equally important to become part of the corporate culture.

Engage with Domain Experts and Business Specialists

Understanding what data is needed may be easier when analytics practitioners work closely with domain experts within their organizations. Teaming data scientists with domain experts and business specialists who understand data sources and how they can be automated should be a best practice for all companies. The team members would complement one another with different expertise and perspectives, uncovering different patterns and identifying different opportunity pathways that may have been missed without the collaboration. This is particularly true when teaming data scientists with marketing and demand planners to develop driver-based consumption models and forecasts. A marketing expert would expect certain consumer shopping patterns and check with the analytic results as compared to assumptions, and know what questions to ask that were not part of the initial analytics investigation. On the other hand, a demand planner would be able to spot and examine anomalies arising due to field sales activities and know what adjustments to the model would help provide insights as to why those activities should be included or discarded.

Data governance ensures that the right data gets utilized so that results are trusted and actionable. Ensuring that the data is high quality will allow data scientists (demand analysts) to start with good building blocks for their models. There needs to be a common truth in the data, including the underlying data, how it's been processed, and how it's analyzed, which will ultimately create more accurate, scalable analytics. However, companies may find that the challenge may be more complex due to legacy systems architecture and fragmented data sources and data repositories. Many corporate leaders agree that data is an important asset; however, those who back up that view with committed organizational resources will gain competitive advantage from deploying predict analytics and AI/machine learning. As a result, many companies now employ a chief data officer (CDO) or chief analytics officer (CAO).

Why Is Downstream Data Important?

Downstream data has been electronically available on a weekly basis since the late 1980s. But most companies have been slow to adopt downstream data for forecasting and planning purposes.

Downstream consumption data is stored within the commercial side of the value chain. Examples include retailer POS data, syndicated scanner data from Nielsen, Information Resources Inc. (IRI), and Intercontinental Marketing Services (IMS). Prior to electronically available downstream data, manufacturers received this type of data in hard copy format as paper decks (after a 4- to 6-week lag). Once received, the data points were entered manually into mainframes via a dumb terminal (a display monitor that has no processing capabilities; it is simply an output device that accepts data from the CPU). In fact, downstream consumption data has been available to consumer goods companies for many decades. Nevertheless, the quality, coverage, and latency of downstream data has improved significantly, particularly over the past 30 years, with the introduction of universal product barcodes (UPC) and retail store scanners.

In fact, companies receive daily and weekly retailer POS data down to the SKU/UPC level through electronic data interchange transfers (an electronic communication method that provides standards for exchanging data via any electronic means). These frequent data points can be supplemented with syndicated scanner data across multiple channels (retail grocery, mass merchandiser, drug, wholesale club, liquor, and others) with minimal latency (1- to 2-week lag) by demographic market area, channel, key account (retail chain), brand, product group, product, and SKU/UPC. Downstream data are the closest source of consumer demand above any other data, including sales orders and shipments. Unfortunately, most companies primarily use downstream data in pockets to improve sales reporting, uncover consumer insights, measure their market mix performance, conduct price sensitivity analysis, and gauge sell through rates. Very few manufacturers, including consumer goods companies, however, have designed and implemented an end-to-end value supply chain network to fully utilize downstream data.

It is apparent that companies' supply chains have been slow to adopt downstream data, although POS data from retailers has been available for decades. Initially, the decision to use downstream data was a matter of data availability, storage, and processing. Today, this is primarily a question of change management due to corporate culture, particularly from a demand forecasting and planning perspective. So, what are the barriers? In large part, the adoption rate is slow because supply chain organizations have not approached the use of these new forms of downstream data from a holistic standpoint. Instead of mapping new processes based on a strong data and analytics strategy, supply chain organizations have tried to force-fit this data into existing processes using Excel, with no real analytics and a lot of manual manipulation. This is happening even though channel data are now available for anywhere from 50% to 70% of the retail channels, and the delivery time has improved significantly with data latency of 1–2 weeks. Given all these improvements, however, many companies still feel downstream data cannot be used for demand forecasting and planning.

Meanwhile, confronted with demand challenges due to the digital economy and COVID-19, companies have been looking for new ways to predict future demand. Under these changing conditions, the traditional demand planning technology has been ineffective at predicting shifting consumer demand patterns. The main reason for the unsuccessful performance of traditional technology is directly related to the fact that most companies are not forecasting demand, but rather supply—that is, shipments—which is proven to be much more volatile as a result of the digital economy, not to mention disruptions like COVID-19. Customer orders, sales invoices, and shipments are not true demand. They are the reflection of retailer and manufacturer replenishment policies—as seen in the bullwhip effect. To make matters more difficult, traditional demand forecasting processes and enabling technology were not designed to accommodate POS/syndicated scanner data, let alone the more sophisticated analytical methods (predictive and AI/machine learning) required to sense shifting consumer demand patterns and to shape future demand to create a more accurate demand response.

To reduce volatility and gain insights into current shifting consumer demand patterns, companies are turning to POS/syndicated scanner data, but are finding that using those data sources for demand forecasting and planning is complex. Perhaps it is because they are not familiar with downstream data, as most demand planners report into operations planning too far upstream removed from the customer/consumer, rather than downstream in sales and marketing. Another reason is that most companies do not collect downstream data on an ongoing basis over multiple years. They normally collect it on a rolling 104-week basis. This trend is no longer a barrier as companies are creating demand signal repositories (DSRs) to capture downstream data by week for more than 104 weeks, which allows them to compare downstream consumer demand to upstream shipments and/or sales orders.

Others will admit that what retailers sold last week, or even yesterday, is not necessarily a good predictor for what they will order next week or tomorrow. This is a myth, as companies have been keeping inventories low with more frequent replenishment because retailers are no longer allowing manufacturers to load up their warehouses at the end of every quarter with their products. As a result, there is a direct correlation with weekly POS/syndicated scanner data at the product level, and in many cases at the SKU/UPC levels (see Figures 3.5 and 3.6). As you can see, there is a strong correlation between weekly shipments and syndicated scanner data. Most consumer goods companies now capture their shipment data daily/weekly by channel and key account, down to the SKU warehouse distribution point.

Another barrier has been scalability. Many companies feel that adding downstream channel data to demand forecasting involves a magnitude of effort and complexity that is well beyond what most demand planners can handle to source, cleanse, and evaluate. However, new processing technology (e.g. parallel processing, grid processing, in-memory processing, and cloud computing) leveraging advanced analytics in a business hierarchy along with exception filtering has enabled several visionary companies to overcome these difficulties. The promise of downstream data can be realized with the right technology on a large scale using advanced analytics, as well as AI/machine learning.

The real barriers are internal corporate culture, lack of analytical skills, and the desire to take that first step. The best way to incentivize sales and marketing to participate in your Sales & Operations Planning (S&OP) and Integrated Business Planning (IBP) processes is to integrate downstream data into the demand forecasting and planning process. Without it there is no real reason for sales and marketing to participate. Downstream data are the sales and marketing organizations' “holy grail” to consumer demand. Furthermore, if demand planners were using POS/syndicated scanner data, they would have seen the shifting consumer demand patterns due to COVID-19 within the first or second week of the crisis.

Demand Management Data Challenges

Most demand forecasting and planning initiatives are abandoned or considered failures due in part to data quality challenges. The right data input has several important dimensions that need to be considered for success of any process. Harnessing the right data always appears to be straightforward and relatively simple; however, bad data, or use of the wrong data, often is the real reason behind process failure.

Almost all demand forecasting and planning systems use some form of statistical forecasting methods that require historical data. In most cases, companies choose to use shipment data, or sales orders data, to predict future product demand, as both are readily available and best understood by demand planners who are responsible for the demand planning process. According to recent research close to 70–80% of companies still use either shipments or customer orders data for demand forecasting and planning, which are their supply and replenishment signals, not their true demand signal. Unfortunately, product shipments data, as well as sales orders, contain several undesirable components, including incomplete or partial order fills, delivery delays, and retail channel load effects due to promotions, supply policies that do not always reflect true demand, and sales/marketing strategies designed to generate incremental consumer demand (e.g. sales promotions, in-store merchandising—displays, features, feature/display, temporary price reductions (TPRs) on shelf, and others). Consequently, shipments data represents how operations planning responded to customer demand (sales orders—replenishment), not consumer demand itself (POS/syndicated scanner data—consumer purchases selling through retailer cash registers).

Demand forecasting and planning systems must build plans off a forecast and use shipments data as a measure of effectiveness in meeting those plans. Sales orders from retailers less any customer returns are the next best data representing customer (retailer) demand, but not necessarily the best demand data input for the statistical forecasting process. The closest data to consumer demand is POS and/or syndicated scanner data. Although many companies collect and store POS/syndicated scanner data, less than 40% of companies use POS data for demand forecasting and less than 10% use syndicated scanner data, according to recent surveys. Everyone agrees that POS/syndicated scanner data is the closest data to true consumer demand, yet both these data streams are among the most underutilized for demand forecasting and planning. To make matters worse, roughly 70–80% of companies are using historical shipments “adjusted” for trend and seasonality cleansed of promotions and outliers, separating the data into historical baseline volume and promoted volume creating two separate data streams, which is a bad practice.

Demand-Signal Repositories

There are many challenges to implementing a consumption-based forecasting and planning process. Many of the data sources are new for most companies to manage, so ownership, expertise, and governance are important—not to mention change management requirements to not only gain adoption, but also governance to make it sustainable. Many executives say that their organizations are still in the basic stages of data integration due to data quality and internal lack of skills about how to maximize analytic benefits. In today's business environment, there are significant financial implications to ineffectively managing all the data streaming from multiple sources. These challenges can take the form of higher costs and lower revenue due to conditions such as ineffective demand planning, non-optimized inventory, strained product launches, and higher out-of-stocks on shelf.

Companies across a myriad of industries understand the importance of transforming into a data-driven company. The challenge is to gather, cleanse, and access the vast amount of downstream demand data, and normalize and harmonize it with upstream transactional information. Downstream data can be difficult to master and govern without a cohesive approach across the organization. Nevertheless, companies are investing in demand signal repositories as an integrated information hub that provides the foundation for breakaway analytics and optimization across the enterprise. Utilizing data to its fullest potential includes such sources as retailer POS; syndicated scanner sources (Nielsen, Information Resources [IRI], and others); loyalty programs; consumer panels; mobile, online, and social media; as well as ERP systems, finance, and internal systems—all of which are available to drive powerful analytics. Downstream consumption data are now more widespread, with many retailers sharing POS data with consumer goods companies daily. As a result, they expect enhanced knowledge from their consumer products suppliers. The demand-signal repository is a central data mart that houses all this information.

The successful implementation of a demand signal repository, supercharged by consumption-based analytics, is dependent upon managing POS/syndicated scanner data effectively, and complementing it with specific internal data, such as the company's product hierarchy. This exercise makes the POS/syndicated scanner data a more robust source of information to analyze with more dimensions that can be “sliced and diced” to gain more actionable insights. This type of data comes from internal corporate systems, local repositories, IoT, and spreadsheets. Furthermore, it is usually manually maintained and thus not subject to good data governance.

What Are Demand Signal Repositories?

A demand signal repository (DSR) is a data repository designed to integrate consumer demand data and leverage that data by consumer goods companies, automotive manufacturers, electronics manufacturers, pharmaceuticals, and others to service retailers and end user consumers efficiently. The focus has been on synchronizing POS/syndicated scanner data with internal shipment and replenishment data, which allows companies to provide business users with a more complete view of their retail performance. The DSR itself is a data repository that stores the information in a format that allows for easy retrieval so that users can quickly query the database to identify what's selling, where, when, and how. Identifying marketing opportunities, “demand performance,” and out-of-stock (OOS), along with control tower tracking and monitoring, are the key requirements for demand forecasting and planning. Leveraging demand signal analytics (DSA) data for demand forecasting and planning using predictive analytics is where the real benefits of such applications can help to identify and measure past, current, and future impacts on consumer demand. With the right architecture, DSRs will continue to grow with the business needs. They will be leveraged across multiple business groups, including demand management, channel marketing, supply chain management, inventory management, promotion, and event management.

Demand signal repositories are defined as centralized data repositories that store and harmonize attributes, and organize large volumes of demand data such as POS data, wholesaler data (electronic data interchange [EDI]), inventory movement, promotional data, and customer loyalty data for use by decision support technologies (channel marketing analysis, shopper insight analysis, demand forecasting and planning, inventory replenishment, and more). Demand signal visualization (DSV) dashboards provide companies with faster decisions and scenario modeling for outcomes and accurate decisions. Furthermore, demand signal analytics (DSA) combines DSV with predictive analytics, allowing companies in real time to conduct root-cause visualization and exploration. DSR, DSV, and DSA are at the heart of being analytics driven. To reap the maximum benefit from a true consumption-based planning process takes commitment and a well-conceived plan, which requires a best-in-class demand signal repository at the core.

Benefits of a Demand Signal Repository

The augmentation of DSR data into the demand planning process improves visibility and control. POS/syndicated scanner data can be a tremendous asset when used properly. By integrating POS/syndicated scanner data with company-specific attributes, consumer goods companies can leverage that data by collaborating more effectively across the organization and with their retailer (customer) networks. POS data can then drive commercial and operational improvements, such as:

• Improving demand forecast accuracy, and enhancing demand-sensing and shaping activities;
• Sensing shifting consumer demand patterns faster and more effectively;
• Improving evaluation of new product information via integration of sentiment analysis;
• Increasing trade promotion effectiveness;
• Reducing out-of-stocks; and
• Lowering inventory and safety stock levels.

Why Is It Important?

If you want to be more proactive than simply basing replenishment on shipments data, you need access to downstream data, analysis, and insights to make decisions that put you ahead of the demand curve. There is more to it than just forecasting trends and seasonality. Demand sensing is about identifying and measuring market signals, and then using those signals to shape future demand.

Using consumer data effectively requires making an investment in a demand signal repository to harmonize and cleanse POS/syndicated scanner data so that it is usable for data analytics. Consumption-based analytics using downstream data, as well as inventory, shipments, and replenishment data, are current examples of using “structured” data. While the term “downstream data” is most often connected to consumption and inventory data, “unstructured data”—such as loyalty data, social sentiment, consumer perception attitudinal data—is starting to be used for targeting consumers, shaping demand, and improving new product launch effectiveness. Using text analytics, we can now transform unstructured data into structured data that can be combined with the breadth of our data in predictive and machine learning models, and then visualized using control towers and dashboards to create value by enabling faster, more effective business decisions.

What Is Consumption-Based Analytics?

Consumption-based analytics uses the combination of visual analytics and predictive analytics to access the data in DSRs to uncover actionable insights with minimal latency. You can think of DSA as being composed of three layers. The foundational layer is a demand signal repository (DSR), an integrated database of essential data that companies need to provide insight into sales, marketing, inventory, price, demand performance, and operations. The DSR harmonizes, normalizes, and integrates the raw demand and supply data from any source (POS, wholesalers, mobile, online, social media, weather, EDI, inventory, syndicated scanner data, promotional/marketing information, customer loyalty data, and more). It works with any data type or source format from multiple retailers, distributors, and their respective disparate systems to make that data available for retrieval, query, reporting, alerts, and analysis.

In summary, the second layer uses visual analytics to transform the DSR data into DSV to allow for exploration, analysis, and insights that suggest areas of focus, improvement, and action. While typical marketing research only provides answers to predefined questions, DSV provides insight into questions companies didn't initially know to ask. The third layer combines DSR and DSV, creating DSA. The addition of predictive and anticipatory analytics complements the descriptive analytics of DSV, and quantifies the direction, size, scope, and variability of supply chain replenishment. Figure 3.7 illustrates a typical interactive visualization combined with predictive analytics, which allows the demand planner to review both demand and replenishment.

Chapter 4 discusses in depth how using consumption-based forecasting companies can predict shifting consumer demand patterns and their impact on supply by linking POS/syndicated scanner data to shipments using predictive analytics. Effectively adjusting to changing market conditions using consumer demand requires aligned strategies across a brand's personnel, processes, measurement, and tools, as well as a demand analyst (data scientist) working closely with product management and demand planning. Product managers provide insights into their sales/marketing programs, as well as the overall dynamics of the marketplace, while the demand analyst models consumption, providing marketing with what-if simulation capabilities that test marketing strategies to optimize marketing investment. The results help brands navigate the current volatile market environment caused by the COVID-19 crisis while driving sustained optimization over the long term. Once consumption has been forecasted and linked to shipments, the demand planner can begin to determine the actions needed to meet that demand signal from a supply planning perspective, creating a final constrained demand plan (supply plan).