Background

A study conducted by A.T. Kearney found that nearly 40 percent of retailers with loyalty programs fail to use all of the data to fully understand their customers.⁵ Even when they do, some retailers who gained that insight failed to ever translate it to action.

With changing times in an evolving market, this customer created an initiative to evaluate data mining tools and advanced analytics to meet the business needs. Specifically, the project focuses on the ability to analyze customers and household purchasing data at the granular level. The project has five goals:

1. Effectively manage and analyze the growing data volumes
   With the increasing number of stores opening and an expanding customer base, the volume of data being collected is growing at a much faster pace. The stores are also adding more merchandise to their shelves to attract buyers with national and private brands. In addition, they want to expand their loyalty customer program with specific offers and incentives for shopping at their stores. Not only has the volume of the data for their data warehouse increased, the types of data collected has grown as well. The size of their enterprise data warehouse has expanded by 25 percent. Thus, a new way of managing the volume growth is needed. In addition, the enterprise data warehouse system is overburdened with queries and analysis that it is taking the retailer longer than expected to execute its analytical processes and reporting needs. With the current infrastructure, the data are not managed effectively to enable a data-driven organization.
2. Streamline the analytic workflow
   The current analytic workflow is too complex. Out of the five goals, the customer is laser focused on this one because it is the most important area for the company. The current analytic approach is very elementary, and the retailer wants to widely adopt analytics across the enterprise to deliver data-driven decisions. This one-stop shop wants to evolve from basic reporting and analysis to advanced analytics that include data model development, deployment, and data visualization by using all of its data that reside in the enterprise data warehouse. The current analytics process is too cumbersome, and they can only analyze a subset of the data because of the analytic application. In addition, the analytic team consists of data scientists and statisticians who are spending too much time manipulating the data and not really doing the core of analytics. For example, it takes several days for the team to extract the data from the data warehouse to the user's personal computer for analysis. During this process, the data scientists and statisticians are having to preparing the data in a form that is compatible for analytics. Preparing the data includes standardizing the data sources, integrating all data sources for consistency, and formatting the data for analytical purposes. The analytical team is spending about 80 percent of its time dealing with these mundane tasks while it should be focusing on the analytics side. Transforming the business into a data-driven company with advanced analytics is the ultimate goal for this retailer.
3. Improve scalability and performance
   The amount of data stored in a database has a great impact on its performance. As mentioned earlier, the enterprise data warehouse is the system that stores all of the data for this customer. It is, however, no surprise that the queries get slower when the data volume grows. All analytics, reporting, and analysis rely on the enterprise data warehouse, and there are many users that touch this system through the day, weeks, and year. Since many of its stores operate 24 hours a day and 364 days in a year (with only Christmas off), the scalability and performance for the data analysis is an issue. Moving the data through the network from the data warehouse to their personal computer is a common practice that creates many headaches for the company. Not only does it create bottlenecks in the network but also analyses of the data can take weeks and days to complete. The personal computer has very limited memory and applications to do what is needed. Concurrent users are querying the data warehouse and constantly fighting for the same resources from the data warehouse. Now that the company wants to embark on advanced analytics, it will require more computations and data-intensive models to accurate analyze the customer base and loyalty program. The current infrastructure is struggling to scale and perform, and the executive team acknowledges that it needs to be improved if the company wants to adopt advanced analytics.
4. Enhance the need for data consistency
   It is common to expect that the data sources come from multiple systems. Once extracted, data should be integrated and loaded in the data warehouse. For this retailer, it is not a standard process. Staff members spend a lot of time extracting data from the warehouse and then manually manipulating it in an effort to meet the reporting and analytics requirements. However, the data source may not be structured to support current analytics and reporting needs. It may not be consistent and formatted in a way that it can be consumed by the business analyst or statistician. More often than not, the condition and granularity of the data are inadequate to support downstream applications if data are not proactively handled properly at the data source. Once the data are extracted and circulated throughout the organization, it is challenging to manage the considerable amount of complexity, and meaningful analyses and reports are eroded. In addition, this customer admits that data quality is not at the forefront and is likely at the back burner (as mentioned in Chapter 2). To enhance data integrity and consistency, data must be cleansed, standardized, and integrated with other data sources. During the data cleansing process, missing values are determined and bad data are corrected. Only after all of these processes are executed should the data be uploaded into the data warehouse to enhance the data consistency. Because this customer has endured the data quality and inconsistency issue, it has impacted the performance of the marketing and sales campaigns and wasted the organization's money, time, and resources.
5. Invest in human capital
   While working with this customer, this is an unexpected initiative that many companies tend to overlook because the bottom line and profitability tend to overshadow everything else. Many organizations say that their employees are their most strategic asset, but this company truly means it and stand behinds this initiative. The investment behind human capital extends across its thousands of employees. In its corporate office, IT and analytical team members were leaving for other companies. One of the reasons is that they were not really doing what they were hired to do. The IT team was not really focused on strategic tasks—instead, it was maintaining the ailing infrastructure and outdated systems. More attrition was endured from their analytical team. The staff spent more time doing data preparation and manipulation than analytics. These data scientists and statisticians were not applying their skills to develop and fine-tune the data models. Instead, they were forced to format the data sets, scrub the data, and prepare the data. These tedious and boring tasks were wearing them out, to the point that they wanted more exciting careers elsewhere. This effect trickled down to the business analysts, marketing, and other departments because many of their marketing and sales campaigns were ineffective. Losing good talent and personnel to competitors and other local businesses left a big void at this retailer. It was suffering from having to hire new employees and train them on what needed to be done. The executive sponsor for this analytical project decided that it was time to invest in human capital to keep employees challenged and on board with the company so that they could execute its long-term plans. The analytical team got more involved with the IT architecture so that whatever tools and technology the company adopted, it would meet their needs to support the company's vision. The transformation from just analytics and reporting is now shifted to more predictive analytics for this company.

Overview

The competitive nature of the business and the evolution of the retail industry have directed this customer to expand its reliance and use of analytics for insights that provide a competitive edge.

As the company yearns to adopt advanced analytics with the focus of predictive analytics, it needed a new, modern architecture and innovative technologies to support it. The process and journey led to the adoption of in-database and in-memory processing. It has been several years of evaluating and exploring different technologies and vendors.

In the traditional environment, all of the business analysts, data scientists, and statisticians have to do all of their analytics at their desk. There are a number of challenges with this process:

• Moving or copying the data from the data warehouse to the desktop can be very time consuming.
• Only a subset or small data set can be analyzed due to the capacity of the desktop.
• Everyone has a copy of the data that persists on desktops, which creates data silos.
• The analytical applications that are available on the desktops have limited functionality.
• Data models cannot be developed and scored in a timely manner.

Figure 5.9 shows the process. As illustrated, there are many steps that consume both time and resources. The customer can only transfer a sample of the data set from the data warehouse to the desktop to explore, develop models, and score. By analyzing a sample of the data, it does give you the big, holistic picture of the customer data and profile. In addition, data-driven decisions are executed partially due to the analysis from a small segment. Having a partial view of their customer data is not effective when it comes to executing marketing and sales campaigns of their customer loyalty program. With this current architecture, the team can only develop less than five data models and score them per week. This customer wants to expand its analytical capabilities to develop and score tens of models and eventually increase to hundreds of models.

New Architecture with In-Database and In-Memory

By working closely with the customer and understanding their goals, the direction was to explore and adopt a combination of in-database and in-memory technologies, as shown in Figure 5.10.

The shift for this organization is going from just doing analytics to more descriptive and predictive analytics. This retailer wants to leverage descriptive analytics to answer questions such as:

• What happened?
• How many, how often? And where?
• What exactly is the problem?
• What is happening?

Descriptive analytics assess the past performance and comprehend that performance by mining the historical data to look for the reasons behind past success or failure of a decision. In this case, this one-stop shop wants to use descriptive analytics for business and management reporting—such as sales, marketing, operations, and finance.

Predictive analytics takes a leap beyond descriptive analytics. As the name suggests, it is an attempt to forecast an outcome based on the data analysis. It is using a combination of historical performances from descriptive analytics with rules, algorithm, and various external data sources to determine the probable outcome of an event or the likelihood of a situation going to occur. Like descriptive analytics, organizations apply predictive analytics to answer complex questions such as:

• Why is this happening?
• What if these trends continue?
• What will happen next?
• What's the best that can happen?

Predictive analytics is more sophisticated and requires involvement from statisticians and/or data scientists who can develop algorithms or data models to answer such difficult questions. As previously mentioned, this customer has dabbled in mode development but with limited scope and data to truly call it predictive analytics.

With that objective in mind, the customer wanted an analytical tool that is capable of delivering descriptive and predictive analytics and well integrated with the data warehouse. In Figure 5.10, it got just that. In this success story, the details will be around a specific focus, which is their loyalty program.

Most retailers, especially grocers, offer loyalty programs. Any loyalty program generates more data and, it hopes, more insights from consumers. The loyalty programs provide a win-win and reward both for the retailer (data) and consumers (in-store discounts). However, it is all about translating this data to understand “individual” purchase behaviors and behaviors outside of the purchase to gain the true meaning of the word loyalty.

Let's examine the process in detail to see how this retailer uses advanced analytics to improve its loyalty program by providing personalized offers and rewards to their customers:

• For the loyalty program from this retailer, the data sources going into the enterprise data warehouse include customer relationship management system, market basket, product, and group data. For example, customer data include first name, last name, email address, zip code, and phone number. Product data span a wide range because this is a one-stop shop where customers can buy literally anything they need from food to electronics to sporting goods. As this retailer expands, the data continue to grow in volumes and variety.
• As the data are collected and get loaded in the data warehouse, the data preparation process begins. A combination of in-database functions are applied as well as the traditional methods. Because not all the data processes can be enabled in-database, this retailer needs to maintain some of its traditional processes and complements them with what functions are available in-database. Let's focus on what this customer does in-database to prepare the data for analytics as illustrated in Figure 5.10, step 2.

When preparing the data for analytics, the analysts do the following:

• Sort the data: For example, you can sort the data by products or customers.
• Summarize the data: For example, summarize the number of customers shopping at a specific store.
• Rank the product: For example, rank the top five products sold from the weekly ad.
• Standardize the data: For example, standardize the first and last name of the customer by removing variations.
• Determine the gender: For example, classify the gender of the customer in categories, such as female, male, and other.
• Analyze the frequency: For example, deliver a count of coupons used per trip to the store.
• Deliver the right format: For example, ensure that the data set is in the right format for analytics.

The end result of this data preparation process provides the statistician and data scientist an analytical data set (ADS). Refer to Chapter 2 for more details about ADS. The streamlined process with the new architecture from Figure 5.10 shows the technologies adopted and what tasks are performed with the complemented and integrated environment.

• Data exploration (as shown in Figure 5.10, step 3) is executed in-memory using a data visualization tool. This visualization tool helps to further explore the data set for any correlation or relationship between variables. For example, the visualization tool can provide a graphical depiction of average money spent per trip by customer and store. This retailer explores the data in many ways:
  • Examine the most busy stores and peak hours.
  • Show what offers are used by customer.
  • Determine the number of trips made by a customer at a specific store.
  • Uncover what store generates the most revenue.
  • Reveal what products need to be stocked.

  In this scenario, data exploration allows the business analysts to keep what data they want to include in the data model development process.

• Model development (as shown in Figure 5.10, step 4) is the most critical process in this architecture. It is the science that delivers critical results for this retailer. The data model is used to predict the likelihood that a “household” will make a purchase within a specified sub-category/brand combination. This requires complex algorithm and many data points to support the model. There are two types of data model:
  • Current customer model: Retain customers in the loyalty program.
  • Potential buyer model: Expand market basket size by incenting customers to purchase in sub-categories they normally do not purchase within.

  The data development is executed on a dedicated analytics appliance. This appliance's primary purpose is to develop complex data models with large volumes of data and many variables. With a dedicated analytics appliance, this retailer is able to develop many more data models compared to the traditional architecture. Currently, it is producing hundreds of models for its loyalty program. All of the power inside this appliance is dedicated to processing the analytics. The advantage of this setup is that it alleviates some of the burden from the enterprise data warehouse because the model development process is on a separate hardware. The team of statisticians and data scientists is able to test their hypothesis and instantly know whether the data models are working or failing. It is a way to easily test a combination of variables and data.

  Compared to how the analytical data models were developed, it was taking days and weeks to complete due to the cumbersome process. Now that it is greatly streamlined, model development only takes minutes and hours to complete, depending on the complexity of the model. In addition, this architecture enables a highly governed environment because all of the data used to build the model come from the data warehouse.

• Once the model development is complete, the model can be scored inside the data warehouse (as shown in Figure 5.10, step 5). For this retailer, the results from scoring the model provide:
  • Personalized offers for a specific customer. For instance, customer A is likely to use a coupon for diapers based on past purchases of baby items.
  • Rewards for their loyalty customer for meeting certain conditions. For example, customer B receives a reward if the total purchase is over a certain amount based on five previous trips.
  • Predictions for the propensity that a customer responds to a specific campaign by purchasing a product solicited in a marketing campaign. In this case,
    • The customer has shopped and bought merchandise identified as solicited items within the allocated response period OR
    • The customer has shopped and bought any items within the allocated response period.
• Predict those customers that are likely to respond to a specific campaign regardless of the season such as summer or winter. Thus, the score will select a population to receive the winter campaign since the holiday season is the busiest season for this retailer.

The ability to execute many of the functions in the database and have a dedicated analytic appliance has truly streamlined the end-to-end process for this retailer.

Summary of Benefits and Future

• Achieve faster and more robust analytic insights in less time.
• Accommodate additional workload with existing staff.
• Become more strategic instead of tactical with analytics.
• Execute models within minutes and hours versus days and weeks.
• Develop data models much quicker against large data sets.
• Enable fail-fast capability of the data models.

This retailer has made significant human investment. The plan is to hire more data scientists and statisticians to expand the support for enterprise analytics. With the expanded staff, it intends to develop many more data models, beyond hundreds of models to ultimately deliver superior customer experience through its loyalty program.

Background

Information systems for governance and public administration have been a high-priority concern in almost all countries. Governance and public administration cannot be productive, effective, and efficient without the support of modern data management and analytic technology. The use of analytics to analyze data and distribute information has been an absolute requirement for public administration and management development. Governments tend to partner with local agencies to provide advisory services and technical assistance in this important area. In practice, however, effective use of information technology and successful development of analytical systems in public sector are not simple. This is substantiated by the numerous government digitization and information systems projects that have failed across the world.

The tremendous speed with which information technology is changing causes misperception, delays, and false expectations for these solutions by many decision makers and government officers. Like the private sector, the public sector has similar issues:

• Confusions about the capabilities and limitations of technologies
• Incorrect development strategies
• Vague development priorities
• Insufficient resource planning
• Inadequate cost estimates
• Misinterpreted user needs
• Selection of wrong products

All of the above often cause information projects to fail. At the heart of the failure is the combination of the technical and management side of the government. Understanding what is the purpose of a government information system and how to develop it successfully is fundamental to government decision makers and managers responsible for their critical areas. The critical issues include: appreciation and understanding of the advanced analytics and data management, awareness of the trends of modern information technologies and their impacts on development strategies, and cognizance of government information infrastructure and information systems development. Similar to the private sector, the public sector is recognizing the value and role of promoting information technology to be used in public administration. In particular, analyzing data to detect fraud is a major focus in the public sector, particular for government agencies to reduce costs and waste of taxpayer's money.

Government programs and operations are vulnerable to a wide variety of criminal activities. Fraud has been a major issue if left unchecked and undetected and can seriously undermine the public's perception and confidence in their government. Taxpayers rightly mandate that public officials wisely use and preserve the money and power entrusted in their government. Social benefit programs such as health care, government loans, and grant programs are all exposed to false claims fraud. Individual acts of fraud and organized ones all add up to huge losses for the government.

Unfortunately, government officials and investigators are largely in reactive mode, chasing the money and the fraudsters after the act has been executed. The need for governments to do a much better job in protecting the integrity of these social programs is essential by proactively stopping the frauds before the money exits the account and before they become widespread and organized. Too often, too many programs are constrained by antiquated or archaic information technology systems. Modernization of these systems should include sophisticated data management and analytics that can enable better, faster detection, and fraud prevention controls to regain the public's faith and trust.

Overview

This customer is an international information and communication technology company for the ministry of economy and finance located in Europe. With more than 30 years of service, it is committed to ensuring the modernization of the public sector by simplifying administrative procedures and increasing integration between public administrations. This IT company promotes the development and innovation through technology to ensure unity and effectiveness of strategic data-driven actions for the government. With thousands of employees, it has designed and built information systems for taxation, self-service analytics, and applications for government departments. This company has cutting-edge technology infrastructure and network to support nearly 100,000 workstations and direct connection with external entities, citizens, businesses, and professionals. Through their efforts, it has created advanced decision-making tools supporting economic and financial policy from a centralized but complex database system. This innovative company cooperates with its customers to ensure high standards of quality and safety and invests heavily in technology, training, and research and development to prepare for the digital age of big data. Its focus in data management and analytics delivers concrete opportunities for growth and rationalization of spending to enhance efficiency of the public information systems.

The customer collaborates with the local authorities for assistance in developing a system to better identify where, when, and how fraud occurs. Working side-by-side, the agency identifies the issues and challenges with the current infrastructure:

1. Need of insights and timely responses
   The government agencies are not getting information that yield to insights in order to respond to critical issues in the tax office, customs office, state property, and minister of finance in a timely manner. Much of their responses to prevent fraud, for example, are weeks if not months after the fraudulent act is complete.
2. Lack of predictive analytics and self-service capabilities
   Business analysts and power users of analytics are limited to what they can analyze. The current tools and technology allow them to analyze only what has happened, and they need the tools and technology to analyze what is likely to happen. Predictive analytics strengthen the ability to detect fraud. In addition, the government agencies lack the self-service capabilities to run reports and share information across different organizations.
3. Unable to detect anomalies in transactions
   The first step to detecting fraud is the ability to report anomalies or strange behavior of the person or business applying for the relevant service or benefit. This process often involves collecting and analyzing the data effectively by using the right tools and technology to alert officials of suspicious claims such as social care, improper health benefits, housing entitlement, and so on.
4. Dynamically explore large volumes of data
   There are many data points to consider when analyzing for fraud that amalgamate to large volumes of data. Considering a sample or partial data sets, which provide a limited perspective of the activities, will not give you a holistic view of the transactions. The ability to explore data dynamically and generate a series of reports will help to find, verify, and investigate people or businesses before fraud can occur.
5. Centralize analytics and reporting across agencies
   Standardizing and centralizing analytics can deliver consistent views of information. It is often an unfortunate case where different analytics and reporting tools are used from silo data and create inconsistent results. Preventing fraud requires a robust process that delivers reliable and dependable information and will alleviate budgetary and resource pressures within the public sector.

Let's examine how this international IT company is helping the local government to improve its process with cutting-edge technology and innovation.

Driving Innovation with In-Memory Analytics, Hadoop, and Enterprise Data Warehouse

The goal of the IT company is to promote and consolidate a culture of transparency and integrity for the public administration. By understanding their needs, it is architecting a system to conduct activities to improve performance, economics, and governance by identifying the areas that are most at risk and providing support for acquiring a growing awareness around fraud. This company is taking strong measures to prevent and repress criminal acts in the public administration around fraudulent activities. The development of advanced technological solutions will help to achieve its goal (see Figure 5.11).

Data sources for this government agency are structured, more on the traditional side with rows and columns. The data warehouse is the definitive place for all of the data. The enterprise data warehouse has been developed with all kinds of data for decades. Data in the warehouse include census information, household, income, tax collection, and health status, for example. Instead of placing all of the analytical functions and processes on the data warehouse, it has adopted Hadoop to help alleviate some of the workload and performance augmentation. During peak hours of operations, hundreds of users are accessing the data warehouse for various queries and analysis. Hadoop in this scenario is used to extend the value and purpose of the data warehouse for storing data used for fraud-specific analysis. The data from the data warehouse are copied into Hadoop to prepare for in-memory analytics data exploration. This is where predictive analytics come into play. The data are lifted into memory to do the following:

• Provide hundreds of users technology to quickly explore and visualize data for trends and deeper analysis.
• Identify anomalies, irregularities, and fraud in health, gaming, and tax declarations for corporations and individuals.
• Generate reports on fiscal activities, tax collection, health care economics, billing, and use of IT resources by applying predictive and descriptive analytics.

Selecting the right technology and consulting services to integrate data management and analytics with the enterprise data warehouse, Hadoop and in-memory data exploration have allowed the public administration process to be streamlined and effective. The agency is now able to capture anomalies for fraud detection. It is able to do a cost comparison between generic drug and standard product for best and worst behavioral analysis to improve the health care program. It has developed an external web application for all types of users within the government agency to gather more information and define an outlier or anomaly to identify housing tenancy fraud, procurement fraud, payroll fraud, and pension fraud. More applications are being built as they adopt a vision for deeper analytical approach.

Best Practices and Lessons Learned

When the government was considering options in consulting services and technology, a proof of concept was conducted using the resources from the agency. It also leveraged the agency's network and expertise to advise on a standard practice and technology for data management and analytics. The need to centralize and standardize the process and analytics help the hundreds of users to look at the same data and decipher the same results across the government departments. The proof of concept consisted of hardware, software, and services.

Once the government selected the IT company to develop the architecture with bundled software, hardware, and services, it quickly learned that it needed to build the process from the ground up. Many of the users across the government agency were not used to exploring and visualizing the data that they had. They simply took what was there and applied basic analytics to it. Of course, it was often not integrated and/or had missing values in the data to make the analysis effective. In recent years, many customers have leveraged the data exploration process to summarize the characteristics of the data and extract knowledge from the data. They often use newer technologies such as data visualization tools with easy-to-use interfaces at their fingertips to quickly reveal trends and correlations. Visualization tools not only offer users the flexibility of the click-drag-drop user interface, they also have analytical functions for deep analysis of the data.

Building a strong foundation is key. Once you know what data you have to work with, you can use it to build a strong footing for any applications, such as fraud. Basic fraud-detection rules were needed to build an effective system, and the old infrastructure and process did not have it. A comprehensive fraud solution is complex, and many business rules need to be considered to detect the anomalies. Focus on a specific fraud case such as benefit overpayment so that you can show success in one area before expanding.

Let's examine another success story with all three components: in-database, in-memory, and Hadoop centered on an enterprise data warehouse.

Background

The railroad is a critical element that fuels the economy. No technology, grain for food, or automobile is worth a thing if it cannot move from the manufacturing floor to the consumer's hand. According to the most recent statistic from Commodity Flow Survey, railroads carry more than 40 percent of freight volume in the United States—more than any other mode of transportation—and provide the most fuel- and cost-efficient means for moving freight over land. To support this finding, this customer is able to move a ton of freight almost 500 miles on a single gallon of fuel. This is possible thanks to the technological advancements and innovative sensors in locomotives.

Today, the U.S. freight rail network is widely considered one of the most dynamic transportation systems in the world. The multibillion-dollar industry consists of 140,000 rail miles operated by national, regional, and local railroads. Not only does the railroad system move more freight and goods nationwide, it also offers numerous public benefits including reductions in road congestion, highway fatalities, fuel consumption and greenhouse gasses, logistics costs, and public infrastructure maintenance costs.

The U.S. freight railroads are private organizations that are responsible for their own maintenance and improvement projects. Compared with other major industries, they invest one of the highest percentages of revenues to maintain and add capacity to their system with technological advances. In general, bulk freight such as grain and coal ships in rail cars, while consumer goods such as products found at a neighborhood grocery or department store ship in containers or trailers called intermodal traffic. Intermodal traffic refers to the transport of goods on trains before and/or after transfers from other modes of transportation, such as planes, vessels, or trucks.

Almost anything can be shipped by rail. Bulk commodities products such as agriculture and energy products, automobiles and it components, construction materials, chemicals, coal, equipment, food, metals, minerals, and pulp and paper make up over 90 percent of the rail freight business. Intermodal traffic is less than 10 percent, which consists of consumer goods and other miscellaneous products. Compared to other modes of transportation, the rail option offers many benefits. It is efficient at moving heavy freight over long distances. While the trucking industry excels in providing time-sensitive delivery services for high-value goods being transported over medium- and short-haul distances, the rail industry delivers goods and commodities across the country. Raw materials and heavy freight going long distances are likely to continue their journey by rail, or some combination of truck, rail, and water. With the future growth in freight, it is anticipated that freight rail will continue to make investments in technology to increase capacity for long-distance shipments.

Similar to other industries, the rail sector is going through some significant changes and challenges to meet customers' demands. This company wants to offer multiple transportation solutions in a constant effort to be competitive. One area where it sees growth is intermodal transportation, which allows for containers consisting of all types of goods to be placed on rail and redeployed across the country to facilities where they are transferred to other forms of transportations (such as truck, ship, airplanes, etc.) to their final destination. The efficiencies gained from intermodal transportation are even greater with the emergence of double-stacked containers on railcars. With improvements in service and facilities, rail intermodal will become more competitive and absorb the projected increases in freight movement caused by population growth and the growth of the intermodal movement of goods into the future.

To compete in the global marketplace and to enhance quality of life, this customer has made significant investments in technology specifically in analytics.

Use of Advanced and Predictive Analytics

Preparing for growth and the future, this rail company is using advanced analytics to help drive data-driven decisions made by management and the leadership team. With a network of thousands of miles of rail tracks and tens of thousands of locomotive movements to transport millions of containers and products, there is an enormous amount of data to analyze and make sense of data. Advanced analytics can transform data into insights that the business leaders can use to determine how they can increase efficiency and expand the business.

Advanced analytics have become an essential and integral part of the business in the decision-making process. This company is using predictive analytics to uncover the most efficient and safest routes for their trains, adequately staff train crews, proactively order parts and equipment, inspect the trains for preventive maintenance, and so much more. Because the railroad operations can be very complex with so many moving parts, this freight company uses all its data and applies the science to help solve problems. It is also leveraging data to anticipate the changing needs of the business and adhere to government regulations. Another use of advanced analytics is analyzing the shipments of merchandise trains. These trains carry containers of mixed commodities such as lumber, paper, machinery, industrial parts, and various types of chemicals used in manufacturing. By using advanced analytics, the company is able to facilitate efficient movements of these shipments by grouping them based on their final destinations to minimize costs and sorting. In addition, it is able to improve the flow of the products, reduce the amount of time it takes to its final destination, and enhance their customer experience.

Analytics is also used in crew assignment and planning. Like any industry, labor costs are the biggest expense and it continues to grow. For many years, crew planning has been a manual task that takes a lot of time, energy, and resources. Planning and assigning a train crew is a complicated process that involves the consideration of many factors in a short amount of time. The train crew planner has to assign crews located across a large geography in the most efficient and cost-effective manner possible so trains are not delayed or canceled by any means.

This task becomes even more complicated with all the rules and regulations imposed by the government to observe and maintain safe rail operations. An analytical algorithm was developed to assign crew planning in real time in an effort to reduce the overall cost of positioning crews to operate trains within a territory. Once the appropriate resource is assigned, it is also using predictive analytics to accurately predict the arrival times of its trains. This information is shared with customers to track their shipments of products in real time. Customers can now know when the train has left the station, when it is in route, and when it will arrive to the final destination.

Timing is everything. In an industry of transporting goods, their customers are heavily relying on the railroad to get their shipments on time so that they can provide these products to us, the consumers.

Big Data Architecture with In-Memory Analytics and Hadoop

Even though this company is already using advanced and predictive analytics for many of its operations, it is continuously exploring innovative technologies and approaches to analyzing new data sources. Managing big data is nothing new for this company. It has been analyzing large amounts of data for years and relies on advanced and predictive analytics to strengthen data-driven decisions. For example, this company is gathering months of weather data and correlating it to track conditions and/or to track volume to forecast expected conditions. It also leverages video data sources to assess any number of conditions, such as missing or misplaced signs or vegetation growth to alert its field teams where to focus its efforts. To meet regulations or pursue its own continuous improvement objectives, many railroads are combining technology, processes, and shipper training—along with expertise from field-based damage prevention managers—to reduce cargo damage or loss. Rail safety begins with the quality of the underlying rail infrastructure. Maintaining and investing in tracks and equipment is key to preventing accidents. As they move more and more to real-time collection and analysis of critical data, big data technologies allow us to be even more responsive to our business and, in turn, our customers.

In today's data-diverse environment, everything in the rail industry can create an electronic trail of data about its operations and logistics. The analytics team utilizes the mounds of data to generate better inputs for its complex data models and improve its daily operations. New data sources come from sensors that monitor and assist in determining when would be the best time to schedule railcars and locomotives for inspections and maintenance. In addition, it uses data from GPS tools to help track the exact location of their assets. These data sources are detail-rich in nature and can help the company to create effective solutions and positively impact the profitability and its bottom line. Figure 5.12 illustrates the new architecture to support additional data sources.

In this scenario, this customer has invested in Hadoop, in-database, and in-memory technologies to modernized its architecture for the twenty-first century.

1. The data has two paths: Depending on the type, the data can go into the enterprise data warehouse if it is structured data. If it is semi-structured data such as detectors and sensors, they go into Hadoop that acts as a data lake to collect a sample of the following sources of data:
   • Wayside detectors: These sensors spot defects such as overheated bearings and damaged wheels, dragging hoses, deteriorating bearings, cracked wheels, and excessively high or wide load on passing rail cars.
   • Acoustic detector systems: These trackside systems use acoustic signatures to gauge the sound and condition of internal bearings and identify those nearing failure. They supplement or replace systems that measure the heat that bearings generate to identify those in the process of deteriorating.
   • Track geometry cars: These technology-rich cars utilize sophis-ticated electronic and optical devices to inspect track alignment, gauge, curvature, and so on. A new vehicle track interaction system is also used to locate difficult-to-find track geometry defects. This information helps railroads determine when track needs maintenance.
   • Ground-penetrating radar: This technology assists in isolating below-ground problems—such as excessive water penetration that hampers track stability.
2. If the data go directly into the data warehouse, the in-database process is used to prepare the data for analytics such as joining, merging, and creating the right format. In addition, this customer is taking advantage of the massively parallel processing of the data warehouse to enable bulk load, update, and export large data sets.
3. The data development process combines the data from Hadoop and data warehouse. The data models consist of data from detectors, sensors, shipments, routes, time, crew, and so on. Statisticians look at many variables and situations to develop a comprehensive view of the business. Much of the focus is on detectors and sensors since this provides real-time perspective of the operations.
4. In this situation, scoring the data and models is taking place in Hadoop. Scoring of sensors and detectors data creates alerts for the crews to proactively react to scenarios that can save the company money for outages and issues.

The rail and freight industry uses detectors and sensors to help to improve safety and service, and it is an area of technology with significant investment with high expectation on return of investment. These detectors and sensors generate a lot of data for this company. Detectors are installed under, alongside, above, and attached to the track to find defects in passing equipment, which adds another layer of safety to prevent unplanned incidents. The primary purpose of these detectors is to see, hear, or feel locomotives and railcars as they pass and determine if there are defects. Detector technologies also look for missing, damaged, or defective components from wheels to bearings.

These detectors are generating large data sets daily. There are five types of detection technology, and this company has more than 2,000 detectors collect data 24 hours a day and seven days a week across the system. The data collected from these detectors are analyzed along with other data sources, and the analytics team takes these inputs to develop a data model for predicting patterns and the likelihood of an issue happening to one of its trains. The analysis will help to tweak procedures and business rules criteria for operational improvements. In conjunction with detectors, sensors are the cutting-edge technology that produces many important data points to better manage and identify potential track and equipment defects. By integrating with existing and other types of data, the analysis can provide crucial and critical information about the rail network. By leveraging predictive analytics patterns can be revealed and derailments and outages can be prevented.

Collecting and analyzing detector data is another step toward proactive rather than reactive maintenance. Of course, being proactive can tremendously decrease the unplanned set outs of equipment that can impact velocity and improve safety of responders without increasing risk of derailment. The data from these detectors can augment manual inspections. The inspectors now have additional data points on what to look for and where to look. Ultimately, these detectors focus on derailment prevention and improved safety for the employees and our communities.

The data from the detectors and sensor are helping this company to manage its network to be more on the proactive stance instead of on the reactive position. In addition, it collects a vast array of fixed and mobile sensors to get a real-time view of its assets' condition and usage. Condition-based maintenance and engineering inspection systems will drive reliability in track and equipment, further optimizing maintenance practices. With advanced analytics by using in-memory technology, they will have the ability to model and predict the network impact of day-to-day decisions such as holding or rerouting a train.

The setup in Figure 5.12 serves many purposes for this organization and its customers. For the organization, it will be able to get the information from wherever with any device that includes dashboard and mobile devices. Manual inputs have been replaced with automation. Carrying paper documents and using outdated information is a thing of the past. Documents are automated, smart, and accessible to data and information from any time, anywhere, and any place. Instead of entering lots of data, information is distributed to line of business, managers, and executives. Systems will provide insight to what tomorrow will look like by integrating many different sources for more precise and accurate day-to-day decision making. Touch-, speech-, and gesture-driven applications are in use to better manage the operations. The result of all of this effort behind predictive analytics is that our network should hit new records for safety, velocity, and on-time performance. Working with its customers and partners, it continues to work not only hard but also smart to provide a safe, reliable freight delivery service for our customers.

In addition to organizational benefits, the customers of this freight company are also reaping advantages from predictive analytics. For the customers, data from GPS systems can be analyzed to identify real-time shipment status and more accurate estimated time of arrival. The end goal is to capture customers' needs and proactively know any potential concerns and issues as well as opportunities to enhance customer experience.