7. Applying Analytics at Production Scale

James Taylor

One of the most powerful uses of data mining and predictive analytics is to apply these techniques to operational, transactional systems. This means applying analytic models and results on a production scale—to all customers, for all products, and embedded in production transactions. Organizations that succeed in applying analytics at such a scale see tremendous results. The benefit of analytics is applied to every one of a large number of transactions for a powerful multiplicative effect. To make this possible, analytics must be augmented with a family of technologies called automated decision systems or decision management systems.

Some industries use data mining and predictive analytics widely in production systems, but most do not. Industries that use the credit risk of an individual consumer in their decision-making are the most well-established. Companies such as credit card issuers and mobile telecom service providers routinely automate decisions that rely on sophisticated predictive analytic models to predict consumer credit risk. In addition, a growing number of companies are using propensity and profitability models in marketing and customer treatment.

In all these cases, the analytic models are applied to critical decisions within a business process. The risk models are applied to help decide which products can be profitably sold to a particular consumer as part of the origination or onboarding process. The propensity models are applied to improve decisions such as what to cross-sell or up-sell a particular consumer during the customer service process. What they have in common is a focus on decisions within these processes and on the use of analytical insights to determine how best to make these decisions automatically, quickly, and accurately.

To succeed at using analytics in this way, however, organizations must address technology and organizational challenges that otherwise can derail these attempts. In particular, they must deal with organizational and attitudinal differences between business units, information technology (IT) departments, and analytics experts, as well as challenges involving compliance, time to deploy, and data consistency. These issues are described in this chapter. A detailed example of a company that has applied analytics successfully at production scale is then given.

Decisions Involve Actions

Data mining and predictive analytic techniques deliver insight for decisions. But decision-making also involves action: Making a decision involves making a commitment to taking a specific action. For analytical insight to improve decisions, it must result in the selection of a more profitable or otherwise “better” action. When analytical insight is delivered to human decision-makers, the expertise and know-how of those decision-makers can determine the appropriate action(s) from those available. When analytics are applied in a high-volume, low-latency production environment, typically no decision-maker is available. If humans are involved, they are unlikely to be qualified to make the decision; high-volume processes rarely employ experts, and those involved are likely to be junior staff. Even when humans are involved, they are there to deal with exceptions rather than routine decisions.

In either case, there is a need to formally define the available actions and to define how to select from among those actions so that the system can make the decision. The decision the system makes must use the predictions derived analytically, but it must also apply regulations, best practices, tribal knowledge, and company policies. For the decision to be made quickly and accurately, it must combine the analytic insight with the business rules—typically in an if/then format—that represent these policies and practices. Analytical insight alone is not enough.

Time to Business Impact

Analytical models, by their nature, tend to degrade over time. Analytical techniques typically are applied to a snapshot or extract of data from an organization’s systems. Analyzing this data results in analytical insight, such as a prediction. When the prediction is made, it is accurate as of the time the data was extracted. When the delivery of an analytical model to a human decision-maker simply involves telling the person what the model said, there is little additional decay in the quality or accuracy of the model. When the model must be embedded into a production system, many additional steps might be necessary:

• Datasets used to build the analytical model may not match the data available in the production system, and this must be reconciled.

• The model must be applied to a whole series of transactions, often in real time. Therefore, the model’s structure must be coded into an IT component that will execute the model against each transaction.

• As mentioned, the actions to be taken based on the model must be defined and likewise coded into an IT component. These are likely to change every time the model is updated, and it is common for them to change much more often than that.

• Although the model may already have been tested analytically, after it is deployed into an information system, it must be tested as an IT component.

All these steps take time. If they take too long, the model’s accuracy and value will degrade before it can be put into production. In fact, if this process is difficult and too costly, the model will not make it into production and will never be used. In a recent survey I conducted, over 40% of organizations took more than six months to deploy models. Given that six months is also a typical time frame for updating a model, this implies that 40% of organizations are not getting a model into production before the analytical team would want to replace it with a new, more accurate model. Therefore, the time taken to deploy analytic models into production is critical.

In addition, the environment must be constantly monitored to ensure that the distribution and quality of data processed is as expected by the model and that the model’s effectiveness does not degrade over time. Constant updates to the model are to be expected. If deploying a model is time-consuming and costly, updates will be impossible.

Business Decisions in Operation

When analytics are deployed into operational systems, organizational challenges arise. Anyone seeking to use analytics to improve decision-making must address the different ways in which business and analytical people think and talk. For example, an analytic model that cannot produce better business results may be interesting, but it is not useful. When analytic models are embedded in information systems, the IT department also must be able to participate.

This means more than just explaining the model. In the case of business people, the model must predict something they can use. For instance, predicting which consumers are a churn risk seven days before their contract expires is not useful in a business context where churning customers typically pick their new service provider 14 days before their contract expires.

In the case of IT people, the practical implications of implementing a model in the organization’s IT environment must be taken into account. A model that is more accurate but requires more data sources or program interfaces—and therefore takes more time and money to implement—may be less valuable than a less-accurate model that can easily and quickly be put to work. Successful use of analytical insight in operational systems requires close and frequent cooperation between analytics, business, and IT people as models are being developed.

Compliance Issues

Because many of the decisions where analytics play a role in production systems and processes impact consumers, regulatory compliance is a real issue. Many decisions about consumers are regulated, often at multiple levels. This is especially true when it comes to applying risk models or when making pricing or eligibility decisions based, in part, on analytical insights. When a decision is regulated, the analytical team needs to consider how the model will be explained to regulators. Saying that a consumer was rejected or charged a higher price “because the model said so” is unlikely to be acceptable. Decisions with compliance issues need models that have explicable results. As a result, the use of techniques such as decision trees, association rules, and additive scorecards may be preferred over more opaque “black box” techniques, such as neural networks and machine learning.

Data Considerations

As noted, data issues arise when models are deployed into production around mapping the data available in production to the analytical datasets used to develop the models. This is not the only data issue that needs to be addressed, however. In addition, the three groups involved in implementing analytical production environments (business, IT, and analytics) all have quite different perspectives on data. IT departments typically think in terms of object models, of storing and managing data. Business users, meanwhile, see their data through the reports and dashboards they use to view it, often considering historical data only in terms of monthly or quarterly roll-ups. Analytical people tend to work with flat-file datasets and rarely use summary data, usually preferring raw transaction data in its full detail.

This difference in perspectives must be resolved when analytics are developed for use in production systems. Analytical and reporting data must be synchronized to ensure a common baseline for business and analytics teams. Analytics and IT departments must effectively manage the mapping of analytical data to production data sources.

Example of Analytics at Production Scale: YouSee

YouSee is Denmark’s leading provider of cable TV and broadband services and aims to excel in terms of content, quality, and customer value. YouSee offers cable TV, IP telephony, mobile broadband, and digital TV services. In Denmark the company is first in digital TV market share, second in broadband, and third in telephony. YouSee’s strategy includes developing an extended portfolio of HDTV and on-demand TV services and launching broadband services with speeds greater than 100Mbps. YouSee employs just over 1,200 people in Denmark and had revenues of 4.012 BDKK (US$775 million) in 2010. YouSee is a division of the TDC Group, Denmark, and was known as TDC Cable TV until it was spun off as an independent brand in 2007.

YouSee has a large number of cable TV customers—1.2 million households, representing approximately 46% of Danish households. Thirteen percent of these are YouSee Plus customers, paying for on-demand packages of TV shows and movies. Its broadband services reach 400,000 households and include web-based TV and the music service YouSee Play, with more than 10 million tracks. IP telephony reaches 69,000, and mobile broadband 3,000.

YouSee faced two challenges that were consistent with those of other telecom firms in other countries:

• Customers of TV and broadband products do not tend to be very loyal, so customer churn is a constant problem.

• Multiproduct customers are more profitable and more loyal, but many YouSee customers have only a single product.

These challenges were being exacerbated by new regulations that will allow competitors to sell broadband services over the YouSee cable network. Other organizations in Denmark are installing increasingly large fiber networks, and these too have the potential to offer competitive products. More competitors means more offers to consumers and therefore higher rates of churn.

With a strong leadership position in product innovation, YouSee identified customer service as a critical area of focus and its call center as the front line in its efforts to retain and develop customers.

Potential Solutions

YouSee believed that a 360-degree customer view would allow it to develop a solution that would generate additional sales and increase loyalty. The company hoped to take advantage of its many contact points with customers and effectively manage the customer dialog. A project to introduce predictive analytics to the call center was started. This project was sponsored at a high level by the president of sales and marketing due to its technical scope, importance to the business, and expected impact on customer handling.

The project involved a large number of people, both internal and external, and took approximately 14 months from start to go-live. Consultants participated in some elements of the project, but YouSee’s internal IT and analytical departments took primary responsibility. The project was divided into technical and analytical infrastructure to develop predictive analytic models, and a Salesforce.com CRM implementation effort.

YouSee was clear on what it wanted to improve—the decision that a call center agent makes about a cross-sell or retention offer when talking to a customer. The company began with the belief that all customer data is relevant in addressing these problems. YouSee identified all the possible sources of information to build this rich customer view. Some 80 or 90 data sources were identified, with many containing data about a single product’s customers.

Identifying all the data sources was a major effort because of the range of people who need to be involved to ensure completeness. Although the expertise required to identify and understand these data sources was largely internal, an industry-specific data model helped bring everything together. Not all this data is relevant to developing predictive analytic models, of course, but YouSee system designers believe that the models should determine what data is relevant.

The next step was to define and build an extract, transform, and load (ETL) process for these data sources. YouSee’s IT department created an infrastructure for extracting the data from the original sources and loading it into what YouSee calls the Detailed Data Store. The Detailed Data Store contains all the data organized around a unique ID for every household, creating a 360-degree view of YouSee customers.

As soon as the Detailed Data Store was available, the analytics team created an analytical base table to support the development of predictive analytic models. This table takes all data collected and formats it to be suitable for analytic modeling. Data in the analytical base table is organized relative to dates when customers churned or made an additional product purchase. YouSee analysts hypothesized that the prior 180 days of historical data can predict the next 90 days, and this drives the data that is included.

Two initial models were then developed in SAS Enterprise Miner®, a data mining tool. For each broadband customer, the models deliver a probability (0 to 1) of a given event. The events are the likelihood of a successful cross-sell of cable TV services to a broadband subscriber in the next 90 days, and the likelihood that a broadband subscriber will churn in the next 90 days.

YouSee knew all along that simply displaying these predictive probabilities in the call center application would be unsatisfactory. Integrating the predictive analytic models into the call center CRM application involved two steps—making the predictive scores available to the application and then developing dynamic scripts that used those scores.

Deploying the predictive scores involved generating SAS code and running this code against the database every night to score each existing customer. Within the call center CRM system (Salesforce. com), YouSee created a set of business rules. These rules use the scores and many other data attributes to generate a suitable dynamic script for use with the customer. Only the most at-risk customers are selected. The business rules ensure that a customer doesn’t get the same offer twice within a given period of time. Specific product campaigns override the models and manage eligibility of products based on the customer’s location.

The combination of customer data, predictive models, and business rules decides what script the call center agents will see in Sales-force. Customers who are equally likely to churn will not necessarily get the same script because the script depends on their use of other services and other elements of their customer record. A unique “micro decision” is made for each customer to determine the best script.

The final stage of the project was maintenance and education. The models have to be monitored and updated when necessary, and the technical infrastructure also must be monitored constantly. In addition, the project educates and monitors the performance of call center agents.

YouSee Results

The implementation of dynamic and differentiated model-driven scripts has delivered an improvement in cross-sell and a reduction in churn for YouSee. Individual call center agents have achieved up to a 40% success rate on the cross-sell suggestions, for instance. Across the organization as a whole, a success rate of between 13% and 18% has been sustained even as usage rates have risen. The new scripts do contribute to a reduction in churn, but an exact number is difficult to estimate because several new set-top box and broadband services have been launched. These also influence customer churn.

The call center has changed its focus as a result of the solution. Script usage has risen steadily in the 18 months since the system went live. Some teams have much higher adoption than others—up to 3 times the teams with the lowest usage. The teams with supportive team leaders have higher adoption rates, which are strongly correlated to better sales numbers. Wait time and the time taken to handle a call are still key metrics, but the focus has now broadened to include retaining customers and selling them additional products.

With the solution deployed, the call center agents see only the scripts generated by the system. The agents don’t see the predictions or necessarily know why a specific script has been generated. This allows even new, inexperienced agents and those with no analytic skills to use the predictive analytic models—truly pervasive analytics. In addition, YouSee now has a data warehouse for reporting and analysis purposes and has seen a big improvement in customer insight. In addition to the call center system, the CRM department as a whole is using the models to understand its customers and to help define campaigns. The product teams are also using the analytical results to help understand customers.

Challenges and Lessons Learned from YouSee

Getting to production scale from an IT and analytics perspective has been straightforward. The biggest challenge that YouSee faced in achieving the desired results was the human aspect of the solution. In particular, educating the call center agents on how the new system would work and why they should use it has been challenging. In retrospect, the implementation team feels it spent too much time on IT and analytics and not enough time on the broader aspects of the business problem. In the 18 months since the system went live, usage has grown steadily. However it was not until 12 months after the system was first deployed that most of the teams reached an acceptable level of usage. Some teams are still struggling with adoption, but as a whole, usage levels are satisfactory.

Increasing the general understanding throughout YouSee of business analytics, including predictive analytic models and educating the call center agents, have been important tasks. Adoption and effective use of the solution remain an ongoing focus. The education of the call center agents needs to begin earlier in the project. The relatively high churn rate of call center agents also means that this education must be an ongoing task. Without this education, experienced agents prefer their “gut feelings” over the generated scripts. A number of initiatives are proving effective in this regard:

• Getting team leaders to spend more time on the solution has helped. Some teams have shown much higher usage and better results.

• Focusing on targets related to the solution (such as how often the scripts are used and their hit rate) has increased adoption.

• Comparing teams and showing that teams using scripts are outperforming those that do not has created internal competition and increased interest in the scripts.

• Hiring Salesforce quality consultants and instituting a Sales-force task force to increase adoption has had a positive impact.

• Using more dynamic scripts that are related to explicit campaigns known to the call center staff also has helped.

Because the solution requires a new interface in the call centers, change and adoption must be managed—After all, most people don’t like change. The analytic team has also had to answer questions about how the predictive analytic models work to help those who must rely on the models trust them. Initial skepticism has given way to belief as the model-driven scripts have demonstrated their effectiveness.

The team has also found that it is essential to be able to continuously improve the decisions and the resulting scripts. Market conditions, competitors, and customer behavior are constantly evolving, and this leads to continuous change in data and conditions.

YouSee now knows that the first steps should have involved the call center and the call center agents more. An effort to develop and evolve the script interface and business rules in parallel with developing the predictive analytic models would have helped ensure more rapid adoption of the models when they were complete. Not involving the call center agents until a very late stage of the project resulted in a rough start to the “go-live” process.

Future Plans for Analytics at Scale

Three more churn and cross-sell models related to other YouSee products are in the development stage and will be used to drive additional scripts in the call center. Scripts are also being developed to see if a “next best action” approach should replace the current offer-centric approach. YouSee has some data to suggest that asking for email addresses and mobile phone numbers, so that more regular communication is possible in the future, might be more helpful in deepening the relationship with a customer than simply trying to sell something right now.

YouSee also plans to move to real-time scoring. In particular, the company plans to use real-time scoring to integrate predictive analytic model results into its web applications and set-top boxes. These boxes will allow its customers to rent movies and other content, and real-time scoring will support an analytically sophisticated recommendation engine. The current ad hoc use of the predictive analytic models to drive marketing campaigns will also be upgraded, using the models to drive systematic outbound campaigns.

Lessons Learned from Other Successful Companies

The issues YouSee faced are typical of those encountered by other organizations. These challenges are real and require different analytical development processes, organizational implementation techniques, and tools. There are certainly variations across different businesses and groups, but lessons can be learned from successful companies:

• Establish governance processes and technology to ensure that data used in analytical modeling will be available in production systems and reporting infrastructure. Do not allow these to get out of sync.

• Bring IT and business people into a multidisciplinary team early in the process. Do not allow the analytics team to work on the models alone.

• Invest early and consistently in teaching IT and business users the basics of analytics, especially what models can and cannot do.

• Ensure that your enterprise architecture contains an explicit description of how it will support decision-making IT components and the deployment of analytics. Don’t allow decision-making components, or decision services as they are often called, to be treated like a generic IT component.

• Consider business rules management systems or applications with a strong business rules component as a deployment infrastructure for analytical models.

• Establish a separate IT life cycle and methodology for building, deploying, and evolving analytical decision-making components rather than using the standard IT software development life cycle.

• Focus on decisions. Ensure that you know which decisions are at issue, what their characteristics are, what their value is, and how they impact the company’s business.

YouSee and other early adopters of production-scale analytics typically had to integrate products themselves and act without frameworks to support particular decisions at scale. More recently, however, companies that are well along in the use of decision technologies have begun to introduce the concept of “decision services” and have given them a role in their enterprise architectures.¹ In addition, business rules technology is increasingly being embedded within analytical systems. In the future, these previously separate worlds of business rules and analytical technology will increasingly be combined.

Endnote

1. For more on decision services, see James Taylor, Decision Management Systems: A Practical Guide to Using Business Rules and Predictive Analytics. Indianapolis: IBM Press, 2011.