9. Analytical Technology and the Business User
Thomas H. Davenport
It’s clear that long-term changes are taking place in the technology environment for business analytics. However, to understand the technology environment of the next five years, it’s useful to understand that during the last five—actually, the last 10 or 20—analytics have been a relatively stable technology. In this chapter I’ll briefly describe nine different attributes of the past business analytics technology environment, each of which is likely to change in the future. Later in this chapter I’ll describe how these attributes will change in the future analytical technology environment.
The past technology environment for business analytics and business intelligence was a relatively monolithic environment, with both quantitative analysts and business users being expected to employ the same tools and data sources. That environment worked relatively well for professional analysts, but the much larger group of business users generally were not served well by it.
Separate but Unequal
The technological environment for business analytics is largely separate from the rest of the application environment for most organizations. It was intentionally separated from the transaction system environment because organizations didn’t want to risk problems with transaction systems by directly analyzing their data. Analysis functions were kept separate from transaction functions, and data was kept separate from transaction databases in a warehouse. The two environments were unequal in that companies almost always spent far more on implementing transaction systems than on capabilities to analyze their data.
Staged Data
The preceding discussion of separated transaction systems and analytics suggests that the data for analysis in the current paradigm comes from one source: a data warehouse or mart. This acted as a staging area for access by analytical applications and tools. If you wanted data in your warehouse, you first had to follow an extract, transform, and load (ETL) process to get the data out of your transaction system and into your warehouse or mart. If you wanted to employ data originating in multiple business systems, extensive integration activities typically preceded even the ETL process. If you are working in a big-data environment with massive volumes of data or highly unstructured data, the efforts needed to get your data in a position to be analyzed typically dwarf the efforts to actually analyze the data.
Multipurpose
Since at least the 1970s, analytical capabilities have been multipurpose. Users were provided an extensive toolbox of analytical methods and tools. It was the job of the analyst or decision-maker to decide what tools were appropriate for what analytical context. This, of course, required a high degree of sophistication—one that many analysts and almost all decision-makers lacked. Many data environments for analytics were also multipurpose. The idea behind an enterprise data warehouse is to support a variety of analyses and decisions. Less-popular data marts were intended to support a single type of analysis, or at least a narrow range.
Generally Complex
For both avoidable and unavoidable reasons, analytical technology environments typically are complex. As mentioned, the tool choices typically are quite large, which increases complexity. Serious analytical tools tend to have less-than-ideal user interfaces, although leading vendors have made inroads into solving this problem. Large data warehouses are complex as well. Finally, analytical technology environments will probably always be somewhat difficult to use because analytics themselves are a complex discipline. How many students have difficulty with statistics in school? Despite all the efforts that analytical software providers have made to insulate business users from statistics-driven complexity, an underlying element of quantitative skill is required to succeed with analytics.
Premises- and Product-Based
Analytical tools generally have been based on the customer premises and have been sold primarily as products, rather than as services or solutions. This may contribute to the complexity problem, in that having to worry about memory and storage limitations and other implementation issues are problems for users. Despite the enthusiasm for software as a service (SaaS) and cloud computing, however, this has been one of the relatively less problematic aspects of the analytical environment.
Industry-Generic
Historically, the analytical tools sold to a customer in one industry were the same tools provided to a customer in another. There was little or no tailoring of tools for particular industries. This is despite the fact that each industry has business problems that are best solved with a particular analytical approach. And transaction software (such as ERP) vendors have long customized their products by industry. Although a solution to almost any industry-specific problem can be cobbled together with generic analytical tools, the skills to do this are not widely available. This has begun to change in the past several years, as vendors have offered products specifically designed for some industries, such as money laundering and fraud prevention in banking.
Exclusively Quantitative
Analytics in the past have been almost exclusively quantitative, relying on structured quantitative data and employing algorithms and models to analyze it. This was not a major problem when there were no tools for analyzing other types of data (such as text, voice, and video) and no other approach to supporting decisions (such as business rules). As noted in Chapter 1, even when text and images are being analyzed (as in many big-data contexts), the data about them needs to be converted into structured quantitative data before it can be analyzed.
Business Unit-Driven
Historically, many decisions about the technology to be used for business analytics in organizations were made by managers in charge of particular business units or functions. Their unit had particular decisions to make or problems to solve, so they acquired the software (and often the data storage) to address that particular issue. In effect, they created departmental systems that ran on departmental servers and were maintained by semiprofessional IT people in the business units. This approach ensured that the technology met the specific business need, but it also meant that different departments acquired different analytical technologies, leading to a proliferation of tools and vendors throughout the organization.
Specialized Vendors
Historically, analytics software has been available only through stand-alone, dedicated analytics vendors. Those vendors sold only (or at least primarily) analytics software. One implication of this vendor segregation is the separation of analytics from other software-based activities mentioned earlier. This makes it more difficult to integrate analytical tools and capabilities into transaction systems.
Problems with the Current Model
The current environment worked well for some user classes but not so well for others. Professional analysts generally were well-served in the past environment. It was often a separate world in which an analyst could explore to his or her heart’s content. All the data and analytical methods that he or she could ever need were available, and any question could be answered, any decision supported. This was an excellent environment for those with high levels of analytical, technological, and information management skills.
However, for business users, the analytical technology environment has been overly complex. There have been too many tools in the toolbox and too much data in the warehouse. Business users find it difficult to master a broad array of tools, and they don’t know how to get the data they need out of the warehouse. The BI industry has spoken for years about “self-service” queries and reporting, and about executives “drilling down” into data, but it simply doesn’t happen very often.
Another problem for the current technology environment for business analytics is that the technology is either too close to or too far from the decisions it is supposed to support. In the case of highly departmental applications, the technology does support a specific decision, but it may be difficult to scale or be shared across the company. In the case of highly automated decisions, the analytical tools and the decision are so close as to be one and the same, but developing automated decision engines is difficult. In the multitool, multi-data, multipurpose environment described earlier, the technology is too far from the decisions it is designed to support. Software is too difficult for business users to employ, and enterprise data warehouses are too vast.
Because of these problems, there is clearly a need for a new technology environment for business analytics. However, the change will take place in different ways for different user groups. In particular, the technology environment for the business user—the nonprofessional analyst who still needs to create reports and analyses from data—is the one in greatest need of simplification and change.
Changes Emerging in Analytical Technology
This section describes how the analytical technology environment is either already beginning to change, or promises to change in the future, for business users. The most important aspect of the future environment is that it is no longer monolithic. There is not one future environment, but rather three different ones, as shown in Figure 9.1. The BI starburst in the upper left is the multipurpose, multitool environment of the past, which was intended to service both professional analysts and business users. Because the latter was a much larger group, the figure portrays it primarily in that group’s camp.
Figure 9.1. The changing analytical technology environment.
Because the multipurpose BI environment didn’t serve business users well, future environments won’t be in that cell of the matrix. Instead, they’ll evolve into three other primary environments:
• The single-purpose environment for business users, which I call analytical applications because of their resemblance to apps on iPhones and other smartphones. This environment is simple to use and allows business users to easily find the data and produce the queries and reports they need to make specific decisions. Because of their simplicity and small size, these apps should accelerate the cycle of insights-to-decisions-to-action for many managers and organizations. This is the newest analytical environment. These apps will also guide business users through a decision process; some have called this function guided analytics.
• The multipurpose environment for professional analysts, which I’ll call the analyst sandbox. This environment provides multiple tools and data sources for analysts who can understand them all and effectively choose from among them. It consists of multifunction statistical “packages” such as SAS, SPSS, or R, as well as complex data warehouses with multiple types of data. It is similar to the “old BI” environment, except that there is no longer an assumption that it serves business users, so it doesn’t have to be simplified. Its primary purpose can be the creation of advanced analytics, rather than standard or ad hoc reports. In many organizations this environment already exists (although it could always be improved), so it will undergo the least change of the three.
• The single-purpose environment for professional analysts, which I’ll call embedded analytics. The primary reason for involving professional analysts in single-purpose applications is to achieve scale and real-time delivery. This environment also includes automated decision applications, which almost always need to be developed by analytical professionals. It is also the province of big-data applications requiring professional data science skills. Relatively few such embedded analytics environments exist today, but when they do, they require the technological and analytical skills of professional analysts. They are growing rapidly in the big-data context.
Creating the Analytical Apps of the Future
In contrast to the analytical environment of the past, the next set of attributes characterize the emerging environment for analytical technology as it relates to business users.
Single-Purpose, Industry-Specific, and Simple
Going forward, for relatively simple analytical applications for business users requiring human exploration and interpretation (that is, nonembedded analytics), multipurpose analytical packages are not appropriate. Instead, we’ll see analytical “apps,” or single-purpose tools that are linked to a particular type of decision. If you need to do a sales forecast, the app will do that and nothing more.
Moreover, I believe that these tools will be tied closely to a particular industry. The sales forecasting tool will be designed to forecast retail sales or discrete manufacturing sales. If you want to do shipment load optimization in a transportation firm, there will be an app for that. The industry-specific apps will know what data is typically employed in an industry and will be able to link to that data easily with only a modicum of system integration work. The combination of decision types and industries will eventually yield thousands of discrete apps.
Like iPhone apps, these tools will be relatively simple to use and will be “guided.” They will have intuitive, touch-based interfaces. They will guide users through the process of analyzing the data and even making the resulting decision. Not only will they do the needed calculations on the data, but they also will steer the user through the process of ensuring that the data are well-suited to it, interpreting the results, and making a decision based on the data. They should provide a faster and better return on information in many business analytics domains.
These analytical apps may be developed by software vendors, consultants and integrators, or internal developers (typically IT professionals and professional analysts) within organizations. For example, several “analytic applications” are now available from SAP, although the company envisions that they will eventually be created by a broad ecosystem. Many of the applications thus far have been “co-created” with particular customers. The following are some early applications and their intended industries:
• Health care: quality management
• Health care: nursing productivity
• Telecommunications: customer management and retention
• Banking: enterprise risk and reporting
• Government: planning and consolidation
• Defense: readiness assessment
If this model takes off, there will ultimately be thousands of these applications, some industry-specific and some horizontal. Some will be developed by vendors and some by companies that are users of analytics.
For an example within a company, professional analysts in the Commercial Analytics groups at Merck have developed an analytical app for determining whether a vacancy in the sales force should be filled. The tool accesses the data necessary to perform the analysis and leads the business user—normally a regional sales manager—through the decision process. Also in the pharmaceutical industry, a consultant makes available single-purpose, industry-specific tools for sales forecasting and promotion analysis. Perhaps at some point there will be an “app exchange” for companies to sell or exchange analytical apps that are not deemed to provide competitive advantage.
Service- and Solution-Based
It would be consistent with the analytical apps environment to have application services delivered as services, rather than as premises-based products. That’s a simpler approach to providing such apps, and business users wouldn’t have to worry about new versions and updates. Service-based applications would also facilitate the use of analytics on mobile devices for industry and process contexts that require them. Not surprisingly, many vendors are beginning to offer analytics as a service, and I expect this trend to continue and accelerate—particularly for analytical apps environments. Deloitte, for example, offers a Managed Analytics service with a variety of single-function analytical applications, including the following:
• Transportation analysis
• Aftermarket services revenue growth
• Transportation contract compliance
• Services operations and warranty analysis
Solutions consisting of bundled products and professional services may not be as necessary in the future as they are today because apps will be simpler for business users to use. However, it is possible that services will still be necessary to configure apps and ensure that they are drawing on the correct data sources. For this reason, it’s reasonable to expect some degree of solutions orientation on the part of major vendors.
Centrally Coordinated
It seems ironic that in a shift to analytical apps for business users, there will be more coordination by a central IT function. After all, there is little or no central coordination for iPhone apps. However, even with analytical apps, there will be a need for some central coordination, although business users will probably initiate their use. They will need to be developed and integrated, and some of that work will be done by internal IT organizations. They will also require data, and IT and data management professionals will need to help provide it. And for apps that are popular across enterprises, vendors may well provide site-license pricing that would require central coordination and distribution. Finally, to avoid the “multiple versions of the truth” problem, these experts need to ensure that different analytical applications don’t overlap and that similar applications use similar data.
Of course, for embedded analytics and analytical sandboxes, IT organizations typically have played important roles in the past, and they will continue to do so. In big-data environments, technology-oriented professionals will be even more important than in the past.
Integrated Vendors
For both analytical apps and embedded analytics applications, separate analytics vendors are becoming part of larger integrated firms offering transaction processing software and services. Of course, this transformation is already largely complete: Large software and hardware providers have already acquired most of the freestanding analytical and business intelligence software vendors. These large, integrated vendors are beginning to introduce offerings that integrate analytical capabilities with other software tools. Examples of this integration include the following:
• Creating small analytical apps that link to particular modules of transaction software. An example is a trade promotion analysis application linked to the trade promotion transaction system for a retail ERP system.
• Embedding analytics and algorithms into transaction software. An example is introducing an automatically calculated customer lifetime value analysis into the order management function of an ERP system.
• Implementing in-database processing of calculations for more rapid processing of data-intensive analytics. (Independent analytics vendors are pursuing this same approach through partnerships and alliances.)
• Inclusion of reporting—if not advanced analytics—capabilities in the in-memory versions of transaction software, which offer rapid response and click-based report design.
• Incorporation of data warehouse, data mart, and on-demand data assembly by traditional database and storage vendors.
The remaining independent analytics vendors will attempt to match this integration by focusing primarily on the analytical sandbox and by increased emphasis on partnerships and alliances for embedded analytics. Large services and systems integration vendors are also incorporating analytics into their practices in a substantial way. These firms also focus heavily on transactional and other enterprise software capabilities and are likely to be active in integrating analytical functions into those environments.
Summary
These changes in the technology environment for business user-centric analytics are already happening and will become more widely distributed over time. Some organizations may need to emphasize one of the future environments more than others. Those with primarily reporting needs will probably emphasize analytical apps, and firms needing a lot of advanced analytics may emphasize the analytical sandbox. Firms with a strong process and transaction orientation may emphasize the embedded analytics environment. Although analytical apps may represent the bulk of business analytics activity because of the large size of the user base, most large organizations will probably need elements of all three environments to support their key decisions with data and analysis. Particularly for business user analytics, we are likely to see more change in the next few years of analytical technology than we have seen in the last few decades. This change is long overdue. It promises a much closer and more effective link between information and decision-making than ever before.