AI-driven Analytics

Today, AI is becoming a key driver of analytics. BI remains out of the technical reach of the average business person, and data volumes have exploded. When Teradata was born in 1979, most business leaders could never imagine amassing an entire terabyte of data. Today, many people store terabytes in their homes and the cloud. And we continue to create more data all the time with things as common as our phones, as well as with connected devices such as smart homes, cars, and planes and trains—the Internet of Things (IoT)—to name but a few data sources.

In a recent McKinsey analytics survey, nearly half of all respondents said “data and analytics have significantly or fundamentally changed business practices in their sales and marketing functions, and more than one-third say the same about R&D.”

The challenge for traditional BI—in which data experts summarize and aggregate data from a data warehouse or data mart and then load it to a BI server for exploration and reporting—is that it cannot support the agility and deeper insights businesses require, nor the data volumes. Still, organizations recognize the need to be data-driven to keep up with existing competitors and fend off new digital natives.

This is where AI presents a significant opportunity. Thanks in part to the parallel explosions of data, affordable compute resources, and advanced algorithms, AI now can gather the amount of inputs necessary for it to make reasonable decisions and deliver the results of analyses in a timely fashion so that they are valuable.

AI-driven analytics can help users reveal insights in seconds in multiple ways. One example is the use of natural language processing (NLP). Analytics solutions with strong AI capabilities can understand and translate queries such as, “What are sales for each category and region?” to identify the appropriate underlying data, calculate the sums, and visually present a best-fit chart, as shown in Figure 1-1. The user never needs to think about the rows and columns and calculations.

Figure 1-1. Modern analytic solutions support NLP to enable you to use everyday language to ask questions of your data

Automated analytics are another example of AI augmenting analytics to accelerate time-to-insight. In this case, a user can simply point their analytics solution at a dataset, a field, or even a specific data point and ask AI to identify key drivers of and anomalies within that data. Thanks to modern compute power and programming techniques, the AI can run thousands of analyses on billions of rows in seconds. Through natural language generation, the system can present the AI-driven insights to the user in an intuitive fashion—including results to questions that the user might not have thought to ask. With user feedback and machine learning, the AI can become more intelligent about which insights are most useful.

This notion of augmented analytics—applying AI techniques such as machine learning and natural language generation to analytics—presents such a disruption to the data and analytics market that industry thought leaders are encouraging their adoption. The opportunity is so significant that analyst firm Gartner, Inc. says that augmented analytics are “crucial for unbiased decisions, impartial contextual awareness and acting on insights”.

AI: uniting database and analytics technologies

Fundamentally, AI requires both database and analytics technologies that operate at massive volume and speed. AI requires significant storage to hold all of the data that its models require for training and learning. And AI needs analytics technology to do something useful with all that data, whether the end result is identifying a person by their face or predicting which products will be hot sellers in the next month. All of this must be combined with massive processing power to return results in a timely fashion.

Essentially, data is the crude oil of our digital economy. There is great value to be gained in data, but it requires very significant resources to turn massive volumes of dirty data into shiny insights. Data in its raw form is often useless. Like oil refinement, data refinement is difficult and expensive. As a portion of the population that can benefit from data insights, those who know how to process and analyze data are relatively small in number. And, like crude oil, there are millions of consumers waiting to use the completed data products.

This is where AI comes in—presenting accurate, relevant answers at the time that they matter to the business user.

AI requires an extremely tight integration between data storage and computation. Even though databases and analytics have long been closely connected (with database innovations often enabling new user interactions and analytic modeling techniques), there have been fewer efforts to jointly, inextricably develop storage, computation, analytics, and visualization together. Instead, enterprises have combined and integrated various components to build best-of-breed solutions based on their use cases (and existing vendor contracts and budgets).

In this paradigm, AI could never integrate with BI beyond the simplest use cases, as BI was not built for scale. Traditional BI relies on cubes and data aggregates loaded to a single BI server. The minute someone—or something, like AI—needs to learn more by drilling deeper into a detail that is more granular or outside the scope of the cube, the process breaks.

This is not to say that AI-driven analytics require every component and feature of traditional databases. But it does, at a minimum, require tighter integration between storage, compute, and analytics, along with a visualization layer or some other publication technique for the intelligence to be delivered in a timely enough fashion to be of value.

On a related note, in our always-connected world, we now expect that information always be available to us no matter where we are located or what we are doing. Therefore, the serving layer for AI-driven insights and results must be planet-scale. This was not possible prior to the widespread adoption of cloud technologies. Amazon Web Services (AWS), which holds the largest share of the cloud-based computing and storage market, is only a dozen years old. Hence, AI-driven analytics is a relatively young, though already proven, technological advancement.

The role of memory

The evolving market for in-memory storage and processing also has played an important role in recent advances in AI-driven analytics. The second generation of BI tools were invented prior to the popularization of 64-bit computing and could only scale up to a few gigabytes of random-access memory (RAM). As the cost of RAM has decreased, enterprises are finding it more feasible to store and process increasingly large volumes of data in-memory rather than on less expensive but significantly slower disk drives.

“To become insight-driven or insight-centric, the goal is to get from data to analytics to action with a latency of only subseconds in the pipeline,” writes Nadav Finish, CTO of GigaSpaces. “Businesses must advance beyond traditional analytics perspectives, which separate data inputs and transactional systems from the analytics systems.”

Indeed, developers of memory-based, AI-driven analytics measure their code optimizations in nanoseconds—one billionth of a second. InfoWorld says that “nanosecond latency is at the bleeding edge of real-time computing,” and “the value of time has never been higher and therefore speed has never been more critical to business applications.”

Recent advances in AI

AI-driven analytics is a relatively young concept, but it is not the only area in which AI has made advances in recent years. Many organizations have actively embraced various forms of machine learning, the aforementioned subset of AI in which machines become progressively smarter or better at performing specific tasks. Essentially, machine learning is the use of algorithms for statistical analysis on input data to predict outputs. Machine learning is often broken into three categories: supervised, unsupervised, and reinforcement. Let’s take a moment to look at each of these:

Supervised machine learning

A data scientist or analyst provides both the inputs and a desired output, including feedback on the results to help the models “learn” so that they can make better predictions. The expert iterates and the machine tweaks the models until there are ultimately no or very few wrong outputs.

A popular application occurs on social media websites in which users identify people in pictures. When a user loads a new photo, the site can make a very accurate suggestion of who should be tagged in the photo.

Unsupervised machine learning

Computers rely on deep learning similar to neural networks (rather than feedback from a data expert) to make their predictions. By looking at extremely large numbers of data points, machines can identify trends and correlations between variables on their own and then use this training to recognize new data points or make predictions.

Marketers use unsupervised machine learning algorithms such as clustering to identify similar groups of customers or prospects for targeted marketing campaigns.

Reinforcement machine learning

Machines take actions in an environment to maximize a “reward.” This is typically done through a Markov Decision Process when there is no exact mathematical model of the environment and experts are not involved in providing the inputs or feedback on outputs. The goal is to maximize the reward based on existing knowledge while simultaneously acquiring new knowledge.

A popular example is that of a gambler with a row of slot machines from which to choose. Common applications include financial portfolio optimization, network routing, and clinical trials. Reinforcement machine learning is often applied in video games and robotics.

Many companies have invested heavily in deep learning, a subset of machine learning, which is itself a subset of AI. Deep learning and artificial neural networks enable image recognition, voice recognition, NLP, and other recent advancements. We have already come to take these for granted in our personal lives in the age of the internet and big data, but such features are hardly commonplace in analytics software.

Common AI algorithms used in analytics

Although AI-driven analytics is still too nascent to describe the algorithms behind it as “popular,” there are algorithms that are becoming more widely used across the AI-for-analytics landscape. Let’s examine a few of these here:

Linear regression

Linear regression (Figure 1-2) models the response of a dependent to an independent variable or set of independent variables. The model is an equation with the dependent on one side and a weight for each variable on the other side. The equation can be used to generate insights on customer behavior or profitability.

Figure 1-2. An example of linear regression

Logistic regression

Logistic regression (Figure 1-3) is similar to linear regression in that it builds a linear model for an independent and a dependent variable. However, in a logistic regression, the dependent is binary—0 or 1, true or false, yes or no. It can be used for image segmentation and processing or categorical predictions.

Figure 1-3. An example of logistic regression

Decision trees

Decision trees are tree-like models of decisions and consequences or outcomes, often with the likelihood of those outcomes modeled as weights. They are popular in logistics, project management, health care, and finance. Figure 1-4 shows an example.

Figure 1-4. An example of a decision tree

Naive Bayes Classification

Naive Bayes Classification is a machine learning technique, shown in Figure 1-5, that assumes that features or predictors are independent of one another to calculate the likelihood that an item is classified into various categories. It is very popular in text analytics for use cases such as spam recognition and news category tagging.

Figure 1-5. Results of a Naive Bayes Classification model

Clustering algorithms

These types of algorithms attempt to group together items that are more similar to each other. K-means, depicted in Figure 1-6, is probably the most popular clustering algorithm. To begin, you select the number of classes or groups that you want to create and the centers of those groups. As the model trains, it will shift the center of the groups until ultimately it finds the center with the shortest distance between the members of its group and the farthest distance from members of the other group. This is a very fast method because there are few computations—you are only calculating the distance between data points and the center. Clustering algorithms are used in customer segmentation, bioinformatics, medical imaging, social network analysis, and web search.

Figure 1-6. Results of a clustering model

Principal Component Analysis (PCA)

PCA is most commonly used for dimension reduction. In this case, PCA measures the variation in each variable (or column in a table). If there is little variation, it throws the variable out, as illustrated in Figure 1-7, thus making the dataset easier to visualize. PCA is used in finance, neuroscience, and pharmacology.

Figure 1-7. Results of a principal component analysis

Advances in processing

Though some of the algorithms outlined in the previous section require relatively few calculations, they will require massive volumes of data to effectively train. As discussed earlier, lack of significant volumes of compute power and data have historically been impediments to AI development. Large volumes of data are necessary for models to learn, and large compute resources are necessary to quickly perform any complex calculations.

Fortunately, developers have made significant advances in processing over the past few decades, and the cost to take advantage of these advancements has considerably decreased. Graphic processing units (GPUs) and RAM are now affordable enough that enterprises can incorporate them into mission-critical systems. Companies now store terabytes of their enterprise data in RAM for fast access and analytics. And thanks to massively parallel processing, software can intelligently combine the resources of many machines to multiply processing power.

This has led to new programming frameworks such as deep learning. In deep learning, computers learn how features are represented rather than relying on task-specific algorithms. Deep learning has risen to prominence in speech recognition, NLP, bioinformatics and other fields. This can be supervised or unsupervised.

Deep neural networks, perhaps the most popular subfield of deep learning, examine inputs and determine outputs. “Deep” means that a neural network has multiple hidden layers between the input and output, which deep neural networks use to learn features of the data to apply weights—the likelihood or probability of each output—to various factors to create a feature hierarchy. This requires both large amounts of data and large amounts of compute power. Through training, eventually the machine will settle on the appropriate weight for each feature and be able to correctly classify new inputs without any human assistance.

Transforming the way your organization operates with AI

Perhaps one of the most important considerations in implementing AI technologies is ensuring that teams have the skillsets to use something so advanced. In fact, 51% or organizations see staffing as their primary challenge regarding AI and machine learning, according to Harvard Business Review. In the public sector, AI is likely to free up significant hours—up to 30%—from mundane tasks such as paperwork. But it won’t necessarily reduce the need for staff hours. Rather, agencies likely will shift staff time to focus on tasks that require human judgement, according to Deloitte.

Fortunately, in the world of analytics, AI can actually reduce complexity. Thanks to advances in speed and scale as we discussed earlier, AI-driven analytics can traverse massive volumes of data and perform thousands of analyses to uncover outliers and other valuable data points and insights. But how does a machine distinguish between interesting insights and noise from the end user’s perspective? Machine learning, the bedrock of AI, can learn what areas of a dataset are most commonly inspected by the user or their team in order to expose only the most relevant facts. Figure 1-8 shows that auto transactions account for the highest revenues. However, AI reveals the hidden pattern that in ZIP code 80017, dining transactions are 10 times the average.

Figure 1-8. In analytics, AI can reduce complexity by exposing the most relevant facts such as the SpotIQ Insights

In addition, AI is improving data literacy across the organization. Historically, a data expert needed to understand the details of the data, determine the best way to visualize analytics results, and typically copy and paste those visualizations into a PowerPoint presentation, where they would explain what was being shown. AI-driven analytics solutions greatly reduce the need for human intervention and intelligence throughout this process. Now, a user asks a question, the software determines how to best visually represent the results, and then the system uses natural language to explain those results.

In the traditional BI paradigm, business people struggled to get access to their data, but even if they crossed that hurdle, they still needed to know what questions to ask. AI can be an accelerator and educator for data literacy by helping business people know what to look for in their data in the first place.

This automation of analytics through AI can minimize the opportunity for errors and reduce human bias by presenting valuable insights to questions that users might not have considered asking in the first place. Because a machine can run thousands of correlation calculations on data points and even on the results of its own analysis, it can uncover key contributing factors that users have not yet considered.

By reducing complexity and the need for expert skills, AI-driven analytics can be a significant contributor to the growth of the new class of citizen data scientists. Gartner defines a citizen data scientist as “a person who creates or generates models that use advanced diagnostic analytics or predictive and prescriptive capabilities, but whose primary job function is outside the field of statistics and analytics.”

“Citizen data science (CDS) fills the gap between mainstream self-service analytics by business users and the advanced analytics techniques of data scientists. Data and analytics leaders should use CDS to explore new data sources, apply new analytics capabilities and access a larger user audience,” according to Gartner.

The opportunity for AI to transform your organization while enabling the new class of citizen data scientists cannot be overlooked.