3.1. Basic ideas of machine learning

1. 1. Providing all relevant variables and excluding all irrelevant variables. This may include some elementary processed variables via feature engineering, for example, when we know that the ration between two variables is very relevant, we may want to explicitly supply that ratio as a column in the data table.
2. 2. Providing empirical data that is significant and representative of the situation.
3. 3. Assessing the results of candidate models to make sure that they output what is expected.
4. 4. Explicitly adding any essential restrictions that must be obeyed.

Supervised methods deal with datasets for which we possess empirical data for the inputs to the model as well as the desired outputs of the model. Unsupervised methods deal with datasets for which we only have the inputs. Imagine teaching a child to add two numbers together. Supervised learning consists of examples like $1+2=3$ and $5+4=9$ whereas unsupervised learning consists of examples like $1+2$ and $5+4$. It is clear from this difference that we can expect supervised methods to be much more accurate in reproducing the outcome. Unsupervised methods are expected to learn the structure of the input data and recognize some patterns in them.

This distinction is closely related to the difference between a categorical variable and a continuous variable. A categorical variable is equal to one of several values, like 1, 2, or 3, where the values indicate membership in some group. The difference between two values is therefore meaningless. If datapoint A has value 3 and datapoint B has value 2, all this means is that A belongs to group 3 and B belongs to group 2. Taking the difference of the values, $3-2=1$, has no meaning, that is, the difference between the two datapoints does not necessarily belong to group 1 nor does group membership necessarily even make sense for the difference.

A continuous variable is different in that it measures a quantity and taking the difference does mean something real. For instance, if datapoint A has a flow rate of 1 ton per hour and datapoint B has a flow rate of 2 tons per hour, we can take the difference $2-1=1$ and the difference is 1 ton per hour of flow, which is a meaningful quantity that we can understand.

1. 1. Do you have data for the result you want to compute, or only for the factors that go into the computation?
2. 2. Is the result a membership in some category, or a numerical value with intrinsic meaning?

3.2. Bias-variance complexity trade-off

3.3. Model types

3.3.1. Deep neural network

The neural network is perhaps the most famous and most used technique in the arsenal of machine learning (Hagan, Demuth, & Beale, 1996). The recent advent of deep learning has given new color to this model type through novel methods of training its parameters. Training methods are beyond the scope of this book, however.

While it took its inspiration from the network of neurons in the human brain, the neural network is merely a mathematical device. A schematic diagram like the one shown in Fig. 3.7 is commonly displayed for neural networks. On the left side, the input vector $\underset{\_}{x}$ enters the network. This is called the input layer and has several neurons equal to the number of elements in the input vector. On the right side, the output vector $\underset{\_}{y}$ exits from the network. This is called the output layer and has several neurons equal to the number of elements in the output vector; this is the result of the calculation. In between them are several hidden layers. The number of hidden layers and the number of neurons in each are up to us to choose and is referred to as the topology of the neural network.

In bringing the data from one layer to the next, we multiply it by a matrix, add a vector, and apply a so-called activation function to it. These are the parameters for each layer that the machine learning algorithm must choose from the data so that the neural network fits to the data in the best possible way. A neural network with two hidden layers, therefore, looks like

$\underset{\_}{y}=A_2·f\left.{\underset{\_}{A}}_1·f\left.{\underset{\_}{A}}_0·x+{\underset{\_}{b}}_0\right)+{\underset{\_}{b}}_1\right)+{\underset{\_}{b}}_2$

The activation function $f\left.\dots \right)$ is not learnt but decided on beforehand. Considerable research has been put into this choice. While older literature prefers $\tanh \left.\dots \right)$, we now have evidence that $ReLu\left.\dots \right)$ performs significantly better in virtually all tasks for all layers except the outermost for which we usually choose the identity function. They are both displayed in Fig. 3.8.

$ReLu\left.x\right)=\begin{array}{ll}0\hfill & x\le 0\hfill \\ x\hfill & x>0\hfill \end{array}$

The neural network is not some magical entity. It is the above equation. After we choose values for the parameter matrices ${\underset{\_}{A}}_i$ and vectors ${\underset{\_}{b}}_i$, the neural network becomes a specific model. The main reason it is so famous is that it has the property of universal approximation. This property states that the formula can represent any continuous function with arbitrary accuracy. There are other formulae that have this property, but neural networks were the first ones that were practically used with this feature in mind. In practice, this means that a neural network can represent your data. What the property does not do is tell us how many layers and how many neurons per layer to choose. It also does not tell us how to find the correct parameter values so that the formula does in fact represent the data. All we know is that the neural network can represent the data.

Machine learning as a research field is largely about designing the right learning algorithms, that is, how to find the best parameter values for any given dataset. There are multiple desiderata for it, not just the quality of the parameter values. We also want the training to be fast and to use little processing memory so that training does not consume too many computer resources. Also, we want it to be able to learn from as few examples as possible. Beyond the hype of “big data,” there is now significant research into “small data.”

Many practical applications provide a certain amount of data. Getting more data is either very difficult, expensive, or virtually impossible. In those cases, we cannot get out of machine learning difficulties by the age-old remedy of collecting more data. Rather, we must be smarter in getting the information out of the data we have. That is the challenge of small data and it is not yet solved. We mention it here to put the property of universal approximation into proper context. It is a nice property to be sure, but it has little practical value.

The neural network is the right model to use if the empirical observations $\underset{\_}{x}$ are independent of each other. For example, if we take a manual fluid sample from a well once per day and perform a laboratory analysis on it, we can assume that today's observation is largely independent of yesterday's observation but may well depend on the pressure and temperature right now. In this context, we may use pressure and temperature as inputs and expect the neural network to represent the laboratory measurement.

If, however, time delays do play a role, then we must incorporate time into the model itself. Let's say we change the choke setting on a well and measure its wellhead pressure, we will see that the wellhead pressure responds to our action a few minutes later. If we measure these quantities every minute, then the observations are not independent of each other because the observation a few minutes ago causally brought about the current observation. Applying the neural network to such datasets is not a good idea because the neural network cannot represent this type of correlation. For this, we need a recurrent neural network.

3.3.2. Recurrent neural network or long short-term memory network

A recurrent neural network is very similar in spirit to the regular, the so-called feed-forward, neural network. Instead of having connections that move strictly from left to right in the diagram, it also include connections that amount to cycles, see Fig. 3.9. These cycles act as a memory in the formula by retaining—in some transformed way—the inputs made at prior times.

As mentioned in the previous section, the purpose of recurrence is to model the interdependence between successive observations. In practice such interdependence is virtually always due to time and represents some mechanism of causation or control. If my foot touches the brake pedal in my car, the car slows down a short time later. Modeling the dependence of car speed on the position of the brake pedal can lead to important models, that is, the advanced process control behind the automatic braking system in your car. The time delay involved is important as the car will have moved some distance in that time and so braking must begin sufficiently early so that the car stops before it hits something. It is cases such as this for which the recurrent neural network has been developed.

There are many different forms of recurrent neural networks differing mainly in how the cycles are represented in the formula. The current state-of-the-art is the long short-term memory network, or LSTM (Hochreiter and Schmidhuber, 1997; Yu, Si, Hu, & Zhang, 2019; Gers, 2001). This network is made up of cells. A schematic of one cell is shown in Fig. 3.10. These cells can be stacked both horizontally and vertically to make up an arbitrarily large network of cells. Once again, it is up to the human data scientist to choose the topology of this network.

The inner working of one cell is defined by the following equations that refer to Fig. 3.10.

$\begin{array}{ll}\underset{\_}{f}\left.t\right)=\hfill & \sigma \left.{\underset{\_}{W}}_{fh}·\underset{\_}{h}\left.t-1\right)+{\underset{\_}{W}}_{fx}·\underset{\_}{x}\left.t\right)+{\underset{\_}{b}}_f\right)\hfill \\ \underset{\_}{i}\left.t\right)=\hfill & \sigma \left.{\underset{\_}{W}}_{ih}·\underset{\_}{h}\left.t-1\right)+{\underset{\_}{W}}_{ix}·\underset{\_}{x}\left.t\right)+{\underset{\_}{b}}_i\right)\hfill \\ \underset{\_}{\tilde{c}}\left.t\right)=\hfill & \tanh \left.{\underset{\_}{W}}_{\tilde{c}h}·\underset{\_}{h}\left.t-1\right)+{\underset{\_}{W}}_{\tilde{c}x}·\underset{\_}{x}\left.t\right)+{\underset{\_}{b}}_{\tilde{c}}\right)\hfill \\ \underset{\_}{c}\left.t\right)=\hfill & \underset{\_}{f}\left.t\right)·\underset{\_}{c}\left.t-1\right)+\underset{\_}{i}\left.t\right)·\underset{\_}{\tilde{c}}\left.t\right)\hfill \\ \underset{\_}{o}\left.t\right)=\hfill & \sigma \left.{\underset{\_}{W}}_{oh}·\underset{\_}{h}\left.t-1\right)+{\underset{\_}{W}}_{ox}·\underset{\_}{x}\left.t\right)+{\underset{\_}{b}}_o\right)\hfill \\ \underset{\_}{h}\left.t\right)=\hfill & \underset{\_}{o}\left.t\right)·\tanh \underset{\_}{c}\left.t\right)\hfill \end{array}$

The various $\underset{\_}{W}$ and $\underset{\_}{b}$ are the weight and bias parameters of the LSTM cell. All weights and biases of the entire network must be tuned by the learning algorithm. The evolution of the internal state of the network $\underset{\_}{h}\left.t\right)$ is the memory of the network that can encapsulate the time dynamics of the data.

The network is trained by providing it with a time-series $\underset{\_}{x}\left.t\right)$ for many successive values of time $t$. The model outputs the same time-series but at a later time $t+T$, where $T$ is the forecast horizon. This is the essence of how LSTM can forecast a time-series,

$\underset{\_}{x}\left.t+T\right)=L_t\left.\underset{\_}{x}\left.t\right)\right)$

It is generally not a good idea to model the next timestep and then to chain the model

$\underset{\_}{x}\left.t+3\right)=L_{t+2}\left.\underset{\_}{x}\left.t+2\right)\right)=L_{t+2}\left.L_{t+1}\left.\underset{\_}{x}\left.t+1\right)\right)\right)=L_{t+2}\left.L_{t+1}\left.L_t\left.\underset{\_}{x}\left.t\right)\right)\right)\right)$

because this compounds errors and makes for a very unreliable model if we want to predict more than one or two timesteps into the future. It is much better to choose the forecast horizon to be some larger number of timesteps. Having done that, the intermediate values can always be computed using the same model with input vectors from earlier in time.

3.3.3. Support vector machines

The support vector machine (SVM) is a technique mainly used for classification both in a supervised and unsupervised manner and can be extended to regression as well (Cortes & Vapnik, 1995). The idea, as illustrated in Fig. 3.11 is that we have a set of points that belong to one of two categories and we draw a non-linear plane through the space dividing the data into two pieces. Everything above the plane is classified as one category and everything below the plane is classified as the other. The plane itself is an interpolated spline-like object that is based on a few selected data points—the support vectors—that must be found.

The plane chosen is that which has the maximum distance from all points possible. The points at the boundary of this space form the support vectors. In practice, the classification problem may require multiple planes for a good result. As the space of all points gets divided into sections by a collection of planes, we may find that a few data points are isolated in one area. This is another way of identifying outliers.

3.3.4. Random forest or decision trees

The decision tree is familiar perhaps from management classes where it is used to structure decision making. At the base of the tree is some decision to be made. At each branching, we ask some question that is relatively easy to answer. Depending on the answer, we go down one branch as opposed to another and eventually reach a point on the tree that has no more branches, a so-called leaf node. This leaf node represents the right answer to the original decision.

Fig. 3.12 illustrates this using a toy example. The question is whether the equipment needs maintenance or not. The left-hand tree starts by analyzing the equipment age in years. If this is larger than 10 years, we go down the left-hand path and otherwise the right-hand path. The left-hand path terminates at a leaf node with the value 1. The right-hand path leads to a secondary question where we ask for the number of starts of the equipment. If this is larger than 500, we go down that left-hand path leading us to the number 1/2 and otherwise we go down the right-hand path leading to the number 1/3. The number that we end up with is just a numerical score for now.

The numerical score now needs to be interpreted, which is usually done by thresholding, that is, if the score is greater than some fixed threshold, we decide that the equipment needs preventative maintenance and otherwise we decide that it can operate a little longer without inspection. This is an example of a decision tree in the industrial context (Breiman, Friedman, Olshen, & Stone, 1984).

We can combine several decision trees into a single decision-making exercise. Fig. 3.12 displays a second tree that operates in the same way. Having gone through all the trees, we add up the scores and threshold the final result. If the left tree leads to ½ and the right tree leads to 1, then the total is a score of 3/2. This combination of several decision trees is called a random forest.

Constructing the trees, their structures, the decision factors at each branching, and the values for each branching are the model parameters that must be learnt from data. There are sophisticated algorithms for this that have seen significant evolution in recent years making random forest one of the most successful techniques for classification problems. Incidentally, the forest is far from random, of course. The forest is carefully designed by the learning method to give the best possible classification result possible. Random forests can also be used for regression tasks, but they seem to be better suited to classification tasks.

3.3.5. Self-organizing maps (SOM)

A self-organizing map (SOM) is an early machine learning technique that was invented as a visualization aid for the human expert (Kohonen, 2001). Roughly, it belongs to the classification kind of techniques as it groups or clusters data points by similarity. The idea is to represent the dataset with special points that are constructed by the learning method in an iterative manner. These special points are mapped on a two-dimensional grid that usually has the topology of a honeycomb. The topology is important because the learning, that is, the moving around of the special points, occurs over a certain neighborhood of the special point under consideration.

When this process has converged, the special points are arrayed in the topology in a certain way. Any point in the dataset is associated with the special point that it is closest to in the sense of some metric function that must be defined for the problem at hand. This generates a two-dimensional distribution of the entire data. It is now usually a task for the human expert to look at this distribution and determine to what extent the cells in the topology (the honeycomb cells) are similar to each other. Frequently it turns out that several cells are very similar and can be associated with the same macroscopic phenomenon.

For example, in Fig. 3.13 we begin with a topology of 22 honeycomb cells. Having done the learning and the human examination, it is determined that there are only four macroscopically relevant clusters present. Each cluster is represented by several cells. These cells can either have a more subtle meaning within the larger group, or they can simply be a more complex representation of a single cluster than one special point; as one would do using algorithms like k-means clustering (Seiffert & Jain, 2002).

Ultimately, when one encounters a new data point, one would compute the metric distance between it and all the special points learnt. The special point closest to the new point is its representative and we know which cluster it belongs to. The SOM method has turned out to be a very good classification method in its own right. In addition to this however, the current position of a system can be plotted on the graphic and one can see the temporal evolution on the state chart, which is an elegant visual aid for any human working with the data source.

3.3.6. Bayesian network and ontology

Bayesian statistics follows a different philosophy than standard statistics and this has some technical consequences (Gelman, Carlin, Stern, & Rubin, 2004). Without going into too much detail, standard statistics assesses the probability of an event by the so-called frequentist approach. We will need to collect data as to how frequently this event occurs and does not occur. Usually, this is no problem but sometimes the empirical data is hard to come by, events are rare or very costly, and so on.

Bayesian statistics concerns itself with the so-called degree of belief. This encapsulates the amount of knowledge that we have. If we manage to gather additional knowledge, then we may be able to update our degree of belief and thereby sharpen our probability estimate. The probability distribution after the update is called the posterior distribution. There is a specific rule, Bayes theorem, on how to perform this updating and there is no doubt or difficulty in its application. The tricky part is establishing the probability distribution at the very beginning, the prior distribution. Usually we assume that all events are equally likely if we have no knowledge at all or we might prime the probabilities using empirical frequencies if we do have some knowledge.

Supposing that we have gathered quite a bit of knowledge about the general situation, we can start to create multiple conditional probability distributions. These are conditional on knowing or not knowing certain information.

Let's take the case of a root-cause analysis for a mechanical defect in a gas turbine, see Fig. 3.14. We know from past experience that certain measurements provide important information relating to the root cause of this problem, for example, the axial deviation, which is the amount of movement of the rotor away from its engineered position. This, in turn, is influenced by the bearing temperature and the maximum vibration. These two factors are influenced by other factors in turn. We can build up a network of mutually influencing events and factors. At one end of this matrix are the events we are concerned about such as the mechanical defect. At the other end are the events we can directly cause to happen such as the addition of lubricant into the system.

Based on our past experience we can establish that if the maximum vibration is above a certain limit, the chance of a rotor imbalance increases by a certain amount, and so on for all the other factors. We can then use statistics to obtain a conditional probability distribution for the mechanical defect as a function of all the factors. This cascading matrix of conditional probabilities is called a Bayesian network (Scutari and Denis, 2015). Once we have this network, it can be used like so: Initially, the chance of a mechanical defect is very small; we assess this from the frequency of observations of such defects. Then we encounter a high bearing temperature and a high vibration for a low rotation rate. Combining this information using the various conditional probabilities updates our probability of defect. If this is sufficiently high, we may release an alarm and give a specific probability for a problem. We can also compute what effect it would have on the probability if we added lubricant and, based on the result, may issue the recommendation for this specific action.

Please note that this is very different from an expert system. An expert system looks similar at first glance, but it is a system of rules written by human experts as opposed to the Bayesian network that is generated by an algorithm based on data. The expert system has the advantage that all components are designed by people who deeply understand the system. However, combinations of rules often have logical consequences that were not intended by those experts and these are difficult to detect and prevent. Due to this complexity, expert systems have usually failed to work, or at least be commercially relevant given all the effort that flows into them. At present, expert systems are considered an outdated technology. In comparison, Bayesian networks are constructed automatically and therefore can be updated automatically as new data becomes available. An understanding of each individual link in the chain is possible, just like an expert system, but the totality of a network can usually not be understood, as a practical network is usually quite large.

Bayesian networks are great as decision support systems. How much will this factor contribute to an outcome? If we perform this activity, how much will the risk decrease? In our asset intensive industry, the main use cases are: (1) root-cause analysis for some problem, (2) deciding what to do and to avoid based on a concrete numerical target value, (3) scenario planning as a reaction to changes in consumer demand, market prices, and other events in the larger world.

A related method is called an ontology (Ebrahimipour and Yacout, 2015). This is not a method of machine learning but rather a way to organize human understanding of a system. We may draw a tree-like structure where a branch going from one node to another has a specific meaning such as “is a.” For example, the root node could be “rotating equipment.” The node “turbine” then has a link to the root node because a turbine is an example of a rotating equipment. The nodes “gas turbine,” “steam turbine,” or “wind turbine” are all examples of a turbine. This tree can be extended both by adding more nodes as well as adding other types of branches such as “has a” or “is a part of” and so on. Having such a structure allows us to draw certain inferences on the fly. For example, being told that something is a gas turbine, immediately tells us that this object is a rotating equipment that has blades, requires fuel, and generates electricity and so on. This can be used practically in many asset management tasks. Combined with the Bayesian network approach, it reveals causal links between parts of an asset.

3.4. Training and assessing a model

The original dataset is divided into several, roughly equally sized, parts. It is typical to use 5 or 10 such parts and they are called folds. Let's say that we are using $N$ folds. We now construct $N$ different training datasets and $N$ different testing datasets. The testing datasets consists of one of the folds and the training datasets consist of the rest of the data. Fig. 3.16 illustrates this division.

We now go through $N$ distinct trainings, resulting in $N$ distinct models. Each model is tested using its own testing dataset. The performance is now estimated to be the average performance over all the trained models. Generally, only the best model is kept but its performance has been estimated in this averaged fashion. This way of assessing the performance is more independent of the dataset bias than assessing it using any one choice of testing dataset and so it is the expected performance estimation method.

Having done all this, we could get the idea that we could use all $N$ models, evaluate them all in each real case and take the average of them all to be the final output of an overarching model. This is a legitimate idea that is called an ensemble model. The ensemble model consists of several fully independent models that may or may not have different architectures and that are averaged to yield the output of the full model. These methods are known to perform better on many tasks as compared to individual models, at least in the sense of variance. However, it comes at the expense of resources as several models must not only be trained but maintained and executed in each case. Especially for industrial applications, that often require real-time model execution, this may be prohibitive.

3.5. How good is my model?

3.6. Role of domain knowledge

• • Precision: How much uncertainty is in a value?
• • Accuracy: How much deviation from reality is there?
• • Representativeness: Does the dataset reflect all relevant aspects of the domain?
• • Significance: Does the dataset reflect every important behavior/dynamic in the domain?

3.7. Optimization using a model

So, we have a model $y=f\left.\underset{\_}{x}\right)$ where the function $f\left.\cdots \right)$ is potentially a highly complex machine learning model, $\underset{\_}{x}$ is potentially a long vector of various measurements, and $y$ is the target of the optimization, that is, the quantity that we want to maximize. Of the elements of $\underset{\_}{x}$, there will be some that the operators of the facility can change at will. These are called set-points because we can set them. There will also be values that are fixed by powers that are completely beyond our control, for example, the weather, market prices, raw material qualities and other factors determined outside our facility. These are fixed by the outside world and so correspond to constraints.

Now we can ask: What are the values of the set-points, given all the boundary conditions and constrained values, that lead to the largest possible value of $y$? This is the central question of optimization. Let us call any setting of the set-points that satisfies all boundary conditions, a solution to the problem. We then seek the solution with the maximum outcome in $y$.

3.8. Practical advice

1. 1. Do you need to calculate values at a future time? You will probably get the best results from a recurrent neural network such as an LSTM as it is designed to model time dependencies.
2. 2. Do you need to calculate a continuous number in terms of others? This is the standard regression problem and can be reliably solved using deep neural networks.
3. 3. Do you need to identify the category that something belongs to? This is classification and the best general classification method is random forest.
4. 4. Do you need to compute the likelihood that something will happen (risk analysis) or has happened (root-cause analysis)? Bayesian networks will do this very nicely.
5. 5. Do you need to do several of these things? Then you will need to make more than one model.