Machine learning is regarded as a subfield of artificial intelligence that deals with algorithms and technologies to squeeze out knowledge from data. Its fundamental ingredient is Big Data, since without help of a machine, our attempt to manually process huge volumes of data would be hopeless. As a product of computer science, machine learning tries to approach problems algorithmically rather than purely via mathematics. An external spectator of a machine learning module would admire it as some sort of magic happening inside a box. Eager reductionism may lead us to say that it all is just “bare” code executed on a classical computer system. Of course, such a statement would be an abomination. Machine learning does belong to a separate branch of software, which learns from data instead of blindly following predefined rules. Nonetheless, for its efficient application, we must know how and what such algorithms learn as well as what type of algorithm(s) to apply in a given context. No machine learning system can notice that it is misappropriated. The goal of this chapter is to lay down the foundational concepts and principles of machine learning exclusively through examples.
Supervised learning: Here an algorithm is exercised on known observations until it achieves a desirable level of performance. The main challenge is to acquire enough high-quality marked data for appropriate training. Some members of this group are linear regression, logistic regression, support vector machine, naive Bayes classifier, etc.
Unsupervised learning: These algorithms try to autonomously discover hidden structures in data for the purpose of grouping them, finding interesting relationships (a.k.a. association rule learning), or reducing inherent dimensionality (describe effects with fewer features). Some members of this group are K-Means clustering, principal component analysis, manifold learning, Aprori algorithm, etc.
Semi-supervised learning: These algorithms try to decipher hidden structures based on guidance from labeled specimens. It isn’t uncommon that unsupervised learning algorithms are run on half-marked data, as a preprocessing step, to label remaining data points (a technique known as label propagation).
Full-batch learning (a.k.a. statistical learning): We feed an algorithm all training data at once. After initial training, model parameters remain fixed. This scheme is also most popular for demonstrating various algorithms in action due to its simplicity.
Mini-batch learning: We feed an algorithm in a chunked manner. For example, with an enormous training sample, the standard gradient descent optimization method is prohibitive. This is where the stochastic gradient descent alternative becomes attractive, which works with a reduced chunk at any moment of time.
Online learning: This is an extreme case of mini-batching in which the batch size is reduced to a single observation. The usual setup is that the system is warmed up on historical data and left to update its parameters while running in production. This learning method has many subvariants, which are listed below as separate groups to avoid nesting.
Streaming: Here, an algorithm also works on a single observation at a time but cannot revisit past records (generic online learning algorithms are allowed to go over data multiple times). It is OK for a streaming system to cache recent items or keep running statistics (like moving average), although these are miniscule in comparison to the training corpus used in previous sets. Finally, many streaming approaches actually rely on micro-batching (with tiny batches). They are still called streaming since they don’t pass multiple times over older batches.
Active learning: This method actively seeks feedback from other entities while running. For example, an online learning–based spam filter may incorporate user actions (declassify a piec of spam as normal, or vice versa) to continuously update itself. A semi-supervised system could ask for help to label conflicting (confusing) observations.
Reinforcement learning: This special dynamic learning method builds upon interaction between the system and its environment. Using an efficient feedback loop and rewarding mechanism (with positive and negative rewards), an algorithm learns what actions are proper in a given context. Robots typically learn in this manner.
We can also embark on separating algorithms by their similarity (e.g., neural networks, spectral methods, Bayesian networks, etc.). Nevertheless, any classification can only broadly enumerate knowledge areas, with blurred borders and overlaps. However, some notions are shared by all of them. These will be the topic in the rest of this chapter.
Irrespectively of what approach you choose, at one point in time, you will need to peek under the hood of many algorithms to tweak advanced parameters, combine them to work in a unified fashion (a.k.a. ensemble), or connect them as a pipeline for staged processing. You cannot eventually escape deep mathematics. The good news is that you can survive this encounter in incremental fashion.
Exposition of Core Concepts and Techniques
data_generator.py Module
Our intimate knowledge about each underlying data generator process allows us to illuminate concepts in an exact manner. The real world simulated by generate_response has the following vector form: y = f (β0 + Xβ1 : n) + ε (see also the sidebar “Linear Regression Varieties”), where f is a discretionary function. The error term represents an inherent noise, which isn’t encompassed by the model. We cast it as a Gaussian random variable 
; that is, a term that symmetrically fluctuates around the output following the Normal distribution. For simplicity, we produce homoscedastic outputs, meaning the errors are uncorrelated and uniform (they don’t depend on X). The vector β is arbitrary.
1From outside we can only gather observations and try to establish sound relationships between features and outputs
Establishing relationships between features and output allows us to either predict future outcomes or better understand what is going on in the system. In Figure 7-1, the frontal digits represent assumed predictors and fitted values (what we closely examine and crunch in production), while the digits on the globe represent real-world data. We can state that our model closely resembles the real world if there is an acceptable match between them.
Linear Regression Varieties
Parametric: m = β0 + Xβ1 : n is smooth, defined, and easily interpretable.
Nonparametric: m is smooth, must be discovered from data, flexible, and hard to interpret.
Semiparametric: Somewhere between the preceding two cases (partially structured).
You must evaluate the cost-benefit ratio before deciding which version to use. For example, if you have outliers (extreme but valid observations that you cannot eradicate), then simple parametric linear regression isn’t a good choice. The nonparametric version is more robust and less vulnerable regarding outliers.
Imagine that you encounter a helpful wizard, who reveals everything about the real world’s processes. With that knowledge, how accurately may you predict future outcomes? You definitely cannot talk in absolute terms, as you have to contend with randomness in making your predictions; you may only speak about probabilities of seeing particular responses. In other words, you can develop a conditional probability distribution of an output given an observation. This can be expressed as 
The truth is, you are not straying far from reality (and sanity) if you are hoping for such a wizard. The scikit-learn framework ( https://scikit-learn.org ) is a wizard in its own right. You just need to carefully interpret scikit-learn’s words and never misapply the power bestowed on you. The aim of this section is to shed some light on necessary communication skills and language so that you become this wizard’s beloved apprentice.
observer.py Module
First Part of session.py Module (Some Imports Are for Later Use)
Figure 7-2 pictures how error accumulates; the line itself was calculated over the whole dataset, so it appears wrong on this subset. The MSE is simply an average of sum of squared errors (SSE). Each vertical dashed red line designates a single error term (difference between true yi and its predicted value 
). The model’s coefficients minimize this MSE (loss function), and the corresponding expression to calculate the vector β is an
unbiased minimal variance estimator
. It is independent of the inherent noise factor, and MSE is an estimator of this error term’s spread (
). Estimates and values dependent upon them are denoted by a hat symbol.
Note
The intercept only makes sense when the matching predictor is of ratio level data type (the notion of absolute zero exists). Otherwise, it should be turned off (see the constructor for the LinearRegression class).
Hint
The print_parameters method is rudimentary. For a complete R-style summary of model parameters, you may want to utilize the statsmodels package (visit https://www.statsmodels.org ).
Once a model is prepared (trained), it is ready for production usage to deal with unseen data. This is what I refer to as a
runtime model
. All learned (runtime) parameters are its integral part and are usually distributable over various communication channels. This property allows training to be done separately on powerful machines with lots of data. For example, all we need to know in production to handle real data is the corresponding vector 
. Calculating 
can be performed on any constrained device.
Error terms 
, which are squared, summed up, and averaged to calculate MSE. The root MSE, (
), is useful to restore the response’s original unit.
Overfitting
We will augment the linear model with polynomial features of various degrees; the model itself remains linear in terms of features. The aim is to demonstrate overfitting. Since this model is more powerful than the previously investigated model, it has enough capacity to encompass erroneous fluctuations in data; in a deterministic case (without error), you wouldn’t notice any difference. Machine learning algorithms cannot decipher that irregularities in data aren’t critical. There is a mechanism to detect overfitting by splitting historical data into training and testing sets.
Contains Functions to Exemplify Overfitting
When a model matches the truth, the MSEs induced by both the training set and the test set gravitate around the achievable minimum MSE (reflects inherent variance in data), as shown in Figure 7-3. Otherwise, the test set shows a worse performance, as presented in Figure 7-4. This is a sign that the model isn’t generalizing properly and is picking up unimportant details from a training set. As a model’s power increases above the required level, the cross-validation (CV) score considerably deteriorates, too.
MSE of the test set approaches natural variance when the model’s complexity is just right
Furthermore, the variance between CV, testing, and training scores lessens as we use more training data. There is no test score marker when we use all available data for training. The horizontal dashed line denotes the inherent error in data (it may stem from measurement errors). A proper model wouldn’t try to embody this segment. In machine learning we cannot simply instruct the computer to forget about inherent error. The trick is to set your model’s complexity just right, so that it has no capacity to memorize unwanted details. Apparently, the machine first tried to suck in everything, when the training set was small enough, but later gave up and only followed the main trend. Consequently, you must possess enough training data for a given model’s complexity. Monstrous models (especially deep neural networks) must devour a huge amount of training data before being ready for production.
Overfitting and underfitting (demonstrated in the next section) are tightly associated with the central issue of supervised learning known as bias-variance trade-off . A highly biased model may miss important variances in the training data, while a high-variance model may try to capture nonessential properties. Balancing these two forces is one of the most difficult aspects of training machine learning algorithms. For example, extreme care must be given to decision trees, which may easily cover all edge cases in training data.
MSE of the test set rises above the inherent error level when the model’s complexity is too high
The cross-validation score is even out of range here. Observe that the model completely memorized a small training sample at the beginning and always managed to follow unintended fluctuations. One way to combat this situation is to saturate the system with more training data. In our case, it has started to reasonably perform on a test set only from 60% of the training set size.
Underfitting and Feature Interaction
This section introduces another typical situation in which features interact. The goal is to showcase that it isn’t enough to solely identify pertinent features. You must understand how they interrelate in the real world. We assume two models: one that treats individual features as independent, and another that includes their interaction.
The complex model’s prediction line is jagged due to an attempt to encompass inherent error
Function to Exemplify Underfitting via Feature Interaction
MSEs of all sets are far away from inherent error with a weak model
Collinearity
Extension of session.py with the demo_collinearity Function
The only difference between these runs is the significance of x_combined. This is a typical sign of collinearity. The system is confused whether to “work” with x_combined or x_collinear, since one is a direct linear combination of another. There is also a strong relation between x_normal and x_combined. A practical way to check for such interdependence is to generate a scatter matrix plot , as shown in Figure 7-7; this is the final result of running demo_collinearity (the columns are renamed to better fit on the diagram). It is possible to produce a correlation matrix, but it will only reveal linear relationships. A diagram may show you non-linear associations, too.
Independence assumption is common in machine learning algorithms. Naive Bayes method fully relies upon this characteristic. Feature dependence has negative consequences on convergence in logistic regression. Moreover, highly related features are redundant and just complicate a model.
Residuals Plot
As a data scientist, you must constantly seek to inspect problems from multiple angles. Likewise, you should know about complementary plotting techniques, since they could illuminate otherwise invisible aspects of a problem. You have just seen the utility of a scatter matrix plot. In this section we generate two worlds, one linear and one quadratic. Both have tiny coefficients, inherent randomness, and are approximated by truly linear models. Listing 7-7 contains the demo_residuals function to highlight the importance of residuals plots.
Scatter plots of all pairs of variables, with density plots on diagonal
Function That Creates Two Deceptive Offers, Where the Obvious One Isn’t Obviously Wrong
This code reuses the model instance from case 1 in case 2. You must always read carefully the documentation about what is happening when you call the fit method repeatedly. Here is the citation from scikit-learn’s tutorial (see https://scikit-learn.org/stable/tutorial/basic/tutorial.html ): “Hyper-parameters of an estimator can be updated after it has been constructed via the set_params() method . Calling fit() more than once will overwrite what was learned by any previous fit().” This behavior is exactly what we need.
The direction of the slope is wrong for case 1, since it should be positive (see Listing 7-7)
The residuals plot clearly favors case 1 over case 2 with evident visual explanation; the curvature nicely reveals a “quadratic” pattern in residuals
There is no residual pattern for case 1. See Listing 7-2 and the evaluate_model function (this creates a residuals plot with the lowest line for depicting trend).
Regularization
We don’t know in advance which model is optimal. We can either start with a weak model and add more features or start with an overly complex model and try to tone it down. Neither of these approaches is scalable with manual work. The idea is to just err on the side of complexity and utilize some automation to reduce the model’s complexity toward an optimal level. This is all about regularization , an automatic mechanism to eschew overfitting. There are many types of regularization. We will use the Ridge regression with built-in cross-validation.
that is added to the basic loss function (in our case MSE). The vector w contains the model’s coefficients (weights). The parameter α balances minimization attempts between MSE and penalty term. Higher values tend to reward lesser coefficients, and vice versa. Alpha is calculated by some trial-and-error method (such as cross-validation over a list of candidates). Listing 7-8 shows the function to illuminate regularization, while the aftermath is shown in Figure 7-10.Implementation of demo_regularization Function to Showcase Ridge Regression
All the magic happens inside the scikit-learn framework’s RidgeCV class.
Predicting Financial Movements Case Study
The realm of financial modeling will shed some light on time series analysis , where we want to react to events in near real time (for example, predicting stock prices in markets). This is totally different than what we have done thus far using batch processing. My aim here isn’t to develop a new breed of stock market application, but to drive your attention to novel problems and potential solutions with streaming data. You cannot access such data all at once, so stream processing techniques are intrinsically incremental and casual (they act on current knowledge to predict future outcomes). This entails that model parameters evolve over time instead of being fixed at the end of the training stage. Monitoring and regularizing this evolution are also some new tasks compared to classical approaches (see references [1-2]).
Although the model’s complexity, in term of features, is the same as in our example for overfitting, due to regularization it doesn’t overfit as earlier
Data Retrieval
We are relying on Alpha Vantage’s API (see https://www.alphavantage.co/documentation/#daily ) to retrieve daily time series data. You can get a free API key from Alpha Vantage and insert it into the URL in the preceding code where indicated. There are limits on the number of requests that are well documented on the site. The outputsize parameter is set to full, which pulls up to 20 years’ worth of historical data. The output format (datatype parameter) is set to csv.
Data Preprocessing
Discovering Trends in Time Series
Variation of AAPL closing levels over time
Much smoother diagram compared to Figure 7-11
The y axis is now showing scaled values
Transforming Features
Log returns :
, where v denotes the asset’s value (such as closing price, adjusted closing price, volume, adjusted volume, etc.).Scaled percent returns :
. There is a method pct_change in Pandas to calculate this quantity. For daily returns, scaled percent returns are near to log returns. You can see this from Taylor expansion of the log function, when 
is very small, so only the first term (x − 1) matters.
This diagram has a couple of downward spikes that are far away from central values
We could treat spikes as outliers, which are hard to model and account for (unless you are a finance guru). One way to circumvent this problem is to use volatility-normalized log returns.
The half-life of 23 days amounts to a decay factor (weight) of 0.97 (you need to solve for w the equation 
), which controls how long the stock market remembers (or how fast it forgets) old events. The i-th data point has a decay factor of wi. Volatility is calculated as a rolling standard deviation across data points taking into account the exponential decay for old data. Higher decay damps volatility. You should experiment with this factor to match the desired risk level.
Streaming Amounts
For simplicity reasons, we have thus far managed the data in batch mode. Nonetheless, log returns, moving average, and volatility are all amounts that may be calculated in real time. For log returns, you just need to cache the last value to make an update. The same is true for volatility with exponential downweighting, although it is not as obvious as with log returns. To track a moving average, you will need to store and update the last numerator and denumerator.
Suppose that you know the current weighted variance 
, where w is the decay factor; for daily returns, we may assume the mean to be zero. The denominator is the sum of a geometric series whose limit is 
. When a new return r0 arrives, then we have 
. You can compute the current volatility as 
.
To perform volatility normalization in streaming mode, you need to divide the new return with the current volatility: 
Of course, you would do this before updating the current volatility; that is, before executing Vcurrent ← Vnew.
This diagram seems to be properly centered around zero, but we should still eyeball its distribution
Normalization has smoothed out the log returns and made the time series better behaved
Histogram of the non-normalized price log returns, which is heavily left-tailed
Histogram of the normalized price log returns, which is bell shaped
This diagram is normally distributed with a slight right tail
data_preprocessing.py Module , Which Contains Functions to Prepare the Data Frame for the Analysis Phase
data_visualization.py Module , Which Contains Auxiliary Visualizations of Time Series
driver.py Module , Which Connects All the Pieces Together (First Part of File Shown)
Feature Engineering
Currently, we have raw close levels, raw volume levels, volatility-normalized closing log returns, and volume log returns as our features (we will create more). To see how these impacts a potential target (like, predicted stock price change) it is useful to consult Pearson’s correlation coefficient r. It is an indicator for a linear relationship between two features, whose range is [-1, 1]. A positive correlation means as one value increases/decreases the other does the same. A negative correlation denotes the opposite behavior. A value of zero represents the absence of a linear relationship, although the quantities may be interrelated in non-linear ways. Usually, if |r| > 0.3, then we can consider the correlation to be noticeable (this a judgement call, so take this heuristics with a decent pinch of salt).
feature_engineering.py Module , with Function to Investigate Auto-correlation
Function to Create a Scatter Plot to Trace Auto-correlation (data_visualization.py Module)
As current volume log returns increase, future volume log returns decrease, and vice versa
By varying the period’s length, it is possible to find the highest level of auto-correlation, which may serve as a basis for creating features. It seems that 5 days is a good choice (you may experiment with different periods using the accompanying code base). Therefore, out target feature will be the 5-days future price change, while our initial basic features will be the current 5-days price and volume changes.
, where 
. OBV is connecting volume flow with price changes. Listing 7-14 contains the code for creating candidate features.Code to Create Features Described in Text (Part of feature_engineering.py)
Code to Report Correlations Using Seaborn’s heatmap Facility
This diagram shows only the relevant part of the correlation matrix
All diagonal elements are 1, since features are perfectly aligned with themselves. The matrix is symmetric, so the upper-right triangle is redundant. None of the features are strongly coupled with the target, at least not in a linear way.
Implementing Streaming Linear Regression
Content of the streaming_regression.py Module to Showcase Streaming Linear Regression in Online Mode
Last Piece of the Main driver.py Module
At the time of this writing, the StreamingLinearRegressionWithSGD class is missing the setIntercept method . Consequently, we have very “strange” values for R-squared and explained variance. It is also crucial to remember the importance of scaling features before training the model. The RobustScaler scaler is convenient if you aren’t sure about the distribution of your regressors.
Exercise 7-1. Improve Reusability
The plot_mse function presets font sizes for title and axes. This tactic is also repeated in other places to preserve consistency. Nonetheless, this approach is a real maintenance nightmare (notice that explain_sse repeats the same setup). Improve the code to centralize setting common parameters (hint: read about style sheets at https://matplotlib.org/users/customizing.html ). What else can you devise to make the code base more flexible and reusable?
As you work on various machine learning problems, utility functions become very handy. There is no point in reinventing the wheel each time you need to plot MSEs. Try to make your code base reusable as much as you can.
Exercise 7-2. Fix a Bug
Your task is to find out why this marked line is wrong and implement a fix. In reality, the situation is even worse, because nobody will point out to you a wrong section of code. Locating the exact place of an issue is a huge milestone toward correcting errors.
Hint: Figure 7-6 was produced with both lines commented out. Obviously, such resolution is a quick and dirty hack. Try running demo_underfitting with the original plot_mse function and see what happens.
Exercise 7-3. Avoid Side-Effects
In Listing 7-12 you can see a function with side effects. It modifies the input data frame with extra columns. This modification is a prerequisite for calling the function presented in Listing 7-14. All the chaining is driven from the driver module.
Programming in terms of side effects is generally a bad practice. You may wonder why the model is altered in this fashion (see Listing 7-2) after calling its fit method. The crucial difference is that model encapsulates its state inside an object, where you have more control over changes to internal stuff. When ordinary functions are spread out with assumed side effects, then it is easy to lose control and create an unmaintainable mess.
Refactor the feature_engineering.py module to group feature-related manipulations inside a dedicated class. Compare how such object orientation helps to retain control over internals of the system.
Exercise 7-4. Experience Streaming Behavior
This was OK to demonstrate the scaffolding of the solution but makes no sense in a real environment. The whole point of streaming is to allow data to continuously arrive in chunks. Apache Spark even allows streams to be combined, so that you may have data coming over multiple channels (for example, you can also monitor a folder for new data files and parse them in real time).
Refactor the solution to have many parts in the training and test sets. You need to split the df_train and df_test referenced data frames into sections and convert each into an RDD. Finally, provide these parts in a list to the queueStream method. Observe what you get on the output (you should receive as many reports as there are pieces of test data).
Bonus task: The current accuracy report doesn’t reveal too much about performance. You may want to create a scatter plot of predictions vs. actual values (observations from test data). Draw also the ideal line (this is basically the function y=x).
Summary
This chapter barely scratches the surface of the machine learning (ML) domain. Even dozens of books wouldn’t be enough to fully cover (even at an introductory level) all available algorithms and technologies. The major aim of this chapter was to present common concepts that permeate the ML knowledge area. Without being aware of underfitting, overfitting, regularization, scaling, and similar topics, there is no way to be efficient with any ML approach. To be proficient in ML, you must also recall the golden tenet of data science and engineering: “Keep it simple, stupid!” (a.k.a. KISS principle). Typically, you don’t even need ML to solve a problem, and rarely will you ever need to fire up complicated deep neural networks.
Another potent message of this chapter is the rather blurred borderline between science and art regarding parameter tuning. There are some rules of thumb (for a good overview, I suggest the document at http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf ), but you will need lots of experimentation and trial. Consequently, to make all this happen in reasonable amount of time, you will need powerful hardware (readily available in a cloud as infrastructure or platform as a service).
ML is all about mathematics. There is no way to escape this fact. Neural networks appear to give you a seemingly good escape route (we will cover them in detail in Chapter 12), but eventually you will need to dig under the hood to understand what is going on. This is tightly associated with interpretability of your model. With our ordinary least-square method of regression, coefficients can be easily explained. A particular coefficient represents how much the target changes for a unit change in the corresponding feature. With more complex models, the situation is different. At any rate, I will talk more about inner characteristics of models in Chapter 9.
ML is getting huge attention from both research institutions and industry. This isn’t surprising, since as we delve more into the realm of Big Data, there is a greater need to handle such large amounts of data in an efficient manner. Only with the help of machines can we hope to seize control of massive data. One hot topic is transfer learning, which is an attempt to boost reuse of trained models. The idea is to leverage models optimized for one task to perform well on other tasks, too (maybe with minor extra tweaking and training). This will surely trigger a whole bunch of new algorithms and technologies.
One urgent need in the area of machine learning is an efficient and standardized way to exchange prediction models. One such standard is the Predictive Model Markup Language (PMML). There is a Python library for converting scikit-learn pipelines to PMML at https://github.com/jpmml/sklearn2pmml .
References
- 1.
Christopher Bishop, Pattern Recognition and Machine Learning, Springer, 2006.
- 2.
Nathan George, “Machine Learning for Finance in Python,” DataCamp, https://www.datacamp.com/courses/machine-learning-for-finance-in-python .
- 3.
Ian H. Witten, Eibe Frank, Mark A. Hall, and Christopher J. Pal, Data Mining: Practical Machine Learning Tools and Techniques, Fourth Edition, Morgan Kaufmann, 2016.