Getting the Data

Therefore, lets observe the data with a 2D scatter plot Figure 8-1.

In[2]:= ListPlot[Transpose[{x,y}],AxesLabel→{"X axis","Y axis"},PlotStyle→Red]

Out[2]=

Algorithm Implementation

Let us now proceed to the implementation of the algorithm with the Wolfram Language. The algorithm will consist of defining the constants, the number of iterations, and the learning rate. Then we will create two lists containing initial values of zero, in which the values of the coefficients for each iteration will be stored. Later, we will perform the calculation of the coefficients through a loop with Table, which will not end until we reach the number of iterations. In our case, we will establish a number of iterations of 250 with a learning rate of 1.

In[3]:= itt=250;(*Number of iterations*)

α=1;(*Learning rate*)

θ0=Range@@{0,itt};(* Array for values of Theta_0*)

θ1=Range@@{0,itt};(* Array for values of Theta_1*)

Table[{

θ0[[i+1]]=θ0[[i]]- *Sum[(θ0[[i]]+θ1[[i]]* x[[j]]-y[[j]]),{j,1,Length@x}];

θ1[[i+1]]=θ1[[i]]- *Sum[( θ0[[i]]+θ1[[i]]*x[[j]]- y[[j]])*x[[j]],{j,1,Length@x}];},{i,1,itt}];

Let us look at how the line fits the data in Figure 8-2.

In[6]:= Show[{Plot[F[X],{X,0,1},PlotStyle→Blue,AxesLabel→{"X axis","Y axis"}],ListPlot[Transpose[{x,y}],PlotStyle→Red]}]

Out[6]=

Since we have built the linear model, we can make a graphical comparison of the variation of the learning rate with the number of iterations and the loss value given by the function J.

But first we must declare the loss function J. For the summation we can either use the special symbols of sigma or write or Sum [expr, {i,i_max}].

In[7]:=J[Theta0_,Theta1_]:=1/(2*Length[x])*Sum[(Theta0 + (Theta1*x[[i]]) - y[[i]])^2,{i,1,Length@x}]

Below is the graph of loss vs. each interaction for learning rate values of α1=1, α2=0.1, α3=0.01, α4=0.001, and α5=0.001, when repeating the process Figure 8-3.

Multiple Alphas

In the previous graph (Figure 8-3) we can visualize the size of iterations with respect to cost and how it varies depending on the value of alpha. With a high learning rate, we can cover more ground at each step, but we risk exceeding the lowest point. To know whether the algorithm works, we must see that the loss function is decreasing in each new iteration. The opposite case would be an indicator that the algorithm is not working properly; this can be attributed to various factors such as a code error or an incorrect value of the learning rate. As we see in the graph, adequate values of alpha correspond to small values between a scale of 1 to 10^-4. It is not necessary to have to use these same values; you can use values that are within this range. Depending on the form of the data it is possible that the algorithm may or may not converge with different alpha values as the same for the iteration steps. If we choose very small alpha values, the algorithm can take a long time to converge, as we can see for alpha values 10^-3 or 10^-4.

Predict Function

The Predict function helps us predict values by creating a predictor function using the training data. It also allows us to choose different learning algorithms, the purpose of which is to be able to predict a numerical, visual, categorical value or a combination. The methods to choose from are decision tree, gradient boosted tree, linear regression, neural network, nearest neighbors, random forest, and gaussian process. For each method, there are options within it; the options vary depending on the algorithm chosen to train the predictor function. Let us look at the linear regression method. The input data for Predict can be in the form of a list of rules, associations, or a dataset.

Boston Dataset

Let’s look at the first example loading the Boston Homes data from the Wolfram Data Repository (Figure 8-4). The Boston dataset contains information about housing in the Boston Mass area. To look for more in-depth information, visit the article by David Harrison, and Daniel Rubinfeld, “Hedonic Housing Prices and the Demand for Clean Air” which appears in the. Journal of Environmental Economics and Management, (1978; 5[1], 81-102. https://doi.org/10.1016/0095-0696(78)90006-2) or the book Regression Diagnostics: Identifying Influential Data and Sources of Collinearity: 546 by David Belsley, Edwin Kuh, and Roy Welsch, (2013; Wiley-Interscience).

In[1]:= Bstn=ResourceData[ResourceObject["Sample Data: Boston Homes"]]

Out[1]=

Model Creation

We will try to create a model that is capable of predicting housing prices in the Boston area through the number of rooms in the dwelling. To achieve this, the columns of interest correspond to RM (average number of rooms per dwelling) and MEDV (median value of owner-occupied homes), since we want to find out if there is a linear relationship between the number of rooms and the price of the house. Applying a bit of common sense, the houses with the largest number of rooms are larger and therefore have the capacity to store more people, making the price go up.

By observing the matrix plot (Figure 8-6), it can be concluded that there is a good linear relationship between RM and MEDV.

When entering the code, depending on the option added to TrainingProgressReporting, a progress bar and panel report should appear (Figure 8-8). The time of the panel displayed depends on the training time of the model. To set a specific time for the training time, add as an option TimeGoal, which specifies how long the training should last for the model. Time values are seconds of CPU time—that is, the number with no units. With units of time (seconds, minutes, and hours), the use of Quantity command is needed, like TimeGoal → Quantity [“time magnitude”, #] & / @ {“Second”, “Minute”, “Hour”}.

The information panel (Figure 8-9) includes data type, root mean squared (StandardDeviation), method, batch evaluation speed, loss, model memory, number of examples for training, and training time. The graphics at the bottom of the panel are for standard deviation, model learning curve, and learning curve for the other algorithms. If you hover the cursor pointer over the numerical parameters, it will show the confidence intervals and units. If it’s done by the name of the method, it will show the parameters of the linear regression method. Since we did not select a specific optimization algorithm within the LinearRegression method, Mathematica tries to search through the algorithms for the best one (this can be viewed in the learning curve for all algorithms). We will see how to access these options further down the line.

Model Measurements

This report (Figure 8-11) shows different parameters, such as the root mean square (standard deviation), mean cross entropy, among others. And it shows us a graph of the fit of the model along with the current values and predicted values. We see that the model is good for most cases, with the exception that there are still some outliers that affect performance.

To better understand the precision of the model, let’s look at the root mean squared error (RMSE) and RSquared (coefficient of determination) shown in Figure 8-12. To display the associated uncertainties, use the option ComputeUncertainty with true value.

In[11]:= Dataset[AssociationMap[PRM[#,ComputeUncertainty→True]&,{"StandardDeviation","RSquared"}]]

Out[11]=

This gives us a slightly high RMSE value and not a good r-squared value. Remembering that the value of r squared indicates how good the model is for making predictions. These two values would indicate that although there may be a linear relationship between the number of rooms and prices, this is not necessarily explained by a linear regression. These observations are also consistent, remembering that we obtained a correlation value of 0.7.

Model Assessment

The graphs made within the model are the model graph and the target variable (ComparisonPlot). To check the distribution of the variance, use the ResidualHistogram function, and to check the residual plot, use ResidualPlot. These are shown in Figure 8-13.

In[12]:= PRM[#]&/@{"ResidualHistogram","ResidualPlot","ComparisonPlot"}

Out[12]=

As we can see, there are terms such as L1Regularization, L2Regularization, and OptimizationMethod. The first two terms are associated with regularization methods, and L1 refers to the Lasso regression name and L2 to the Ridge regression name. Regularization is used to minimize the complexity of the model, in addition to reducing the variation; it also improves the precision of the model, solving problems of overfitting. This is accomplished by adding a penalty to the loss function; this penalty is added to the sum of the absolute value of the coefficient , whereas for L2, it is given by the expression , where the function to minimize is the loss function . For more mathematical depth, visit Artificial Intelligence: A Modern Approach. by Stuart Russell and Peter Norvig (2010 Upper Saddle River, NJ: Prentice Hall) and An Introduction to Statistical Learning: With Applications in R by Gareth James, Trevor Hastie, Robert Tibshirani, and Daniela Witten (2017; 1st ed. 2013, Corr. 7th printing 2017 ed.: Springer). The third term is the option of which optimization method we want to choose; the existing methods are NormalEquation, StochasticGradientDescent, and OrthantWiseQuasiNewton. That said, it must be emphasized that when using the vector of coefficients with the L1 and L2 standards, this is known as an Elastic Net regression model. Elastic Net might be used in circumstantces when there is correlation in the parameters. For more theory, use the next reference, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Second Edition by Trevor Hastie, Robert Tibshirani, and Jerome Friedman(2nd 2009, Corr. 9th Printing 2017 ed.: Springer).

Retraining Model Hyperparameters

To see the properties related to an example, type properties after the input data for the Predictorfunction—for example, PF2[“example”, “Properties”].

Now, let’s compare the performance of the new model by showing the graphs and metrics like before (Figure 8-14 and Figure 8-15).

In[16]:= PRM2=PredictorMeasurements[PF2,test];

PRM[#]&/@{"ResidualHistogram","ResidualPlot","ComparisonPlot"}

Dataset[AssociationMap[PRM2[#,ComputeUncertainty→True]&,{"StandardDeviation","RSquared"}]]

Out[16]=

Making observations in the graphs, we see the model merely decrease to a certain degree; this agrees with the new value of r squared, which decreases to 0.51. However, it is still a poor model when it comes to making future predictions. This can be attributed to the optimization choice, the L1 and L2 parameters choice.

Titanic Dataset

For the following example we will use the titanic dataset , which is a dataset that describes the survival status of the passengers. The variables used are class, age, sex, and survival condition. We will load the data directly as a dataset (Figure 8-16) from the ExampleData and enumerate the rows of the dataset.

These numbers represent the rows that contain the missing data for the age column. To eliminate them we use the DeleteMissing command, considering that there is missing data at level 1. The final dataset is seen in (Figure 8-17)

In[5]:= Titanic=DeleteMissing[Titanic,1,1]

Out[5]=

This means that there is no content associated with key 16. If you want to check all keys, use the row list of the missing data.

Data Exploration

After eliminating the rows with the missing elements, we see that the dataset consists of 284 elements in first class, 261 in second class, and 501 in third class (Figure 8-19). Also note that more than half of the registered passengers were male and that there were more deaths than survivors. It is possible to verify this graphically by showing the percentages (Figure 8-19). The same approach is applied to the columns class and sex.

In[8]:= Row[{PieChart[{N@(#[[1]]/Total@#),N@(#[[2]]/Total@#)}&[Counts[Query[All,"survived"][Titanic]]], PlotLabel→Style["Percentage of survival",#3,#4], ChartLegends→ {"Survived", "Died"}, ImageSize→#1,ChartStyle→#2,LabelingFunction→(Placed[Row[{SetPrecision[100#,3],"%"}],"RadialCallout"]&)],

PieChart[{N@(#[[1]]/Total@#),N@(#[[2]]/Total@#)}&[Counts[Query[All,"sex"][Titanic]]], PlotLabel→Style["Percentage by sex",#3,#4], ChartLegends→{"Female", "Male"}, ImageSize→#1,ChartStyle→#2,LabelingFunction→(Placed[Row[{SetPrecision[100#,3],"%"}],"RadialCallout"]&)],

PieChart[{N@(#[[1]]/Total@#),N@(#[[2]]/Total@#),N@(#[[3]]/Total@#)}&[Counts[Query[All,"class"][Titanic]]], PlotLabel→Style["Percentage by class",#3,#4], ChartLegends→{"1st", "2nd","3rd"}, ImageSize→#1,ChartStyle→#2,LabelingFunction→(Placed[Row[{SetPrecision[100#,3],"%"}],"RadialCallout"]&)]},"----"]&[200,{ColorData[97,20],ColorData[97,13],ColorData[97,32]},Black,20]

Out[8]=

For this case , we are going to predict the survival of the Titanic passengers. We will build a model that will classify whether the given class, age, and sex will survive or not. The features will be class, age, and sex, and the target will be the survival status. We are going to use these variables as the features, which the model will then use to classify whether their class, age, and sex survive or not, which is our target variable. For this we divide the dataset into 80% training (837 elements) and 20% test (209 elements). To split the dataset, first we will do a random sampling; after we will extract the keys of the IDs and create the new dataset divided by train and test set (Figure 8-20).

In[9]:= BlockRandom[SeedRandom[8888];

RandomSample[Titanic]];

Keys@Normal@Query[All][%];

{train,test}={%[[1;;837]],%[[838;;1046]]};

dataset=Query[<|"Train"→{Map[Key,train]},"Test"→{Map[Key,test]} |> ][Titanic]

Out[9]=

Classify Function

The Classify command is another super function used in the Wolfram Language machine learning scheme. This function can be used in tasks that consist of solving a classification problem. The data that this function accepts are numerical, textual, sound, and image data. The input data of this function can be in the same way as with the Predict function {x → y}. However, it is also possible to enter data as a list of elements, as an association of elements, or as a dataset. In this case we will introduce it as a dataset.

In this case we will extract the data from the dataset format by specifying that the columns input (class, age, sex) pointing to the target (survived). Now let’s build the classifier function (Figure 8-21), with the following options, Method → {LogisticRegression, L1 → Automatic, L2 → Automatic}. When choosing Automatic, we let Mathematica choose the best combination of L1 and L2 parameters. For the OptimizationMethod set the StochasticGradientDescent method. And for performance goal set Quality. Finally, choosing a seed with a value of 100,000 and the CPU unit as the target device.

After training, like with the Predict function, the Classify function returns a classifier function object (Figure 8-21) instead of a predictor function. Inspecting the classifier function we can see the input data types, which are two—nominal and numerical. The classes, which is the survival status—either false or true. The method used (Logistic Regression); and the number of examples (837). To obtain information on the model use the Information command. Let’s look at the model report (Figure 8-22).

In[11]:= Information[CF]

Out[11]=

The probabilities of the latter example show that survival status of the passenger may be more inclined to the False status.

To see the full properties of a new classification, type the example followed by Properties. The properties included are Decision (best choice of class according to probabilities and its utility function) and Distribution (categorical distribution object). Probabilities of each class are displayed as associations, ExpectedUtilities (expected probabilities), LogProbabilities (natural logarithm probabilities), Probabilities(probabilities of all classes), and TopProbabilities (most likely class). This is displayed in the following dataset (Figure 8-24).

In[14]:= Dataset@

AssociationMap[CF[{"3rd",23,"male"},#] &,{"Decision","Distribution","ExpectedUtilities","LogProbabilities","Probabilities","TopProbabilities"}]

Out[14]=

Testing the Model

The object returned is called a ClassifierMeasurementsObject (Figure 8-25), which is used to look for the properties of the ClassifierFunction after testing the test set. Let’s now look at the report (Figure 8-26).

In[16]:= CM["Report"]

Out[16]=

The report seen in the figure shows information such as the number of test examples, the accuracy, and the accuracy baseline, among others. It also shows us the confusion matrix, which shows us the prediction results for the classification model, showing the number of correct and incorrect predictions; these being broken down by class in this case return either false or true, which gives us an idea of the errors the model is making and the type of error it is making. Basically, it shows us the true positives and true negatives and false-positives and false-negatives for each class.

Let’s look at the graph (confusion matrix) in a concrete way (Figure 8-27).

In[17]:= CM["ConfusionMatrixPlot"]

Out[17]=

To get the values of the confusion matrix, use CM[“ConfusionMatrix”] or class CM[“ConfusionFunction”].

Looking at the plot, we see that the model classified, starting from left to right at the top, 106 examples of false correctly classified, 21 examples of false as true, 34 examples of true as false, and 48 examples of true correctly. To better visualize the performance, let’s look at the ROC curves (Figure 8-28) for each class, their respective values, and the Matthews correlation coefficient and AUC values.

In[18]:= {CM["ROCCurve"],Dataset@<|{"AUC"→CM["AreaUnderROCCurve"]},{"MCC"→CM["MatthewsCorrelationCoefficient"]}|>}

Another way to see if the model behaves consistently in predictions is to look at key metric values like Accuracy, Recall, F1Score Precision, and the Accuracy rejection plot (Figure 8-29). Let’s look at these metrics for the model.

In[21]:= CM[{"Accuracy","Recall","F1Score","Precision","AccuracyRejectionPlot"}]//TableForm

Out[21]//TableForm=

To see related metrics about the accuracy, type the following properties: Accuracy (number of correctly classified examples), AccuracyBaseline (accuracy of predicting the common class), and AccuracyRejectionPlot (ARC plot, accuracy rejection curve). However, to find information about probability and the predicted class of the test set, use the following properties: DecisionUtilities (value of the utility function for every example in the test set), Probabilities (probabilities for every example in the test set), and ProbabilityHistogram (histogram of class probabilities).

Let’s see how the probability behaves by plotting the probability of a passenger survival status (Figure 8-30). Remembering that the false state means that a passenger did not survive, and True means that a passenger did survived.

In[22]:= TruPlot=

{Plot[{CF[{#1,age,#4},"Probability"→ #6 ],CF[{#2,age,#4},"Probability"→ #6 ],CF[{#3,age,#4},"Probability"→ #6 ]}, {age,0,90},PlotLegends→{"Male in 1st class", "Male in 2nd class ", "Male in 3rd class"},FrameLabel→ {Style["Age in years",Bold,15], Style["Probability",Bold,15]}, Frame→#6,FrameTicks→#7,GridLines→ {{20,40,60,80}},ImageSize→#8],Plot[{CF[{#1,age,#5},"Probability"→ #6 ],CF[{#2,age,#5},"Probability"→ #6 ],CF[{#3,age,#5},"Probability"→ #6 ]}, {age,0,90},PlotLegends→{"Female in 1st class", "Female in 2nd class ", "Female in 3rd class"},FrameLabel→ {Style["Age in years",Bold,15], Style["Probability",Bold,15]}, Frame→#6,FrameTicks→#7,GridLines→ {{20,40,60,80}},ImageSize→#8]}&["1st","2nd","3rd","male","female",True,All,250];

FalsPlot={Plot[{CF[{#1,age,#4},"Probability"→ #6 ],CF[{#2,age,#4},"Probability"→ #6],CF[{#3,age,#4},"Probability"→ #6]}, {age,0,90},PlotLegends→{"Male in 1st class", "Male in 2nd class ", "Male in 3rd class"},FrameLabel→ {Style["Age in years",Bold,15], Style["Probability",Bold,15]}, Frame→True,FrameTicks→#7,GridLines→ {{20,40,60,80}},ImageSize→#8],Plot[{CF[{#1,age,#5},"Probability"→ #6 ],CF[{#2,age,#5},"Probability"→ #6 ],CF[{#3,age,#5},"Probability"→ #6]}, {age,0,90},PlotLegends→{"Female in 1st class", "Female in 2nd class ", "Female in 3rd class"},FrameLabel→ {Style["Age in years",Bold,15], Style["Probability",Bold,15]}, Frame→True,FrameTicks→#7,GridLines→ {{20,40,60,80}},ImageSize→#8]}&["1st","2nd","3rd","male","female",False,All,250];

Headings={Style["True class",Black,20,FontFamily→"Arial Rounded MT"],Style["False class",Black,20,FontFamily→"Arial Rounded MT"]};

Grid[{{Headings[[1]],Headings[[2]]},{TruPlot[[1]],FalsPlot[[2]]},{TruPlot[[2]],FalsPlot[[1]]}},Alignment→{{Center,Center},{None,None}},Dividers→{False,1}]

Out[22]=

In the graphs shown in Figure 8-30, clearly it is seen that the probability of survival decreases for males as the age goes up, to even hit values below 20% of chance, whether 1st, 2^nd, and 3rd class. This is contrary to the probability of survival for females, where it starts with values above 60% of chance and decreases as age increases too, hitting values above 50% for 1st class.

Clusters Identification

This tells us that the first cluster contains 145 elements, the second cluster contains 138 elements, and the third cluster contains 167 elements; these are the same number of points we created earlier, which equal 450. Each cluster is comprised of a two-point coordinate system. The FindClusters command returns the points where it identifies the clusters. Let's see the plot of the clusters generated; this is exhibited in Figure 8-32.

In[5]:= ListPlot[Clusters,PlotStyle→{Red,Blue,Green},PlotLegends→Automatic,Frame→True,FrameTicks→All,PlotLabel→Style["Cluster Plot",Italic,20,Black],Prolog→{LightYellow,Rectangle[Scaled[{0,0}],Scaled[{1,1}]]}]

Out[5]=

As we can see, Find Clusters automatically found the clusters and colored them. To explicitly establish the number of clusters to search, we add the desired number as the second argument—that is, in the form FindCluster [“points”, “number of clusters”]. In the previous example we set the method option to automatic. The different methods for finding the clusters are shown here. Agglomerate (which is the algorithm of single linkage clustering), density-based spatial clustering of applications with noise (DBSCAN), NeighborhoodContraction (nearest-neighbor chain algorithm), JarvisPatrick (Jarvis[Dash]Patrick clustering algorithm) , KMeans (k-means clustering), MeanShift (mean-shift clustering), KMedoids (k-medoids partitioning), SpanningTree (minimum spanning tree clustering), Spectral (spectral clustering), and GaussianMixture (Gaussian mixture model).

Choosing a Distance Function

In addition to the method option, there is also the DistanceFunction , which was given the value of Automatic. This option is used to define how the distance between the points is calculated. In general when we choose automatic, the square Euclidean distance is used ( ∑ (y_i − x_i)², SquaredEuclideanDistance). There are also other values for the distance function, EuclideanDistance , ManhattanDistance (∑|x _ {i} − y _ {i}|), ChessboardDistance, or ChebyshevDistance (max(|x _ {i} − y _ {i}|)), among others.

Now that we know how the clusters are identified, we want to know the centroid of each one. For this it is necessary to calculate the mean of the points of the clusters. The centroid of a series of points is obtained from the following expression, , which can be interpreted as the average of the points. For the calculation, we extract the data from each cluster and calculate its arithmetic mean.

In[6]:= {Cluster1Centroid,Cluster2Centroid,Cluster3Centroid}={N@Mean@Clusters[[1,All]],N@Mean@Clusters[[2,All]],N@Mean@Clusters[[3,All]]}

Out[6]= {{182.807,0.815713},{115.935,0.300888},{353.108,0.39227}}

Let´s plot the clusters together with their centroids to visualize how the points are classified with respect to each centroid (Figure 8-33).

In[7]:= ClusterPlot=ListPlot[Clusters,PlotStyle→{Red,Blue,Green},PlotLegends→{"Cluster 1","Cluster 2","Cluster 3"}];

CentroidPlot=ListPlot[{Cluster1Centroid,Cluster2Centroid,Cluster3Centroid},PlotStyle→Black];

Show[{ClusterPlot,CentroidPlot},Prolog→{LightYellow,Rectangle[Scaled[{0,0}],Scaled[{1,1}]]},Frame→ True,FrameTicks→ All,PlotLabel→Style["Cluster Plot",Italic,20,Black]]

Out[7]=

To make sure the first cluster corresponds to the red points, try using ListPlot to plot the points contained in Clusters[[1, All]], as well as those in the second cluster (blue) and third cluster (green).

As an alternative we can highlight the area of the centroids by adding the option Epilog to the plot. Epilog is another graphic option like Prolog, but we will use it to highlight the area of the centroid points (Figure 8-34).

In[8]:= Show[{ClusterPlot,CentroidPlot},Prolog→{LightYellow,Rectangle[Scaled[{0,0}],Scaled[{1,1}]]},Frame→ True,FrameTicks→ All,Epilog→{Opacity[0.2],PointSize[0.1],Point[Cluster1Centroid],Point[Cluster2Centroid],Point[Cluster3Centroid]}]

Out[8]=

Identifying Classes

The command returns us that class one contains 174, class two contains 132, class three contains 144. One point to clarify is that why the clusters identified with FindClusters and ClusteringCompnents defer. Well, this is because by setting the automatic option in the distance function, we are telling Mathematica to find the optimal distance function. And depending on the data one function might gather elements in different forms as we will see later on.

K-Means Clustering

At the moment we have seen how to search for clusters in a generic way. In this part we will focus on the K-means method.

The K-means is a technique to find and classify groups (k) of data so that the elements that share similar characteristics are grouped together and in the same way for the opposite case (not similar characteristics). To distinguish whether the data contain similarities or not, the method calculates the distance between the data with respect to a centroid. The elements that have less distance between them will be those that share similarities. This technique is carried out as an iterative process in which the groups are adjusted until they reach a convergence. Basically, the K-means method, which is a simple algorithm, consists of making a classification by means of specific partitions, in different groups, where each point or observation belongs to the group. Clustering is done by minimizing the sum of the distances between each object and the centroid of its group. The k-means clustering technique tries to build the clusters so that they have the least variation within a group. This is done by minimizing the expression , where C_i represents the i-th cluster, x_j represents the points, and ?_? represents the centroid of each cluster C_i. The square term of the function is the distance function; the most used is the square Euclidean distance, as in this case. To learn more about the mathematical foundation behind this technique, consult the reference An Introduction to Statistical Learning: With Applications in R by Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. (1st ed. 2013, Corr. 7th printing 2017 ed.: Springer).

Dimensionality Reduction

There are two ways to do the process, either using the DimensionReduce command or the DimensionReduction command, which are used to reduce the dimensions of the data. The difference between the two is that the first returns the values as a list. The second returns a DimensionReducerFunction (Figure 8-35) as output as in the case of Predict and Classify. Both belong to the Wolfram Language special functions for machine learning. For this case we will use the DimensionReduction command. Since we have the data, we introduce the standardized data as arguments, followed by specified target dimensions (2), with the as “PrincipalComponentAnalysis” method. This will give us the DimensionReducerFunction that will assign us the name of DR.

In[15]:= DR=DimensionReduction[ST,2,Method→"PrincipalComponentsAnalysis"]

Out[15]=

This calculates the variance of each component , followed by the total to find the proportion of variance explained. Observing that PC1 seems to represent 76% of the data dispersion, and PC2 seems to represent 23%. To obtain the accumulated percentage we add the variations of each component. To view more depth about the proportion of variation refer to An Introduction to Statistical Learning: With Applications in R (James, G., Witten, D., Hastie, T., & Tibshirani, R. ; 1st ed. 2013, Corr. 7th printing 2017 ed.: Springer).

We look at the plot (Figure 8-36) of the main components made by the previous process. If you look over the complete iris data from the ExampleData, the first 50 elements correspond to the setosa specie, the next 50 to versicolor, and the last 50 to virginica.

In[18]:= Labels={Style["First principal component",Black,Bold],Style["Second Principal component",Black,Bold]};

ListPlot[{PCA[[1;;50]],PCA[[51;;100]],PCA[[100;;150]]},PlotLegends→Placed[{Placeholder["setosa"],Placeholder["versicolor"],Placeholder["virginica"]},Right],PlotMarkers→"OpenMarkers",GridLines→All,Frame→True,Axes→False,FrameTicks→All,FrameLabel→Labels]

Out[18]=

Applying K-Means

In Figure 8-37, it appears that the method clearly identifies the left points as a single cluster (setosa specie), whereas some of the points between clusters 2 and 3 might be misclassified.

Chaining the Distance Function

Changing the DistanceFunction can modify how the clusters are arranged, the next code shows the plot for k = 3 and choosing a different distance function. In the next block of code, the computation of the clusters is made for the same k (3), with a different distant function and stored into their respective variables. Then the clusters are plotted (Figure 8-38) for each of the different distance functions, and finally they are displayed within a graphic grid.

In[20]:= {ED,MhD,ChD,CosD}={FindClusters[PCA,3,PerformanceGoal→#1,Method→#2,DistanceFunction→EuclideanDistance,RandomSeeding→#3],FindClusters[PCA,3,PerformanceGoal→#1,Method→#2,DistanceFunction→ManhattanDistance,RandomSeeding→#3],FindClusters[PCA,3,PerformanceGoal→#1,Method→#2,DistanceFunction→ChessboardDistance,RandomSeeding→#3],FindClusters[PCA,3,PerformanceGoal→#1,Method→#2,DistanceFunction→CosineDistance,RandomSeeding→#3]}&["Quality","KMeans",8888];

{EDplt,MhDplt,ChDplt,CosDplt}={

ListPlot[ED,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@ED[[1,All]]],Point[Mean@ED[[2,All]]],Point[Mean@ED[[3,All]]]},PlotLabel→ Style["Euclidean Distance",Black]],

ListPlot[MhD,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@MhD[[1,All]]],Point[Mean@MhD[[2,All]]],Point[Mean@MhD[[3,All]]]},PlotLabel→ Style["Manhattan Distance",Black]],

ListPlot[ChD,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@ChD[[1,All]]],Point[Mean@ChD[[2,All]]],Point[Mean@ChD[[3,All]]]},PlotLabel→ Style["Chessborad Distance",Black]],

ListPlot[CosD,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@CosD[[1,All]]],Point[Mean@CosD[[2,All]]],Point[Mean@CosD[[3,All]]]},PlotLabel→ Style["Cosine Distance",Black]]

}&[True,0.8,"OpenMarkers",{ColorData[97,1],ColorData[97,2],ColorData[97,3]},All,Automatic,300,Labels,False,All,1,0.03];

LegendsText={Placeholder[Style["Cluster 1",Bold,Black,10]],Placeholder[Style["Cluster 2",Bold,Black,10]],Placeholder[Style["Cluster 3",Bold,Black,10]]};

Labeled[Legended[GraphicsGrid[{{EDplt,MhDplt},{ChDplt,CosDplt}},Frame→All,Background→White,Spacings→1],PointLegend[{ColorData[97,1],ColorData[97,2],ColorData[97,3]},LegendsText,LegendMarkers→"OpenMarkers"]],Style["K-means clustering for K=3",FontFamily→"Times",Black,20,Italic],Top]

Out[20]=

As seen in Figure 8-38, the clusters can have different arrangements with different distance functions; one thing to note also is that the clusters centroids change in each of the subfigures (Figure 8-38).

Different K’s

Having seen that for different distance functions the clusters can vary, let’s now construct the process but with different K’s—that is, for k= 2, 3, 4, and 5, as exhibited in Figure 8-39.

In[21]:= {K2,K3,K4,K5}={FindClusters[PCA,2,PerformanceGoal→#1,Method→#2,DistanceFunction→#3,RandomSeeding→#4],FindClusters[PCA,3,PerformanceGoal→#1,Method→#2,DistanceFunction→#3,RandomSeeding→#4],FindClusters[PCA,4,PerformanceGoal→#1,Method→#2,DistanceFunction→#3,RandomSeeding→#4],FindClusters[PCA,5,PerformanceGoal→#1,Method→#2,DistanceFunction→#3,RandomSeeding→#4]}&["Speed","KMeans",SquaredEuclideanDistance,8888];

{PK2,PK3,PK4,PK5}={

ListPlot[K2,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@K2[[1,All]]],Point[Mean@K2[[2,All]]]},PlotLabel→ Style["K=2",Black]],

ListPlot[K3,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@K3[[1,All]]],Point[Mean@K3[[2,All]]],Point[Mean@K3[[3,All]]]},PlotLabel→ Style["K=3",Black]],

ListPlot[K4,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@K4[[1,All]]],Point[Mean@K4[[2,All]]],Point[Mean@K4[[3,All]]],Point[Mean@K4[[4,All]]]},PlotLabel→ Style["K=4",Black]],

ListPlot[K5,Frame→#1,AspectRatio→#2,PlotMarkers→#3,PlotStyle→#4,GridLines→#5,PlotRange→#6,ImageSize→#7,FrameLabel→#8,Axes→#9,FrameTicks→#10,

Epilog→{Opacity@#11,PointSize@#12,Point[Mean@K5[[1,All]]],Point[Mean@K5[[2,All]]],Point[Mean@K5[[3,All]]],Point[Mean@K5[[4,All]]],Point[Mean@K5[[5,All]]]},PlotLabel→ Style["K=5",Black]]

}&[True,0.8,"OpenMarkers",{ColorData[97,1],ColorData[97,2],ColorData[97,3],ColorData[97,4],ColorData[97,5]},All,Automatic,260,Labels,False,All,1,0.015];

LegendsText2={Placeholder[Style["Cluster 1",Bold,Black,10]],Placeholder[Style["Cluster 2",Bold,Black,10]],Placeholder[Style["Cluster 3",Bold,Black,10]],Placeholder[Style["Cluster 4",Bold,Black,10]],Placeholder[Style["Cluster 5",Bold,Black,10]]};

Labeled[Legended[GraphicsGrid[{{PK2,PK3},{PK4,PK5}},Frame→All,Background→White,Spacings→1],PointLegend[{ColorData[97,1],ColorData[97,2],ColorData[97,3],ColorData[97,4],ColorData[97,5]},LegendsText2,LegendMarkers→"OpenMarkers"]],Style["K-means clustering for K=2,3,4,5",FontFamily→"Times",Black,20,Italic],Top]

Out[21]=

Cluster Classify

Getting the classifier function (Figure 8-40), we can see the details of the classifier, and we can see the input vector is a numerical vector, the number of classes (three), the method, and the number of training examples.

We can see that there are three classes: class 1, class 2 and class 3. The distance function used is EuclideanDistance, and the numeric vector features are referred to by the name f1. A simple example is used, by choosing a sepal length of 1 and sepal width of 2, to show the different properties that can be used when testing the data; this is shown in the dataset form (Figure 8-42). The example is first written followed by the properties Decision (cluster that belongs the example), Distribution (categorical distribution object for histogram plots), ExpectedUtilities (expected probabilities and indeterminate threshold), LogProbabilities (log probabilities), Probabilities (probabilities of the test data based on classes), and TopProbabilities (best probabilities for the test data).

In[29]:= Dataset[AssociationMap[CC[{1,2},#]&,{"Decision","Distribution","ExpectedUtilities","LogProbabilities","Probabilities","TopProbabilities"}]]

Out[29]=

We can see that the example belongs to the third cluster and that the associated probability is 1 → 0.976148. Let’s look at the rest of the data and plot the cluster classification.

The classified data plot is shown in Figure 8-43.

In[30]:= ListPlot[Pick[TsT,CC[TsT],#]&/@{1,2,3},PlotMarkers→"OpenMarkers",GridLines→Automatic,PlotLegends→{Placeholder[Style["Cluster 1",Bold,Black,10]],Placeholder[Style["Cluster 2",Bold,Black,10]],Placeholder[Style["Cluster 3",Bold,Black,10]]},Frame→True,FrameTicks→All,FrameLabel→{"Sepal Lenght","Sepal Width"}]

Out[30]=

As a complement, a probability restriction for values below an established probability value can be added, with IndeterminateThreshold, as depicted in Figure 8-44.

In[31]:= ListPlot[Pick[TsT,CC[TsT,IndeterminateThreshold→ 0.6],#]&/@{1,2,3,Indeterminate},PlotMarkers→ "OpenMarkers",PlotLegends→{Placeholder[Style["Cluster 1",Bold,Black,10]],Placeholder[Style["Cluster 2",Bold,Black,10]],Placeholder[Style["Cluster 3",Bold,Black,10]],Placeholder[Style["Indeterminate",Bold,Black,10]]},Frame→True,FrameTicks→All,FrameLabel→{"Sepal Lenght","Sepal Width"},GridLines→Automatic]

Out[31]=