This section is a prequel to the following chapters, where we will discuss different machine learning techniques in detail. At a high level, there are only a handful of tasks that machine learning tries to address. However, for each of such tasks, there are several approaches and algorithms in place.

The typical tasks in any machine learning are one of the following:

In classification, the objective is to assign a new data point to one of the predetermined classes. Typically, this is either a supervised or semi-supervised learning problem. The well-known machine learning algorithms used for classification are logistic regression, support vector machines (SVM), decision trees, Naïve Bayes, neural networks, Adaboost, and random forests. Here, Naïve Bayes is a Bayesian inference-based method. Other algorithms, such as logistic regression and neural networks, have also been implemented in the Bayesian framework.

Regression is probably the most common machine learning problem. It is used to determine the relation between a set of input variables (typically, continuous variables) and an output (dependent) variable that is continuous. We discussed the simplest example of linear regression in some detail in the previous section. More complex examples of regression are generalized linear regression, spline regression, nonlinear regression using neural networks, support vector regression, and Bayesian network. Bayesian formulations of regression include the Bayesian network and Bayesian linear regression.

Clustering is a classic example of unsupervised learning. Here, the objective is to group together similar items in a dataset based on certain features of the data. The number of clusters is not known in advance. Hence, clustering is more of a pattern detection problem. The well-known clustering algorithms are K-means clustering, hierarchical clustering, and Latent Dirichlet allocation (LDA). In this, LDA is formulated as a Bayesian inference problem. Other clustering methods using Bayesian inference include the Bayesian mixture model.

Association rule mining is an unsupervised method that finds items that are co-occurring in large transactions of data. The market basket analysis, which finds the items that are sold together in a supermarket, is based on association rule mining. The Apriori algorithm and frequent pattern matching algorithm are two main methods used for association rule mining.

Forecasting is similar to regression, except that the data is a time series where there are observations with different values of time stamp and the objective is to predict future values based on the current and past values. For this purpose, one can use methods such as ARIMA, neural networks, and dynamic Bayesian networks.

One of the fundamental issues in machine learning is called the curse of dimensionality. Since there can be a large number of features in a machine learning model, the typical minimization of error that one has to do to estimate model parameters will involve search and optimization in a large dimensional space. Most often, data will be very sparse in this higher dimensional space. This can make the search for optimal parameters very inefficient. To avoid this problem, one tries to project this higher dimensional space into a lower dimensional space containing a few important variables. One can then use these lower dimensional variables as features. The two well-known examples of dimensional reduction are principal component analysis and self-organized maps.

Often, the probability distribution of a population is directly estimated, without any parametric models, from a small amount of observed data for making inferences. This is called density estimation. The simplest form of density estimation is histograms, though it is not adequate for many practical applications. The more sophisticated density estimations are kernel density estimation (KDE) and vector quantization.