4.1. Introduction

The study of linear classifiers with supervised training has a long history. As early as 1936, Fisher's discriminant had already laid the groundwork for statistical pattern recognition. Later, Rosenblatt's perceptron (1956) was the first neural classifier proposed; it shares many similarities with the learning techniques used in the support vector machine developed by Vapnik [358] and in an earlier paper by Boser, Guyon, and Vapnik [33].

Section 4.2 begins by introducing a simple two-class classifier, then goes on to derive the least-squares classifier and the classical Fisher discriminant linear analysis. The decision boundary of the least-squares classifier and the Fisher classifier is dictated (i.e., supported) by all of the training data. In contrast, in Section 4.3, the decision boundary of the support vector machine (SVM) hinges on a specially selected set of training data that serve as support vectors. In a linear SVM, some "important" training data are considered as support vectors that are divided into two groups—one group for each class. The objective of the SVM is to maximize the minimal separation margin of these two groups of support vectors. A notion of fuzzy SVMs is proposed to cope with training data that are not linearly separable. The notion of margin of separation is conveniently replaced by a new notion of fuzzy separation region. Mathematically, slack variables are introduced so that linearly separable constraints can be relaxed. To further enhance the SVM's classification capability, nonlinear kernels are adopted in Section 4.5. The use of nonlinear kernels leads to more general and flexible nonlinear decision boundaries. For more in-depth treatment of SVMs for pattern classification, readers are referred to these references [40, 55, 257, 337].