A generative model aims to generate all the values of a phenomenon, both those that can be observed (input) and those that can be calculated from the ones observed (target). We try to understand how such a model can succeed in this goal by proposing a first distinction between generative and discriminative models.
Often, in machine learning, we need to predict the value of a target vector y given the value of an input x vector. From a probabilistic perspective, the goal is to find the conditional probability distribution p(y|x).
The conditional probability of an event y with respect to an event x is the probability that y occurs, knowing that x is verified. This probability, indicated by p(y|x), expresses a correction of expectations for y, dictated by the observation of x.
The most common approach to this problem is to represent the conditional distribution using a parametric model, and then determine the parameters using a training set consisting of pairs (xn, yn) that contain both the values of the input variables and the relative vectors of corresponding outputs. The resulting conditional distribution can be used to make predictions of the target (y) for new input values (x). This is known as a discriminatory approach, since the conditional distribution discriminates directly between the different values of y.
As an alternative to this approach, we can look for the joint probability distribution p(x∩ y), and then use this joint distribution to evaluate the conditional probability p(y | x) in order to make predictions of y for new values of x. This is known as generative approach, because by sampling from the joint distribution, it is possible to generate synthetic examples of the vector of characteristics x.
The joint probability distribution p(x, y) is a probability distribution that gives the probability that each of x, y vectors falls in any particular range or discrete set of values specified for that variable.
A generative approach, regardless of the type of data and the theoretical model used, is divided into two basic steps:
- The first step involves the construction of the generative model. The input data is processed with the aim of deducing their distribution. To do this, input data can simply be reorganized into a different structure, or it can represent new information extracted from input data from specific algorithms. The result of the construction of the generative model is the presentation of data according to the distribution to which it has been approximated.
- Once the generative model has been built on the input data, this allows sampling, which leads to the formation of new data that shares the same distribution with the input data.
The construction of a generative model allows highlighting features and properties implicitly present in the initial data. The individual approaches are then distinguished by the type of processing performed on the data to explain these characteristics, and consequently for the type of variables on which an approximate data distribution is obtained.
Why are AI researchers so excited about generative models? Let's take a simple example: suppose we provide the system with a series of images of cats. Suppose then, that after seeing these images, the computer is able to generate new photos of cats in a completely independent manner. If the computer were able to do it and the images that were produced had the right number of legs, tails, ears, and so on, it would be easy to prove that the computer knows which parts make up the cat, even if no one has ever explained cat anatomy to it. So, in a sense, a good generative model is proof of the basic knowledge of concepts by computers.
This is why researchers are so enthusiastic about building generative models. These models seem to be a way to train computers to understand concepts without the need for researchers to teach them a priori concepts.