So, the inevitable question now arises: how do these things work?
In a nutshell, we have neurons on the grid; gradually, via iterations, they adapt themselves to the shape of our data (in our example, shown in the following image on the left-hand side in the Points panel). Let's talk a bit more about the iterative process itself.
- The first step is to randomly position data on the grid. We will randomly be placing our grid's neurons in our data space, as follows:
- The second step is where our algorithm will select a single data point.
- In the third step, we need to find the neuron (data point) that is closest to the chosen data point. This then becomes our best matching unit.
- The fourth step is to move our best-matching unit towards that data point. The distance that we move is determined by our learning rate, which will ultimately decrease after each iteration.
- Fifth, we will move the neighbors of our best-matching unit closer to it, with the farther-away neurons moving less than those that are closer. The Initial radius variable you see on the screen is what we use to identify neighbors. This value, just like the Initial learning rate, will decrease over time. If you have ReflectInsight (RI) up and running, you can watch the Initial learning rate decrease over time, as shown in the following screenshot:
- Our sixth and final step will be to update the Initial learning rate and Initial radius, as we have described so far, and then repeat. We will continue this process until our data points have stabilized and are in the correct position.
Now that we've introduced you to a little bit of intuition on SOMs, let's talk a little more about what we're going to do in this chapter. We have chosen a very common mechanism for teaching our principals, which is the mapping of colors. The colors themselves are 3D objects represented by red, green, and blue, but we will be organizing them into two dimensions. There are two key points you will see here with the organization of colors. First, the colors are clustered into distinct regions, and second, regions of similar properties are usually found adjacent to each other.
Our second example, which is a little bit more advanced, will use an artificial neural network (ANN) as we described before; this is an advanced form of machine learning, used to create an organizational mapping that matches the one presented to it. Let's look at our first example.
Here's a screenshot of our example. As you can see, we have a random pattern of colors, which, when finished, will be organized by clusters of similar colors:
If we are successful—and we will be—here's what our result should look like:
Let us begin by following the steps of the process as follows:
- We will start out by using 500 iterations to achieve our goal. Using a smaller number may not produce the blend that we are ultimately after. As an example, if we have used 500 iterations, here is what our result would look like:
- As you can see, we are far from being where we need to be. Being able to change the iterations allows you to experiment with exactly the right setting. I can tell you that 500 is higher than we need, so I will leave it as an exercise for you to figure out the number where the progression stops and you are satisfied with the organization.
- After setting the number of iterations, all we must do is make sure we have the random color pattern we want, which can be achieved by clicking on the Randomize button.
- Once you have the pattern that you want, you simply click on the Start button and watch the results.
- Once you click on Start, the Stop button will be activated, and you can stop the progression anytime you like. The organization will automatically stop once you reach the number of iterations that you specified.
Before we get into the actual code, let me show you some screenshots of what some organizational patterns look like. You can accomplish wonderful results by simply changing different parameters, which we will describe in detail further on. In the following screenshot, we have set the number of Iterations to 3000 and the Initial radius as 10:
In the following screenshot we are using 4000 iterations and an Initial radius of 18:
In the following screenshot we have set the number of Iterations to 4000 and the Initial radius as 5:
Here, we have set the number of Iterations to 5000, the Initial learning rate as 0.3, and the Initial radius as 25, as shown in the following screenshot, to obtain the desired result:
As promised, let's now dive into the code.
In this example, we are going to work with AForge and use the DistanceNetwork object. A distance network is a neural network of only a single distance. As well as being used for an SOM, it is used for an ElasticNet operation, which is what we will be using to show the elastic connections between objects during progression.
We will create our distance network using three input neurons and 1000 neurons that will be doing the work under the hood:
network = new DistanceNetwork(3, 100 * 100);
When you click on the Randomize button to randomize the colors, here's what happens under the hood:
private void RandomizeNetwork()
{
if (network != null)
{
foreach (var neuron in (network?.Layers.SelectMany(layer
=> layer?.Neurons)).Where(neuron => neuron != null))
neuron.RandGenerator =
new UniformContinuousDistribution
(new Range(0, 255));
network?.Randomize();
}
UpdateMap();
}
You will notice that the randomization range we are dealing with stays within the range of any color's red, green, or blue characteristic, which is 255.
Next, we will look at our learning loop, which looks like this. We'll do a deep dive into it in a moment:
SOMLearning trainer = new SOMLearning(network);
double[] input = new double[3];
double fixedLearningRate = learningRate / 10;
double driftingLearningRate = fixedLearningRate * 9;
int i = 0;
while (!needToStop)
{
trainer.LearningRate = driftingLearningRate
* (iterations - i)
/ iterations + fixedLearningRate;
trainer.LearningRadius = radius * (iterations - i)
/ iterations;
if (rand != null)
{
input[0] = rand.Next(256);
input[1] = rand.Next(256);
input[2] = rand.Next(256);
}
trainer.Run(input);
// update map once per 50 iterations
if ((i % 10) == 9)
{
UpdateMap();
}
i++;
SetText(currentIterationBox, i.ToString());
if (i >= iterations)
break;
}
If we look closer, the first object we create is an SOMLearning object. This object is optimized for square space learning, meaning it expects that the network it is working on has the same height as its width. This makes it easier to find the square root of the network's neuron counts:
SOMLearning trainer = new SOMLearning(network);
Next, we need to create variables to hold our red, green, and blue input colors, from which we will continually randomize the input colors in order to achieve our goals:
if (rand != null)
{
input[0] = rand.Next(256);
input[1] = rand.Next(256);
input[2] = rand.Next(256);
}
Once we enter our while loop, we will continually update our variables until we reach the total number of iterations we selected. In this update loop, there are several things happening. First, we will update the learning rate and learning radius, and store it in our SOMLearning object:
trainer.LearningRate = driftingLearningRate * (iterations - i) /
iterations + fixedLearningRate;
trainer.LearningRadius = radius * (iterations - i) / iterations;
The learning rate determines our speed of learning. The learning radius, which can have a pretty dramatic affect on the visual output, determines the number of neurons to be updated relative to the distance from the winning neuron. The circle of the specified radius consists of neurons, and they are updated continually during the learning process. The closer a neuron is to the winning neuron, the more updating it will receive. Please note that if, during your experiments, you set this value to zero, then only the winning neurons' weights are updated and no others.
Although we will have a very pretty visual effect to watch, we'll still need to know what's going on within our application, and that's where RI comes in:
RILogManager.Default.ViewerSendWatch("Learning Rate", $"{trainer.LearningRate}");
RILogManager.Default.ViewerSendWatch("Learning Radius", $"{trainer.LearningRadius}");
RILogManager.Default.ViewerSendWatch("Red", $"{RGBInput[0]}");
RILogManager.Default.ViewerSendWatch("Green", $"{RGBInput[1]}");
RILogManager.Default.ViewerSendWatch("Blue", $"{RGBInput[2]}");
RI, as we mentioned earlier, has a watch panel that lets you continually track whatever variables you are interested in. In our case, we are interested in watching the learning rate, learning radius, and each RGB color that is randomized. All we need to do is supply the label and the value, and RI will do the rest, as we will see in a moment.
Finally, as they relate to RI, we want to see the RGB values in our message window as well, so we will add a debug message for that:
RILogManager.Default.SendDebug($"Red {RGBInput[0]}, Green {RGBInput[1]}, Blue
{RGBInput[2]}");
We now make a training Run for this iteration and pass to it the RGBInput array:
trainer.Run(RGBInput);
Let's talk about learning for a moment. As we mentioned, each iteration will try and learn more and more information. This learning iteration returns a learning error, which is the difference in the neurons' weights and the input vector RGBInput. As mentioned previously, the distance is measured according to the distance from the winning neuron. The process is as follows.
The trainer runs one learning iteration, finds the winning neuron (the neuron that has weights with values closest to those provided in the RGBInput), and updates its weights. It also updates the weights of the neighboring neurons. As each learning iteration occurs, the network gets closer and closer to the optimal solution.
Up next is a screenshot of our application running. In the background is RI, so that you can see how we are logging each iteration, what color values we are using when we update the map, as well as the learning rate and learning radius. As your machine learning programs and algorithms get more and more complex, you will realize that this kind of insight into your applications becomes incredibly invaluable. It is also an indispensable real-time debugging and diagnostic tool!
Since SOM are, well, self-organizing, our second example is going to be more graphical. Our hope is that it will help you to better understand what's happening behind the scenes.
In this example, we'll again use AForge.NET and build a 2D plane of objects, organized into a few groups. We will, starting from a single location, visually arrive at the location of those shapes. This is conceptually the same as our color example, which used points in a 3D space, except that this time, our points are in 2D. The visualization occurs in the Map panel and is a top-down view of what is happening in 2D space, so as to get to a 1D graphical view.
Initially, neurons in the SOM grid start out at random positions, but they are gradually massaged into a molded outlining that is in the shape of our data. This is an iterative process, and although putting an animated .gif into the book is a feat that we have not yet achieved, I have taken screenshots at various points in the iteration to show you what happens. You can run the example for yourself to see it in real time.
We start out with all our objects in the positions on the left. We will run through 500 iterations to show the evolution. We will go from a blank white panel to one that, hopefully, resembles the Points panel:
Now we click on the Start button and away it goes! You will see the points beginning to organize themselves by moving to their correct locations, which (hopefully) will mirror that of the Points we have specified:
After 199 iterations:
After 343 iterations:
And, after completion, you can see that the objects have organized themselves like the pattern that we initially created. If you imagine that you are looking down at the map, even though you are on a flat piece of paper, you can see the 3D experience if you look hard enough. The blue dots are the active neurons, the light gray dots are the inactive neurons, and the lines drawn are the elastic connections between neurons.
The checkboxes below the map allow you to easily choose whether to display either or both of these:
If you take a screenshot with the connections and inactive neurons not shown, you will see that the organizational patters in the map arrive at the same clustering as our objective, which for us means success:
How exactly all this works is the next topic we will investigate. As always, let's take a look at our main execution loop. As you can see, we'll be using the same DistanceNetwork and SOMLearning objects that we previously discussed:
DistanceNetwork network = new DistanceNetwork(2, networkSize
* networkSize);
// set random generators range
foreach (var neuron in network.Layers.SelectMany(layer =>
layer.Neurons))
neuron.RandGenerator = new UniformContinuousDistribution(
new Range(0, Math.Max
(pointsPanel.ClientRectangle.Width,
pointsPanel.ClientRectangle.Height)));
// create learning algorithm
SOMLearning trainer = new SOMLearning(network, networkSize,
networkSize);
// create map
map = new int[networkSize, networkSize, 3];
double fixedLearningRate = learningRate / 10;
double driftingLearningRate = fixedLearningRate * 9;
// iterations
int i = 0;
// loop
while (!needToStop)
{
trainer.LearningRate = driftingLearningRate
* (iterations - i) / iterations + fixedLearningRate;
trainer.LearningRadius = (double)learningRadius *
(iterations - i) / iterations;
// run training epoch
trainer.RunEpoch(trainingSet);
// update map
UpdateMap(network);
// increase current iteration
i++;
// set current iteration's info
SetText(currentIterationBox, i.ToString());
// stop ?
if (i >= iterations)
break;
}
As we mentioned earlier, the LearningRate and LearningRadius continue to evolve through every iteration. This time, let's talk a bit about the RunEpoch method of the trainer. This method, although very simplistic, is designed to take a vector of input values and then return a learning error for that iteration (as you can now see, also sometimes called an epoch). It does this by calculating against each one of the input samples in the vector. The learning error is the absolute difference between the neurons' weights and inputs. The difference is measured according to the distance from the winning neuron. As mentioned earlier, we run this calculation against one learning iteration/epoch, find the winner, and update its weights (as well as neighbor weights). I should point out that when I say winner, I mean the neuron that has weights with values closest to the specified input vector, that is, the minimum distance from the network's input.
Next, we will highlight how we update the map itself; our calculated projects should match the initial input vector (points):
// get first layer
Layer layer = network.Layers[0];
// lock
Monitor.Enter(this);
// run through all neurons
for (int i = 0; i < layer.Neurons.Length; i++)
{
Neuron neuron = layer.Neurons[i];
int x = i % networkSize;
int y = i / networkSize;
map[y, x, 0] = (int)neuron.Weights[0];
map[y, x, 1] = (int)neuron.Weights[1];
map[y, x, 2] = 0;
}
// collect active neurons
for (int i = 0; i < pointsCount; i++)
{
network.Compute(trainingSet[i]);
int w = network.GetWinner();
map[w / networkSize, w % networkSize, 2] = 1;
}
// unlock
Monitor.Exit(this);
//
mapPanel.Invalidate();
As you can see from this code, we get the first layer, calculate map for all the neurons, collect the active neurons so that we can determine the winner, and then update map.
Since we have talked about the winner so much, let me show you just how much code is involved in calculating the winner:
public int GetWinner()
{
// find the MIN value
double min = output[0];
int minIndex = 0;
for (int i = 1; i < output.Length; i++)
{
if (output[i] < min)
{
// found new MIN value
min = output[i];
minIndex = i;
}
}
return minIndex;
}
That's it! All we are doing is looking for the index of the neuron whose weights have the minimum distance from the network's input.