As discussed in the first part of the chapter, we optimize for two different sources of loss.

The first loss we optimize for is the actual difference between the input image and the output image; this is ideal for us if the difference is minimal. To do this, we expose the output layer and then calculate the difference to the input. For this example, we are using the sum of the squared errors between the input and output, nothing fancy. In pseudocode, this looks like the following:

valueLoss = sum(squared(input - output))

In Gorgonia, we can implement it as follows:

m.out = l7
valueLoss, err := gorgonia.Sum(gorgonia.Must(gorgonia.Square(gorgonia.Must(gorgonia.Sub(y, m.out)))))
if err != nil {
    log.Fatal(err)
}

Our other loss component is the KL divergence measure, for which the pseudocode looks like the following:

klLoss = sum(mean^2 + exp(sd) - (sd + 1)) / 2

Our implementation in Gorgonia is more verbose, with a generous use of Must:

valueOne := gorgonia.NewScalar(g, dt, gorgonia.WithName("valueOne"))
valueTwo := gorgonia.NewScalar(g, dt, gorgonia.WithName("valueTwo"))
gorgonia.Let(valueOne, 1.0)
gorgonia.Let(valueTwo, 2.0)

ioutil.WriteFile("simple_graph_2.dot", []byte(g.ToDot()), 0644)
klLoss, err := gorgonia.Div(
    gorgonia.Must(gorgonia.Sum(
        gorgonia.Must(gorgonia.Sub(
            gorgonia.Must(gorgonia.Add(
                gorgonia.Must(gorgonia.Square(m.outMean)),
                gorgonia.Must(gorgonia.Exp(m.outVar)))),
            gorgonia.Must(gorgonia.Add(m.outVar, valueOne)))))),
    valueTwo)
if err != nil {
    log.Fatal(err)
}

Now, all that's left is a little bit of housekeeping and tying everything together. We will be using the Adam's solver for this example:

func (m *nn) learnables() gorgonia.Nodes {
    return gorgonia.Nodes{m.w0, m.w1, m.w5, m.w6, m.w7, m.estMean, m.estSd}
}

vm := gorgonia.NewTapeMachine(g, gorgonia.BindDualValues(m.learnables()...))
solver := gorgonia.NewAdamSolver(gorgonia.WithBatchSize(float64(bs)), gorgonia.WithLearnRate(0.01))

Let's now assess the results.