In this section, we'll build a deep belief network (DBN) based on the RBM, as shown in the following diagram. The network consists of four layers. The first layer recedes the 748 inputs to 500 neurons, then to 250, followed by 200, and finally to the last 10 target values:

As the code is the same as in the previous example, let's take a look at how to configure such a network:

MultiLayerConfiguration conf = new 
   NeuralNetConfiguration.Builder() 

We will define the gradient optimization algorithm, as shown in the following code:

    .seed(seed) 
    .gradientNormalization( 
    GradientNormalization.ClipElementWiseAbsoluteValue) 
    .gradientNormalizationThreshold(1.0) 
    .iterations(iterations) 
    .momentum(0.5) 
    .momentumAfter(Collections.singletonMap(3, 0.9)) 
    .optimizationAlgo(OptimizationAlgorithm.CONJUGATE_GRADIENT) 

We will also specify that our network will have four layers:

   .list(4) 

The input to the first layer will be 748 neurons and the output will be 500 neurons. We'll use the root mean squared error cross entropy, and the Xavier algorithm to initialize weights by automatically determining the scale of initialization based on the number of input and output neurons, as follows:

.layer(0, new RBM.Builder() 
.nIn(numRows*numColumns) 
.nOut(500)          
.weightInit(WeightInit.XAVIER) 
.lossFunction(LossFunction.RMSE_XENT) 
.visibleUnit(RBM.VisibleUnit.BINARY) 
.hiddenUnit(RBM.HiddenUnit.BINARY) 
.build()) 

The next two layers will have the same parameters, except for the number of input and output neurons:

.layer(1, new RBM.Builder() 
.nIn(500) 
.nOut(250) 
.weightInit(WeightInit.XAVIER) 
.lossFunction(LossFunction.RMSE_XENT) 
.visibleUnit(RBM.VisibleUnit.BINARY) 
.hiddenUnit(RBM.HiddenUnit.BINARY) 
.build()) 
.layer(2, new RBM.Builder() 
.nIn(250) 
.nOut(200) 
.weightInit(WeightInit.XAVIER) 
.lossFunction(LossFunction.RMSE_XENT) 
.visibleUnit(RBM.VisibleUnit.BINARY) 
.hiddenUnit(RBM.HiddenUnit.BINARY) 
.build()) 

Now, the last layer will map the neurons to outputs, where we'll use the softmax activation function, as follows:

.layer(3, new OutputLayer.Builder() 
.nIn(200) 
.nOut(outputNum) 
.lossFunction(LossFunction.NEGATIVELOGLIKELIHOOD) 
.activation("softmax") 
.build()) 
.pretrain(true).backprop(false) 
.build(); 

The rest of the training and evaluation is the same as in the single-layer network example. Note that training a deep network might take significantly more time compared to a single-layer network. The accuracy should be around 93%.

Now, let's take a look at another deep network.