We proceed with the recipe as follows:
- Clone the code from github:
git clone https://github.com/TengdaHan/GAN-TensorFlow
- Define a Xavier initializer as described in the paper Understanding the difficulty of training deep feedforward neural networks (2009) by Xavier Glorot, Yoshua Bengio, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.207.2059&rep=rep1&type=pdf The initializers are proven to allow better convergence for GANs:
def xavier_init(size):
in_dim = size[0]
xavier_stddev = 1. / tf.sqrt(in_dim / 2.)
return xavier_stddev
- Define the generator for the input X. First we define a matrix W1 with the dimension [100, K=128], initialized according to a normal distribution. Note that 100 is an arbitrary value for Z, the initial noise used by our generator. Then, we define the bias B1 with the dimension [K=256]. Similarly, we define a matrix W2 with the dimension [K=128, L=784] and a bias B2 with the dimension [L=784]. The two matrices W1 and W2 are initialized by using the xavier_init defined in step 1, while B1 and B2 are initialized by using tf.constant_initializer(). After that, we compute the multiplication between matrices X * W1, sum the bias of B1, and pass it through a RELU activation function for obtaining fc1. This dense layer is then connected with the next dense layer which was created by multiplying the matrices fc1 with W2 and summing the bias of B2. The result is then passed via a sigmoid function. These steps are used to define our two layers' neural network used for the generator:
def generator(X):
with tf.variable_scope('generator'):
K = 128
L = 784
W1 = tf.get_variable('G_W1', [100, K],
initializer=tf.random_normal_initializer(stddev=xavier_init([100, K])))
B1 = tf.get_variable('G_B1', [K], initializer=tf.constant_initializer())
W2 = tf.get_variable('G_W2', [K, L],
initializer=tf.random_normal_initializer(stddev=xavier_init([K, L])))
B2 = tf.get_variable('G_B2', [L], initializer=tf.constant_initializer())
# summary
tf.summary.histogram('weight1', W1)
tf.summary.histogram('weight2', W2)
tf.summary.histogram('biases1', B1)
tf.summary.histogram('biases2', B2)
fc1 = tf.nn.relu((tf.matmul(X, W1) + B1))
fc2 = tf.matmul(fc1, W2) + B2
prob = tf.nn.sigmoid(fc2)
return prob
- Define the discriminator for input X. In principle, this is very similar to the generator. The main difference is that if the parameter reuse is true, then we call scope.reuse_variables() to trigger a reuse. Then we define two dense layers. The first layer uses a matrix W1 of dimensions [J=784, K=128] with a bias B1 of dimension [K=128], and it is based on the standard multiplication of X by W1. This result is added to B1 and passed to a RELU activation function for getting the result fc1. The second one uses a matrix W2 of the dimension [K=128, L=1] with a bias B2 of dimension [L=1], and it is based on the standard multiplication of fc1 by W2. This result is added to B2 and passed to a sigmoid function:
def discriminator(X, reuse=False):
with tf.variable_scope('discriminator'):
if reuse:
tf.get_variable_scope().reuse_variables()
J = 784
K = 128
L = 1
W1 = tf.get_variable('D_W1', [J, K],
initializer=tf.random_normal_initializer(stddev=xavier_init([J, K])))
B1 = tf.get_variable('D_B1', [K], initializer=tf.constant_initializer())
W2 = tf.get_variable('D_W2', [K, L],
initializer=tf.random_normal_initializer(stddev=xavier_init([K, L])))
B2 = tf.get_variable('D_B2', [L], initializer=tf.constant_initializer())
# summary
tf.summary.histogram('weight1', W1)
tf.summary.histogram('weight2', W2)
tf.summary.histogram('biases1', B1)
tf.summary.histogram('biases2', B2)
fc1 = tf.nn.relu((tf.matmul(X, W1) + B1))
logits = tf.matmul(fc1, W2) + B2
prob = tf.nn.sigmoid(logits)
return prob, logits
- Now let's define some useful additional functions. First, we import a bunch of standard modules:
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import os
import argparse
- Then we read the data from the MNIST dataset and define an auxiliary function for plotting samples:
def read_data():
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("../MNIST_data/", one_hot=True)
return mnist
def plot(samples):
fig = plt.figure(figsize=(8, 8))
gs = gridspec.GridSpec(8, 8)
gs.update(wspace=0.05, hspace=0.05)
for i, sample in enumerate(samples):
ax = plt.subplot(gs[i])
plt.axis('off')
ax.set_xticklabels([])
ax.set_yticklabels([])
ax.set_aspect('equal')
plt.imshow(sample.reshape(28, 28), cmap='Greys_r')
return fig
- Now let's define the training function. First, let's read the MNIST data, and then define a matrix X of shape 28 x 28 with one channel for a standard MNIST handwritten character. Then let's define a z noise vector of size 100—a common choice proposed in the seminal GANs paper. The next step is to call the generator on z and assign the result to G. After that, we pass X to the discriminator without reuse. Then we pass the forged/fake G result to the discriminator, reusing the learned weight. One important aspect of this is how we chose the loss function for the discriminator, which is a sum of two cross entropies: one for real characters, where all the real MNIST characters have a label set to one, and one for forged characters, where all the forged characters have a label set to zero. The discriminator and the generator run in an alternate sequence for 100,000 steps. Every 500 steps, a sample is drawn from the learned distribution for printing what that generator has learned so far. This is what defines a new epoch, and the results are shown in the next section. Let's see the code snippet which implements what we just described.
def train(logdir, batch_size):
from model_fc import discriminator, generator
mnist = read_data()
with tf.variable_scope('placeholder'):
# Raw image
X = tf.placeholder(tf.float32, [None, 784])
tf.summary.image('raw image', tf.reshape(X, [-1, 28, 28, 1]), 3)
# Noise
z = tf.placeholder(tf.float32, [None, 100]) # noise
tf.summary.histogram('Noise', z)
with tf.variable_scope('GAN'):
G = generator(z)
D_real, D_real_logits = discriminator(X, reuse=False)
D_fake, D_fake_logits = discriminator(G, reuse=True)
tf.summary.image('generated image', tf.reshape(G, [-1, 28, 28, 1]), 3)
with tf.variable_scope('Prediction'):
tf.summary.histogram('real', D_real)
tf.summary.histogram('fake', D_fake)
with tf.variable_scope('D_loss'):
d_loss_real = tf.reduce_mean(
tf.nn.sigmoid_cross_entropy_with_logits(
logits=D_real_logits, labels=tf.ones_like(D_real_logits)))
d_loss_fake = tf.reduce_mean(
tf.nn.sigmoid_cross_entropy_with_logits(
logits=D_fake_logits, labels=tf.zeros_like(D_fake_logits)))
d_loss = d_loss_real + d_loss_fake
tf.summary.scalar('d_loss_real', d_loss_real)
tf.summary.scalar('d_loss_fake', d_loss_fake)
tf.summary.scalar('d_loss', d_loss)
with tf.name_scope('G_loss'):
g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits
(logits=D_fake_logits, labels=tf.ones_like(D_fake_logits)))
tf.summary.scalar('g_loss', g_loss)
tvar = tf.trainable_variables()
dvar = [var for var in tvar if 'discriminator' in var.name]
gvar = [var for var in tvar if 'generator' in var.name]
with tf.name_scope('train'):
d_train_step = tf.train.AdamOptimizer().minimize(d_loss, var_list=dvar)
g_train_step = tf.train.AdamOptimizer().minimize(g_loss, var_list=gvar)
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
merged_summary = tf.summary.merge_all()
writer = tf.summary.FileWriter('tmp/mnist/'+logdir)
writer.add_graph(sess.graph)
num_img = 0
if not os.path.exists('output/'):
os.makedirs('output/')
for i in range(100000):
batch_X, _ = mnist.train.next_batch(batch_size)
batch_noise = np.random.uniform(-1., 1., [batch_size, 100])
if i % 500 == 0:
samples = sess.run(G, feed_dict={z: np.random.uniform(-1., 1., [64, 100])})
fig = plot(samples)
plt.savefig('output/%s.png' % str(num_img).zfill(3), bbox_inches='tight')
num_img += 1
plt.close(fig)
_, d_loss_print = sess.run([d_train_step, d_loss],
feed_dict={X: batch_X, z: batch_noise})
_, g_loss_print = sess.run([g_train_step, g_loss],
feed_dict={z: batch_noise})
if i % 100 == 0:
s = sess.run(merged_summary, feed_dict={X: batch_X, z: batch_noise})
writer.add_summary(s, i)
print('epoch:%d g_loss:%f d_loss:%f' % (i, g_loss_print, d_loss_print))
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Train vanila GAN using fully-connected layers networks')
parser.add_argument('--logdir', type=str, default='1', help='logdir for Tensorboard, give a string')
parser.add_argument('--batch_size', type=int, default=64, help='batch size: give a int')
args = parser.parse_args()
train(logdir=args.logdir, batch_size=args.batch_size)