Introduction

Ever wish you could paint like Picasso or the famous Indian painter M.F. Husain? It looks like neural networks have made your every wish come true. In this chapter, you would learn one such technique that uses neural networks to compose your own clicked picture in the style of a famous artist or rather in a style of your own choice. The technique is called neural style transfer, which is outlined in Leon A. Gatys’ famous paper – “A Neural Algorithm of Artistic Style.” Though the paper is a great read, you would not need all those details given in the paper to understand this chapter.

Neural style transfer is an optimization technique that blends the contents of an image in the style of another image. The TensorFlow Hub that you have learned to use in a previous chapter contains a pre-trained model for style transfers. First, I will show you how to use this model to get quickly started on your style transfer learning. This will be followed by a do-it-yourself example that will teach you how to extract the contents and the style from two different images and then perform the transformation on the content image to create another stylized image.

The first image on the left is the content image, the image in the middle is the style image, and the image on the right is the stylized image. Note how the style of the middle image is applied to the contents of the image on the left to produce a new stylized image.

The theory behind the style transfer is trivial, and I will be covering it in the custom style transfer proGram later in this chapter.

So, let us get started with fast style transfer.

Fast Style Transfer

The TF Hub provides a pre-trained model for doing quick style transfers. The module name is “arbitrary-image-stylization-v1-256/2”. This performs a fast artistic style transfer as compared to the original work for artistic style transfer with neural networks. The model may work on any arbitrary painting styles. The model is based on the technique proposed by Golnaz et al. in their famous paper “Exploring the structure of a real-time, arbitrary neural artistic stylization network” (https://arxiv.org/abs/1705.06830). The proposed model combines the flexibility of the neural algorithm of artistic style with the speed of fast style transfer networks. This facilitates the real-time stylization using any content/style image pair.

You will be using this pre-trained model hosted on TensorFlow Hub for getting a quick understanding of the effects of style transfer.

Downloading Images

The project requires two images at any given instance – the content image and the style image. The content image will be modified to adapt the style given in the style image. For your testing, I have uploaded a few images on the book’s repository. The URL for an image takes the following form:

https://raw.githubusercontent.com/Apress/artificial-neural-networks-with-tensorflow-2/main/ch12/ferns.jpg

Write a function to extract the filename, download the file, and set a new path to the image. The function definition is trivial and is shown in Listing 12-1.

def download_image_from_URL(imageURL):

  imageName = re.search('[a-z0-9-]+.

                        (jpe?g|png|gif|bmp|JPG)',

                        imageURL, re.IGNORECASE)

  imageName = imageName.group(0)

  urllib.request.urlretrieve(imageURL, imageName)

  imagePath = "./" + imageName

  return imagePath

Listing 12-1

Function for creating an image URL

Call this function to create a path to the target image.

# This is the path to the image you want to transform.

target_url = "https://raw.githubusercontent.com/Apress/artificial-neural-networks-with-tensorflow-2/main/ch12/ferns.jpg"

target_path = download_image_from_URL(target_url)

Likewise, download the image for styling and set its path.

# This is the path to the style image.

style_url = "https://raw.githubusercontent.com/Apress/artificial-neural-networks-with-tensorflow-2/main/ch12/on-the-road.jpg"

style_path = download_image_from_URL(style_url)

The user interface as seen in the Colab notebook is shown in Figure 12-1.

Select the desired target and style images from the drop-down list on the right-hand side of the cell.

You can display both the selected images using the matplotlib imshow function with the following code fragment:

content = Image.open(target_path)

style = Image.open(style_path)

plt.figure(figsize=(10, 10))

plt.subplot(1, 2, 1)

plt.imshow(content)

plt.title('Content Image')

plt.subplot(1, 2, 2)

plt.imshow(style)

plt.title('Style Image')

plt.tight_layout()

plt.show()

The output is shown in Figure 12-2.

So, we have two images with different dimensions.

Displaying Output

To display the image, we will need to convert the tensor to the image format using the following code:

tensor = stylized_image*256

tensor = np.array(tensor, dtype=np.uint8)

tensor = tensor[0]

PIL.Image.fromarray(tensor)

We need to multiply the image data to the scale of 256 as the stylized_image contains the data that is in the scale of 0 to 1. The output image is shown in Figure 12-3.

If you want to display a scaled-down image, shown in Figure 12-4, simply call the imshow method as follows:

plt.imshow(tensor)

Do It Yourself

Having learned how to do a quick style transfer, it is time to learn the techniques behind these projects. The principle behind the style transfer is to extract the style of an image, usually a famous painting, and apply it to the contents of an image of your choice. Thus, there are two input images, namely, content image and style image. The newly generated image is generally called a stylized image. The generated image contains the same contents as the content image, but has acquired the style similar to the style image. As you understand, this is obviously not done by just superimposing the images. So, our proGram must be able to separately distinguish between the content and the style of a given image. That is where we will use the VGG16 pre-trained network model to extract this information and build our own network to create a stylized image based on these inputs. The Android apps such as Prisma and Lucid do such style transfers. Though you will not be taught to develop a similar Android application, this project will teach you the internals of such apps.

Let us first look at the VGG16 architecture to understand how to extract the content and style from an image.

VGG16 Architecture

Gatys et al. (2015) came up with the core idea behind the style transfers. The core idea is to say that the CNNs (Convolutional Neural Networks) pre-trained for image classification know how to encode the perceptual and semantic information about an image. There are many such pre-trained CNNs available in the world. We will use VGG16 to extract the features of an image and then work independently on its content and style. The original paper uses the 19-layer VGG network model from Simonyan and Zisserman (2015). The VGG16 model architecture is shown in Figure 12-5.

As we are not doing an image classification and are interested in just the feature extraction, we do not need the fully connected layers or the final softmax classifier of the VGG network. We only need the part of the model. So, how do we extract only a certain portion of the model? Fortunately for us, this is a very trivial task as Keras provides a pre-trained VGG16 model in which you can separate out the layers. Keras does provide many other models including the latter VGG19. To remove the top most fully connected layers, you need to set the value of the include_top variable to False while extracting the model layers.

Displaying Images

To display the two images, we use a code similar to the one in the previous project. The code is given in Listing 12-4.

content = Image.open(target_path)

style = Image.open(style_path)

plt.figure(figsize=(10, 10))

plt.subplot(1, 2, 1)

plt.imshow(content)

plt.title('Content Image')

plt.subplot(1, 2, 2)

plt.imshow(style)

plt.title('Style Image')

plt.tight_layout()

plt.show()

Listing 12-4

Displaying content and style images

The output is shown in Figure 12-6.

Model Building

To build the model, we feed the VGG16 with our image tensor data and extract the feature maps, the content, and style representations. The model will be loaded with pre-trained ImageNet weights. The model building code is given here:

target = K.constant(preprocess_image(target_path))

style = K.constant(preprocess_image(style_path))

# This placeholder will contain our generated image

combination_image = K.placeholder

                      ((1, img_height, img_width, 3))

# We combine the 3 images into a single batch

input_tensor = K.concatenate([target,

                              style,

                              combination_image],

                                  axis=0)

# Build the VGG16 network with our batch of 3 images as input.

model = vgg16.VGG16(input_tensor=input_tensor,

                    weights='imagenet',

                    include_top=False)

In this code, we first build the input tensor for our content and style images by calling our earlier defined preprocess method. We create a placeholder for the destination image and then create a tensor for three images by calling the concatenate method. This is then used as a parameter to the VGG16 method to extract our desired model.

You can now examine the model summary.

model.summary()

The summary output is shown in Figure 12-7.

Content Loss

We will compute the content loss at each desired layer and add them up. At each iteration, we feed our input image to the model. The model will compute all the content losses properly, and as we have an eager execution, all gradients will also be computed. The content loss is an indication of how similar the randomly generated noisy image (G) is to the content image (C). The content loss is computed as follows.

Assume that we choose a hidden layer (L) in a pre-trained network (VGG network) to compute the loss. Let P and F represent the original image and the image that is generated. Let F[l] and P[l] be the feature representation of the respective images in layer L. Then, the content loss is defined as follows:

We code this formula of the content loss as follows:

def content_loss(base, combination):

    return K.sum(K.square(combination - base))

Style Loss

To compute the style loss, we first need to compute the Gram matrix. The Gram matrix is an additional preprocessing step that is added to find the correlation between the different channels which will later be used for the measure of the style itself.

We define the Gram matrix as follows:

def gram_matrix(x):

    features = K.batch_flatten

                (K.permute_dimensions(x, (2, 0, 1)))

    gram = K.dot(features, K.transpose(features))

    return gram

The style loss computes the Gram matrix for both the style and the generated image and then returns the cost to the caller. The cost is the square of the difference between the Gram matrix of the style and the Gram matrix of the generated image. Mathematically, this is expressed as

The definition of the style loss function is given here:

def style_loss(style, combination):

    S = gram_matrix(style)

    C = gram_matrix(combination)

    channels = 3

    size = img_height * img_width

    return K.sum(K.square(S - C)) /

                (4. * (channels ** 2) * (size ** 2))

Displaying Images

We now display all three images using the following code:

plt.figure(figsize=(50, 50))

plt.subplot(3,3,1)

plt.imshow(load_img(target_path, target_size=(img_height, img_width)))

plt.subplot(3,3,2)

plt.imshow(load_img(style_path, target_size=(img_height, img_width)))

plt.subplot(3,3,3)

plt.imshow(img)

plt.show()

The output image is shown in Figure 12-8.

Summary

In this chapter, you learned another important technique in neural networks, and that is neural style transfer. The technique allows you to transform the contents of an image of your choice into a style of another image. You learned how to do this style transfer in two different ways. The first one was to use a pre-trained model provided in tfhub to perform a fast artistic style transfer. The style transfers using this method are quick and do a very good job. The second approach was to build your own network to do the style transfers at the core level. We used a VGG16 pre-trained model for image classification to extract the contents and style of images. You then learned to create a network which learned how to apply the style to the given contents over several iterations. This method allows you to perform your own experiments on style transfer.

In the next chapter, you will learn how to generate images using Generative Adversarial Networks (GANs).