Similar to what we did in the previous chapter, in this section we will be converting a trained Keras model into a Core ML model using the Core ML Tools package. To avoid any complications of setting up the environment on your local or remote machine, we will leverage the free Jupyter cloud service provided by Microsoft. Head over to https://notebooks.azure.com and log in (or register if you haven't already).

Once logged in, click on the Libraries menu link from the navigation bar, which will take you to a page containing a list of all of your libraries, similar to what is shown in the following screenshot: 

Next, click on the + New Library link to bring up the Create New Library dialog:

Then, click on the From GitHub tab and enter https://github.com/packtpublishing/machine-learning-with-core-ml in the GitHub repository field. After that, give your library a meaningful name and click on the Import button to begin the process of cloning the repository and creating the library. 

Once the library has been created, you will be redirected to the root. From there, click on the Chapter6/Notebooks folder to open up the relevant folder for this chapter, and finally click on the Notebook FastNeuralStyleTransfer_Keras2CoreML.ipynb. Here is a screenshot of what you should see after clicking on the Chapter6 folder:

With our Notebook now loaded, it's time to walk through each of the cells to create our Core ML model; all of the required code exists and all that remains is executing each of the cells sequentially. To execute a cell, you can either use the shortcut keys Shift + Enter or click on the Run button in the toolbar (which will run the currently selected cell), as shown in the following screenshot: 

I will provide a brief explanation of what each cell does. Ensure that you execute each cell as we walk through them so that we all end up with the converted model, which we can then download and import into our iOS project:

import helpers
reload(helpers)

We first import a module that includes the a function that will create and return the Keras model we want to convert:

model = helpers.build_model('images/Van_Gogh-Starry_Night.jpg')

We then use our helpers method build_model to create the model, passing in the style image that the model was trained on. Remember that we are using a feedforward network that has been trained on a single style; while the network can be reused for different styles, the weights are unique per style. 

Talking of weights (previously trained model), let's now load them by running the following cell:

model.load_weights('data/van-gogh-starry-night_style.h5')

Similar to the aforementioned code, we are passing in the weights for the model that was trained on Vincent van Gogh's Starry Night painting for its style.

Next, let's inspect the architecture of the model by calling the summary method on the model itself:

model.summary()

Calling this will return, as the name suggests, a summary of our model. Here is an extract of the summary produced:

____________________________________________________________________
Layer (type) Output Shape Param # Connected to 
====================================================================
input_1 (InputLayer) (None, 320, 320, 3) 0 
____________________________________________________________________
zero_padding2d_1 (ZeroPadding2D) (None, 400, 400, 3) 0 input_1[0][0] 
____________________________________________________________________
conv2d_1 (Conv2D) (None, 400, 400, 64) 15616 zero_padding2d_1[0][0] 
____________________________________________________________________
batch_normalization_1 (BatchNorm (None, 400, 400, 64) 256 conv2d_1[0][0] 
____________________________________________________________________
activation_1 (Activation) (None, 400, 400, 64) 0 batch_normalization_1[0][0] 
____________________________________________________________________
...
...
____________________________________________________________________
res_crop_1 (Lambda) (None, 92, 92, 64) 0 add_1[0][0] 
____________________________________________________________________
...
... 
____________________________________________________________________
rescale_output (Lambda) (None, 320, 320, 3) 0 conv2d_16[0][0] 
====================================================================
Total params: 552,003
Trainable params: 550,083
Non-trainable params: 1,920

As previously mentioned, it's out of scope to go into the details of Python, Keras, or the specifics of this model. Instead I present an extract here to highlight the custom layers embedded in the model (the bold lines). In the context of Core ML Tools, custom layers are layers that have not been defined and, therefore, are not handled during the conversion process, so it is our responsibility to handle these. You can think of the conversion process as a process of mapping layers from a machine learning framework, such as Keras, to Core ML. If no mapping exists, then it is left up to us to fill in the details, as illustrated in the following figure:

 

The two custom layers shown previously are both Lambda layers; a Lambda layer is a special Keras class that conveniently allows writing quick-and-dirty layers using just a function or a Lambda expression (similar to a closure in Swift). Lambda is useful for layers that don’t have a state and are commonly seen in Keras models for doing basic computations. Here, we see two being used, res_crop and rescale_output.

res_crop is part of the ResNet block that crops (as implied by the name) the output; the function is simple enough, with its definition shown in the following code: 

def res_crop(x):
    return x[:, 2:-2, 2:-2] 

Essentially, all that this is doing is cropping the outputs with a padding of 2 for the width and height axis. We can further interrogate this by inspecting the input and output shapes of this layer, by running the following cell:

res_crop_3_layer = [layer for layer in model.layers if layer.name == 'res_crop_3'][0] 

print("res_crop_3_layer input shape {}, output shape {}".format(
    res_crop_3_layer.input_shape, res_crop_3_layer.output_shape))

This cell prints the input and output shape of the layer res_crop_3_layer; the layer receives a tensor of shape (None, 88, 88, 64) and outputs a tensor of shape (None, 84, 84, 64). Here the tuple is broken down into: (batch size, height, width, channels). The batch size is set to None, indicating that it is dynamically set during training.

Our next Lambda layer is rescale_output; this is used at the end of the network to rescale the outputs from the Convolution 2D layer, which passes its data through a tanh activation. This forces our data to be constrained between -1.0 and 1.0, where as we want it in a range of 0 and 255 so that we can convert it into an image. As we did before, let's look at its definition to get a better idea of what this layer does, as shown in the following code:

def rescale_output(x):
    return (x+1)*127.5 

This method performs an element-wise operation that maps the values -1.0 and 1.0 to 0 and 255. Similar to the preceding method (res_crop), we can inspect the input and output shapes of this layer by running the following cell:

rescale_output_layer = [layer for layer in model.layers if layer.name == 'rescale_output'][0]

print("rescale_output_layer input shape {}, output shape {}".format(
    rescale_output_layer.input_shape, 
    rescale_output_layer.output_shape))

Once run, this cell prints the layer's input shape of (None, 320, 320, 3) and output shape of (None, 320, 320, 3). This tells us that this layer doesn't change the shape of the tensor, as well as shows us the output dimensions of our image as 320 x 320 with three channels (RGB). 

We have now reviewed the custom layers and seen what they actually do; the next step is to perform the actual conversion. Run the following cell to ensure that the environment has the Core ML Tools modules installed:

!pip install coremltools

Once installed, we can load the required modules by running the following cell:

import coremltools
from coremltools.proto import NeuralNetwork_pb2, FeatureTypes_pb2

In this instance, I have prewarned you that our model contains custom layers; in some (if not most) instances, you may discover this only when the conversion process fails. Let's see exactly what this looks like by running the following cell and examining its output:

coreml_model = coremltools.converters.keras.convert(
    model, 
    input_names=['image'], 
    image_input_names=['image'], 
    output_names="output")

In the preceding snippet, we are passing our model to the method coremltools.converters.keras.convert, which is responsible for converting our Keras model to Core ML. Along with the model, we pass in the input and output names for our model, as well as setting image_input_names to inform the method that we want the input image to be treated as an image rather than a multidimensional array.

As expected, after running this cell, you will receive an error. If you scroll to the bottom of the output, you will see the line ValueError: Keras layer '<class 'keras.layers.core.Lambda'>' not supported. At this stage, you will need to review the architecture of your model to identify the layer that caused the error and proceed with what you are about to do.

By enabling the parameter add_custom_layers in the conversion call, we prevent the method from failing when the converter encounters a layer it doesn't recognize. A placeholder layer named custom will be inserted as part of the conversion process. In addition to recognizing custom layers, we can pass in a delegate function to the parameter custom_conversion_functions, which allows us to add metadata to the model's specification stating how the custom layer will be handled. 

Let's create this delegate method now; run the cell with the following code:

def convert_lambda(layer):
    if layer.function.__name__ == 'rescale_output':
        params = NeuralNetwork_pb2.CustomLayerParams()
        params.className = "RescaleOutputLambda"
        params.description = "Rescale output using ((x+1)*127.5)"
        return params
    elif layer.function.__name__ == 'res_crop':
        params = NeuralNetwork_pb2.CustomLayerParams()
        params.className = "ResCropBlockLambda"
        params.description = "return x[:, 2:-2, 2:-2]"
        return params
    else:
        raise Exception('Unknown layer')
    return None 

This delegate is passed each custom layer the converter comes across. Because we are dealing with two different layers, we first check which layer we are dealing with and then proceed to create and return an instance of CustomLayerParams. This class allows us to add some metadata used when creating the model's specification for the Core ML conversion. Here we are setting its className, which is the name of the Swift (or Objective-C) class in our iOS project that implements this layer, and description, which is the text shown in Xcode 's ML model viewer.

With our delegate method now implemented, let's rerun the converter, passing in the appropriate parameters, as shown in the following code: 

coreml_model = coremltools.converters.keras.convert(
    model, 
    input_names=['image'], 
    image_input_names=['image'], 
    output_names="output",
    add_custom_layers=True,
    custom_conversion_functions={ "Lambda": convert_lambda })

If all goes well, you should see the converter output each layer it visits, with no error messages, and finally returning a Core ML model instance. We can now add metadata to our model, which is what is displayed in Xcode 's ML model views:

coreml_model.author = 'Joshua Newnham'
coreml_model.license = 'BSD'
coreml_model.short_description = 'Fast Style Transfer based on the style of Van Gogh Starry Night'
coreml_model.input_description['image'] = 'Preprocessed content image'
coreml_model.output_description['output'] = 'Stylized content image' 

At this stage, we could save the model and import into Xcode , but there is just one more thing I would like to do to make our life a little easier. At its core (excuse the pun), the Core ML model is a specification of the network (including the model description, model parameters, and metadata) used by Xcode to build the model when imported. We can get a reference to this specification by calling the following statement:

spec = coreml_model.get_spec() 

With reference to the specification of the models, we next search for the output layer, as shown in the following snippet:

output = [output for output in spec.description.output if output.name == 'output'][0]

We can inspect the output simply by printing it out; run the cell with the following code to do just that:

output

You should see something similar to this:

name: "output"
shortDescription: "Stylized content image"
type {
  multiArrayType {
    shape: 3
    shape: 320
    shape: 320
    dataType: DOUBLE
  }
}

Take note of the type, which is currently multiArrayType (its iOS equivalent is MLMultiArray). This is fine but would require us to explicitly convert it to an image; it would be more convenient to just have our model output an image instead of a multidimensional array. We can do this by simply modifying the specification. Specifically, in this instance, this means populating the type's imageType properties to hint to Xcode that we are expecting an image. Let's do that now by running the cell with this code:

output.type.imageType.colorSpace = FeatureTypes_pb2.ImageFeatureType.ColorSpace.Value('RGB') 
    
output.type.imageType.width = width 
output.type.imageType.height = height
    
coreml_model = coremltools.models.MLModel(spec) 

We first set the color space to RGB, then we set the expected width and height of the image. Finally, we create a new model by passing in the updated specification with the statement coremltools.models.MLModel(spec). Now, if you interrogate the output, you should see something like the following output:

name: "output"
shortDescription: "Stylized content image"
type {
  imageType {
    width: 320
    height: 320
    colorSpace: RGB
  }
}

We have now saved ourselves a whole lot of code to perform this conversion; our final step is to save the model before importing it into Xcode . Run the last cell, which does just that:

coreml_model.save('output/FastStyleTransferVanGoghStarryNight.mlmodel')

Before closing the browser, let's download the model. You can do this by returning the Chapter6/Notebooks directory and drilling down into the output folder. Here you should see the file FastStyleTransferVanGoghStarryNight.mlmodel; simply right-click on it and select the Download menu item (or do it by left-clicking and selecting the Download toolbar item):

With our model in hand, it's now time to jump into Xcode and implement those custom layers.