There are a few rules of thumb to consider according to the specific classification task:

• If the new dataset is small and similar to the ImageNet dataset, then we can freeze all the VGG16 networks and retrain only the custom network. In this way, we also minimize the risk of overfitting for the juxtaposed network:

# Freeze all lower layers for layer in base_model.layers: layer.trainable = False

• If the new dataset is large and similar to the ImageNet dataset, then we can retrain the whole juxtaposed network. We have still maintained the precomputed weights as a starting point and perform a few iterations for the fine-tuning:

# UnFreeze all lower layers for layer in model.layers: layer.trainable = True

• If the new dataset is very different from the ImageNet dataset, in practice it might be still good to initialize with weights from a pretrained model. In this case, we would have enough data and confidence to fine-tune through the entire network. More information can be found online at http://cs231n.github.io/transfer-learning/.