If your target domain is at least somewhat similar to the source domain, transfer learning tends to work well. For example, imagine you were classifying an image as containing either a cat or a dog. There are many ImageNet trained image classifiers that would be ideal to use for this type or problem.

Instead, let's imagine that our problem is to classify a CT scan or MRI as containing a tumor or not. This target domain is very different from the ImageNet source domain. As such, while there might be (and probably will be) a benefit in using transfer learning, we will need much more data and probably some fine-tuning to adapt the network to this target domain.