Instance segmentation is the last thing we will look at in this chapter. In many ways, it can be thought of as a fusion of object detection and semantic segmentation. However, it is definitely a step up in difficulty compared to those two problems.

With instance segmentation, the idea is to find every occurrence, what is called an instance, of a desired object or objects within an image. Once these are found, we want to segment off each instance from the other, even if they belong to the same class of objects. In other words, labels are both class-aware (such as car, sign, or person) and instance-aware (such as car 1, car 2, or car 3).

The result of instance segmentation will look something like this:

The similarity between this and semantic segmentation is clear; we still label pixels according to what object they belong to. However, while semantic segmentation has no knowledge of how many times a certain object occurs within an image instance, segmentation does.

This ability to know how many instances of an object are in an image also makes this problem similar to object detection. However, object detection produces a much coarser object boundary, which means occluded objects are more likely to be missed, which shouldn't happen with instance segmentation.