In this section, we will discuss the most important and famous recent datasets used in image classification. This is necessary, because it is likely that any perusal into Computer Vision will overlap with them (including in this book!). Before the arrival of convolutional neural networks, the two main datasets used in image classification competitions by the research community were the Caltech and PASCAL datasets.

The Caltech dataset was established by California Institute of Technology and was released in two versions. Caltech-101 was published in 2003 with 101 categories of about 40 to 800 images per category, and Caltech-256 in 2007 with 256 object categories, containing a total of 30607 images. The images were collected from Google images and PicSearch, and their size was roughly 300x400 pixels.

The Pascal Visual Object Classes (VOC) challenge was established in 2005. Organized every year till 2012, it provides a famous benchmark dataset of a wide range of natural images for Image category Classification, Object detection, Segmentation, and Action Classification. It is a diverse dataset that includes images from flickr of various sizes, pose, orientation, illumination, and occlusion. It has been developed in stages from the year 2005 (only four classes: bicycles, cars, motorbikes, and people, train/validation/test: 1578 images containing 2209 annotated objects) to year 2012 (twenty classes, The train/validation data has 11,530 images containing 27,450 ROI annotated objects and 6,929 segmentations).

The major change came with the PASCAL (VOC) 2007 challenge, when the number of classes increased from 4 to 20 and have been fixed since then. Evaluation metrics for the Classification task changed to average precision. The annotation for test data are only provided until the VOC 2007 challenge.

With the arrival of more sophisticated classification methods, the preceding datasets were not sufficient, and the ImageNet dataset along with the CIFAR dataset, described in the following sections, became the new standards in classification testing.