Let us define the “Training Set”, “Development Set” and “Test Set”, before discussing the partitioning of the data into these.

• Training set: The set of data/examples used to train machine learning algorithm. In machine learning, this data is used to find the ‘optimal’ weights for the model/classifier. Typically, the majority of data used goes to the training set.
• Development (dev)/validation set: The portion of data which is used to evaluate the model/classifier at intermediate stages of training. This set is used to fine tune hyperparameters and evaluate model architecture with various configurations. It is used during the development of the model, not in the final model evaluation.
• Test set: Once the model is fine tuned and fully trained (we are happy with our loss on the training/dev sets), we consider it fully trained. This model is then evaluated. The data that this is evaluated on is called the test set. The test set consists of the unseen portion of the data, therefore providing an unbiased estimate of the final model performance.

In order to achieve a high performance neural network, it is very important to properly partition the dataset into training, development and test sets. It helps to iterate quicker. In addition, it allows to more efficiently measure the bias and variance of the algorithm so that we can select the ways to improve it in an efficient manner.

In previous eras when we had smaller datasets, say up to 10,000 examples, and simpler classifiers, we would split the dataset into a training and testing set. The training set would usually be split into smaller sets to train the classifier with a technique called cross-validation. It was also good practice to split the dataset at a ratio of 60/20/20 (i.e. 60% training data, 20% dev data, 20% test data). However, the modern era of Big Data has changed this rule of thumb. In cases where we have 1,000,000 examples, the ratio of split has changed to be 98/1/1 (i.e. 98% training data, 1% dev data, 1% test data).

The ratio of dev and test sets becomes smaller as we have even more data.