We are using CIFAR-10 for our example this time instead of MNIST. As such, we do not have the convenience of using the already convenient MNIST loader. Let's quickly go through what it takes to load this new dataset!

We will be using the binary format for CIFAR-10, which you can download here: https://www.cs.toronto.edu/~kriz/cifar.html.

This dataset was put together by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. It consists of 60,000 tiny images 32 pixels high by 32 pixels wide. The binary format of CIFAR-10 is laid out as follows:

<1 x label><3072 x pixel>
<1 x label><3072 x pixel>
<1 x label><3072 x pixel>
<1 x label><3072 x pixel>
<1 x label><3072 x pixel>
<1 x label><3072 x pixel>
 ...
<1 x label><3072 x pixel>

It should be noted that it is not delimited or does not have any other information for validation of the file; as such, you should ensure that the MD5 checksum for the file that you have downloaded matches that on the website. As the structure is relatively simple, we can just pull the binary file straight into Go and parse it accordingly.

The 3,072 pixels are actually three layers of red, green, and blue values from 0 to 255, over a 32 x 32 grid in row-major order, so this gives us our image data.

The label is a number from 0 to 9, representing one of the following categories respectively:

CIFAR-10 comes in six files, five training set files of 10,000 images each and one test set file of 10,000 images:

case "train":
    arrayFiles = []string{
        "data_batch_1.bin",
        "data_batch_2.bin",
        "data_batch_3.bin",
        "data_batch_4.bin",
        "data_batch_5.bin",
    }
case "test":
    arrayFiles = []string{
        "test_batch.bin",
    }
}

Importing this in Go is easy—open the file and read the raw bytes. As every single underlying value is an 8-bit integer within a single byte, we can just cast it to whatever we want. If you wanted the single integer values, you could just convert them all into unsigned 8-bit integers; this is useful for when you want to convert the data into an image. You'll find, however, we've made some slightly different decisions in the code, as follows:

f, err := os.Open(filepath.Join(loc, targetFile))
if err != nil {
    log.Fatal(err)
}

defer f.Close()
cifar, err := ioutil.ReadAll(f)

if err != nil {
    log.Fatal(err)
}

for index, element := range cifar {
    if index%3073 == 0 {
        labelSlice = append(labelSlice, float64(element))
    } else {
        imageSlice = append(imageSlice, pixelWeight(element))
    }
}

As we are interested in using this data for our deep learning algorithm, it is prudent to not stray too far from our happy medium between 0 and 1. We're reusing pixel weight from the MNIST example, as shown here:

func pixelWeight(px byte) float64 {
    retVal := float64(px)/pixelRange*0.9 + 0.1
    if retVal == 1.0 {
        return 0.999
    }
    return retVal
}

This will convert all our pixel values from 0 to 255 to a range between 0.1 and 1.0.

Similarly, for our labels, we will be using one-hot encoding again, encoding the desired label at 0.9 and everything else at 0.1, as shown in the following code:

labelBacking := make([]float64, len(labelSlice)*numLabels, len(labelSlice)*numLabels)
labelBacking = labelBacking[:0]
for i := 0; i < len(labelSlice); i++ {
    for j := 0; j < numLabels; j++ {
        if j == int(labelSlice[i]) {
            labelBacking = append(labelBacking, 0.9)
        } else {
            labelBacking = append(labelBacking, 0.1)
        }
    }
}

We've packaged this into a convenient Load function so we can call it from our code. It'll return two conveniently shaped tensors for us to work with. This gives us a function that can import both the train and test sets:

func Load(typ, loc string) (inputs, targets tensor.Tensor, err error) {

    ...

    inputs = tensor.New(tensor.WithShape(len(labelSlice), 3, 32, 32),        tensor.WithBacking(imageSlice))
    targets = tensor.New(tensor.WithShape(len(labelSlice), numLabels), tensor.WithBacking(labelBacking))
    return
}

This allows us to load the data in my main by calling the following:

if inputs, targets, err = cifar.Load("train", loc); err != nil {
    log.Fatal(err)
}