Now we need to specify a data pipeline in standard JSON. Here, we are mapping a repository to a directory and executing our network in either training or inference mode.

Let's look at our cifar_cnn.json file:

{
 "pipeline": {
    "name": "cifarcnn"
  },
  "transform": {
    "image": "cifarcnn:latest",
    "cmd": [
  "go run main.go"
    ]
  },
  "enable_stats": true,
  "parallelism_spec": {
    "constant": "1"
  },
  "input": {
    "pfs": {
      "repo": "data",
      "glob": "/"
    }
  }
}

The options we've chosen here are straightforward, and you can see the references to the Docker image, commands, and switches, as well as repo and the mount point we're specifying. One thing to note is the parallelism_spec option. Setting this above the default of 1 allows us to scale a specific pipeline stage as required; for example, during the inference phase.

We can now create the pipeline from the preceding template:

pachctl create pipeline -f cifar_cnn.json

This returns you to Command Prompt if there is no error. You can then check the status of the pipeline:

pachctl list pipeline 
NAME INPUT CREATED STATE / LAST JOB
cifarcnn data:/ 8 seconds ago running / running

We can change the level of parallelism dynamically and push the configuration out to our cluster by updating our template:

 "parallelism_spec": {
    "constant": "5"
  },

Then, we can update our cluster and check the status of our job and the k8s cluster pods:

#pachctl update pipeline -f cifar_cnn.json
#pachctl list job 
ID PIPELINE STARTED DURATION RESTART PROGRESS DL UL STATE
9339d8d712d945d58322a5ac649d9239 cifarcnn 7 seconds ago - 0 0 + 0 / 1 0B 0B running

#kubectl get pods
NAME READY STATUS RESTARTS AGE
dash-5c54745d97-gs4j2 2/2 Running 2 29d
etcd-b4d789754-x675p 1/1 Running 1 35d
pachd-fbbd6855b-jcf6c 1/1 Running 1 35d
pipeline-cifarcnn-v1-bwfrq 2/2 Running 0 2m

After giving it some time to run (and using pachctl logs to inspect progress), we can see our successful job:

#pachctl list job
ID OUTPUT COMMIT STARTED DURATION RESTART PROGRESS DL UL STATE
9339d8d712d945d58322a5ac649d9239 cifarcnn 2 minutes ago About a minute 0 1 + 0 / 1 4.444KiB 49.86KiB success