A lot of excellent documentation for Pachyderm is available, and we won't attempt to rehash all of that here. Instead, we will take you through the basics and build a tutorial for managing a simple data pipeline to provide versioned image data to the CNN we built in Chapter 6, Object Recognition with Convolutional Neural Networks.
First, you need to install Docker Desktop and enable Kubernetes for your respective OS. For this example, we are using macOS.
Full instructions can be found at https://docs.docker.com/docker-for-mac/install/, but let's go over them in brief now:
- Download the Docker .dmg file
- Install or launch the file
- Enable Kubernetes
To install and run Pachyderm, follow these steps:
- To enable Kubernetes, select the appropriate checkbox after launching the Docker settings, as follows:
- Ensure that you have a couple of green blobs indicating that your Docker and Kubernetes installations are running. If so, we can confirm that things look okay under the hood by dropping into a Terminal and running the following command:
# kubectl get all
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7m
- Before installing Pachyderm itself, ensure that the cluster is running. We are using Homebrew to install Pachyderm by using the following command (please note that you will need to have the latest version of Xcode installed):
brew tap pachyderm/tap && brew install pachyderm/tap/[email protected]
Updating Homebrew...
...
==> Tapping pachyderm/tap
Cloning into '/usr/local/Homebrew/Library/Taps/pachyderm/homebrew-tap'...
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 13 (delta 7), reused 2 (delta 0), pack-reused 0
Unpacking objects: 100% (13/13), done.
Tapped 7 formulae (47 files, 34.6KB).
==> Installing [email protected] from pachyderm/tap
...
==> Downloading https://github.com/pachyderm/pachyderm/releases/download/v1.9.0rc2/pachctl_1.9.0rc2_d
==> Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/23653453/0d686a0
######################################################################## 100.0%
/usr/local/Cellar/[email protected]/v1.9.0rc2: 3 files, 62.0MB, built in 26 seconds
- You should now be able to launch the Pachyderm command-line tool. First, check that the tool has been installed successfully by running the following command and observing the output:
pachctl help
Access the Pachyderm API.
..
Usage:
pachctl [command]
Administration Commands:
..
- We are almost done setting up our cluster so that we can focus on getting and storing data. The last thing to do is deploy Pachyderm on Kubernetes with the following command:
pachctl deploy local
no config detected at %q. Generating new config...
/Users/xxx/.pachyderm/config.json
No UserID present in config. Generating new UserID and updating config at /Users/xxx/.pachyderm/config.json
serviceaccount "pachyderm" created
clusterrole.rbac.authorization.k8s.io "pachyderm" created
clusterrolebinding.rbac.authorization.k8s.io "pachyderm" created
deployment.apps "etcd" created
service "etcd" created
service "pachd" created
deployment.apps "pachd" created
service "dash" created
deployment.apps "dash" created
secret "pachyderm-storage-secret" created
Pachyderm is launching. Check its status with "kubectl get all"
Once launched, access the dashboard by running "pachctl port-forward"
- Execute the following command to check the status of your cluster. If you run the command just after deploying, you should see the containers being created:
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/dash-8786f7984-tb5k9 0/2 ContainerCreating 0 8s
pod/etcd-b4d789754-x675p 0/1 ContainerCreating 0 9s
pod/pachd-fbbd6855b-jcf6c 0/1 ContainerCreating 0 9s
- They then transition to Running:
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/dash-8786f7984-tb5k9 2/2 Running 0 2m
pod/etcd-b4d789754-x675p 1/1 Running 0 2m
pod/pachd-fbbd6855b-jcf6c 1/1 Running 0 2m
The following section looks at how the data will be prepared.