Managing persistence in IBM Cloud Private
While initially Kubernetes environments were used primarily to run stateless applications, benefits of the platform attracted also workloads which require data persistence. Additionally, by design IBM Cloud Private requires persistent storage for running multiple platform services added on top of the regular Kubernetes cluster like: logging, monitoring, identity and access management, and so forth.
IBM Cloud Private cluster needs to be prepared for the data persistence and in this chapter we discuss options available to the IBM Cloud Private administrator regarding data persistence for running containerized applications.
We assume that the reader is familiar with persistent volume and persistent volume claim terminology and in this chapter we are not going to discuss basic storage concepts which you can find in the Kubernetes documentation (https://kubernetes.io/docs/concepts/storage/persistent-volumes/).
This chapter has the following sections:
4.1 Designing the cluster for data persistence
Starting with version 1.9 Kubernetes implements Container Storage Interface (CSI) specification, which was designed to provide a standard and consistent way of attaching persistent volumes to running containers. As other container-related specifications, CSI is pretty open and allows for wide variety of implementations. Some of the storage provisioners are provided with the Kubernetes, while others need to be installed additionally. See https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner for a list of Kubernetes supported provisioners. Thanks to the pluggable architecture users can define multiple storage providers and multiple storage classes, but it is important to understand that there are significant differences related to the performance and stability of available options.
The following sections discuss the key persistent storage aspects that need to be considered.
4.1.1 Workload specific requirements
The following paragraphs list the workload specific requirements when designing the cluster for data persistence.
Size and performance
The fundamental question related to running workloads that require persistent volumes is related to the size of the data volumes and performance in terms of the required I/O operations per second (IOPS). Obviously the storage available to the containers will not be faster than the underlying disks, so the equipment used for building the cluster is a consideration factor as well.
Data sensitivity and isolation
The next question that needs to be considered is the sensitivity of the data that will be stored and the isolation level required for different workloads. If the data is classified as sensitive (in any sense), you should choose the storage technology that provides encryption at rest and in transit. Most of the dynamic Kubernetes provisioners do not provide the possibility to specify the per-volume encryption, which means that the encryption must be managed at the storage provider level.
For GlusterFS and Ceph you can use the disk volume encryption using dm-crypt. For FlexVolume based drivers that use external storage providers, the encryption level can often be part of the storage class specification (If supported by the back-end hardware). If data isolation is required you can define multiple dedicated hostgroups and create a separate data cluster that uses one of the supported distributed file systems (in such case storageclass used will determine the data placement).
Access mode
Another question is related to concurrency while accessing the data. Kubernetes PersitentVolumesClaim can specify one of the three access modes: ReadWriteOnce (RWO), ReadOnlyMany (ROX) and ReadWriteMany (RWX). If any of the latter two are required, then the underlying storage technology must support concurrent access to the storage volume from multiple worker nodes.
At the time of writing multi-attach was not supported by storage providers implementing FlexVolumes based on the iSCSI protocol as well as the VMware vSphere provider. If your application requires ReadOnlyMany and ReadWriteMany modes, then you should select either any of supported distributed file systems (such as GlusterFS or IBM Spectrum Scale), container native storage provider like Portworx or NFS as your storage technology.
Dynamic versus static provisioning
Storage volumes can be provisioned either manually by cluster administrator or dynamically when the PersistentVolumeClaim object is created. At the time of writing IBM Cloud Private did not support dynamic provisioning for NFS volumes, however there is an open source nfs-client-provisioner implementing this functionality, which can be installed in an IBM Cloud Private cluster. The detailed procedure is described in section “Configuring the dynamic NFS provisioner” on page 123.
For production clusters it is highly recommended to pick the storage option that supports dynamic provisioning.
Snapshots and Data cloning
Finally, do you need ability to easily snapshot and clone persistent volumes? This functionality can prove very useful for testing the applications as well as for performing backup of the data on persistent volumes. At the time of writing such functionality was available from specialized commercial storage solutions like Trident by NetApp, Portworx (with Stork), or IBM Storage Solutions for Containers.
System architecture heterogeneity
IBM Cloud Private is one of the Kubernetes distributions that supports a variety of processor architectures (x86, ppc64le and s390). Some of the distributed file systems may be not available on the specific architecture that you want to use for your work. In case you plan to have heterogeneous worker nodes, we recommend that you check the latest support statement (https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/supported_system_config/supported_os.html).
4.1.2 Maintainability requirements
Another consideration area is about ease and cost of maintaining the storage solution. Is your organization familiar with the selected technology? Do you have a skilled personnel available to monitor, tune and troubleshoot the selected storage provider? Do you have a backup solution in place for the selected provider type?
The answers to the above mentioned questions affect also another choice: whether to use storage provider external to the cluster or the internal one, installed directly on the IBM Cloud Private. When using the distributed file systems installed on IBM Cloud Private nodes the appropriate configuration of the provider is done automatically during the installation and will be automatically upgraded with the future releases of IBM Cloud Private. The drawback of this option is related to the additional CPU load that storage containers will introduce to your worker nodes.
For additional backup considerations see “Backing up PersistentVolume data” on page 99.
4.1.3 Windows worker node support
With IBM Cloud Private version 3.1.2 the initial support for Windows worker nodes was introduced (as a Technology Preview). If any of the worker nodes in your cluster is Windows based, this affects the storage technologies that you may use for your persistent volumes. At the time of writing the Windows worker nodes provided no support for dynamic storage provisioning and the only supported volume types were: local, emptyDir and hostPath.
So in case you want to use persistent volumes on Windows worker nodes, you must provide the shared storage statically with one of the Windows supported technologies like CIFS or NFS and mount the volumes as hostPath.
4.2 Persistent storage for platform services
IBM Cloud Private uses persistent volumes for multiple platform services like MariaDB and MongoDB that are required by Identity and Access Management service or ElasticSearch Data Node used by the Logging service. Those volumes are created at the time of installation as Hostpath and LocalVolume types on the designated master and management nodes as shown in Figure 4-1.
Figure 4-1 Persistent volumes created by IBM Cloud Private during default installation
If you are installing IBM Cloud Private in a high availability topology with multiple master nodes, certain directories on the master nodes must use a shared storage. Those directories are used for storing the platform internal image registry as well as some platform logs.
You can either provide an existing shared storage that uses the NFS protocol or use any other distributed file system like GluserFS. The section “Configuring persistent storage for application containers” discusses mounting a volume from an external GlusterFS storage cluster.
4.3 Configuring persistent storage for application containers
Apart from using persistent storage for platform services, another hot topic is the selection of the storage option for running containerized applications. Which one should you choose? There is only one right answer to this question: It depends on your specific circumstance.
Luckily Kubernetes provides an open architecture which allows for multiple storage options to coexist within the same cluster. Using the storageclass definition users can pick the storage option that is best for their specific workload.
However, it should be noted that while multiple storage technologies provide more flexibility, it comes at the cost of increased operational complexity. In an enterprise environment, the assumption that users will perform backups on their own is a risky one. The same is true for troubleshooting the performance or access problems, so for the operations team careful selection of the storage technology is crucial.
In this section, we describe how to configure some of the popular storage technologies that are supported in IBM Cloud Private environments:
•In “Configuring vSphere storage provider for IBM Cloud Private” on page 119 we discuss how to set up and use the VMware storage provider.
•In “Configuring NFS Storage for IBM Cloud Private” on page 120 we present a short cookbook on setting up the NFS storage provider for IBM Cloud Private.
•In “Configuring GlusterFS for IBM Cloud Private” on page 125 we present how to configure a GlusterFS distributed file system.
•In “Configuring Ceph and Rook for IBM Cloud Private” on page 131 we show you how to configure a Ceph RBD cluster.
•In “Configuring Portworx in IBM Cloud Private” on page 140 we present how to configure a Portworx cluster.
•Finally in “Configuring Minio in IBM Cloud Private” on page 147 we present how to configure Minio, a ligthweight S3-compatible object storage in your IBM Cloud Private cluster.
4.3.1 Configuring vSphere storage provider for IBM Cloud Private
Since IBM Cloud Private is often installed on machines running in VMware vSphere environments, using the VMware cloud provider for dynamic storage provisioning comes as a natural choice. The product documentation on configuring the VMware cloud provider is quite extensive (see https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/vsphere_land.html for more information) so we are not going to replicate it in this book.
We mention a few caveats below that are often omitted.
•Pay attention to the prerequisites of the VMware cloud provider. See https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/vsphere_prereq.html. All of these requirements must be met and the provisioning will not work if any of them are omitted. In vSphere 6.7 or newer, the disk.EnableUUID property must be added to each virtual machine configuration (for all cluster nodes) and set to true.
The best option is to set this property on a template used to create virtual machines for the cluster nodes. This way when adding additional nodes to the cluster later, you will not forget to perform this step.
•Be aware of limitations. Even though the documentation states that ReadWriteMany access mode can be used with the VMware storage when pods are collocated on a single worker node, persistent volumes with RWX and ROX access modes will not be provisioned dynamically.
If you create a persistent volume claim with access mode ReadWriteMany or ReadOnlyMany specifying the storageclass that uses the VMware cloud provider, the provisioner just ignores such a request and your claim will be unbound until you create the appropriate persistent volume manually. This may also happen when provisioning services from IBM Cloud Private catalog, as some of the Helm charts available there may use the RWX access mode for persistent volumes.
4.3.2 Configuring NFS Storage for IBM Cloud Private
Below we present a short cookbook on setting up an NFS server on RedHat Linux server that can be used for test installations of IBM Cloud Private. It is not recommended to use such an NFS server for performing the HA installation, because the NFS server will be a single point of failure in such a setup. If you plan to use NFS as a shared storage for master nodes, consider using a proper NFS server with replication.
Prerequisites
In order to setup an NFS server you must meet the following prerequisites:
•Linux machine (in this case, we will use a Red Hat Enterprise Linux system) with a sufficient disk space available.
•Yum repository configured to install NFS packages.
•The designated NFS server and its clients (in our case the cluster nodes) should be able to reach each other over a network.
Setting up the NFS Server
On the NFS server, run the following command to install the required NFS packages:
sudo yum install -y nfs-utils nfs-utils-lib
Start the NFS service and enable it to persist after reboots:
sudo systemctl start nfs
sudo systemctl enable nfs
In case your Linux machine has a firewall turned on, open the ports required for NFS.
sudo firewall-cmd --permanent --zone=public --add-service=nfs
sudo firewall-cmd --permanent --zone=public --add-service=mountd
sudo firewall-cmd --permanent --zone=public --add-service=rpc-bind
sudo firewall-cmd --reload
Create the directories on the local file system
On the NFS server create a directory that will be used by the clients.
mkdir -p <path_to_share>
Run the following commands to add the entry to /etc/exports.
echo '<path_to_share> <subnet_address>/24(no_root_squash,rw,sync)' >> /etc/exports
Replace the <path_to_share> with the path on the file system that you have sufficient disk space and <subnet_address> with the subnet of your worker nodes. In case you are using a different mask than 255.255.255.0 for your worker nodes, replace the ‘24’ with the correct value. Optionally, instead of a subnet you may specify multiple entries with IP addresses of worker nodes and /32 mask as shown below:
echo '<path_to_share> <node1_IP>/32(no_root_squash,rw,sync)' >> /etc/exports
echo '<path_to_share> <node2_IP>/32(no_root_squash,rw,sync)' >> /etc/exports
...
echo '<path_to_share> <nodeX_IP>/32(no_root_squash,rw,sync)' >> /etc/exports
 
Tip: Do not put any spaces between the netmask value and the opening bracket ‘(‘ -, otherwise you will allow the NFS share to be mounted by any IP address.
Refresh the NFS server with the following command.
exportfs -r
You can verify if the directory was successfully exported by running the exportfs command without parameters as shown in Example 4-1.
Example 4-1 Verification of the exported directories on NFS server
exportfs
 
/storage/vol001 10.10.99.0/24
/storage/vol002 10.10.99.0/24
/storage/vol003 10.10.99.0/24
/storage/vol004 10.10.99.0/24
/storage/vol005 10.10.99.0/24
Configuring NFS clients
On all of the worker nodes install the NFS packages.
sudo yum install -y nfs-utils nfs-utils-lib
You can verify that the shared directories are visible from the NFS client by running the showmount command on the worker node as shown in Example 4-2.
Example 4-2 Verification of shared directories from NFS client
showmount -e 10.10.99.30
 
Export list for 10.10.99.30:
/storage/vol005 10.10.99.0/24
/storage/vol004 10.10.99.0/24
/storage/vol003 10.10.99.0/24
/storage/vol002 10.10.99.0/24
/storage/vol001 10.10.99.0/24
Testing the NFS configuration
On any worker node try mounting the exported directory.
mount -t nfs <nfs_server_IP>:<path_to_share> /mnt
If the mount works, you are ready to use NFS in your IBM Cloud Private cluster.
Create a persistent volume
Create a new persistent volume in IBM Cloud Private with the path to the shared directory (Example 4-3). Using the Create Resource menu option, paste the code into the dialog box.
Example 4-3 YAML file for creating an NFS PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
name: demo-pv
labels:
app: demo-app
spec:
accessModes:
- ReadWriteMany
capacity:
storage: 1Gi
nfs:
path: <path_to_share>
server: <nfs_server_IP>
persistentVolumeReclaimPolicy: Retain
Replace the <path_to_share> and <nfs_server_IP> with your own values.
Create a persistent volume claim for the new volume
Using the Create Resource menu option, paste the following into the dialog box as shown in Example 4-4.
Example 4-4 YAML file for creating a PersitstentVolumeClaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: demo-pvc
 namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
selector:
matchLabels:
app: demo-app
Testing the volume
Create a new deployment to use the new volume. See Example 4-5.
Example 4-5 YAML file for creating a Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-nfs
 namespace: default
labels:
app: nginx-nfs
spec:
replicas: 1
selector:
matchLabels:
app: nginx-nfs
template:
metadata:
labels:
app: nginx-nfs
spec:
volumes:
- name: demo-nfs-storage
persistentVolumeClaim:
claimName: demo-pvc
containers:
- name: nginx-nfs-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: demo-nfs-storage
Once the deployment is ready, get the pod name. Execute a shell session on the pod and create a test file as shown in Example 4-6.
Example 4-6 Create a test file
kubectl get pods -l app=nginx-nfs
 
NAME READY STATUS RESTARTS AGE
nginx-nfs-5b4d97cb48-2bdt6 1/1 Running 0 4m
 
kubectl exec -it nginx-nfs-5b4d97cb48-2bdt6 /bin/bash
 
root@nginx-nfs-5b4d97cb48-2bdt6:/# cd /usr/share/nginx/html/
 
root@nginx-nfs-5b4d97cb48-2bdt6:/usr/share/nginx/html# touch pod-file-check
A file pod-file-check should be created on your NFS server.
Configuring the dynamic NFS provisioner
In Example 4-6 we have created a static NFS share and then manually created the PersistentVolume and PersistentVolumeClaim objects. This approach is good for testing, but does not scale well and is not useful in real-life environments. In order to allow for dynamic creation of NFS based PerststentVolume based on the claims, you can install the NFS dynamic provisioner using the Helm chart from the public Kubernetes Helm repo:
 
Note: Dynamic provisioning of NFS persistent volumes is not supported by IBM, so use it at your own risk.
The nfs-client-provisioner requires existing NFS share to be precreated, so we will reuse the one defined in the section “Create the directories on the local file system” on page 120.
Whitelisting nfs-client-provisioner image
Before installing the Helm chart with the provisioner, we need to allow the provisioner image to be run in our cluster. Create the YAML file with the ClusterImagePolicy as shown in Example 4-7.
Example 4-7 ClusterImagePolicy yaml definition
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ClusterImagePolicy
metadata:
name: nfs-client-provisioner-whitelist
spec:
repositories:
- name: quay.io/external_storage/nfs-client-provisioner:*
Use the following command:
kubectl create -f nfs-provisioner-imagepolicy.yaml
Installing a nfs-client-provisioner Helm chart
In order to follow the procedure below you need cloudctl and Helm CLIs installed. Login to your IBM Cloud Private cluster using cloudctl.
cloudctl login -a https://<cluster_address>:8443 --skip-ssl-validation
Install the Helm chart with the following command:
helm install --name nfs-client-provisioner --tls --set nfs.server=<nfs_server_address> --set nfs.path=<path_to_share> --set podSecurityPolicy.enabled=true --set rbac.pspEnabled=true stable/nfs-client-provisioner
Setting the values podSecurityPolicy.enabled and rbac.pspEnabled to ‘true’ is required in default IBM Cloud Private 3.1.x installations that have the podSecurityPolicies enabled by default. See the output in Example 4-8.
Example 4-8 Sample output of nfs-client-provisioner Helm chart installation
NAMESPACE: kube-public
STATUS: DEPLOYED
 
RESOURCES:
==> v1beta1/PodSecurityPolicy
NAME DATA CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
nfs-client-provisioner false RunAsAny RunAsAny RunAsAny RunAsAny false secret,nfs
 
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
nfs-client-provisioner-5f7bf7f77d-7x7cn 0/1 ContainerCreating 0 0s
 
==> v1/StorageClass
NAME PROVISIONER AGE
nfs-client cluster.local/nfs-client-provisioner 0s
 
==> v1/ServiceAccount
NAME SECRETS AGE
nfs-client-provisioner 1 0s
 
==> v1/ClusterRole
NAME AGE
nfs-client-provisioner-runner 0s
 
==> v1/ClusterRoleBinding
NAME AGE
run-nfs-client-provisioner 0s
 
==> v1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
nfs-client-provisioner 1 1 1 0 0s
As shown in Example 4-8 on page 124, the Helm chart creates a new storage class named “nfs-client”. You can test the dynamic provisioning of the NFS persistent volumes by creating a new persistent volume claim, as shown in Example 4-9.
Example 4-9 YAML file for creating new PersistentVolumeClaim using nfs-client storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-nfs-client
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: nfs-client
As a result you should get the new persistent volume claim bound to the dynamically provisioned persistent volume using a subdirectory from your NFS server as shown in Example 4-10.
Example 4-10 Verification of PersistentVolumeClaim status
kubectl get pvc test-nfs-client -n default
 
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-nfs-client Bound pvc-9dfa0156-349a-11e9-9666-0687b75cc59f 1Gi RWO nfs-client 9m52s
When you delete the persistent volume claim the nfs-client-provisioner will automatically remove the associated persistent volume, however the subdirectories created on the NFS share will not be deleted, but renamed with the “archived.” prefix. This is the default behavior of the nfs-client-provisioner and can be changed with the storageClass.archiveOnDelete parameter.
4.3.3 Configuring GlusterFS for IBM Cloud Private
In this section. we demonstrate the steps that are required to enable the GlusterFS dynamic storage capabilities in an existing IBM Cloud Private cluster. IBM Cloud Private supports configuring GlusterFS both during and after installation and the configuration steps are the same, except that when configuring GlusterFS on existing worker nodes they must be manually labeled.
What is GlusterFS?
GlusterFS is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace. For more information on GlusterFS see https://docs.gluster.org/en/latest/
What is Heketi?
Heketi is a dynamic provisioner for GlusterFS that exposes REST API and is capable of creating storage volumes on request. More information on Heketi can be obtained from the project page: https://github.com/heketi/heketi
GlusterFS prerequisites
The following prerequisites have to be met when configuring GlusterFS within an IBM Cloud Private cluster.
•There must be a minimum of 3 worker nodes.
•Each worker node should have at least one spare disk volume of at least 25GB.
•Each worker node must be connected to a yum repository or have the glusterfs-client package already installed.
Preparing the GlusterFS storage nodes
GlusterFS requires at least 3 nodes in the cluster to be designated as GlusterFS peers. While it is possible to install GlusterFS on master nodes, it is not recommended because GlusterFS will introduce additional workload on the nodes. The following steps describe how to prepare worker nodes for installing GlusterFS. These steps should be repeated for each spare volume that will be used by GlusterFS on each of the worker nodes.
Enabling the GlusterFS Client on the worker nodes
Each cluster node that will use a persistent volume created on GlusterFS needs to have a glusterFS client installed. To install and configure a glusterFS client on Red Hat run the following commands:
sudo yum install glusterfs-client
sudo modprobe dm_thin_pool
echo dm_thin_pool | sudo tee -a /etc/modules-load.d/dm_thin_pool.conf
For other operating systems you can find the appropriate commands here: https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/prepare_nodes.html.
Identifying the device to be used by GlusterFS
Access the terminal on worker nodes and run the following command:
fdisk -l
The output will be a list of attached disk volumes, along with their details.
The sample output is shown in Example 4-11.
Example 4-11 Sample output of fdisk command
fdisk -l
 
Disk /dev/loop0: 870 MiB, 912261120 bytes, 1781760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x13982071
 
Device Boot Start End Sectors Size Id Type
/dev/loop0p1 * 0 1781759 1781760 870M 0 Empty
/dev/loop0p2 448612 453347 4736 2.3M ef EFI (FAT-12/16/32)
 
Disk /dev/sda: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x9cc95f36
 
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 419430366 419428319 200G 83 Linux
 
Disk /dev/sdb: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
In this example, the disk name we are looking for is /dev/sdb. You may see other names such as /dev/vdb, /dev/sdc as this depends on the hypervisor type and number of disks that are attached to the virtual machine.
Wiping the disk
Run the following command to wipe the disk ready for use.
sudo wipefs --all --force /dev/sdb
Getting the SymLink
The config.yaml file requires a symlink for the hard disk device to use on the virtual machine. Run the following command to retrieve the symlink path.
ls -ltr /dev/disk/* | grep ‘sdb’
This should return a line similar to the following:
lrwxrwxrwx 1 root root 9 Mar 5 21:34 pci-0000:00:10.0-scsi-0:0:1:0 -> ../../sdb
This gives the /dev/disk/by-path symlink, which we will use in this example. Note that there are other methods available such as /dev/disk/by-id, /dev/disk/by-uuid, or /dev/disk/by-label, but only by-path has been used in this example.
 
Note: In some environments, such as IBM Cloud Virtual Servers or SUSE Linux Enterprise Server (SLES), no symlinks are automatically generated for the devices. In such case, you have to manually create the symlinks. See https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/prepare_disks.html#manual for the detailed procedure.
Make a note of the symlink and its link path. For the example device sdb, /dev/disk/by-path is the link path and pci-0000:00:10.0-scsi-0:0:1:0 is the symlink. For each device that you are using for the GlusterFS configuration, you need to add the line in the config.yaml file. For the example device sdb, you would add /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0 in the config.yaml file.
 
Important: While symlinks may be identical on different nodes, the actual value will depend on the number and type of the disks in the particular node. So it is better to double-check the symlink path on each node individually.
Configuring the GlusterFS hostgroup in the hosts file
When the nodes are prepared, the next step is to modify the hosts file in the installation directory on the boot node. Create a new stanza named [hostgroup-glusterfs] and add the IP addresses of the worker nodes that you prepared before, as shown in Example 4-12.
Example 4-12 Adding hostgroup stanza for glusterfs nodes
[master]
...
[management]
..
 
[worker]
...
 
[proxy]
...
 
[hostgroup-glusterfs]
<worker_1_ip>
<worker_2_ip>
<worker_3_ip>
Labeling worker nodes for GlusterFS
When installing GlusterFS on an existing cluster you must manually label the worker nodes which are listed in the [hostgroup-glusterfs] stanza in the previous step. Run the following command:
kubectl label nodes <worker_1_ip> <worker_2_ip> <worker_3_ip> hostgroup=glusterfs --overwrite=true
 
Note: In some environments where the nodes names use hostnames instead of the IP addresses, replace the worker node IPs with the appropriate worker node names, as listed in the output of kubectl get nodes command.
Configuring the GlusterFS volume information in config.yaml
The GlusterFS configuration needs to be added to the config.yaml. This is applicable to both new installations, or when adding GlusterFS to an existing IBM Cloud Private cluster.
In the default configuration, glusterfs is disabled in the config.yaml supplied by IBM in the icp-inception image. To enable GlusterFS in the config.yaml file located in the installation directory on the boot node find the management_services section and change the value storage-glusterfs to enabled.
Example 4-13 Enabling the glusterfs management service
management_services:
...
  storage-glusterfs: enabled
...
Next, edit the GlusterFS Storage Settings section (which is commented out by default) by providing the following properties as shown in Example 4-14.
Example 4-14 Sample GlusterFS configuration section in the config.yaml
## GlusterFS Storage Settings
storage-glusterfs:
nodes:
- ip: <worker_1_ip>
devices:
- /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0
- ip: <worker_2_ip>
devices:
- /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0
- ip: <worker_3_ip>
devices:
- /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0
storageClass:
create: true
name: glusterfs
isDefault: false
volumeType: replicate:3
reclaimPolicy: Delete
volumeBindingMode: Immediate
volumeNamePrefix: icp
additionalProvisionerParams: {}
allowVolumeExpansion: true
gluster:
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
heketi:
backupDbSecret: heketi-db-backup
authSecret: heketi-secret
maxInFlightOperations: 20
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
cpu: 1000m
memory: 1Gi
nodeSelector:
key: hostgroup
value: glusterfs
prometheus:
enabled: true
path: "/metrics"
port: 8080
tolerations: []
podPriorityClass: system-cluster-critical
 
Tip: The entries shown in Example 4-14 are already present in the config.yaml, but are commented out. Uncomment and change them to suit your needs. The resources specified for gluster pods are minimal.
You will find below the description of the important elements of the configuration that were marked as bold in Example 4-14 on page 129:
ip is the IP address of the worker node where you want to deploy GlusterFS (you must add at least three worker nodes).
device is the full path to the symlink of the storage device.
storageClass this section defines if the storageclass should automatically be created (created: true) and the properties of a storage class for GlusterFS, such as the name, reclamationPolicy, replicacount, and so forth.
nodeSelector this section refers to the labels that are added to the worker nodes. Make sure that the key and value are identical to those used in the step “Labeling worker nodes for GlusterFS” on page 128.
prometheus this section enables automatic collection of the performance metrics related to glusterFS in the Prometheus database. It is recommended that you turn it on by providing the enabled: true value.
The GlusterFS cluster volumes, as well as the hosts and the config.yaml on the boot node, are now ready. You can run the installation with the following command:
docker run --rm -t -e LICENSE=accept --net=host -v $(pwd):/installer/cluster <icp_inception_image_used_for_installation> addon
The GlusterFS cluster nodes are now ready. You can test the storage provisioning with the following yaml file:
Example 4-15 Testing glusterfs storageclass
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: glusterfs-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: glusterfs
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-glusterfs
labels:
app: nginx-glusterfs
spec:
replicas: 2
selector:
matchLabels:
app: nginx-glusterfs
template:
metadata:
labels:
app: nginx-glusterfs
spec:
volumes:
- name: demo-glusterfs-storage
persistentVolumeClaim:
claimName: glusterfs-pvc
containers:
- name: nginx-glusterfs-container
image: nginx
ports:
- containerPort: 80
name: "http-server"
volumeMounts:
- mountPath: "/usr/share/nginx/html"
name: demo-glusterfs-storage
To test the volume, follow the steps described in “Testing the volume” on page 122 by using the selector app=nginx-glusterfs and creating a file on one pod and verifying its existence on the other one.
4.3.4 Configuring Ceph and Rook for IBM Cloud Private
In this section we demonstrate how to create a Ceph RBD cluster and a Rook management agent in IBM Cloud Private.
What is Ceph?
Ceph is open source software designed to provide highly scalable object, block and file-based storage under a unified system.
Ceph storage clusters are designed to run on commodity hardware, using an algorithm called CRUSH (Controlled Replication Under Scalable Hashing) to ensure data is evenly distributed across the cluster and that all cluster nodes can retrieve data quickly without any centralized bottlenecks. See https://ceph.com/ceph-storage/ for more information.
What is Rook
Rook is an open source orchestrator for distributed storage systems running in cloud native environments.
Rook turns distributed storage software into self-managing, self-scaling, and self-healing storage services. It does this by automating deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management. Rook uses the facilities that are provided by the underlying cloud-native container management, scheduling and orchestration platform to perform its duties.
Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience.
See https://rook.io for more information.
Prerequisites
In order to configure a Ceph storage with Rook on an IBM Cloud Private cluster, the following prerequisites have to be met:
•IBM Cloud Private cluster must be up and running.
•Cluster nodes that will be part of the Ceph storage must have additional raw disks available or enough space in existing filesystems.
•You must have at least a Cluster Administrator role.
 
Attention: While Ceph provides file based and object storage interfaces, the content provided by IBM creates a distributed storage cluster for block devices (RADOS Block Devices or RBD in short). At the time of writing Rook did not support mounting RBD volumes to multiple nodes at the same time, so this class of storage cannot be used for RWX and ROX access modes.
This guide assumes you have a cluster with internet access to pull the required Helm packages and images.
At the time of writing this book, the installation of a Rook Ceph cluster was a three-step process:
1. Configure role-based access control (RBAC).
2. Install the Rook Ceph Operator Helm chart.
3. Install the Rook Ceph cluster (ibm-rook-rbd-cluster) chart.
Configure role-based access control (RBAC)
A Rook Ceph Operator chart is provided by the Rook project. It does not include definitions of the appropriate RBAC roles required in the IBM Cloud Private environment. Thus, as an initial step you must configure some of the security objects. Authenticate to your cluster with the admin user using the cloudctl command and run the following steps.
Create a PodSecurityPolicy
PodSecurityPolicy, as shown in Example 4-16, is required for the Rook Operator and Rook Ceph chart to install properly.
Example 4-16 Sample YAML file defining PodSecurityPolicy for Rook
apiVersion: extensions/v1beta1
kind: PodSecurityPolicy
metadata:
name: rook-privileged
spec:
fsGroup:
rule: RunAsAny
privileged: true
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- '*'
allowedCapabilities:
- '*'
hostPID: true
hostIPC: true
hostNetwork: true
hostPorts:
# CEPH ports
- min: 6789
max: 7300
# rook-api port
- min: 8124
max: 8124
 
Tip: Copying the content from the book PDF may mess up indentation, which results in parsing errors. Access the examples source code at GitHub: https://github.com/IBMRedbooks/SG248440-IBM-Cloud-Private-System-Administrator-s-Guide.git.
Create the file rook-priviledged-psp.yaml with the content shown in Example 4-16 and run the following command:
kubectl create -f rook-priviledged-psp.yaml
As the result you should see the following output:
podsecuritypolicy.extensions/rook-privileged created
Create a ClusterRole
Next, you need to create a ClusterRole that uses the PodSecurityPolicy which was defined above, as shown in Example 4-17.
Example 4-17 Sample YAML file with ClusterRole definition for Rook
# privilegedPSP grants access to use the privileged PSP.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: privileged-psp-user
rules:
- apiGroups:
- extensions
resources:
- podsecuritypolicies
resourceNames:
- rook-privileged
verbs:
- use
Create a file rook-priviledged-psp-user.yaml with the content of Example 4-17 and run the following command:
kubectl create -f rook-priviledged-psp-user.yaml
As the result you should see the following output:
clusterrole.rbac.authorization.k8s.io/privileged-psp-user created
Create a namespace for Rook operator
The ClusterRole defined above must be bound to a namespace in which you install the Rook operator chart. In our example we will create a new namespace for that purpose with the following command:
kubectl create namespace rook
As the result you should see the following output:
namespace/rook created
Create a ClusterRoleBinding for Rook operator
To bind the privileged-psp-user role to the namespace created above, you need to create a ClusterRoleBinding object. See Example 4-18.
Example 4-18 Sample YAML file for ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: rook-agent-psp
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: privileged-psp-user
subjects:
- kind: ServiceAccount
name: rook-agent
namespace: rook
Create a file rook-clusterrolebinding.yaml with the content of Example 4-18 and run the following command:
kubectl create -f rook-clusterrolebinding.yaml
As the result you should see the following output:
clusterrolebinding.rbac.authorization.k8s.io/rook-agent-psp created
Create RBAC for Pre-validation checks (Optional)
Rook Ceph Cluster (ibm-rook-rbd-cluster V 0.8.3) Helm chart includes a pre-validation check that requires additional RBAC permissions. You can either create the resources listed in Example 4-19 or set preValidation.enabled=false during the installation.
Example 4-19 Sample YAML file for creating RBAC for pre-validation checks
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: pod-reader-binding
subjects:
- kind: ServiceAccount
name: default
namespace: rook
roleRef:
kind: ClusterRole
name: pod-reader
apiGroup: rbac.authorization.k8s.io
You need a ClusterRoleBinding for each namespace in which you install the Rook Ceph Cluster chart. In our example, we use the same namespace rook we have created in step “Create a namespace for Rook operator” on page 134. Create a rook-pre-validation.yaml file with content of the Example 4-19 on page 134 and run the command:
kubectl create -f rook-pre-validation.yaml
As the result you should see the following output:
clusterrole.rbac.authorization.k8s.io/pod-reader created
clusterrolebinding.rbac.authorization.k8s.io/pod-reader-binding created
Install the Rook Ceph Operator Helm chart
At the time of writing this book IBM Cloud Private supported the Rook Ceph Operator in version 0.8.3. Follow the steps below to install Rook Ceph Operator chart using the Helm CLI.
Add the rook Helm repo with the following command:
helm repo add rook-beta https://charts.rook.io/beta
Create the rook-values.yaml file with the content shown in Example 4-20.
Example 4-20 The values.yaml file used for Rook Ceph Operator chart installation
image:
prefix: rook
repository: rook/ceph
tag: v0.8.3
pullPolicy: IfNotPresent
 
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
 
rbacEnable: true
pspEnable: true
Then, run the following command to install the Rook Ceph Operator chart:
helm install --tls --namespace rook --name rook-ceph rook-beta/rook-ceph
--version v0.8.3 -f rook-values.yaml
Example 4-21 shows the sample output.
Example 4-21 Sample output
helm install --tls --namespace rook --name rook-ceph rook-beta/rook-ceph -f rook-values.yaml
 
NAME: rook-ceph
LAST DEPLOYED: Fri Feb 22 00:29:10 2019
NAMESPACE: rook
STATUS: DEPLOYED
 
RESOURCES:
==> v1beta1/PodSecurityPolicy
NAME DATA CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
00-rook-ceph-operator true * RunAsAny RunAsAny RunAsAny RunAsAny false *
 
==> v1beta1/CustomResourceDefinition
NAME AGE
clusters.ceph.rook.io 0s
volumes.rook.io 0s
pools.ceph.rook.io 0s
objectstores.ceph.rook.io 0s
filesystems.ceph.rook.io 0s
 
==> v1beta1/ClusterRole
rook-ceph-system-psp-user 0s
rook-ceph-global 0s
rook-ceph-cluster-mgmt 0s
 
==> v1beta1/Role
rook-ceph-system 0s
 
==> v1beta1/RoleBinding
NAME AGE
rook-ceph-system 0s
 
==> v1/ServiceAccount
NAME SECRETS AGE
rook-ceph-system 1 0s
 
==> v1beta1/ClusterRoleBinding
NAME AGE
rook-ceph-global 0s
rook-ceph-system-psp-users 0s
 
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
rook-ceph-operator 1 1 1 0 0s
 
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-f4cd7f8d5-ks5n5 0/1 ContainerCreating 0 0s
 
 
NOTES:
The Rook Operator has been installed. Check its status by running:
kubectl --namespace rook get pods -l "app=rook-ceph-operator"
 
Visit https://rook.io/docs/rook/master for instructions on how
to create & configure Rook clusters
You can verify the successful installation, as shown in Example 4-22.
Example 4-22 Verification of Rook Ceph Operator pods status
kubectl get pods -n rook
 
NAME READY STATUS RESTARTS AGE
rook-ceph-agent-g2r9g 1/1 Running 0 2m49s
rook-ceph-agent-nscs5 1/1 Running 0 2m49s
rook-ceph-agent-qdwz7 1/1 Running 0 2m49s
rook-ceph-operator-6d7f98d49d-shtp9 1/1 Running 0 3m1s
rook-discover-976cj 1/1 Running 0 2m49s
rook-discover-gmkj5 1/1 Running 0 2m49s
rook-discover-jgd7q 1/1 Running 0 2m49s
Install the Rook Ceph cluster (ibm-rook-rbd-cluster) chart
As the last step we install the Rook Ceph cluster chart. You can do this from the IBM Cloud Private catalog. However, below we show the command to install it with the Helm CLI.
Prepare the values.yaml file for Rook RBD Cluster chart
Create a rbd-values.yaml file that contains the nodenames and disk devices as shown in Example 4-23 or node names and directory paths as shown in Example 4-24.
Example 4-23 Sample rbd-values.yaml file specifying target disk devices
rookOperatorNamespace: "rook"
cluster:
storage:
nodes:
- name: "10.73.147.205"
devices:
- name: "xvde"
- name: "10.73.147.237"
devices:
- name: "xvde"
- name: "10.73.147.243"
devices:
- name: "xvde"
Example 4-24 Sample values.yaml file specifying target paths
rookOperatorNamespace: "rook"
cluster:
storage:
nodes:
- name: "1.2.3.4"
directories:
- path: "/rook/storage-dir"
- name: "1.2.3.5"
directories:
- path: "/rook/storage-dir"
- name: "1.2.3.6"
directories:
- path: "/rook/storage-dir"
 
Important: You should use only one of the above examples either raw devices or directory paths.
Install the Rook RBD Cluster using the command line
To install the charts from a remote repository using the Helm CLI, you must add this repo on the workstation from which you run the Helm commands. Add the ibm-charts repo with the following commands:
export HELM_HOME=~/.helm
helm init --client-only
helm repo add ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable
You can verify that this step has been completed successfully as shown in Example 4-25.
Example 4-25 Sample output of helm repo list command
helm repo list
 
NAME       URL
stable     https://kubernetes-charts.storage.googleapis.com
local      http://127.0.0.1:8879/charts
rook-beta  https://charts.rook.io/beta
ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable
Install the ibm-rook-rbd-cluster chart as shown in Example 4-26. It will be deployed to the namespace selected in kubectl context. To target different namespace add --namespace <namespace> to the command.
Example 4-26 Installation of ibm-rook-rbd-cluster Helm chart
helm install --name rook-rbd-cluster -f rbd-values.yaml ibm-charts/ibm-rook-rbd-cluster --tls
 
NAME: rook-rbd-cluster
LAST DEPLOYED: Fri Feb 22 00:33:16 2019
NAMESPACE: kube-system
STATUS: DEPLOYED
 
RESOURCES:
==> v1beta1/Cluster
NAME AGE
rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-cluster 0s
 
==> v1beta1/Pool
rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-pool 0s
 
==> v1/StorageClass
NAME PROVISIONER AGE
rbd-storage-class rook.io/block 0s
 
==> v1/ServiceAccount
NAME SECRETS AGE
rook-ceph-cluster 1 0s
 
==> v1beta1/Role
NAME AGE
rook-ceph-cluster 0s
 
==> v1beta1/RoleBinding
NAME AGE
rook-ceph-cluster 0s
rook-ceph-cluster-mgmt 0s
 
==> v1/RoleBinding
rook-ceph-osd-psp 0s
rook-default-psp 0s
 
NOTES:
1. Installation of Rook RBD Cluster rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-cluster successful.
 
kubectl get cluster rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-cluster --namespace kube-system
 
2. A RBD pool kube-system-pool is also created.
 
kubectl get pool --namespace kube-system
 
3. Storage class rbd-storage-class can be used to create RBD volumes.
 
kubectl get storageclasses rbd-storage-class
You can verify that the resources were created as shown in Example 4-27. Target namespace may be different depending on to which namespace you have deployed the chart.
Example 4-27 Verify that the resources were created
kubectl get cluster rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-cluster
--namespace kube-system
 
NAME AGE
rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-cluster 3m
 
kubectl get pool --namespace kube-system
 
NAME AGE
rook-rbd-cluster-ibm-rook-rbd-cluster-rook-ceph-pool 4m
 
kubectl get storageclasses rbd-storage-class
 
NAME PROVISIONER AGE
rbd-storage-class rook.io/block 4m8s
Verification of the ceph storage cluster
To verify if everything works fine, create a new persistent volume claim with the storage class rbd-storage-class. Create the ceph-test-pv.yaml file that contains the lines shown in Example 4-28.
Example 4-28 ceph-test-pv.yaml file
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-test
namespace: default
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: rbd-storage-class
Run the following command to create a persistent volume claim:
kubectl create -f ceph-test-pv.yaml
The output of the command should be:
persistentvolumeclaim/ceph-test created
Finally, you can verify that the persistent volume was dynamically created and bound as shown in Example 4-29.
Example 4-29 Verification of the PersistentVolumeClaim status
kubectl get pvc -n default
 
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
ceph-test Bound pvc-e6a6d1e8-3635-11e9-a33b-06d591293f01 1Gi RWO rbd-storage-class 13s
4.3.5 Configuring Portworx in IBM Cloud Private
Portworx is a commercial, cloud native storage and data management solution for Kubernetes. There is a Portworx community helm chart provided in IBM Cloud Private catalog. When installing the helm chart you automatically get a 30 day Trial license for PX-Enterprise product. Limited free PX-Developer version is also available from Portworx. More info on Portworx here: https://portworx.com/products/introduction/.
Portworx Helm chart has multiple options regarding what drives/filesystems to use as well as set of optional components (for example, a dedicated UI to manage Portworx cluster). The described procedure uses default settings and installs the PX-Enterprise Trial version with Storage Operator Runtime for Kubernetes (Stork), as described on github.com:
Stork can be used to co-locate pods with where their data is located. This is achieved by using a kubernetes scheduler extender. The scheduler is configured to use stork as an extender. Therefore, every time a pod is being scheduled, the scheduler will send filter and prioritize requests to stork.
Stork will then check with the storage driver. You can either configure the default Kubernetes scheduler to communicate with stork or launch another instance of kube-scheduler.1
More information on Stork can be found here: https://github.com/libopenstorage/stork.
Prerequisites
To install the Portworx Helm chart you need at least 1 node with 4 CPU cores, 4 GB RAM and available free unmounted volumes or filesystems of at least 8 GB size. (See the full list of prerequisties here: https://docs.portworx.com/start-here-installation/#installation-prerequisites).
Portowrx requires an existing key-value database to be available at the installation time (for example etcd or consul). When installing on IBM Cloud Private it is not recommended to reuse cluster etcd database.
Installing etcd database for portworx
The easiest way to provide etcd database for Portworx is to install one on IBM Cloud Private. In our environment we install single node instance (not recommended for production use) using the Bitnami Helm chart. To install the etcd Helm chart, follow the steps below:
1. Authenticate to IBM Cloud Private cluster using cloudctl login command.
2. Add the Bitnami chart repo:
helm repo add bitnami https://charts.bitnami.com
3. Install the etcd Helm chart to a <namespace> providing existing <storage_class> name. If you don’t yet have the storage class with dynamic provisioning available, you can create a static persistent volume with any name as the storage class and then use it in the command.
helm install bitnami/etcd --version 2.0.0 --name px --namespace <namespace> --set persistence.storageClass=<storage_class> --tls
4. Verify that the etcd service has been created. Note down the clusterIP address.
kubectl get svc px-etcd
Installing the Portworx Helm chart
To install the Portworx Helm chart version 1.0.1 (was available in the community catalog at the time of writing the book) you need the following steps:
1. Add ImagePolicy or ClusterImagePolicy to allow the docker images required by the Portworx chart to be run in your cluster. As the Portworx chart has to be installed into the kube-system we used ImagePolicy. Create a px-imagepolicy.yaml file with the content shown in Example 4-30.
Example 4-30 ImagePolicy for portworx images
apiVersion: securityenforcement.admission.cloud.ibm.com/v1beta1
kind: ImagePolicy
metadata:
name: "portworx-image-policy"
namespace: "kube-system"
spec:
repositories:
- name: "docker.io/portworx/*"
policy:
va:
enabled: false
- name: "docker.io/openstorage/*"
policy:
va:
enabled: false
- name: "gcr.io/google-containers/*"
policy:
va:
enabled: false
- name: "docker.io/lachlanevenson/*"
policy:
va:
enabled: false
- name: "docker.io/hrishi/*"
policy:
va:
enabled: false
- name: "quay.io/k8scsi/*"
policy:
va:
enabled: false
Add the Image policy with the command:
kubectl create -f portwokx-imagepolicy.yaml
2. By default IBM Cloud Private in version 3.1.x uses the restricted mode, which means that the service account used to run pods needs a binding to a cluster role with the proper cluster RBAC permissions. Portworx chart creates a set of service accounts and roles, but these are not sufficient in an IBM Cloud Private environment. Create an additional set of clusterrolebindings as shown Example 4-31.
Example 4-31 Additional ClusterRoleBindings for portworx chart
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: px-account-anyuid-binding
roleRef:
kind: ClusterRole
name: ibm-anyuid-hostaccess-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: px-account
namespace: kube-system
 
---
 
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: px-account-privileged-binding
roleRef:
kind: ClusterRole
name: ibm-privileged-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: px-account
namespace: kube-system
 
---
 
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: stork-sa-anyuid-binding
roleRef:
kind: ClusterRole
name: ibm-anyuid-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: stork-account
namespace: kube-system
 
---
 
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: stork-sched-sa-anyuid-binding
roleRef:
kind: ClusterRole
name: ibm-anyuid-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: stork-scheduler-account
namespace: kube-system
 
---
 
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: portworkx-hook-anyuid-binding
roleRef:
kind: ClusterRole
name: ibm-anyuid-clusterrole
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: portworx-hook
namespace: kube-system
Create a portworx-clusterrolebindings. yaml file with the content of Example 4-31 and install it with the following command:
kubectl apply -f portworx-clusterrolebindings.yaml
If the ibm-anyuid-clusterrole is not bound to service accounts used by Portworx, you will notice that the pods will stall in ConatinerCreatingError state with the following message in kubectl describe pod output:
Error: container has runAsNonRoot and image will run as root
You can now install a Portworx chart either from the IBM Cloud Private catalog page or using the Helm client. Below we show how to install Portworx with the Helm CLI.
3. Add the community chart repo to you local Helm client using:
helm repo add ibm-community-charts https://raw.githubusercontent.com/IBM/charts/master/repo/community
4. Install the Portworx chart as shown in Example 4-32. Use <ClusterIP_of_etcd_service> from the px-etcd service (See step 4 of “Installing etcd database for portworx” section.) In our installation when we used the px-etcd.<namespace>.svc.cluster.local service name, the Portworx pods were not able to resolve the name.
 
Attention: By default, the Portworx chart creates a daemonset that means that the Portworx pods will spread on all of the worker nodes in the cluster, scanning for available disk drives. To prevent this behaviour, label the nodes where you don’t want Portworx to be installed with the following command:
kubectl label nodes <nodes list> px/enabled=false --overwrite
Example 4-32 Installation of the Portworx Helm chart
helm install --name portworx ibm-community-charts/portworx --namespace
kube-system --set etcdEndPoint=etcd:http://<ClusterIP_of_etcd_service>:2379 --tls
 
NAME: portworx
LAST DEPLOYED: Sun Mar 10 17:56:36 2019
NAMESPACE: kube-system
STATUS: DEPLOYED
 
RESOURCES:
==> v1/ConfigMap
NAME DATA AGE
stork-config 1 0s
 
==> v1/ClusterRole
NAME AGE
node-get-put-list-role 0s
stork-scheduler-role 0s
stork-role 0s
 
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
portworx-service ClusterIP 10.0.0.246 <none> 9001/TCP 0s
stork-service ClusterIP 10.0.118.214 <none> 8099/TCP 0s
 
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
stork-scheduler 3 3 3 0 0s
stork 3 3 3 0 0s
 
==> v1beta1/StorageClass
NAME PROVISIONER AGE
portworx-null-sc kubernetes.io/portworx-volume 0s
portworx-db2-sc kubernetes.io/portworx-volume 0s
portworx-db-sc kubernetes.io/portworx-volume 0s
portworx-shared-sc kubernetes.io/portworx-volume 0s
 
==> v1/StorageClass
stork-snapshot-sc stork-snapshot 0s
 
==> v1/ServiceAccount
NAME SECRETS AGE
px-account 1 0s
stork-scheduler-account 1 0s
stork-account 1 0s
 
==> v1/ClusterRoleBinding
NAME AGE
node-role-binding 0s
stork-scheduler-role-binding 0s
stork-role-binding 0s
 
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
portworx 3 3 0 3 0 <none> 0s
 
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
portworx-6qlvp 0/1 ContainerCreating 0 0s
portworx-c4h4t 0/1 ContainerCreating 0 0s
portworx-p86qn 0/1 ContainerCreating 0 0s
stork-scheduler-679f999679-5gbt7 0/1 ContainerCreating 0 0s
stork-scheduler-679f999679-jq6gp 0/1 ContainerCreating 0 0s
stork-scheduler-679f999679-w5mv5 0/1 ContainerCreating 0 0s
stork-86bb9cb55d-467wr 0/1 ContainerCreating 0 0s
stork-86bb9cb55d-969vf 0/1 ContainerCreating 0 0s
stork-86bb9cb55d-khwrv 0/1 ContainerCreating 0 0s
 
NOTES:
 
Your Release is named "portworx"
Portworx Pods should be running on each node in your cluster.
 
Portworx would create a unified pool of the disks attached to your Kubernetes nodes.
No further action should be required and you are ready to consume Portworx Volumes as part of your application data requirements.
 
For further information on usage of the Portworx in creating Volumes please refer
https://docs.portworx.com/scheduler/kubernetes/preprovisioned-volumes.html
 
For dynamically provisioning volumes for your Stateful applications as they run on Kubernetes please refer
https://docs.portworx.com/scheduler/kubernetes/dynamic-provisioning.html
 
Want to use Storage Orchestration for hyperconvergence, Please look at STork here. (NOTE: This isnt currently deployed as part of the Helm chart)
https://docs.portworx.com/scheduler/kubernetes/stork.html
 
Refer application solutions such as Cassandra, Kafka etcetera.
https://docs.portworx.com/scheduler/kubernetes/cassandra-k8s.html
https://docs.portworx.com/scheduler/kubernetes/kafka-k8s.html
We tsted this procedure on an IBM Cloud Private 3.1.2 cluster running Centos 7 nodes. Additional steps might be required while running it on IBM Cloud Virtual Servers and SLES.
If the installation succeeds you can verify the Portworx cluster status as shown in Example 4-33.
Example 4-33 Verification of portworx cluster status
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n kube-system -- /opt/pwx/bin/pxctl status
 
Status: PX is operational
License: Trial (expires in 31 days)
Node ID: dc9ddffb-5fc3-4f5d-b6df-39aee43325df
IP: 10.10.27.120
Local Storage Pool: 1 pool
POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION
0 HIGH raid0 50 GiB 6.0 GiB Online default default
Local Storage Devices: 1 device
Device Path Media Type Size Last-Scan
0:1 /dev/sdb STORAGE_MEDIUM_MAGNETIC 50 GiB 10 Mar 19 10:14 UTC
total - 50 GiB
Cluster Summary
Cluster ID: px-cluster-1a2f8f7b-f4e8-4963-8011-3926f00ac9bc
Cluster UUID: 91f47566-cac4-46f0-8b10-641671a32afd
Scheduler: kubernetes
Nodes: 3 node(s) with storage (3 online)
IP ID SchedulerNodeName StorageNode Used Capacity Status StorageStatus Version Kernel OS
10.10.27.120 dc9ddffb-5fc3-4f5d-b6df-39aee43325df icp-worker3 Yes 6.0 GiB 50 GiB Online Up (This node) 2.0.2.3-c186a87 3.10.0-957.5.1.el7.x86_64 CentOS Linux 7 (Core)
10.10.27.118 85019d27-9b33-4631-84a5-7a7b6a5ed1d5 icp-worker1 Yes 6.0 GiB 50 GiB Online Up 2.0.2.3-c186a87 3.10.0-957.5.1.el7.x86_64 CentOS Linux 7 (Core)
10.10.27.119 35ab6d0c-3e3a-4e50-ab87-ba2e678384a9 icp-worker2 Yes 6.0 GiB 50 GiB Online Up 2.0.2.3-c186a87 3.10.0-957.5.1.el7.x86_64 CentOS Linux 7 (Core)
Global Storage Pool
Total Used : 18 GiB
Total Capacity : 150 GiB
5. The Portworx chart does not automatically create a storage class. To enable dynamic storage provisioning, create a portworx-sc.yaml file containing the lines in Example 4-34.
Example 4-34 Portworx storage class definition
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
name: portworx-sc
provisioner: kubernetes.io/portworx-volume
parameters:
repl: "1"
Apply the definition with command:
kubectl create -f portworx-sc.yaml
There are many other parameters that you can define in the Portworx storage class:
 
Attention: The procedure described in this section will not work in the air-gapped environments as Portworx pods downloads the content and activate Trial license on the Portworx site. For air-gapped installation see Portworx manuals at https://docs.portworx.com/portworx-install-with-kubernetes/on-premise/airgapped.
Your portworx storage cluster should be ready with active Trial license for 30 days. In case you want to purchase a production license visit Portworx website.
4.3.6 Configuring Minio in IBM Cloud Private
Minio is the special kind of persistent storage available in IBM Cloud Private. It does not provide Kubernetes persistent volumes as NFS, GlusterFS or Rook Ceph as described above, but it exposes existing persistent storage for the applications using a S3-compatible object storage interface. Consider using Minio if any of the applications that will run in your IBM Cloud Private cluster require object storage.
Because the IBM Knowledge Center provides detailed step-by-step instructions on deploying Minio Helm chart we will not replicate this in this book. See https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/configure_minio.html.
4.4 Managing the storage hosted on IBM Cloud Private
In this section, we briefly describe the features provided by IBM Cloud Private to help users with managing the second day operations on the storage used for persistent volumes.
4.4.1 Monitoring storage status and performance
Monitoring of the storage performance for storage like vSphere datastores, NFS or any external storage clusters is outside of the scope of this book. Usually these storage technologies have dedicated monitoring tools available to the storage teams. In this section we focus on the monitoring of the storage solution hosted within IBM Cloud Private.
In order to help users manage their storage hosted on an IBM Cloud Private cluster, both GlusterFS as well as Rook Ceph solutions provided by IBM have the Pometheus endpoints enabled. This means that they automatically collect performance metrics related to storage in the platform monitoring service.
1. To see the dashboards that are provided out-of-the-box open the IBM Cloud Private Console and from the lefthand side menu. Select Platform and Monitoring as shown in Figure 4-2. This will open the default Grafana interface.
Figure 4-2 Opening the IBM Cloud Private monitoring UI
2. On the default Grafana dashboard click the Home dropdown in the upper left corner, as shown in Figure 4-3.
Figure 4-3 Opening the list of available dashboards
3. This opens the list of dashboards that are loaded in your environment. IBM Cloud Private V 3.1.2 provides out-of-the-box dashboards for storage solutions as shown in Figure 4-4.
Figure 4-4 List of predefined Grafana dashboards available in IBM Cloud Private
4. Selecting any item on the list will open the dashboard. In Figure 4-5 we show a sample Rook Ceph dashboard.
Figure 4-5 Sample Rook Ceph dashboard
If you do not see any data, make sure that the user account that you have used for authenticating to the IBM Cloud Private Console has the access rights to the namespace hosting the Rook Ceph Helm chart.
4.4.2 Extending the available storage
One of the common tasks related to the storage is extending available capacity. For solutions hosted on the IBM Cloud Private cluster described in this book, it can be achieved by adding more raw disks to the distributed filesystem clusters. New devices can be mounted either on the worker nodes that are already used as storage providers or on new nodes - extending not only the storage space, but also the I/O performance.
IBM Knowledge Center provides detailed step-by-step instructions on configuring additional storage space for GlusterFS. See https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.2/manage_cluster/increase_vol.html.
4.5 Performance considerations
During writing this chapter we set up an IBM Cloud Private environment with different types of persistent storages:
•NFS
•GlusterFS
•Ceph
•VMware vSphere
•Portworx
All of the persistent volumes used for tests were hosted on datastore residing on local drives inside the ESXi server.
We have run series of performance benchmarks using dbnech and pgbench tools. All the volumes used were hosted on SSD drives inside a ESXi server used for the test.
Below we present the results of these tests.
 
Note: All of the storage providers were used with the default settings as of IBM Cloud Private V 3.1.2. It is very likely that some tuning could improve the results, especially for the distributed filesystems.
4.5.1 Performance test using dbench
The first test we performed using a dbench image is available here: https://github.com/logdna/dbench which makes use of the Flexible I/O tester (FIO) tool (https://github.com/axboe/fio).
Table 4-1 shows the results averaged over several runs of the test jobs.
 
Attention: The results are provided on a as-is basis using the output that was produced by the FIO tool. Note that NFS uses the client caching feature, which allows the data to be cached on the NFS client and read out of local memory instead of remote disk. This might affect the results for NFS. Also, Portworx applies some I/O optimization techniques which might affect the results.
Table 4-1 Performance test using dbench
Provider
Random Read
[IOPS]
Random Write
[IOPS]
Random Read
[MB/s]
Random Write
[MB/s]
Vmware
2 184
2 021
121
135
Gluster
1 023
465
55
23
Ceph RBD
855
438
47
77
NFS
16 300
1 664
660
73
Portworx
126 000
31 500
4 277
279
4.5.2 PostgreSQL database performance
The raw IOPS (Input/Output Operations Per Second) test may not be relevant for a real application workload, so in our econd test we used a standard PostgreSQL performance test pgbench. For each of the storage classes in our test environment we installed one instance of the ibm-postgres-dev Helm chart (version 1.1.3) using default settings. Each PostgreSQL deployment was exposed using the NodePort. From an external system we ran the following commands:
pgbench -i -d -s 64 -h $PGHOST -p $PGPORT -U admin postgres
This command creates the sample database of 1 GB size. After that we ran the test command:
pgbench -c 10 -j 2 -t 10000 -h $PGHOST -p $PGPORT -U admin postgres
The command runs 10 clients in parallel with 2 threads each, where each client executes 10000 transactions.
Neither the database nor the storage were not optimized in any way as the goal of this test was to just show the relative performance of different storage options using the same environment.
The results returned by the benchmark are shown in Table 4-2.
 
Table 4-2 Results of pgbench benchmark
Provider
Transactions per Second
Vmware
1898
Gluster
103
Ceph RBD
225
NFS
1283
Portworx
676
The results are in general consistent with IOPS test, showing significant performance advantage of Vmware over the other providers. This is not very surprising in our test setup. However, it is worth noticing that Ceph RBD performed over 2 times better than GlusterFS (while both storage providers reported similar IOPS performances). From the distributed storage providers, Portworx showed almost 6 times better performance than GlusterFS and 3 times better than Ceph RBD.
Additionally, on GlusterFS the benchmark returned few errors such as:
ERROR: unexpected data beyond EOF in block 105200 of relation base/16384/16503
HINT: This has been seen to occur with buggy kernels; consider updating your system.
A brief search on the Internet returns the hint from GlusterFS documentation: “Gluster does not support so called ‘structured data’, meaning live, SQL databases.”
This might change with the newer versions of GlusterFS.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.216.27.251