We’ve already presented various pool layout performance issues, so now it’s time to consider rules of thumb for the given redundancy types.
Note
We won’t be covering striped pools. A striped pool is a pool consisting of two or more disks that provide no redundancy. While the total pool capacity equals the combined capacity of all the disks in the pool, the file system will become corrupted and subject to data recovery if you lose a single drive. A rule of thumb for storage is: don’t use striped pools.
General Considerations
For mirrored pools, a good rule of thumb is to use them only when you really need an incredible read performance or are paranoid about your storage. Disks don’t fail that often and a mirrored pool will halve your total pool capacity . With triple mirrors, your capacity will be the total disk’s capacity divided by three, and so on. A rule of thumb is to use them sparingly and with care.
For RAIDZ (which is a rough equivalent of RAID-5 and RAID-6), go rather for RAIDZ-2 . It gives you quite good resilience while conserving a lot of space. There is also another recommendation and from personal experience I’d adhere to it: for RAIDZ pools, have 2n+1 disks per vdev. That’s three, five, seven, etc., but no more than eleven. This is 2n data disks plus 1 disk for parity data.
With the smallest set of three disks per vdev, you have basically a capacity of a mirrored set with lower read performance. Consider starting with five disks per vdev. For RAIDZ-2, the rule is to use 2x+2 disks, which translates to four, six, eight, etc., and have no more than 12 disks within a vdev. Given this guide, have a typical target maximum of 20 disks in the pool (including ZIL and L2ARC). It’s a good idea to have two eight disks RAIDZ-2 vdevs in the pool, totaling 16 disks of total pool capacity of 12 disks.
Creating a Mirrored Pool
Since I’ve shown you how to create simple pools in previous chapters, there is no need to demonstrate this now. I am therefore going to jump straight to more involved configurations. Bear in mind, however, that with a single node, the setup options are limited.
As a reminder, we are not going to cover striped pools at all. Your pool will have absolutely no resiliency in such a setup and you should never consider hosting data you care for using such a configuration.
Before running any command that may endanger your data, especially in production, i.e., zpool create or zpool destroy, confirm that the disks you want to use are those that you intended to be used by ZFS.
We have already covered a simple mirrored pool, so let’s create bigger one consisting of 10 disks. I am going to follow with zpool status to print the resulting pool configuration :
trochej@ubuntuzfs:~$ sudo zpool create -f datapool mirror /dev/sdb /dev/sdcmirror /dev/sdd /dev/sdemirror /dev/sdf /dev/sdgmirror /dev/sdh /dev/sdimirror /dev/sdj /dev/sdktrochej@ubuntuzfs:~$ sudo zpool statuspool: datapoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMdatapool ONLINE 0 0 0mirror-0 ONLINE 0 0 0sdb ONLINE 0 0 0sdc ONLINE 0 0 0mirror-1 ONLINE 0 0 0sdd ONLINE 0 0 0sde ONLINE 0 0 0mirror-2 ONLINE 0 0 0sdf ONLINE 0 0 0sdg ONLINE 0 0 0mirror-3 ONLINE 0 0 0sdh ONLINE 0 0 0sdi ONLINE 0 0 0mirror-4 ONLINE 0 0 0sdj ONLINE 0 0 0sdk ONLINE 0 0 0errors: No known data errors
The resulting pool total capacity equals half the capacity of all the disks in the pool:
trochej@ubuntuzfs:~$ sudo zpool listNAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTdatapool 9.92G 64K 9.92G - 0% 0% 1.00x ONLINE -
The pool is mounted at /datapool and contains a file system called datapool, as you can see in the following output:
trochej@ubuntuzfs:~$ sudo zfs listNAME USED AVAIL REFER MOUNTPOINTdatapool 58K 9.77G 19K /datapool
Creating a RAIDZ Pool
I am reusing the same disks in all the examples. Before creating a new pool on them, I am going to run zpool destroy on the pool. It does exactly that: it marks a pool as destroyed and disks as free to be used by other ZFS setups. When ZFS adds a disk to the pool, it labels it with its own GUID and some information that allows ZFS to be self-contained. You may move the pool around, export it from the current server, reinstall the server to FreeBSD, and import the same pool without a problem. Thus, if you decide you no longer need the pool and try to reuse disks for other configuration, zpool will refuse to add it to a new one without using the -f switch.
trochej@ubuntuzfs:~$ sudo zpool destroy datapool[sudo] password for trochej:
The virtual machine I am working with has 12 disks for use as storage :
trochej@ubuntuzfs:~$ ls -ahl /dev/sd[a-z]brw-rw---- 1 root disk 8, 0 Feb 12 21:59 /dev/sdabrw-rw---- 1 root disk 8, 16 Feb 15 17:43 /dev/sdbbrw-rw---- 1 root disk 8, 32 Feb 15 17:43 /dev/sdcbrw-rw---- 1 root disk 8, 48 Feb 15 17:43 /dev/sddbrw-rw---- 1 root disk 8, 64 Feb 15 17:43 /dev/sdebrw-rw---- 1 root disk 8, 80 Feb 15 17:43 /dev/sdfbrw-rw---- 1 root disk 8, 96 Feb 15 17:43 /dev/sdgbrw-rw---- 1 root disk 8, 112 Feb 15 17:43 /dev/sdhbrw-rw---- 1 root disk 8, 128 Feb 15 17:43 /dev/sdibrw-rw---- 1 root disk 8, 144 Feb 15 17:43 /dev/sdjbrw-rw---- 1 root disk 8, 160 Feb 15 17:43 /dev/sdkbrw-rw---- 1 root disk 8, 176 Feb 12 21:59 /dev/sdlbrw-rw---- 1 root disk 8, 192 Feb 12 21:59 /dev/sdm
/dev/sda is a system disk, which leaves us with disks from /dev/sdb to /dev/sdm. It means 12 disks for use as storage. Let’s create a RAIDZ pool following the previously noted best practice of five disks per vdev:
trochej@ubuntuzfs:~$ sudo zpool create datapoolraidz /dev/sdb /dev/sdc/dev/sdd /dev/sde /dev/sdfraidz /dev/sdg /dev/sdh/dev/sdi /dev/sdj /dev/sdktrochej@ubuntuzfs:~$ sudo zpool statuspool: datapoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMdatapool ONLINE 0 0 0raidz1-0 ONLINE 0 0 0sdb ONLINE 0 0 0sdc ONLINE 0 0 0sdd ONLINE 0 0 0sde ONLINE 0 0 0sdf ONLINE 0 0 0raidz1-1 ONLINE 0 0 0sdg ONLINE 0 0 0sdh ONLINE 0 0 0sdi ONLINE 0 0 0sdj ONLINE 0 0 0sdk ONLINE 0 0 0errors: No known data errorstrochej@ubuntuzfs:~$ sudo zpool listNAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTdatapool 19.8G 106K 19.7G - 0% 0% 1.00x ONLINE -
The setup shown here can withstand losing a single disk per each vdev at once. With two disks unused, you can add so-called hot spares. Hot spares are idle disks added to a pool for replacement in case any active disk in the pool fails. The replacement is done automatically by ZFS. The hot spare mechanism isn’t intelligent, so it can cause resiliency issues if you care for the physical layout of your pool—spread your pool’s disks in different JBODs so that you can the lose the whole chassis and still retain the pool and data.
In a simple single server setup, this problem isn’t significant. You should be safe adding the spare disk to a pool. I’ll demonstrate this process in Chapter 5.
Creating a RAIDZ2 Pool
Let’s now walk through creating a RAIDZ2 pool, which will consist of 12 disks spread evenly between two vdevs:
trochej@ubuntuzfs:~$ sudo zpool create -f datapoolraidz2 /dev/sdb /dev/sdc /dev/sdd/dev/sde /dev/sdf /dev/sdgraidz2 /dev/sdh /dev/sdi /dev/sdj/dev/sdk /dev/sdl /dev/sdmtrochej@ubuntuzfs:~$ sudo zpool statuspool: datapoolstate: ONLINEscan: none requestedconfig:NAME STATE READ WRITE CKSUMdatapool ONLINE 0 0 0raidz2-0 ONLINE 0 0 0sdb ONLINE 0 0 0sdc ONLINE 0 0 0sdd ONLINE 0 0 0sde ONLINE 0 0 0sdf ONLINE 0 0 0sdg ONLINE 0 0 0raidz2-1 ONLINE 0 0 0sdh ONLINE 0 0 0sdi ONLINE 0 0 0sdj ONLINE 0 0 0sdk ONLINE 0 0 0sdl ONLINE 0 0 0sdm ONLINE 0 0 0errors: No known data errorstrochej@ubuntuzfs:~$ sudo zpool listNAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOTdatapool 23.8G 152K 23.7G - 0% 0% 1.00x ONLINE -
Forcing Operations
There are situations where you will want to conduct two operations with final consequences—such as destroying a pool or forcing an operation on a pool, i.e., a create operation. You may see lots of this especially in the first stages, when you are learning the ZFS administration.
The best practice is to destroy a pool before reusing its components, but there are situations when you may end up with a bunch of healthy disks that someone else disposed of. They may contain disks previously in a ZFS pool, but not enough of them to import it and destroy it properly.
For such occasions, there is the -f switch, meaning force.
Train and test
Remember that creating a pool is largely one way road. You can’t remove drives from it and once you decide on redundancy level, you must add new disks in the same configuration. Play with zpool and zfs commands in virtual machines. It’s a low cost way of getting familiar with ZFS. Get familiar with tools that help you monitor drives: smartctl, ZED, sysstat.