First, you need to identify what drives are in your system and determine which should be formatted and which are system drives that shouldn’t be formatted.

From the command line of the machine, run:

df

It will return something similar to:

Filesystem      1K-blocks       Used    Available       Use%    Mounted on
/dev/sda1       27689156        8068144 18214448        31%     /
none    1536448 672     1535776 1%      /dev
none    1543072 1552    1541520 1%      dev/shm

none    1543072 92      1542980 1%      /var/run
none    1543072 0       1543072 0%      /var/lock

This should list all mounted filesystems. Just note which drive is used for booting so you don’t accidentally format it later; very often it is the sda device mounted on the root / directory. But you can can verify that by poking around in /proc/mounts and finding which directory is the rootfs:

cat /proc/mounts

It will return something similar to:

rootfs / rootfs rw 0 0
none /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
none /proc proc rw,nosuid,nodev,noexec,relatime 0 0
none /dev devtmpfs
rw,relatime,size=1536448k,nr_inodes=211534,mode=755 0 0
none /dev/pts devpts
rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/disk/by-uuid/c9402f9d-30ed-41f2-8255-d32bdb7fb7c2 / ext4 rw,relatime 0 0
none /sys/kernel/debug debugfs rw,relatime 0 0
none /sys/kernel/security securityfs rw,relatime 0 0
none /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
none /var/run tmpfs rw,nosuid,relatime,mode=755 0 0
none /var/lock tmpfs rw,nosuid,nodev,noexec,relatime 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc
rw,nosuid,nodev,noexec,relatime 0 0
gvfs-fuse-daemon /home/joe/.gvfs fuse.gvfs-fuse-daemon
rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0

The output tells you that the root filesystem is /, which you know from the df command is mounted on /dev/sda1. That is where the system will boot.

To find out what other block devices the operating system knows about, look in the /sys/block directory. This should list all the block devices known to the system:

ls /sys/block

The command will return something similar to:

loop0 loop3 loop6 ram1  ram12 ram15 ram3 ram6 sda sdd sdg sdj sdm
loop1 loop4 loop7 ram10 ram13 ram19 ram4 ram7 sdb sde sdh sdk
loop2 loop5 ram0 ram11 ram14 ram2 ram5 ram8 sdc sdf sdi sdl

The devices you’re interested in are the ones starting with sd in this output, but other systems might have different prefixes. Also review which filesystems are formatted by running the following, which you might need to run as root:

blkid

It will return something similar to:

/dev/sda1: UUID=”c9402f9d-30ed-41f2-8255-d32bdb7fb7c2” TYPE=”ext4”
/dev/sda5: UUID=”b2c9d42b-e7ae-4987-8e12-8743ced6bd5e” TYPE=”swap”

The TYPE value will tell you which filesystem is on the device (e.g., xfs or ext4). Naturally, you should not use ones that have swap as their type.

It’s important to keep track of which devices need to be mounted to which system location. This can be done with mounting by labels in a separate upstart script created for Swift. Labeling the drive will ensure the drive gets that name every time the OS reboots. If you don’t do this, the next time the OS boots, a device with your data might come up under a different device name. This would lead to all sorts of unexpected behaviors when the cluster reboots.

We suggest not using the fstab file for this purpose. /etc/fstab is used to automatically mount filesystems at boot time. Make sure that the only items listed there are critical for the booting of the machine, because the machine might hang upon boot if it can’t mount a device. This is likely when running a big box full of disks.

Start building up a list of devices that the system knows about that need to be formatted. Identify all the devices in /sys/bock that are not formatted (or not formatted in the manner specified here).

You should label each device so you can keep an inventory of each device in the system and properly mount the device. To add a label, use the -L option to the mkfs.xfs command. For instance, the following assigns the label d1 to the device /dev/sdb and the label d2 to the device /dev/sdc:

mkfs.xfs -f -L d1 /dev/sdb
mkfs.xfs -f -L d2 /dev/sdc

Go in this manner through all your drives that will be formatted.

Swift uses extended attributes, and the XFS default inode size (currently 256) is recommended.

Notice that we didn’t create device partitions. Partitioning is an unnecessary abstraction for Swift because you’re going to use the entire device.

The next step is to tell the operating system to attach (mount) the new XFS filesystems somewhere on the devices so that Swift can find them and start putting data on them.

Create a directory in /srv/node/ for each device as a place to mount the filesystem. We suggest a simple approach to naming the directories, such as using each device’s label (d1, d2, etc):

mkdir -p /srv/node/d1
mkdir -p /srv/node/d2

For each device (dev) you wish to mount, you will use the mount command:

// _mount -t xfs -o noatime,nodiratime,logbufs=8 -L_ <dev label> <dev dir>

Run the command for each device:

mount -t xfs -o noatime,nodiratime,logbufs=8 -L d1 /srv/node/d1
mount -t xfs -o noatime,nodiratime,logbufs=8 -L d2 /srv/node/d2
...

After the drives are mounted, a user needs to be created with read/write permissions to the directories where the devices have been mounted. The default user that Swift uses is named swift. Different operating systems have different commands to create users, looking something like this:

useradd swift

Next, give the swift user ownership to the directories:

chown -R swift:swift /srv/node

The mount commands we ran earlier take effect only during the current boot. Because we need them to run automatically on reboot, we’ll put the commands in a script called mount_devices that contains the mount commands we just ran. This new script should be created in the /opt/swift/bin/ directory:

mount -t xfs -o noatime,nodiratime,logbufs=8 -L d1 /srv/node/d1
mount -t xfs -o noatime,nodiratime,logbufs=8 -L d2 /srv/node/d2
...
mount -t xfs -o noatime,nodiratime,logbufs=8 -L d36 /srv/node/d36

Next, create an Upstart script in the /etc/init directory called start_swift.conf with the following commands:

description "mount swift drives"
start on runlevel [234]
stop on runlevel [0156]
exec /opt/swift/bin/mount_devices

There are many good resources online that can explain more about Upstart scripts and runlevels.

Be sure the script is executable:

chmod +x /opt/swift/bin/mount_devices

Test the scripts by rebooting the system:

reboot

When the system restarts, log on and make sure everything mounted properly with the configuration settings by running the mount command:

mount

You should see something similar to:

/dev/sda1 on / type ext4 (rw,errors=remount-ro, commit=0)
...
...
/dev/sdc on /srv/node/d1 type xfs
(rw,noatime,nodiratime,logbufs=8)

Each user-defined storage policy is declared with a policy index number and name. If a policy is deprecated or the default, there are additional fields that can be used. The general format for a storage policy is:

[storage-policy:N]
name =
default =
(optional) deprecated=

//Here's an example
[storage-policy:1]
name =  level1
(optional) deprecated=yes

The storage policy index, [storage-policy:N], is required. The N is the storage policy index whose value is either 0 or a positive integer. It can’t be a number already in use, and once in use it can’t be changed.

The policy name (name =) is required and can only be letters (case-insensitive), numbers, and dashes. It can’t be a name already in use, but the name can be changed. Before such a change, however, consider how it will affect systems using the current name.

One storage policy must be designated as the default. When defining one or more storage policies, be sure to also declare a policy 0. This makes it easier to manage the policies and prevents errors. As mentioned earlier, if there are no user-defined polices, then the default will be an internally referenced storage policy 0. A different policy can be designated as the default at any time; however, consider the impact it will have on the system before making the change.

When storage policies are no longer needed, they can be marked as deprecated with deprecated=yes rather than being deleted. A policy can’t be both the default and deprecated. When deprecated, a policy:

Although storage policies are not necessary for this installation, we have included some to provide examples of how they are added and used.

To declare a storage policy, edit the swift.conf file in the /etc/swift/ directory. The suggested placement in the file for storage polices is below the [swift-hash] section:

[swift-hash]
# random unique strings that can never change (DO NOT LOSE)
swift_hash_path_prefix = changeme
swift_hash_path_suffix = changeme

[storage-policy:0]
name = level1

[storage-policy:1]
name = level2
default = yes

[storage-policy:2]
name = level3
deprecated = yes

Once the storage policies are declared, they can be used to create builder files. For each storage policy (storage-policy:N), an object-N.builder file will be created.

Think of a builder file as a big database. It contains a record of all the storage devices in the cluster and the values the ring-builder utility will use to create a ring.

The help menu for swift-ring-builder shows the format for the create command as:

swift-ring-builder (account|container|object|object-n).builder create
<part_power> <replicas> <min_part_hours>

So the commands we will be running are:

swift-ring-builder account.builder create <part_power> <replicas>
<min_part_hours>
swift-ring-builder container.builder create <part_power> <replicas>
<min_part_hours>
swift-ring-builder object.builder create <part_power> <replicas>
<min_part_hours>
swift-ring-builder object-n.builder create <part_power> <replicas>
<min_part_hours>

The three parameters the create command takes are:

These are critical parameters for a cluster, so let’s dig a bit more deeply into each of these configuration settings before using them.

The total number of partitions that can exist in your cluster is calculated using the partition power, an integer picked during cluster creation.

The formula to calculate the total number of partitions in a cluster is:

For example, if you choose a partition power of 15, the number of partitions your cluster can support is 2¹⁵ = 32,768. Those 32,768 partitions are then mapped to the available drives. Although the number of drives in a cluster might change, the number of partitions will not.

If your first thought is to choose a very large number, your instinct would be rewarded. Rings are relatively small compared to memory available on modern systems, so it is best to err on the large side. However, you should keep in mind several things if you choose a large number of partitions. Replication might take longer because fewer files will be grouped together in each partition for replication checks. Also, lookups take a fraction longer because the ring gets that much bigger. Lastly, a larger number of partitions means it takes longer to build the ring when making future changes.

To choose the partition power, estimate the maximum number of drives you expect your cluster to have at its largest size. (It might help to have a crystal ball handy.) In general, it is considered good to have about 100 partitions per drive in large clusters.

If you can make that prediction, you can calculate the partition power as follows:

Let’s look at an example for a large deployment of 1,800 drives that will grow to a total of 18,000 drives. If we set the maximum number of disks to 18,000, we can calculate a partition power that would allow for 100 partitions per disks:

So we judge that a good partition power for this example is 21.

Even if you guess wrong, there is still recourse—it’s always possible to stand up another object storage cluster and have the authentication system route new accounts to that new storage cluster.

The more replicas you have, the more protected you are against losing data when there is a failure. Distributing replicas across data centers and geographic regions protects you even more against hardware failures. The replica count affects the system’s durability, availability, and the amount of disk space used. For example, if you have a replica count of three, then each partition will be replicated for a total of three copies. Each of the three replicas will be stored on a different device in the cluster for redundancy. When an object is put into the cluster, its hash will match a partition and be copied to all three locations of that partition.

You set the replica count for the cluster in each of the builder files using the replicas parameter. When you build a Swift ring, the value of replicas is how many copies of the partition will be placed across the cluster. It is set as part of the ring-building process to any real number, most often 3.0. In the less-common instance where a non-integer replica count is used, a percentage of the partitions will have one additional replica; for instance, a 3.15 replica count means that 15% of the drives will have one extra replica for a total of 4. The main reason for using a fractional replica count would be to make a gradual shift from one integer replica count to another, allowing Swift time to copy the extra data without saturating the network.

Part of partition replication includes designating handoff drives. This means that when a drive fails, the replication/auditing processes will notice and push the data to handoff locations. This dramatically reduces the MTTR (mean time to repair) compared to standard three-way mirroring or RAID. The probability that all replicated partitions across the system will become corrupt (or otherwise fail) before the cluster notices and is able to push the data to handoff locations is very small, which is a main reason why we say that Swift is durable. Depending on the durability guarantees you require and the failure analysis of your storage devices, you can set the replica count appropriately to meet your needs.

While a partition is being moved, Swift will lock that partition’s replicas for a period of time to prevent several from being moved simultaneously. Moving multiple replicas at the same time could risk making data inaccessible.

The minimum part hours, configured through the min part hours setting, ensures that only one partition’s replica is moved at a time. It is best to set the hours to something greater than what it takes for the system to do a full partition replication cycle. A good default setting is 24 hours.

To tune the setting, look at the logs of each replicator on each object store. The log will say how long the replication cycle took. Set the minimum part hours to be comfortably out of reach of the longest replication time.

If the cluster experiences an unusual number of drive failures, or if you are making ring changes, it might be worth increasing the minimum part hours setting temporarily because replication times will be increasing. If they increase outside of the minimum part hours setting, you’ll have an increased risk of unavailable data. This setting can be dynamically tuned at cluster runtime to optimize the rate at which ring changes can be made.

Now we are ready to create the builder files.

The commands we will be running are:

swift-ring-builder account.builder create <part_power> <replicas> <min_part_hours>
swift-ring-builder container.builder create <part_power> <replicas> <min_part_hours>
swift-ring-builder object.builder create <part_power> <replicas> <min_part_hours>
swift-ring-builder object-n.builder create <part_power> <replicas> <min_part_hours>

Now that we understand more about the values needed for part_power, replicas, and min_part_hours, please select your values and run the create command. Here for account, container, and object we use a part_power value of 17, replicas value of 3, and min_part_hours value of 1.

For our storage policies, these values may be different. For this example, we will have storage policy 1 provide increased durability by setting its replica count to four. So we would use a part_power value of 17, replicas value of 4, and min_part_hours value of 1.

For storage policy 2, which we designated as deprecated, we will use the default settings. In general, a storage policy will become deprecated after having been in use by the cluster, so there would be an existing ring. Since that is not the case here, we need to be sure one is created for storage policy 2, as later in this installation the proxy server process will try to load it. The proxy server process loads deprecated policies because they are still able to be accessed.

cd /etc/swift
swift-ring-builder account.builder create 17 3 1
swift-ring-builder container.builder create 17 3 1
swift-ring-builder object.builder create 17 3 1
swift-ring-builder object-1.builder create 17 4 1
swift-ring-builder object-2.builder create 17 3 1

You’ll now see account.builder, container.builder, object.builder, object-1.builder, and object-2.builder in the directory. There should also be a backup directory, appropriately named backups.

To take some mystery out of what was just created, let’s open an interactive Python session and show what is in an object.builder file. The contents are a Python data structure that you can view using a pickle file. Some of the fields will be familiar.

$ python
>>>
>>> import pickle
>>> print pickle.load(open('object.builder'))
{
  '_replica2part2dev': None,
  '_last_part_gather_start': 0,
  'in_part_hours': 24,
  'replicas': 3.0,
  'parts': 262144,
  'part_power': 18,
  'devs': [],
  'devs_changed': False,
  'version': 0,
  '_last_part_moves_epoch': None,
  '_last_part_moves': None,
  '_remove_devs': []
}

Swift allows a physically distinct part of the cluster to be defined as a region. Regions are often defined by geographical boundaries; for example, several racks of servers (nodes) can be placed in higher-latency, off-site locations. When a multi-region cluster receives a read request, Swift will favor copies of the data that are closer, as measured by latency. As explained in Swift Architecture, you can configure how replicas get distributed throughout a multi-region cluster using write affinity.

There are no restrictions on the value you choose for the name of your region label. We will use r1 in this chapter to indicate region 1.

Within regions, Swift allows you to create availability zones to isolate failures. The goal of zones is to allow the cluster to tolerate significant outages of large hunks of storage servers. An availability zone should be defined by a distinct set of physical hardware whose failure will be isolated from the rest of the cluster.

In a large deployment, availability zones might be defined as separate facilities in a large data center campus. In a single data center deployment, the availability zones might be different rooms. For a handful of racks in a data center, each rack could represent a zone. Typically, deployments create isolated failure boundaries within a data center by segmenting the power source, the network that is serving each zone, or both.

There are no restrictions on the value you choose for the name of your zone label. We will use z1 in this chapter to indicate zone 1.

You need to specify the IP address of the node for each device. This should be on the network segment designated for internal communication that will allow the nodes in the cluster to communicate with each other via rsync and HTTP.

Each of the account, container, and object server processes can run on different ports.

By default:

Each device in the system has a weight. As described in Distribution of Data, the weight is a number that determines the relative number of partitions a device (drive) will have. It is set as part of the ring-building process. Because this is a relative number, it’s up to the operator to configure the weights that will be used to represent each drive.

For example, if when starting out each drive is the same size, say 2 TB, you could give each drive a weight of 100.0, so that they all have the same likelihood of receiving a replica of a partition. Then, when it’s time to add capacity and 3 TB drives are available, the new 3 TB drives would receive a weight of 150.0 because they are 50% larger. With this weight, each 3 TB drive would recieve 1.5 times as many partition replicas as one of the 2 TB drives.

Another, often simpler, strategy is to use the drive’s capacity as a multiplier. For example, the weight of a 2 TB drive would be 200.0 and a 3 TB drive would be 300.0.