By now, we need the full stack and probably a fully active database server as well. Follow all the recipes up to Tweaking XFS performance before starting here.

For this recipe, we will need two PostgreSQL servers, pg1 and pg2, where pg1 is the currently active node. Follow these steps as the root user on the system indicated to move an active PostgreSQL service from one node to another:

There is actually very little in this recipe that we have not done in this chapter. What we have actually done here is formalized the steps necessary to tear down and build up an active stack. We start the process by stopping the PostgreSQL service with pg_ctl, as we clearly can't move the data while it's still in use.

Next, we use umount to decouple the /dev/VG_POSTGRES/LV_DATA device from the /db directory. With no locks on the storage volume, we can use vgchange with the -a parameter set to n to deactivate any volume in the VG_POSTGRES group. Since the VG_POSTGRES group actually resides on the DRBD device, it can only be active on one node at a time.

Once the volumes are no longer active, we can set the DRBD status to secondary with drbdadm. After we perform this step, the /dev/VG_POSTGRES directory and any corresponding device will actually disappear. This is because a DRBD device in secondary status is only active within DRBD. Here is what DRBD shows us in /proc/drbd regarding the situation:

DRBD sees the device as Secondary on both nodes; currently, neither node can access our PostgreSQL data. From this point, we merely reverse the process to reactivate all of these resources on pg2 instead.

We begin reactivating PostgreSQL by promoting the storage to primary status with drbdadm on the pg2 node. This causes the requisite VG_POSTGRES volume group to appear on pg2, making it a candidate for activation with vgchange.

Now we simply reuse the mount command that we discussed in the Tweaking XFS performance recipe on the pg2 node, making the data available to us once again. If we start PostgreSQL with the pg_ctl control script, our database will begin running as if it were still on the pg1 node. PostgreSQL does not know anything has changed.

Since data can switch nodes arbitrarily as demonstrated here, upgrades and maintenance to server hardware are much easier. What can we do with the extra node? We can reboot it, apply firmware or kernel updates, apply security patches, or even update the database software to a bug-fix release.

Following any required or suggested changes to the secondary node, we merely promote it to run PostgreSQL in place of the current server. Then, we can repeat modifications on the other node. With this, we can limit outages to a matter of seconds while still providing high uptime guarantees, all without skipping system maintenance.

In fact, this process is so standardized that we will be exploring it in great detail in the next chapter. Once this tear-down and build-up procedure is automated, maintaining or replacing servers is even easier.