Even though Zabbix is a mature product with more than 15 years behind it, it is still very actively developed. Bugs are fixed and new features are added. At some point, accumulated improvements make it worth upgrading. In this section, we will look at the following:
- General version policy: Which versions are stable and which ones are supported for longer periods of time
- The upgrade process: What can be upgraded to what and how it should be done
- Compatibility between Zabbix components: Which versions of the server can be used with which versions of the agent and so on
The Zabbix versioning scheme has changed a few times over the years. In general, the first two numbers have denoted a major version, such as 2.4 and 3.0, while the third number has denoted a minor version number. Previously, an even second number denoted a stable branch, while an odd second number denoted a development branch. Thus, 2.3 was a development branch for 2.4, while 2.4 was the resulting stable branch. This has slightly changed with 3.0. The development releases have moved away from the odd numbering, that is, the 2.5 number. They are now called 3.0.0alpha1, 3.0.0beta2, and so on. This is deemed to be more user friendly, although the internal numbering is still based on 2.5 in several places—the database version, for example, which we will explore in more detail a bit later.
The new version numbering since Zabbix 3.0 could be summed up as follows:
- A version number with just digits (and dots) in it denotes a stable release
- A version number with the
alpha,beta, orrc(release candidate) keywords added is not a stable release
For stable branches, there are even more differences. The release and support policy has changed as well, and the current policy states that there are two types of stable branches:
Historically, the 2.2 branch was designated as an LTS branch, with plans to release 2.4 and 2.6 as short-term support branches. Plans tend to change, and the 2.6 branch was canceled. 3.0 is the current LTS branch, with 3.2 and 3.4 planned as short term support branches, 4.0 following as the next LTS branch, and all further LTS branches aligning to N.0 versioning. Will this hold? That is very hard to predict, so you might want to check the current policy at http://www.zabbix.com/life_cycle_and_release_policy.php.
Tip
This support mostly references commercial services, although it strongly affects all users. We will discuss support options in Appendix B, Being Part of the Community.
How to decide which branch to use? Consider the available features and how quickly you would be able to upgrade. Does the latest LTS version satisfy you and you don't plan to upgrade for years? Stick with it. Really desire a feature in a non-LTS branch and plan to upgrade when the next stable branch comes out? Go with the non-LTS branch. Anything in-between, and you'll have to make a decision based on the support policy that's in effect at that time. Here's a quick lookup table to help you decide:
|
Use a non-LTS branch when… |
Use an LTS branch when... |
|---|---|
|
You need a new feature in the non-LTS branch |
The LTS-branch features satisfy you |
|
You plan to upgrade to every new version quickly |
You prefer to stay with one version as long as possible |
|
You can tolerate slight instability |
You prefer a more stable version |
Note that the slight instability mentioned in the table does not mean that there are serious issues with the non-LTS versions. In some cases, more stable might mean this bug is pretty stable, but has not been fixed for a long time.
Read the upgrade notes.
What was that? Yes, before performing any upgrades, take a little time, go to the Zabbix manual, and read the upgrade notes. If you are jumping over a few major versions, do read all the upgrade notes in between. Even if you have followed Zabbix development a bit, you might have missed some change that could cause problems—removed or added configuration parameters, memory requirement changes, API changes...upgrade notes should list all significant changes.
It is also highly suggested to read the pages on new features and improvements, called What's new. While it's much less risky to miss some of those changes, knowing about them could help you use Zabbix in a more efficient way.
Let's talk about the upgrade process itself now. This process and compatibility will differ depending on the version change you are performing:
- A minor version upgrade inside the same major version is simple and easy to undo
- A major version upgrade is more complicated and hard or impossible to undo
This is the simplest case. For example, upgrading from 3.0.0 to 3.0.1 or from 3.0.1 to 3.0.5 would be considered a minor version upgrade.
When performing a minor version upgrade, we may upgrade any combination of components—server, agents, proxies, Java gateway, and so on. While it is suggested to keep the main components of the same version to reduce confusion, a 3.0.0 server will happily work with a 3.0.1 frontend, 3.0.2 proxies, and 3.0.3 agents. Inside one major version, all components are compatible with each other.
It is also perfectly fine to skip minor versions when upgrading—as mentioned, going from 3.0.1 directly to 3.0.5 is perfectly fine.
While minor versions won't have upgrade notes often, do make sure to check for them. And read those What's new pages.
Zabbix Server, agents, and potentially proxy binaries have to be updated. The exact process will depend on how you installed them in the first place. Compiled from the source? Perform the same steps as during the installation. Installed from packages? Use the distribution package-management tools to perform the upgrade. This process should be fairly simple, and we discussed the details back in Chapter 1, Getting Started with Zabbix.
After starting the upgraded Zabbix server, in some rare cases, you might see a log entry like this:
10852:20151231:094918.820 starting automatic database upgrade
For a minor version upgrade, that could be a change to the database indexes to improve performance, but we will discuss that in more detail when we get to the major version upgrades.
Upgrading the Zabbix frontend from one minor to another minor version should be simple as well. If installed from the sources, copy over the new frontend files. Instead of overwriting the frontend, it might be a good idea to copy the frontend to a separate directory first, verify that it works as intended, and then move over your users.
For example, if your original installation had the Zabbix frontend in the relative path zabbix/, place the new frontend files in zabbix-<new_version>/, rename the zabbix/ directory to zabbix-<old_version>/, and create a symlink called zabbix that points at the new version so that you don't have to use a different URL whenever you upgrade. To skip the configuration wizard, copy over the configuration file:
# cp zabbix-<old_version>/conf/zabbix.conf.php zabbix/conf/
That should be enough. Now, you can refresh the Zabbix frontend in the browser and check the page footer—the new version number should be visible there.
This approach with keeping the old frontend versions is useful if a new version turns out to have a problem and you would like to test whether the old version also had the same problem—just load up a URL in your browser that points to the old frontend directory. If the problem indeed turns out to be a regression, simply change the symlink to point to the old directory, and revert to the old version.
You may keep and use multiple versions of the Zabbix frontend in parallel, as long as they all are of the same major version. While normally not needed, it can be very helpful when some debugging or comparison has to be performed.
A major-level upgrade is slightly different from a minor version upgrade. As a quick reminder, definitely go and read the upgrade notes—major versions will always have some. Remember about the What's new pages, too.
Back to the major version upgrade itself, the most significant differences from a minor version upgrade are as follows:
- Database schema changes
- Compatibility
- Reading the upgrade notes
Let' talk about database schema changes right now, let's discuss compatibility in detail a bit later, and let's always remember to read our upgrade notes.
While the Zabbix team works hard to keep minor version upgrades without database changes, for major releases, it's open season. Changes to the database schema and its contents are made to accommodate new features, improve performance, and increase flexibility. Users wouldn't appreciate it if they couldn't keep gathered data and created configuration in the name of new greatness, so with each new version, a database upgrade patch is provided. This may include adding new tables and columns, removing tables and columns, and changing the data layout.
Given that a major version upgrade changes the database, make sure you have a recent backup. While upgrades are extensively tested, it is not possible for the developers to test all scenarios. What has worked for a thousand people might break in some obscure way for you. Additionally, interrupting the upgrade process because of a hardware or electricity failure is likely to leave your database in a broken state. You have been warned, so get that backup ready.
You are strongly encouraged to test the upgrade on a test installation, preferably using a production dataset (maybe with trimmed history and trend data, if the available hardware does not permit testing with a full copy).
With a fresh backup created, we are ready to engage the major version upgrade. The database upgrade process significantly changed for Zabbix version 2.2. In older versions, we had to apply the database patch manually. If you happen to have an old Zabbix installation—old being pre-2.0—you will have to patch it up to the 2.0 database schema manually. For your reference, the database patches are located in the upgrades/dbpatches directory in the source tree, but if you really want to follow that path, make sure to consult with the Zabbix community via the channels discussed in Appendix B, Being Part of the Community.
For upgrading to Zabbix 2.0 or more recent versions, no manual patching is required. Starting up the new server will automatically upgrade the database schema. Note that this database upgrading happens without a confirmation. Be careful not to start a more recent server binary against an older database version if you do not intend to change the database. One last note regarding the upgrade notes: promise. While the latest Zabbix upgrades are really quick even in large installations, older versions sometimes upgraded historical data tables, and that took a long time—like, really, a long time. In some reported cases, it was days. If such a change is required in any of the future versions, it will be mentioned in the upgrade notes, and you'll be glad you read them.
With all this talk about the database version and schema changes, let's take a closer look at how version information is stored and how we can check the upgrade status. Examine the dbversion table in your Zabbix database:
mysql> select * from dbversion; +-----------+----------+ | mandatory | optional | +-----------+----------+ | 3000000 | 3000000 | +-----------+----------+ 1 row in set (0.00 sec)
This table is the way Zabbix components determine which version of the database schema they are dealing with. There are two numbers in there: the mandatory and optional version. The following rules are important regarding version numbers:
- Inside one major version, the mandatory version number is always the same
- If a more recent server is started, it upgrades the database to the latest mandatory and optional version
- The server and frontend can work with a database as long as its mandatory version matches their mandatory version exactly—the optional version does not affect compatibility
The mandatory version encodes things such as table changes, column changes, and otherwise significant changes that break compatibility. The optional version would usually denote an index change—something that is helpful but does not prohibit older versions from working with a more recent database.
When a new major version of Zabbix Server is started, it is possible to observe the current status and database upgrade progress in the server log file:
10852:20151209:094918.686 Starting Zabbix Server. Zabbix 3.0.0 (revision {ZABBIX_REVISION}).
10852:20151209:094918.729 ****** Enabled features ******
...
10852:20151209:094918.730 TLS support: NO
10852:20151209:094918.730 ******************************
10852:20151209:094918.730 using configuration file: /usr/local/etc/zabbix_server.conf
10852:20151209:094918.820 current database version (mandatory/optional): 3000000/ 3000000
10852:20151209:094918.820 required mandatory version: 3000000
10852:20151209:094918.820 starting automatic database upgrade
...
10852:20151209:094918.866 completed 20% of database upgrade
...
10852:20151209:094918.937 completed 100% of database upgrade
10852:20151209:094918.937 database upgrade fully completedNotice how it prints out the current mandatory and optional database versions we just examined in the database and the required mandatory version. If the mandatory or optional database version numbers are lower than the required version, the server will upgrade the database. If the database mandatory version is higher than the server version, the server will refuse to start up. During the database schema upgrade, no monitoring happens. Monitoring restarts once the database upgrade is complete.
What happens if you upgrade the fronted before upgrading and starting the server to take care of the database upgrade? You are likely to see a message like this in the frontend:
If you see such a message when upgrading, start the new server and ensure the database upgrade is successful. If that doesn't help, make sure you are not starting some older Zabbix server binary or pointing Zabbix server at a different database. If you see a message like that when not upgrading Zabbix, you likely have a quite significant misconfiguration. Such a situation should never happen during the normal operation of Zabbix or minor version upgrades. Note that the Zabbix frontend stores the major version it is compatible with in the defines.inc.php file in the ZABBIX_DB_VERSION constant.
The database upgrade process can be very quick, but in some cases, it can also take quite some time. It might be required to keep gathering data even during the Zabbix upgrade, but how can we achieve that if the monitoring does not resume until the upgrade is finished?
Remember the additional Zabbix process, the proxy? It was able to gather data and buffer it for sending to the Zabbix server, so no data was lost even if the server was down for some time, which sounds pretty much like our situation. If all your actual monitoring is already done by Zabbix proxies, you are already on the right track.
If you have items that are polled directly by server, you might want to set up a temporary proxy installation, maybe even on the same server, that would be running during the Zabbix server upgrade and removed later. To do this easily, use the mass update functionality in the Configuration | Hosts section in the frontend and set the Monitored by proxy option. Make sure the proxy can actually gather data by testing it first with a single host.
Tip
Setting up a temporary proxy installation will be notably harder if you are using active items. It would be required to reconfigure all Zabbix agents as they connect to the address, specified in the ServerActive parameter. On the other hand, active agents do buffer data for a short while themselves, so a quick server upgrade might not miss that data anyway.
The proxy method sounds great, but it is a bit more complicated than just upgrading the server. Officially, only the same major version is supported for server-proxy compatibility. This means that we should not use proxies of the previous version with our upgraded server. Proxies, if used with a MySQL or PostgreSQL backend, can upgrade their database as well. The suggested path for using proxies to continue data collection through the major version upgrade would be like this:
- Block all proxy-server communication (possibly using a local firewall such as
iptables). - Stop the old Zabbix server, upgrade it, and start the new server.
- Stop one of the old Zabbix proxies, upgrade it, and start the new version to upgrade the local database.
- Restore the communication between the proxy and the new server.
- Proceed the same way with all the remaining proxies.
This should ensure minimum data loss through the upgrade, especially if the steps for an individual proxy upgrade are scripted and happen with no delays.
While the database upgrade is the most important step when moving from one major version to another, it's worth paying a moment of attention to the Zabbix frontend configuration file. It is suggested to compare the old configuration file with the new one and see whether there are any new parameters or significant changes to the existing parameters. The easiest way might be comparing with the zabbix.conf.php.example file in the conf/ subdirectory. This configuration file is pretty small, so spotting the differences should be easy.
We have discussed upgrading the Zabbix server. But there are quite a lot of components, and the compatibility between each of them differs slightly. Actually, the official compatibility rule list is very short:
- All older versions of Zabbix agents are supported
- Zabbix server, proxies, and Java gateways must be of the same major version
Regarding the agents, it really is as great as it sounds. All the old agent versions will work with the latest version of the Zabbix server or proxy, even down to 1.0 from 2001. If you upgrade Zabbix server, you can keep your agents as-is—although you would not benefit from new features, performance, or even security improvements.
Technically, combinations outside of the support rules might work. For example, a more recent agent might work with an older server in some cases, and the Zabbix Java gateway protocol has not changed much, so it is likely to work with different major versions of Zabbix server, too. Such combinations are not tested by Zabbix developers, are not supported, and should be avoided in general.