Planning for Operational Patching and Upgrading

With a fully operational NNM deployment, there comes the task of maintenance. Some or all of the NNM systems may require a patch or an update. The underlying operating system or one of its subsystems (such as DNS) may require patching. A parameter may require tuning. Performance may be poor at some sites, requiring additional disk striping and memory. The IP address of an NNM system may have to be changed because the system must be moved to another location. A third-party package such as CiscoView, Optivity, or NetCool may require a patch. These are routine matters.

Less routine is the annual or biannual upgrade cycle. The operating system may be due for an update, a third-party product update with desirable features may become available, or better yet, a compelling NNM upgrade is now available.

The benefit of the NNM test lab is that all of the above can be tested as much as necessary until the patch, update, or change is demonstrated as safe, functional, and doesn’t break existing functionality. It is inadvisable to test patches on operational NNM systems. To minimize the number of outages, try to batch changes into one outage.

After validating the change, schedule an outage for each NNM system. Perform the required work during this time and verify full functionality. If something goes wrong, either back out of the patch or revert to a backup. Always have a backout plan. Finally, declare the outage over.

For NNM upgrades, additional concerns present themselves. Some file formats (such as snmpCol.conf) may change, requiring conversion. There may be new daemons that ITO should be configured to monitor. Changes in the menu structure is a foregone conclusion, and there will be additional functionality. The NNM upgrade release notes (Help:What’s New in the GUI) is a must-read for learning about new features and changes, and the HP Education Center’s Delta Training class is a valuable resource. Users should be advised of the changes they’ll see after the upgrade and learn how to take advantage of new features. In-house training materials must also be upgraded.

Occasionally, a single site will have a unique problem. For example, a new subnet may be discovered by an NNM system where one of the devices has a badly behaving SNMP agent which returns data that causes netmon to terminate or loop. If a patch is available, it may be appropriate to directly patch that system. In this example, you would turn on full logging for the netmon daemon and wait for the problem to occur. Review the netmon.trace log file for the offending device and put its IP address in the netmon.noDiscover file.