HP OpenView [HPOpenView] is a broad family of network and system management products. This reflects the fact that network management needs in general are sophisticated and wide-ranging, and no single product can generally satisfy any more than a small subset of the overall user requirements. Central to the function of NNM is the management station—the command and control center. This is the computer (UNIX or Windows) that performs the bulk of the data:

For large networks, there can be more than one management station, and these can exchange information with one another. A management station can also delegate NE data acquisition to collection stations. The latter connect to NEs and collect status and configuration data, which is then passed to the management station. Typically, HP suggests that a single management station can handle 5,000 managed objects. Up to 60,000 objects can be managed if collection stations are deployed along with the management station [HPNNMScale2002]. This type of distribution helps to reduce the load on the NMS. In its default-installed state, NNM provides the following main features:

As these are quite generic NMS facilities, each of them is discussed in general terms in the following sections. Each section is then followed by an NNM-specific description. These facilities are all based on manually retrieving or asynchronously receiving data from the network. Pushing data onto the network can be achieved only by using special-purpose, third-party, add-on software. In this sense, NNM is a platform on which additional software can be layered in order to achieve full-featured network management. This again reflects that network management is complex and requires a range of software applications that go together to make up an overall solution. As networks grow, they can be effectively managed only using advanced software.

An interesting aspect of managing modern networks is that there is no single solution to all the network management needs. This is similar to the fact that there is no single desktop software application that satisfies all possible needs (word processing, email, Web browsing, spreadsheet, etc.). Desktop user workflows and business processes are generally too complex and varied for it to be feasible to produce one application capable of handling them all. Instead, a range of software packages are deployed and used. In the same way, the many commercial network management tools go together to make up a continuum of packages. This then forms the basis for the management solution. Substantial teams of people are needed in vendor organizations to service the ongoing (and changing) needs of large enterprise customers.

Before looking at NNM, we take a brief detour into the important area of mediation.

Mediation

Once we start to look at the way NMS are used in practice, the area of mediation becomes relevant. This is another type of layering, similar to the ones we have mentioned in previous chapters. Mediation software exists to protect application layer software from proprietary configuration data. Figure 5-1 illustrates a multivendor network operating in conjunction with a mediation layer that feeds network-originated data into a set of NMS applications. Raw data is processed by the mediation layer and passed up to the applications for further processing.

Figure 5-1. Mediation in a multivendor network.

An example of one of these applications is billing—the A (accounting) in FCAPS. The NEs in Figure 5-1 generate data relevant to billing, such as:

• The number of ATM-encapsulated IP packets received by the MPLS LER

• The number of ATM cells received by the ATM switch

• The number of ATM cells forwarded onto an ATM virtual circuit

• The number of voice calls made using the ISDN switch

Typically, the NEs generate billing data in a proprietary format, and this must be transferred from the network into the mediation layer. The processed data emitted by the mediation software is presented in a standard format such as Billing Automatic Message Accounting Format (BAF) that can be handled using a standard commercial off-the-shelf (COTS) accounting package.

Discovery is a useful NMS feature by which NEs are automatically detected (or learned) and recorded. It fits into the C (configuration) part of the FCAPS areas. Automatic discovery frees the user from the potentially error-prone and tedious task of manually entering and maintaining the details of the deployed NEs. Large service providers tend to deploy NMS from the point in time that they start to build their networks (smaller service providers may wait for some revenues before adding an NMS). So, the burden of data entry may not be so great for large service providers; that is, they add an NE to the network and simultaneously add all the necessary details to the NMS. Once the deployed NEs are known and stored in a persistent repository, they can be managed. Discovery typically follows three main stages:

Initial discovery occurs when a device is encountered about which nothing is known by the NMS. Its details are read by the discovery application and recorded in the database. Examples of such details are:

NE software configuration is equally as important as the hardware details. An example is the set of ATM and MPLS virtual circuits we saw in Chapter 4, “Solving the Network Management Problem,” in Figure 4-9. These circuits facilitated the exchange of a range of traffic types between the two geographically distant sites.

The results of discovery are extremely useful, but even automated, it can be a time-consuming and expensive operation requiring SNMP (not all NEs support SNMP, e.g., many optical devices use the OSI management protocols for historical/technical reasons) messaging, and much database activity. Discovery provides the information needed for the other functional areas of management (such as provisioning, fault handling, performance analysis, and billing), but it should try not to do this at the expense of the overall solution. A balance is needed between supporting all the FCAPS features and the computational discovery effort required in trying to maintain parity between the actual network and the list of discovered entities.

A network-mapping feature further processes the discovered NEs and attempts to understand and depict the logical (and sometimes geographical) interconnections between them. Knowing the interconnections allows for a more comprehensive understanding of the network operation.

Notification processing is an important part of network fault management—this is the F (fault) part of the FCAPS areas—arguably the most critical part of any NMS because faults generally reflect problems in the network. Network problems can in turn affect end users. Notifications are the means by which SNMP agents asynchronously communicate problems with their NMS. From a scalability perspective, notifications provide a cue for remedial action from the NMS in response to some change in the network. This reduces the need for polling by the NMS. A number of issues arise in relation to SNMP notifications:

When an NMS receives an SNMP trap over an unreliable transport, it never acknowledges it. This is in the interests of scalability and keeping the management protocol as lightweight as possible. It also helps avoid exacerbating situations such as network congestion. When an agent detects a problem, it sends a best-effort notification message and delegates resolution of the underlying problem to the NMS. Networks are often designed to leave an absolute minimum of about 25 percent bandwidth free to allow for routing, signaling, and management protocols to continue to operate at all times. If this is adhered to, then in theory agent notifications should always get through to the NMS. This enables the latter to carry out some meaningful remedial action.

Faulty NEs can generate large numbers of notifications; for example, if a node interface is flapping up and down, then each status transition results in a new notification. The NMS user should quickly try to resolve this by downing the associated link or resolving the underlying problem with the interface.

Aggregated services, such as layer 2 VPNs (as we saw in Chapter 3, “The Network Management Problem,” Figure 3-2), may have thousands of underlying connections. If a major fault occurs, such as a fiber cut, then the originating node for each affected connection may legitimately emit a notification. This can result in a great many notifications, particularly for the increasingly dense next-generation NEs (described in Chapter 3). If the NEs are aware they are participating in a VPN, then it should be possible to intelligently reduce the number of notifications, as discussed next.

Reporting is one of the most important features of an NMS. Data is retrieved from the network and presented in a specified report format. This can include:

The data gathered is presented in a manner useful to the operator.

NMS provide a persistent repository, typically, a commercial database product such as Informix, Oracle, or Ingres. It is this database that the NMS tries to keep synchronized with the network state. This is also the database used by remote management clients. By storing most of the managed data centrally, the outlying components of the NMS can be made as thin as required, even hosted on standard COTS applications such as Web browsers. It also facilitates data security to a degree because the database can be hosted in a secure location with access granted only to authorized clients.

Once a network topology has been discovered and stored in the database, the user can execute management operations against it such as:

The database is therefore a key component of the NMS. All of the FCAPS applications use it. Commercial database products are often complex, sophisticated software products in their own right.

Backup and restore are important functions in any network environment. In the case of network management, both firmware and NE configuration are high value items. The firmware version on a given NE is subject to change as new features and technologies are added. It is important to be able to back up the existing version of firmware, allowing for it to be reapplied if required. One application for this would be firmware redistribution as part of disaster recovery, such as a lightning strike.

Configuring NEs is also an increasingly nontrivial task as more and more technologies and protocols are packed into devices. Once the NEs have been configured and are operational, it is important to be able to back up the settings—often called a configuration database. As for the firmware case, the configuration details can then be restored if necessary. In providing the backup and restore capabilities, equipment vendors can use protocols such as File Transfer Protocol (FTP) and Trivial File Transfer Protocol (TFTP).

Standard Java-enabled Web browsers can play an increasingly important role in network management because they are COTS products. This reduces the investment required for producing clientside NMS solutions because almost no development is required for the client application framework. The browser provides the execution environment for client applications. A Java-based client application can be executed using the browser to download a Java applet. The use of Java has a number of additional merits:

The needs of network management solution developers are well served by the object-oriented features of Java [JavaDev]. The multiple views of UML fit easily into a Java environment, and some development tools (e.g., Telelogic Tau and Rational Rose) can generate Java code.

Security is an increasingly important issue in network management as the incidence of attacks increase [CERTWeb], so it is with good reason that security increasingly concerns the operators of all types of networks. All levels of network management infrastructure have to be protected from the NEs/EMS, all the way up to the business-management layer. Multiplatform support is also important, because two of the main competing platforms are:

Java allows for full compatibility across these platforms: Code written for NT should run unmodified on the others. This can help reduce development and testing time.

As NEs deploy increasingly advanced features, it is often necessary to upgrade deployed NMS components. Java provides an elegant means for doing this by the use of packages. These are logical Java code and data entities that can be signed for security and swapped in and out of systems as required. Packages also provide many advantages for developers, such as encapsulation and interfaces.

MIBs are the cornerstone of SNMP-based NMS. They provide details concerning the network managed objects and form the basis for the NMS data model. In an ideal world, the MIBs would be sufficiently flexible to provide the bulk of the data model, but (as discussed in Chapter 3) for the current generation of MIBs, this is generally not the case. NMS should provide a number of baseline features for MIB support, including the ability to:

When new devices are added to networks, their associated MIBs must be loaded into the NMS in order to provide managed-object access. It should be a relatively simple matter of adding new MIB files to the existing set. The new MIB files should be automatically compiled by the NMS and verified for correctness. Following the addition of the files, the associated managed objects should then become visible to the NMS. This latter would almost certainly be a manual configuration step, but the inclusion of marked objects could help in automating it, as described next.

MPLS operates at the network level; that is, the managed objects relevant to MPLS involve more than one node. MPLS nodes have a number of managed objects, the status of which can change over time:

There are other MPLS objects that do not change because they are static in nature:

All of these are highly relevant to managing MPLS; for example, LSPs may span any number of MPLS nodes and can be built using EROs, resource blocks, and cross-connects. Providing MPLS support requires software for managing these and other objects.

Chapter 3 introduced PBNM in the context of automatic traffic engineering and explained some of its advantages. In general network management terms, policies can be used to automatically solve recurring problems, such as certain types of failures or other important events. A simple example is traffic thresholds. The user can set a threshold on an interface for the number of packets received. If the threshold is exceeded, then a notification is generated and the NMS can divert some of the traffic. A simple rule is defined to achieve this based on the setting and crossing of a traffic threshold.

System reliability is an estimate of the probability of failure. There are a number of ways of improving reliability using backup facilities. In the case of NMS, this can take the form of protecting the central database. Failure of the database is usually fatal, and for this reason, many database vendors provide a failover capability. The user can deploy a backup version of the database that runs in parallel with the primary system. Failure of the primary results in a full switchover to the secondary.

Supporting the wide variety of NEs in enterprise networks requires a rich mix of NMS and operational skills. Our primary concern in this chapter is the NMS, and in this section we briefly review the function of some ancillary systems to which it may be attached. These can include software applications that handle:

Business asset analysis can be used for depreciation studies, lifecycle management, and department billing. NEs are expensive items that need to be recorded, managed, and maintained from initial deployment through decommissioning. Issues like upgrade and replacement have an important bearing on the network operator.

Specialized software applications can be used to carry out offline traffic analysis. A snapshot of the discovered topology can be exported into a modeling package, and the user can execute what-if scenarios, for example, increasing the bandwidth on a given link. The effects of any such changes can be viewed offline before making a change in the network. Modeling packages may also allow new topology details to be exported back into the NMS.

Alarms generated by the network can be routed via the NMS to trouble ticket applications. This allows for recording and directing any remedial work required to clear the fault. Another method of alarm annunciation is the simple audible variety, such as sounding a bell or a computer speaker. Visual indication can be a GUI topology object color change. Routing a message to an email recipient (or a short text message to a mobile phone or pager) can be used to indicate text-based fault indication. It is even feasible that a voice phone call could be initiated by the NMS to indicate a particularly serious fault.

Performance analysis can be facilitated by a combination of special reports and third-party applications. Many NEs generate performance data records (PDR) that provide utilization details for:

Mediation software can process PDRs to produce data ready for export to reporting. This is similar to billing. Billing is already a critical SP business requirement, particularly as IP service billing becomes increasingly important [IPDR-ORG]. Enterprises also need billing as SLAs become more common.

Many NEs generate call detail records (CDR) that are preprocessed by mediation software prior to export to billing. The raw NE data blocks in Figure 5-1 are generally CDRs and/or PDRs.

Security is an issue of grave concern to the owners and operators of all large networks. The distributed nature of managed networks provides possible targets for attack:

Security is required at all levels of a managed network, and this may require additional specialized software, such as directories for secure storage of relevant data.

Networks generally do not remain static, and changes need to be applied in a controlled manner. Workflow systems can help achieve this by tracking and managing the tasks required for running a network. This is discussed in more detail later in the chapter.

The facility of adding software to an NMS can greatly extend its usefulness. This can include anything from MPLS/ATM/SONET provisioning to special-purpose NE monitoring. Open source NMS are a relatively new phenomenon and represent what might be called the ultimate in end-user configurability—actually changing the base software itself. Network management requirements are unique to every network, so programmability is an important addition.

Enterprises deploy an increasingly wide variety of NEs as their network requirements and business processes evolve. In this section we look at the workflows and business processes surrounding the way these networks are managed and operated.

Increasingly, NMS are required to assist in rapidly bringing up, keeping up, and downing large networks. One application of this is when a network is rebalanced after a significant hardware addition occurs, such as a new multiservice switch. The operator wants to execute this as smoothly as possible, and this may involve:

Major reconfiguration actions such as these can be recorded by the NMS and stored as command files. These can be reused at a later time when similar operations are required.

It is sometimes said that the network is the database for any NMS. As we've seen, all NMS attempt to maintain parity between their database and the one stored in the network. Saying that the network is the database reflects the fact that the following may all be stored on the NEs and accessed at will via an NMS:

The merit of this is that all such device-resident data is completely up to date because it exists in the network. The problem with retrieving this data is that there is so much of it. Getting the requisite data from the network for processing by an NMS requires expensive and possibly lengthy device retrievals. As we've seen, most NMS struggle to close the gap between their snapshot of the network and the real picture.

When data is discovered from the network, it is important that it is accurately reflected in the NMS. This can lead to anomalous results when NEs support multiple personalities, such as for MPLS SIN. As discussed earlier, SIN provides support for both MPLS and ATM at port level on the same NE. Should this node be described as an MPLS or an ATM entity? Whatever solution is found, the NMS should respect the data stored in the network, because in this context, the network is the true database.

An important point about managed networks is the definition of the boundary. This is the point outside of which the NMS has no jurisdiction. NNM must have some idea about this in order to avoid trying to discover nodes unnecessarily. This can arise when a given network is connected to the Internet and the operator wants discovery to stop its data collection at the last node before the Internet.

The network boundary problem is also seen in multiservice networks that contain numerous clouds of different technologies, including:

As for the Internet case, this may require careful administrative actions, such as specifying a seed file in NNM. In essence, this sets a boundary address for management beyond which it is not permitted (or meaningful) to operate.

One last point about this is that even when the boundary has been carefully and correctly designated, the traffic introduced by network management may be too high.^[1] In this case, it is important to be able to pace the management operations to suit the available bandwidth.

The theory and practice of network management are separate and distinct. Effective network management requires a broad range of software tools. Two approaches can be taken: Build one all-encompassing system or divide and conquer via a number of packages. The latter is the philosophy adopted in NNM and reflects the fact that computing power is increasingly inexpensive. Specialized software can be employed for areas such as mediation, billing, and performance analysis. The cost of ownership and development can be significantly lowered by the use of standard COTS packages. Java-based software can facilitate the use of standard browsers for (thin client-based) network management. Discovery, mapping, and monitoring are often inextricably interwoven, and along with fault management, these are the big “readers” from the network. They are the means by which the NMS attempts to keep pace with changes in the network.

Minimizing the gap between the network situation and that perceived by the NMS is crucial and provides a baseline for defining the quality of a given NMS. Provisioning both writes to and reads from the network as it executes the user's commands. Fault management can perhaps be considered the most crucial of all NMS facilities. The workflows associated with enterprise and SP networks share some similarities, but the financial cost of downtime may be more keenly felt in the latter. We briefly explored the reasons why it is often said that the network is the database. All NMS have a boundary, particularly those that encompass more than one service or technology.