CHAPTER 13
Secure Remote Management and Monitoring
In this chapter, you will learn about
• Integrating IP-enabled AV devices to ease troubleshooting and support new revenue streams
• Network-connected management systems and how they communicate
• Criteria for choosing and designing a remote management and monitoring system
• Security considerations for remote management and monitoring systems
As more audiovisual systems have become IP-enabled—audio processors, video switchers, projectors, displays, room sensors, and more—there is both a need and an opportunity to exploit IP communication for good. Imagine a video projector in a classroom that can communicate over a network that its bulb is about to go out. Or a room sensor that can tell a central system that the room is unoccupied and therefore the lights can be shut off and the video screen retracted.
Many AV professionals hear these possibilities and think of control systems—those increasingly standard components of any AV installation that provide an intuitive interface for operating many different systems in a room or building. But taken to the next level, such systems can do more than just control networked devices—and for more and different users. They can also give the very professionals who design and install them a new revenue stream.
Introduction to the Remote Management and Monitoring System
When it comes to networked AV systems, remote management and monitoring systems help AV professionals—from technology managers to service providers—manage what is an increasingly complex array of technologies. Remote management and monitoring systems also give technology service providers a potentially new source of recurring revenue. Having installed, for instance, a conference room or even an entire campus of AV systems, today’s AV customers may be willing to cede certain management functions to the systems’ designers or integrators (or to an AV services company that didn’t have anything to do with the original system) for a regular service fee. And thanks to IP networking, service providers and technology managers can usually manage and monitor systems without physically interacting with the networked AV devices—all while ensuring enterprise security and accountability.
Remote management and monitoring applications were originally developed for IT administration, but the tools and concepts are now applicable to more technology systems and service providers, including AV. Network connectivity offers a chance to reduce service costs and improve troubleshooting efficiency by preventing unnecessary truck rolls and facilitating better root-cause analysis. A sound remote management and monitoring architecture gives service providers the ability not only to identify problems before the customer does, but also to fix issues without dispatching a technician.
Remote management and monitoring architectures typically fall into two areas. There are the individual device-management systems; and there is a centralized enterprise management console, sometimes referred to as the manager-of-managers (MoM) system. See Figure 13-1.
Figure 13-1 A manager of managers draws data from device-management systems and individual devices.
Each relies heavily on IP networks. A device-management system comprises a network server that aggregates data from a particular type of device or devices by collecting status information and alarms. A manager-of-managers system takes information from both devices and device-management systems and presents it all via a single user interface that users can access over the Internet. This enables management from remote locations, and each type of management system can usually be configured to send alert messages to key personnel through email, short message service (SMS), or other notification protocols.
NOTE Every service target in a service-level agreement (SLA, Chapter 14) must be measurable. Identify targets that you can remotely measure for your customer using a remote management and monitoring system. You’ll gain both the opportunity to validate your service targets and a lucrative ongoing relationship with your customer.
For our purposes, when we talk about remote management and monitoring, we are speaking of a “manager-of-managers” model—a centralized management system that aggregates data from many devices and device-management systems. Devices may be displays with Ethernet ports; device-management systems, often referred to as subsystems, might include a lighting control system, as in Figure 13-2. Depending on a customer’s needs and the extent of an AV installation, it may be adequate to manage and monitor only a fleet of installed projectors, for example. However, true remote management and monitoring in a networked AV world is best handled through a manager-of-managers system that can aggregate disparate data over IP, even if the customer starts small first and adopts more comprehensive systems management later.
Figure 13-2 Manager-of-managers systems can pull together data from various device-management systems.
A centralized AV management system (the manager of managers) has an interface that allows users to add subsystems and devices that they’d like managed and to configure the system so it operates to their specification. Advanced management systems also have an autodiscovery function that can search a network, find subsystems and devices, and add them to its inventory without user input. Once configured, the management system connects to AV devices and device-management systems over IP to retrieve relevant information.
Simple Network Management Protocol
Management systems communicate with devices and subsystems in a variety of ways, but nearly all of them originate in IP networking. The most common means of communication for remote management systems is the Simple Network Management Protocol (SNMP). Although originally used for routers and switches, SNMP now works with almost any device that connects to a network.
SNMP represents a set of IETF standards for network management, including an Application Layer protocol, a database schema, and a set of data objects. SNMP exposes device data in the form of variables on the managed systems, which describe the system configuration. These variables can then be queried, and sometimes set, through management applications. Some device-management systems use proprietary means of communicating with individual devices, but the subsystem itself almost always has an SNMP output interface so it can communicate with a central management system.
SNMP is the most widely used and available interface for device and subsystem communication. It can be used to fetch data on many objects and metrics in many devices and subsystems. Devices and subsystems that have SNMP configuration capabilities (i.e., if the device has “SNMP writes” enabled) can be remotely configured and operated by a manager of managers or central management system, allowing for enhanced configuration management.
But SNMP isn’t without its flaws. SNMP network traffic is uncompressed and very inefficient when it comes to bandwidth consumption. Moreover, SNMP doesn’t provide a mechanism for flow control, so it is susceptible to network flooding.
SNMP generally comes in two flavors: versions 2 and 3. SNMPv2 is extremely common and can be used to represent many kinds of information. However, it sends packets in plain text format, which poses an obvious security risk. SNMPv3 allows for enhanced security and packet encryption and should always be used over connections that traverse the open Internet.
NOTE SNMPv2 sends information in clear text. In general, customers do not want their information sent this way. There may be specific laws or policies that forbid it. For example, Federal Information Processing Standards (FIPS) 140 forbids sending US government proprietary information via clear text. Keep that in mind when designing a remote management and monitoring system for a government client.
Management Information Base and SNMP
If you plan to utilize SNMP for your remote management and monitoring system, you need to determine whether the devices and subsystems you want to manage can support it. Your first clue is a device’s or subsystem’s management information base (MIB) file. A MIB file is a document that defines what a particular device or subsystem offers to an SNMP manager or collector. MIB files are text files, formatted according to the SNMP standard, that define a data interface between managers and devices or subsystems.
It is possible to “read” a MIB file as a text file, but that file format makes the contents difficult to interpret. Instead, you can use a software application called a MIB browser to decode a MIB file. Most management systems have a MIB browser feature.
The objects in a MIB file have unique IDs called object identifiers (OIDs). These identifiers let the management system know exactly where to look in a managed device’s SNMP interface for each piece of data. Identifying the reporting capabilities of monitored devices and device-management subsystems is a critical step in designing a remote management and monitoring architecture. Usually, equipment manufacturers provide documentation that outlines the communications protocols they support. It includes information about the data objects that a management system can poll and collect through its interfaces. The level of documentation can range from a printout of the MIB structure (in the case of SNMP) to a full-fledged API specification.
Unfortunately, sometimes there is no such documentation. After you’ve added a particular device or subsystem to the management system inventory, you may be able to use the system’s user interface to determine the reporting capabilities of the monitored equipment. Although this won’t provide the same level of information as manufacturer documentation would, it should allow you to see what information the central management system can gather from the devices automatically. With SNMP, many management systems automatically grab the MIB from a managed device and extract the available objects, then list them as metrics associated with that device.
Another method to determine reporting capabilities is to inspect the reporting interface manually using a text editor or MIB browser, although this is not the easiest method. If all else fails, do an Internet search for the particular device or subsystem that you are trying to integrate into your remote management and monitoring architecture. There may be available documentation that wasn’t included with the device. There is also a chance that someone else has tried to extract information from the same type of device and posted information about their experience on an industry website.
Alternatives to SNMP Communication
Some manager-of-managers systems can also integrate other types and sources of information, such as syslog messages, HTTP communication, and Extensible Markup Language (XML) data, as shown in Figure 13-3. If SNMP isn’t an option (or if it’s not the only option), you may want to integrate one of these other device-communication technologies into your remote management and monitoring architecture.
Figure 13-3 Other sources of device-management data.
Syslog
Syslog sends real-time messages to a log collection server. Basically, it’s a data dump of a system log. Syslog messages aren’t in a structured format, so they require text parsing and custom programming for use in an automated system. They are also very insecure and inefficient. Log messages can fill up terabytes of hard drives very quickly.
ICMP Echo Request
An Internet Control Message Protocol (ICMP) echo request is a network administration utility used to test network connectivity and response time. It is also referred to as a Ping tool. (You will learn more about ICMP and Ping as they pertain to troubleshooting in Chapter 16.)
Using ICMP, an originating host sends the echo request to a device destination and waits for a response. An ICMP echo request provides very low-level status monitoring. It can tell you if a networked device is currently powered on and connected to the network with a functional network interface card (NIC). It can’t tell you any device-specific information, or even if the device is functioning properly.
Hypertext Transfer Protocol
Using common HTTP-based communication, a technology manager or service provider can log in to a device or subsystem while using a standard web browser. Most networked devices and subsystems have web interfaces. This type of device-reporting capability would be considered manual because a person must log in to the device to get its status. HTTP-based communication isn’t ideal, however. If a user has to go into a subsystem or device to look for data, it defeats the purpose of having an automated remote management and monitoring system.
Web Services Integration
Web services describe an interdevice communication method that treats every device or subsystem on a network as a “service.” Any other device or system on the same network can interact with the service through “Post” or “Get” messages using XML and open standards of message formatting.
Proprietary Communication
Some device-management systems use proprietary protocols to communicate. Such proprietary protocols can (but don’t always) improve the security and efficiency of device communications. However, proprietary control protocols can’t be easily integrated into a manager-of-managers system or with third-party devices.
Scripting
Devices and systems can communicate with each other using server-side scripting languages, such as PHP, and custom APIs. This method requires advanced programming skills, however, and constant code maintenance and customization to manage a variety of devices. It also limits the ability of devices and subsystems to interact if they do not support PHP or other scripting languages.
Basic Reporting
Regardless of what communication protocol a management system uses to exchange information, there are two primary ways that information gets into the system in the first place: polling and traps. Both are essential to remote management and monitoring.
Polling is when a management system sends a request to a subsystem or device for specific information (“Hey, are you on?”). Depending on how frequently the management system is configured to poll devices—and how much data the system requests in each poll—the information it receives can be more or less granular. For example, a management system can tell you more about a device if it asks whether the device is on every 15 minutes than if it asks every hour. And it can tell you even more if it asks whether the device is on and if it’s operating correctly (a determination that likely requires several other data points).
It is important to understand that there can be a delicate balance between how much information you need a management system to gather and how much network congestion the polling process creates. Poll too frequently and you use up network bandwidth; not enough and you may not get usable information.
A trap is a mechanism in a subsystem or device that is programmed to generate an alert if an error or other specified event takes place. This alert travels back to the central management system and represents an alternative means for a manager of managers to gather data. Trap alerts don’t usually contain as much information as polling data, but they can provide detailed situational data for a specific problem.
Typically, a centralized management system can report and collect an unlimited amount of data, ranging from device-specific information, such as what program is currently playing on a network-connected TV, to general network availability information, such as the Ping response time of an IP-enabled security camera. With more potential devices under centralized control, scalability is a chief concern of any remote management and monitoring architecture.
Individual devices or device-management systems can report detailed information to the manager of managers. A device-management system can also report summary information, such as facility-wide power consumption from a power supply management system. Typical remote management and monitoring architectures allow subsystems and devices to report differing levels of detail, depending on the nature of the system and the requirements of the service provider or technology manager. (See Figure 13-4.)
Figure 13-4 Devices and device-management system report differing levels of detail.
TIP When designing a centralized management system, it’s generally a good idea to take advantage of the summary reporting abilities of individual device-management systems, rather than harvesting all the detailed data available. This takes some of the scalability strain off the management system and distributes the load to available subsystems. Each device-management system should still provide a web-based capability to drill down into detail for each managed device, as necessary.
System Configuration
Configuring a remote management system requires focusing on both internal and external configuration. Internal configuration refers to the setup and customization of the actual management or control system. In the case of a manager-of-managers solution, this includes setting up user accounts and defining roles and privileges for people authorized to manage and monitor networked systems; discovering relevant subsystems and devices on the network and adding them into an inventory of systems under management; applying profiles and classifications to each item in the inventory; and configuring any advanced tools available through the central management systems, such as topology maps, event rules, and alarms.
External configuration refers to the ability of the central system to configure other devices and subsystems. Managed devices often require configuration if they’re to work properly with the management system. They may require firmware updates, specific device settings, threshold and trigger profiles, network addresses, user control settings, and more. Keep in mind, most centralized, manager-of-managers systems can integrate with an unlimited number of devices and subsystems, but they don’t typically offer a mechanism for automating the configuration of each item in the inventory. However, most enterprise management systems do provide links to the management interfaces of the devices and subsystems they’re connected to. Once you are connected to the device-management system, or the device in question, you can usually configure its settings directly.
All devices or subsystems in a customer installation may have different interfaces and communication abilities. Their particular sets of abilities are called device profiles. For example, some lighting control systems may include a web interface for remote control and others may not. Unfortunately, there is no universal device profile format, so you may have to conduct some research of your own to determine the remote configuration abilities of devices and subsystems.
TIP In order to configure devices and subsystems, management system users need administrative accounts and passwords for each component. Therefore, access to the central management interface sometimes presents security issues. When designing a remote management and monitoring system, document how you will mitigate security risks for password management issues and user accountability.
Security and Troubleshooting
Advanced management systems come with integrated security features to help automate the login process for each device or subsystem. The system administrator can define which users can control which subsystems or devices. This strategy
• Removes the user administration burden from each device or subsystem
• Allows the system administrator to use complex passwords for better security
• Provides user activity logging, so the administrator always knows who has been doing what to each device under management
Keep in mind, whether you’re a service provider or a technology manager, deploying a remote management and monitoring system means collecting information on systems that either aren’t yours or aren’t used by you. Security measures determine not only who can access devices under management but also how the data that’s collected will be protected. When evaluating a remote management and monitoring architecture, the following questions are important to securing the operation:
Control Systems Growing into Management Systems
AV professionals have spent years laying the groundwork for successful remote management and monitoring solutions, starting in the humble conference room. Thanks to Ethernet and IP networks, the traditional control systems that AV integrators have used to lower projection screens and window shades, power up projectors, and dim lights have expanded beyond a room’s four walls to encompass entire buildings. AV industry stalwarts, such as AMX, with its Resource Management Suite; Crestron Electronics, with its Fusion Enterprise Management platform (shown); and Extron Electronics, with its GlobalViewer Enterprise solution, have broadened their traditional AV control functionality to include site-wide management and monitoring from a central location.
By placing AV control systems on networks, users can manage devices remotely, more efficiently maintain equipment inventories, and respond to customer service issues more effectively. In an enterprise-wide control environment, a user can turn on a projector from another time zone by using a smartphone.
In addition, as AV control systems speak IP—the same language as other network-connected devices—they can incorporate more technology. Today’s networked control systems offer more than AV control; they can monitor light and occupancy sensors, building systems, HVAC controls, and more. Increasingly, an AV control system can play the role of a manager of managers, allowing AV professionals to offer secure remote management and monitoring using the tools and interfaces they’ve grown accustomed to over the years.
That said, placing control systems on a network requires planning and coordination among AV and IT departments. Among other things, IT needs to know how you plan to interface with their network, what data you plan to transmit, and how managed devices may or may not impact other data on the network. The more ambitious the control system, the more you need to ensure that disparate equipment, using potentially different control protocols, is accurately reporting data back to the central management console. Although all the systems you want to incorporate may be capable of IP communication, some may use different protocols. For example, many building systems communicate using BACnet, which some modern AV control systems support natively over IP.
For control signals to find their destinations, each device must have an IP address or a DNS host name. Work with IT to determine which devices can be connected to the network. After that, you will continue to work with IT to assign IP addresses or DNS host names to the devices. The IT manager will appreciate the advance notice, and you’ll be able to work out any issues early.
• Who should have access to the management solution?
• Who should have access to view the user interface?
• Who should be able to log in to remotely managed devices?
• Who should be able to add or remove devices?
• Who should be able to configure the management system?
• Who should receive alarms? Which alarms?
• Who should have access to exports and backups?
It’s important for you and/or your client not only to answer these questions but also to make sure that the system you use can support this level of security.
When it comes to troubleshooting, a central management system offers two basic problem-solving services: root-cause analysis and the ability to take corrective actions. Root-cause analysis is an automated way to isolate and identify a problem and determine its cause using reported data. Think of root-cause analysis like a doctor diagnosing a medical issue. The doctor asks general questions to identify the area of concern, then asks more specific questions to home in on the exact issue. Then the doctor runs very specific tests to try to find out what is causing the problem.
A management system does this analysis by giving service providers and technology managers visibility into the status of their devices and subsystems, allowing them to drill down for more detailed information about any particular item. Management systems typically provide an array of tools to the user to facilitate the root-cause analysis:
• Topology maps
• Alarms and notifications
• Status views of device- or subsystem-specific information
• Customized dashboards
• Reporting and trending
• Event rules engine
Corrective action refers to a management system’s ability to allow users to fix problems. As noted earlier, central management systems typically provide links to the interfaces of individual subsystems and devices so a user can control them remotely and resolve problems. For security purposes, they also automate the login process and limit access to subsystems and devices based on user profiles.
Once a problem has been isolated to a particular device, the most common corrective action is simply to restart it. Modern management systems have power management tools that map devices to particular power outlets, which can be controlled remotely. This means that a service provider or technology manager can identify a problem, determine its root cause, log in to the device, fix the problem, and reboot it without leaving his desk. While this doesn’t solve every possible issue, it prevents costly service calls for common problems.
TIP When troubleshooting any device, try rebooting it. In the majority of cases, this corrective action solves the problem.
Planning a Remote Management and Monitoring System
There are many things to consider when selecting a centralized system to fit your remote management and monitoring architecture. You cannot prioritize all of them, however. Think carefully about which of the following aspects of a remote management and monitoring system are most integral to your customer’s culture and needs.
• Security The solution you choose needs to have robust user account management features and the ability to assign user privileges. It should also provide a secure user interface using common encryption techniques and facilitate logging into subsystems and devices.
• Accountability The system should provide user account control, track user actions, and make available records of any changes made and/or measures taken on the network. This will hold authorized users accountable.
• Scalability As the number of managed devices, subsystems, and collected data grows, scalability becomes a concern. Select a network management solution that allows the system to grow with business operations. Don’t start with the biggest, most powerful management system; choose a solution that allows you to expand as needed.
• Interoperability Because a management system interacts with many subsystems and devices, it has to be compatible with all their communication protocols. SNMP is the most common communications protocol, but your subsystems may have other available protocols that provide better information. Check to see that the management system supports these protocols as well. You should also consider devices that you or your customer will want to incorporate in the future. Will they be able to interface with the central management solution that you choose?
• Extensibility No management system will include all the functions and features you want for your remote management and monitoring architecture out of the box. Most solutions allow you to add views, dashboards, and reports through the user interface. Many include application programming interfaces (APIs) that you (or a contractor) could use to develop add-ons to the system. Consider what sort of extensibility the solution offers and how much work it will take for you to customize or develop new tools and features.
• Event Rule Configuration Alarms and traps are near-real-time methods to identify problems using a remote management and monitoring system. However, many service professionals have learned from experience that simply forwarding every error to support staff is a poor way of monitoring problems. Too many traps and alarms can overload a network, a phenomenon called “trap flooding.” Even if a network has the bandwidth to handle many messages, you don’t want to overload your support staff with information. Select a management system that supports the ability to create event rules—logical combinations of alarms and traps that allow users to configure the system to, for example, “Send me a message only when X, Y, and Z happen” (i.e., when something is really wrong).
• Cost There are many manager-of-managers management systems on the market. You can spend as much or as little as you want. That said, here are some cost-related questions to ask when evaluating technology management systems:
• Is the up-front cost justified? Sometimes you may not need the biggest system immediately.
• Is expansion expensive? You should find a system that grows as you do at a reasonable price.
• How much is yearly service and maintenance? Sometimes a solution can be inexpensive at first but cost a fortune for upgrades and bug fixes.
• How much staff will you need to maintain the system? If you have to dedicate multiple personnel to run the management system, it may end up costing more than you pay for it.
• How is the license structured? You may have to pay per seat (user), per device, per subsystem, or per year, or you may simply pay one large license fee on an annual basis. Examine your needs to find out if the license structure fits your business.
Always remember that one major goal of a remote management and monitoring system is to reduce expenses, so take into consideration its problem-solving capabilities and compare that against the cost of actually dispatching technicians to fix AV and other technology systems.
Management System Architecture
The primary goal of a remote management and monitoring solution is to ensure the performance, security, and uptime of the devices that it monitors. With security being a primary concern, it’s important that the central management system be connected to devices and subsystems over a private network, if possible. Accessing networked devices over the open Internet is too risky, which isn’t to say you shouldn’t use the Internet to manage AV and other networked devices—after all, the Internet is what puts the “remote” in “remote management and monitoring.” But if a management system is to operate over the Internet, it should be via secure, encrypted protocols, such as virtual private networking (VPN).
In addition, any manager-of-managers system should be installed on a certified and reliable hardware platform that has been tested with the software in a configuration supported by the vendor. It should be installed professionally, with power surge protection and backup power. Optimally, you should install redundant management systems so that if one suffers a critical failure, the other will still be available.
Remote Management and Monitoring Access and Reporting Methods
A management system collects data in different ways; it also reports what it collects in different ways. When setting up a management system, consider the ways in which technical staff will need to access device information, based on the tools they’re accustomed to using and the security requirements in place. There are two basic ways that a management system can report information to a user. It can send alarm messages or display data in a user interface.
Alarm messages usually consist of email or SMS text messages. As noted earlier, for messaging to work effectively, the configuration of event rules is important: you don’t want to send thousands of SMS messages based on false alarms. These messages don’t typically contain detailed reports of the issue; they just call attention to a particular failure. Whether sent by email or SMS, the summary needs to make it clear to recipients where they should go to get more information.
The user interface of a central management system usually comprises two parts: a web interface and an application interface. Users access the web interface using a standard web browser and the IP address of the management system. Ideally, the management system would be configured to communicate using Hypertext Transfer Protocol Secure (HTTPS), the secure web protocol. This web interface should let support staff members access reports, alarm notifications, and status views so they can explore problems and figure out what’s wrong. Because it’s a web interface, authorized users should be able to manage and monitor systems through a secure browser wherever they happen to be and on any computing device.
The application interface of a management system is a richer software program that runs on a local workstation or a networked server. As such, it’s more powerful and generally offers the most detailed and powerful reporting interface available. It contains setup, management, configuration functions, and data displays. This interface should only be accessible through a secure VPN connection or a secure remote terminal connection, if not run on a password-protected computer.
Across all reporting methods, user administration is very important. You need to have the ability to assign different access levels and privileges to each user so that you can control who can get emails and SMS messages, who can log in to the web and/or application interfaces, and who can log in to the devices and subsystems to address service issues and/or make changes.
CAUTION When setting up access to a centralized management system, make sure logins are unique. Change any that are in the format “admin/admin.”
Alarm Notification and Response Plan
After you’ve integrated devices and subsystems across an IP network, your management system starts telling you how those systems are operating. If it’s generating too few alerts, you’ll have a hard time knowing when a problem happens. If it’s generating too many, you may miss the important data. The sheer volume of information that a remote management and monitoring system can generate might prevent your organization from actually using the system—unless you systematically approach your alert strategy.
For starters, determine a handful of technology failures that, if they happened, would cause the biggest problems. Every AV installation is different and every customer has their own priorities. A router going down is an obvious choice: if it goes offline, then all communications fail and networked devices stop working.
Once you’ve identified a few critical points of failure, configure them to generate the highest-priority alerts in your remote management and monitoring system. As for the myriad of other alerts, some will require immediate attention, and others just need to be recognized so you can fix the problems later. A few will even be extraneous and can be ignored.
Prioritizing alerts and determining a course of action based on those alerts is something of an art and varies from system to system and customer to customer. Your management system should include an event rules engine that allows you to apply logic to alerts. For example, you may decide that two different low-priority alerts can be ignored, but if they occur simultaneously, it makes for a high-priority event that requires immediate attention.
The following are common event classifications, based on alarms:
• Critical event The alarm is such a high priority that it should kick off an alert right away.
• Persistent event If the alarm is triggered once, it isn’t a high priority. But if it remains an alarm for a specified amount of time, it should generate an alert.
• Intermittent event If the alarm is triggered once, it isn’t a high priority. But if it happens repeatedly a specified number of times over a certain time period, it should generate an alert.
• Combination event The alarm by itself isn’t a high priority, but if it happens at the same time as something else, then it should generate an alert.
You should also determine who will receive different alerts, based on the devices that generate them and the skills of the technical staff, and write those instructions into the event rules engine. If a video switcher generates an alert, make sure an AV technician gets it; if a router generates an alert, the system may automatically alert an IT professional. Depending on the size of the support team, everybody involved may need to receive every alert.
Keep in mind that there may be situations where a device or subsystem fails and no alert is generated. How will you know? Check the logs. When integrating a remote management and monitoring system, make sure it has the ability to log alarms, alerts, and events. Maybe email alerts are landing in a spam filter or SMS text messaging isn’t working properly. By periodically checking the logs, you may be able to determine whether the alerting system is working as it should.
Required Ports and Protocols
In our discussion of remote management and monitoring, we have covered many communication methods and protocols necessary to integrate devices and subsystems into a centralized, manager-of-managers system. As you configure your remote management and monitoring architecture, you will also need to ensure that any network you’re using is configured to allow your management communications to pass. While your requirements may vary, Table 13-1 contains a list of commonly used communication ports and protocols you should be aware of for secure, remote management and monitoring. The list is not exhaustive, and there are sure to be ports active on your network that are not included here.
Table 13-1 Common Remote Management Ports and Protocols
Scoping Remote Management and Monitoring System
Once you have determined how your remote management and monitoring solution is going to report information, you then need to identify the information itself and where, specifically, it resides. Sure, you can monitor virtually any network-connected device, but which are you actually going to integrate into the management system?
Start by identifying the scope of work that you or the user wants the management system to handle. Without getting into all the technical details of each device, figure out what needs to be monitored. Ask the following questions:
• What common support tasks do I want to reduce or eliminate?
• What kinds of events do I need to know about?
• What devices or subsystems have the potential to drive customer complaints? Where are these things located?
• What are the typical technology problems at each location?
• Where do I need visibility and accountability? For which devices or subsystems?
The answer to one question may lead you to the next. The answer to some questions may be “everything, everywhere,” while the answer to others may be very specific. In the needs analysis for a remote management and monitoring system, you are trying to identify what you or the customer needs, not everything the system can do. As with all networked AV systems, remember to consider network bandwidth and processing required for complex automated procedures.
Use drawings and graphical representations of the locations, devices, and subsystems to help conceptualize the scope of work. It’s a good idea to use a geographic map to mark the various locations that you want to monitor and then use drawings or architectural diagrams to mark down the devices and subsystems at each site. You can even use schematics of individual racks to identify each device. Thinking of your systems in terms of their actual locations and components helps you prioritize system functions and features. It also helps you identify how the system will actually help the customer.
Determine What Data to Monitor
Once you’ve worked out the scope of your remote management and monitoring architecture, you need to determine exactly which data or metrics you want to monitor on each device or subsystem. Focus on actual use cases and applications. You might have access to thousands of different data points for each device, but you will probably only care about a subset of those. Your system’s monitoring capabilities will depend, at least in part, on reporting techniques.
The lowest level of detail you can monitor is whether a system is “up,” that is, whether it’s operational. You can monitor this basic information using polling operations, such as Ping. Ping polling tells you if a particular device or subsystem is turned on and if its network interface is working properly. If the component has a web interface, then HTTP polling can tell you if the system is running and if its control interface is accessible.
The next level of detail is to monitor system error logs, which can be done using the syslog protocol. The data available through syslog is limited because log messages are usually reserved for errors only, not regular metrics. Monitoring a system using syslog files can also be extremely labor intensive. You have to custom-configure the management systems to parse the log messages for each data point you’re trying to monitor. Syslog produces a never-ending stream of textual messages with little order or structure.
TIP You should only monitor metrics using the information from syslog if the information isn’t available through SNMP or a web service interface.
SNMP allows you to get a large amount of information from a particular device or subsystem. Just be selective about what you choose to monitor. Focus on the particular use or application of a device or subsystem. Make sure you monitor data that is directly applicable to the user’s needs. For example, if you’re going to manage power consumption using an IP-addressable power management device, you will want to get the power consumption data for the device. However, you may not want to retrieve the power consumption value for each outlet. SNMP can provide you with as much or as little data as you want. Getting too much granular information might overwhelm your team. Where possible, stick to monitoring the big picture.
NOTE The best way to determine what information to monitor via SNMP is to refer to the documentation for the component or to use a MIB browser to explore the available data.
Some devices offer customized interfaces, either via web services or proprietary protocols. Either would require custom integration with your remote management and monitoring system. The goal of web services is to have an open data layer that allows various systems to retrieve information, but they take some custom programming to use. Proprietary protocols are not designed for interoperability. Integrating a proprietary communication protocol into a centralized management system requires cooperation from the vendor. Some vendors are very reluctant to allow third-party integration, but they may offer a product that collects the data and repackages it in a standard format.
NOTE With either web services or a custom communications protocol, it is very important to do a careful study of the data and metrics available through the interface. Integrating these systems into a management system will likely require additional software development. Perform a cost-benefit analysis before proceeding.
Integrate Devices
Once you’ve determined what you wish to monitor and how to retrieve the required data, you must understand how devices and subsystems integrate into management systems. There are actually two things to figure out at this point: how to add devices and subsystems to the system and how to integrate monitoring metrics. In other words, you need to figure out how you want to add components to your inventory of devices under management and then decide how you want to add individual objects and metrics from those components to your list of data points.
Most remote management and monitoring solutions include a discovery process that adds network-connected devices and subsystems to the system inventory automatically. Usually, this means you can input a range of IP addresses and the system searches for devices within that range. When it encounters a device, the system tries to discover what it is and adds it to the inventory. Most systems allow you to do this by IP address range, subnet, VLAN, or individual address. You can also manually add a device or subsystem by inputting its details into the management system and testing the connection to the device.
TIP Before you decide to go with a particular solution for remote management and monitoring, examine how the solution manages inventory. Will its strategy fit with your architecture?
Determine Data Retention and Redundancy Requirements
In the process of integrating devices and determining monitoring requirements, you need to consider data retention policies. Ask yourself or your client: What level of information do I need to log and keep? How long do I need to keep it? What forms of backup are acceptable?
It is unrealistic (and arguably impossible) to permanently store all the monitoring and logging data from a fully configured remote management and monitoring system. However, there may be legal reasons to keep some level of detail for long periods of time. The best way to approach this issue is to differentiate between “all data” and “critical data.” As a general rule of thumb, your management system should be able to store “all data” in its database for at least six months—a realistic period of time. You should also be able to export all data so you can archive it outside the system for emergency purposes.
Next, identify your “critical data,” whether it’s a summary uptime report, user activity log, or any number of metrics. Your remote management and monitoring architecture should allow you to store this critical data indefinitely. Keep in mind that what is considered “critical” is up to you and your customer. It could be that you determine a particular time frame as critical, such as a daily report, as opposed to a second-by-second log. Many remote management and monitoring systems offer the ability to store daily information indefinitely, but real-time, instantaneous information for a defined period.
There are several ways to store and back up management and monitoring data and logs. Ask yourself the following questions:
• How long will I need access to user activity?
• How long do I want to store device-monitoring data?
• How far back do my summary reports need to go?
• What happens if my management system crashes?
• How accessible is my backup?
From there, it will help if you understand the various methods for storing and backing up management information.
• Databases Every software solution for remote management is built on a database back end that stores data dynamically so the user interface can access it quickly and easily. Systems vary in the level of detail they store in their databases as well as how they store it over time. When choosing a solution, find out how long it stores the raw data that comes in from devices.
•Aggregation Aggregation refers to creating summarized data, based on raw data, and storing it in separate tables that can be retained for longer periods. For example, a management system may receive raw data on a second-by-second basis, but it may only store that data for one month. It can then aggregate the data into a daily value, such as “average bandwidth utilization per day,” and store that for a period of several years.
• Exports Many management systems allow you to manually export data or set up automated exports. There are three common types of exports that you should consider when looking at data retention options: CSV (spreadsheet), PDF (protected document), and SQL (database dump). A CSV export basically takes a predefined data set and saves it to a text file in comma-separated values, meaning it can be opened by most spreadsheet applications. A PDF export takes a particular report or display and saves it as a protected, noneditable document, which is usually used for saving summary reports or user activity logs. An SQL data dump is simply a way for you to save the contents of a database and store it somewhere else, so that you could restore it if needed. Saving PDF documents takes a lot less disk space than saving an entire database, but you’re stuck with only the information contained in that particular report; whereas, in the case of a complete database dump, you have the flexibility to run various reports. These are all trade-offs to consider when designing the data retention portion of your system.
• Disk Images Disk images are byte-by-byte copies of an entire system’s storage disk that allow you to clone a particular server. This can be useful for several purposes, including crash recovery, off-site system backup, and even data mining. A standard practice is to schedule monthly disk image backups whereby the entire management system is cloned and the image is stored on a machine built for mass storage and reliability. You could even store these disk images in the cloud with an online storage provider, if you trust the provider’s security systems.
TIP If your data retention policy calls for storing disk images for an extended period of time, consider storing them with the equipment that can read the storage format. For example, older disk images used to be stored on floppy drives. Most modern computers no longer have floppy drives. If a computer with a floppy drive were stored with the disk image, you would retain the ability to view the old disk images. Be sure to store redundant reading equipment as well, in case the original equipment breaks.
Implications of Cloud-Based Monitoring
When designing your remote management and monitoring architecture, you may run across solutions that use the cloud to monitor devices in your installations. If the cloud concept is still nebulous to you, it simply means that computing resources or services are hosted somewhere else—on a service provider’s infrastructure—and you can lease them over the Internet for a regular fee. In the case of cloud-based remote management and monitoring, this is usually accomplished by installing an on-site device that gathers information from the various network-connected devices and then retransmits that information to a server on the Internet. Just be sure to understand the benefits and security risks of cloud-based monitoring and management.
A cloud-based monitoring solution cuts down on the amount of system management you have to do on-site. The cloud service provider does it for you. If you’re comfortable entrusting access to your customer’s network to a third party, then this approach can nearly eliminate your up-front costs and drastically reduce ongoing platform maintenance. However, if this approach does not include an ability to manage data on-site, then intermittent Internet outages will weaken your remote management and monitoring architecture because data won’t always be able to reach the cloud.
Cloud-based services offer simplicity in that you go to one online portal—usually the service’s website—log in, and access all of your (or your customer’s) systems from there. Again, ease of use is a big advantage, but if the provider’s cloud server happens to go offline for whatever reason (think news reports of Google Mail going down), you can’t do the remote management and monitoring you need to do. When evaluating cloud-based solutions, ask providers for policy statements on how they ensure uptime and what they do in the event of a failure. Communicate these factors to your customers, as ultimately it may be their data being stored in the cloud.
Most importantly, if you explore a cloud-based solution, you need to make sure the service provider goes to extra lengths to secure the information that it collects from the customer site. If you want the same level of detail as an on-site management system provides, then you have to transmit a very large amount of information over the Internet to the cloud server. This presents an opportunity for hackers to intercept the data. It must therefore be encrypted in transit and, ideally, “at rest” on the provider’s infrastructure. The good news is, as long as the data is secure in the cloud, you can access it on any mobile device, tablet, laptop, or other Internet-connected workstation, anywhere in the world.
Chapter Review
Networked AV systems, such as streaming, digital signage, and conferencing systems, take advantage of Ethernet and Internet Protocol technologies to improve an organization’s ability to communicate through audiovisual experiences. Remote management and monitoring systems use networks to fulfill a different function.
Once AV manufacturers began including Ethernet ports on their equipment, allowing them to speak IP the same way IT systems do, it created an opportunity for AV service providers and users to communicate with their gear over a network. This allows them to manage settings and monitor performance from afar, saving technology staffs time and resources. Such capabilities are important not only because they make the lives of AV technology users easier but also because AV companies can use the same network of devices to offer AV systems management as a service.
Now that you’ve completed this chapter, you should be able to
• Describe the types and benefits of remote management and monitoring systems
• Describe the ways remote management and monitoring systems communicate, including Simple Network Management Protocol
• Plan a remote management and monitoring architecture, including integrating devices, opening required network ports, and determining the data you want to collect
• Create an alarm notification and response strategy
Review Questions
1. A remote management and monitoring system that aggregates data from devices and device-management systems is called a ____ ____.
A. control system
B. smart building
C. manager of managers
D. subsystem
2. The most common communication protocol for IP-based remote management and monitoring systems is _______.
A. FTP
B. HTTP
C. HTTPS
D. SNMP
3. Why wouldn’t you use version 2 of the Simple Network Management Protocol for a system that will be installed at the Department of Education?
A. The Department of Education doesn’t use SNMP.
B. The Department of Education uses SNMP version 4.
C. Federal information processing standards forbid using SNMP.
D. Federal information processing standards don’t allow the transmitting of government data in plain text.
4. If you don’t know whether a networked device supports SNMP, check its ____ ____.
A. IP address
B. MAC address
C. management information base file
D. ports and protocols
5. There are two primary ways that information from networked devices gets into a remote management and monitoring system: ____ _____ and ____ _____.
A. polls/traps
B. alerts/alarms
C. Pings/echo requests
D. SMS/HTTPS
6. If a remote management and monitoring system must operate over the open Internet, make sure to employ ____ ____.
A. user authentication
B. encryption
C. strong passwords
D. high bandwidth
7. If a remote management and monitoring system will integrate syslog messages, which network port(s) should be open?
A. Port 80
B. Ports 161 and 162
C. Port 514
D. Port 443
Answers
1. C. A remote management and monitoring system that aggregates data from devices and device-management systems is called a manager of managers.
2. D. The most common communication protocol for IP-based remote management and monitoring systems is SNMP (Simple Network Management Protocol).
3. D. SNMPv2 sends information in clear text. Federal Information Processing Standards (FIPS) 140 forbids sending US government proprietary information via plain text.
4. C. If you don’t know whether a networked device supports SNMP, check its management information base (MIB) file.
5. A. There are two primary ways that information from networked devices gets into a remote management and monitoring system: polls and traps.
6. B. If a remote management and monitoring system must operate over the open Internet, make sure to employ encryption.
7. C. If a remote management and monitoring system will integrate syslog messages, make sure port 514 is open.