Problems are a fact of life in all data centers. The sheer complexity of systems and diversity of services offered today all but guarantee that problems will occur. One of the many factors that separate world-class infrastructures from mediocre ones is how well they manage the variety and volume of problems they encounter.

This chapter discusses the entire scope of problem management. It begins as usual with a commonly accepted definition of this process followed by its interpretation and implications. Then we discuss the scope of problem management and show how it differs from change management and request management.

Next we look at the length the key steps required to develop a robust problem management process. Also included is a recent survey of infrastructure managers prioritizing their major issues with problem management in their own shops. The chapter concludes with a discussion on how to develop meaningful metrics for problem management. Just as important are some examples showing how one can easily perform trending analysis on these metrics.

Regardless of how well-designed its processes or how smooth-running its operations, even a world-class infrastructure will occasionally miss its targeted levels of service. The branch of systems management that deals with the handling of these occurrences is called problem management.

The identification of problems typically comes in the form of a trouble call from an end-user to a service desk, but problems may also be identified by programmers, analysts, or systems administrators. Problems are normally logged into a database for subsequent tracking, resolution, and analysis. The sophistication of the database and the depth of the analysis vary widely from shop to shop and are two key indicators of the relative robustness of a problem management process. We will discuss these two characteristics in more detail later in this chapter.

Several of my clients have struggled with distinguishing the three closely related processes of problem management, change management, and request management. While the initial input to these three processes may be similar, the methods for managing a problem, a change, or a request for service typically varies significantly from each other. As a result, the scope of what actually constitutes problem management also varies significantly from shop to shop. Many infrastructures do agree that first-level problem handling, commonly referred to as tier 1, is the minimum basis for problem management. Table 11-1 shows some of the more common variations to this scheme.

The most integrated approach is the last variation in the table, in which all three tiers of problem management are tightly coupled with change management and request management. Among other things, this means that all calls concerning problems, changes, and service requests go through a centralized help desk and are logged into the same type of database.

Problem, change, and request management are three infrastructure processes that are closely related but distinct. Changes sometimes cause problems or can be the result of a problem. Expanding a database that is running out of storage space may be a good, proactive change, but it may cause backup windows to extend into production time, resulting in a scheduling problem for operations.

The same is true of problems causing or being the result of changes. Request management is usually treated as a subset of problem management, but it applies to individuals requesting services or enhancements, such as a specialized keyboard, a file restore, an advanced-function mouse, extra copies of a report, or a larger monitor. Table 11-2 shows a general delineation of problem, change, and service requests.

To further delineate the differences between problem, change, and service requests, Table 11-3 provides examples of the three types of requests taken from actual infrastructure environments.

Some IT shops draw a distinction between the notion of an incident and that of a problem, and by extension, they draw a distinction between incident management and problem management. In these cases, the delineation usually follows the guidelines offered by the IT Infrastructure Library (ITIL), which was discussed in more detail in Chapter 6 “Comparison to ITIL Processes.”

In accordance with ITIL best practices, service desk operators initially treat all calls as incidents. After a brief assessment, the operator may determine that the incident does not reflect any kind of failure in the infrastructure but instead is a request of some kind by the user calling in. In this case, the incident falls into the category of a service request, similar to the criteria described in the previous section.

Service desk operators analyze and attempt to resolve those incidents that are not classified as service requests. If the operator is successful in resolving such an incident, it is considered resolved and is designated as closed. If the service desk operator is not able to resolve the incident, it is turned over to a second level, or tier 2, support group for resolution and is then designated as a problem. In short, some IT shops designate tier 1 issues as incidents and their tier 2 issues as problems. The process described in this chapter can apply to both categories of incidents and problems.

Most all IT organizations have some type of service desk to offer assistance to end-users. In the past, IT managers used various names to designate this function, including help desk, call center, trouble desk, or technical support. As IT evolved to become more customer-oriented, the name of this area changed to that of customer support or the customer service center. With the emphasis today on IT service management, many organizations have renamed their help desks to that of service desk.

The service desk is a function, not a process. A function is defined within a department on an organization chart, has strict organizational boundaries, and is associated with numerous personnel management issues. For example, the implementation of a service desk involves hiring staff, training them, evaluating their performance, offering them career paths and promotions, and rectifying unacceptable behavior. A process has none of these characteristics. Table 11-4 summarizes some of these generic differences between a function and a process.

Because a service desk is a function and not a process, it is not designated as one of the 12 systems management processes. But the service desk does play a very crucial role in problem management. It serves as the single point of contact into IT for all users and customers; also, it is responsible for accurately logging all calls that come into the service desk, classifying them, handling them as best as possible, and tracking them to completion. The relative effectiveness of a problem management process is directly related to the caliber of the individuals who staff the service desk. It is hard to overstate the significant role the service desk plays in the success of a problem management process.

As IT services grow in many companies, the number of service desks frequently grows. When PCs first came onto the scene, they were usually managed by a support group that was totally separate from the mainframe data center, including a separate service desk for PC-related problems. This pattern of multiple service desks was often repeated as networks grew in size, number, and complexity; the pattern was also prevalent when the Internet blossomed.

As IT organizations began recentralizing in the early and mid-1990s, infrastructure managers were faced with the decision of whether to segregate their service desks or integrate them. By this time, many IT shops had grown out of control because they used multiple call centers—companies with 10 or 15 IT service desks were not uncommon. If non-IT services were included in the mix, employees sometimes had as many as 40 or 50 service numbers to choose from for an inquiry. In one extreme example, a bank provided nearly 100 separate service desks for IT alone, having grown a new one each time an application was put into production.

Over time the benefits of integrating service desks gradually prevailed in most instances. This is not to say that segregated service desks are always worse than integrated ones. Highly diverse user populations, remote locations, and a wide range of services are some reasons why segregated service desks can sometimes be a better solution.

Table 11-5 summarizes the advantages and disadvantages of integrating services desks. Since the advantages and disadvantages of segregating service desks are almost the exact reverse of those for integrating them, the table is not repeated for segregated service desks.

An integrated service desk is not only easier for customers to use, it lends itself to the cross-training of staff. Furthermore, there is a cost savings because companies can standardize the call-tracking tool by collecting all call records on a single, centralized database. Even more is saved by utilizing staff more efficiently. Also, since there are fewer and more standardized help desks, management is easier and less complex. Finally, it is easier to integrate other system management disciplines into it, particularly change management.

The primary advantage of a segregated service desk is the ability to customize specialized support for diverse applications, customers, and services. For example, some consolidated service desks have attempted to offer fax, telephone, facilities, office moves, and other IT-related services only to find they were attempting to do too much with too few resources; they were providing too little service to too many users. In this case, multiple service desks tend to work better, although communication among all of them is critical to their combined success.

A compromise hybrid solution is sometimes used in which all IT customers call a single service desk number that activates a menu system. The customer is then routed to the appropriate section of a centralized service desk depending on the specific service requested.

The following 11 key steps are required to develop a robust problem management process. We then discuss each step in detail.

The opening and closing of problems are two of the most critical activities of problem management. Properly opened tickets can lead to quick closing of problems or to appropriately dispatched level 2 support groups for timely resolution. Infrastructures with especially robust problem management processes tie in their call-tracking tool with both an asset management database and a personnel database. The asset database allows call agents to view information about the exact configuration of a desktop and its problem history. The personnel database allows agents to learn the caller’s logistics and their recent call history.

Closing a problem should be separate and distinct from resolving it. A problem is normally said to be resolved when an IT service has been restored that had been interrupted. A problem is said to be closed when the customer sends confirmation in one form or another that the resolution activity was satisfactory and that the problem has not reoccurred. In general, a level 2 problem will be resolved by a level 2 support analyst, but it should be closed by the service-desk analyst who opened it.

Analyzing a variety of trending data can help to continually improve the problem management process. This data may concern the types of problems resolved, the average time it took to resolve, devices involved, customers involved, and root causes. Table 11-9 provides a sense of the variety and distribution of problems handled by a typical service desk. This data shows a recent representative month of IT problem calls to a service desk at a major motion picture studio.

Three items are noteworthy:

1. The inclusion of the accumulative percent field gives an immediate snapshot of the distribution of calls. One-third of the calls are the result of problems with PC software; over half the calls are a result of desktop software problems in general (comprised of both PC and Macintosh software).
2. The overwhelming majority of calls—92.3%—are desktop-related.
3. The recent inclusion of fax service requests is showing a healthy start—over 100 requests a month.

Summary snapshot data such as this can be fed into weekly and monthly trend reports to analyze trends, patterns, and relationships. This analysis can then be used to formulate appropriate action plans to address issues for continual improvement.

I conclude this chapter with the results from a recent survey of infrastructure mangers and senior analysts on what they believed were the greatest exposures with their current problem management processes. The respondents represented more than 40 client companies. They identified, in their own words, 27 separate issues. Table 11-10 lists these issues and their percentage of occurrences.

This chapter began in the usual manner with a formal definition of problem management, followed by a discussion of the differences between problem management, change management, and request management. Following this we showed how problem management differs from incident management and we described the important role that the service desk plays in the problem and incident management processes.

Next was the core of the chapter, in which we looked at the 11 key steps to developing a robust problem management process. These included preferred characteristics of a process owner, areas that should be represented on a cross-functional team, a sample of prioritized requirements, a representation of a problem priority scheme, and examples of service and process metrics.

We concluded by discussing proper methods to open and close problem tickets; advantages and disadvantages of integrated help desks; and candid results of a survey from over 40 infrastructure managers and analysts on what they believed were their major concerns with their own problem management processes.