CHAPTER 2: USING METRICS

There are many ways to utilize metrics and get value from them and no one method is the best. The methods used to develop and implement metrics should be adapted to fit your needs and situation. Developing, implementing, and managing metrics goes well beyond simply purchasing a monitoring tool and collecting measurements. From our standpoint, metrics help us understand:

•   Customers and their behaviors
•   Industries and how they work
•   The current environment (e.g. resources, budgets, locations, etc.)
•   The infrastructure supporting services
•   The value of the services provided
•   Cost-justification for the service (comparing cost of the service to the benefits gained).

Therefore, a methodical process/procedure is required to understand these concepts in a concise and accurate manner that either provides or illustrates benefits to stakeholder groups.

This chapter will review areas that are important to consider when applying metrics. Use the considerations in this chapter to build your own methodology for developing and using metrics.

Understanding metrics

Be prudent when choosing the measurements and metrics as this can become unmanageable very quickly. This can lead to metrics becoming the invisible monster behind the scenes. We have a number of customers that have 10 or more tools to monitor and measure the environment; many tools with duplicate functionality and most of which are not utilized to the fullest extent. Instead, they have several tools utilizing partial functionality to monitor limited aspects of the infrastructure and services. This can be one of the more wasteful facets of IT; think about this waste:

•   Limited use of very expensive tools
•   A host of servers to run the tools
•   Storage to collect and manage the data
•   Agents distributed across the infrastructure
•   Network bandwidth utilization
•   Personnel (installation, maintenance, training, etc.) to manage all of the above.

The costs of underutilized or wasted resources add up very quickly and yet this is not always considered during design. Metrics are often considered during the ‘service transition’ phase which can lead to impulse decisions such as quick tool purchases to fill perceived holes.

When developing metrics the proper theme should be ‘early and often’.

•   Begin developing metrics early in the design phase to ensure all aspects of the design are measureable and what metrics can be collected.
•   Review the metrics often and make sure they will continue to provide value through transition. This will accommodate any changes in build or design found during transition. This will continue through the lifecycle and throughout the life of the service.

Figure 1 Metrics influence

Metrics can and should have an influence in the daily activities of the service provider. Figure 1 demonstrates the sphere of influence metrics should have. Metrics are the centerpiece to ongoing successful delivery of services. Good decisions are made using experience gained from lessons learned and the measurements and metrics available. Those decisions form the measureable activities, and actions taken, to carry out the decision. Measurements from the activities and actions are collected and stored during execution. The activities and actions produce the measureable products or outcomes. The measurements and metrics are then used to:

•   Improve the next set of decisions
•   Improve the activities and actions increasing efficiency
•   Ensure the products meet or exceed the requirements of the customers.

Defining metrics

Before moving forward on any endeavor, it will benefit all stakeholders and participants to develop commonality in communication. This includes creating a lexicon to help everyone stay in the same mindset when discussing certain topics. Establish consistency in your terminology to prevent confusion and misunderstanding. Even simple wording can cause turmoil within a group or project. This is the same when defining measurements and metrics. Therefore, we suggest creating your own definition for these basic terms. Go online and take a look at how many definitions there are for words we assume everyone understands and see how confusion can grow so easily.

With that said, we have provided a few basic definitions to start you down the metrics path. There will be several other terms that will require definition but the terms below are just to get things rolling.

•   A measure is a quantifiable expression of performance collected during the execution of activities.
•   A metric is a specific measurement or calculation associated with performance. Metrics are applied as references for assessing variance as compared to a defined target. Some metrics may reflect Key Performance Indicators (KPIs).
•   A baseline is a known state or performance level that is used as a reference for subsequent measurements. It consists of a set of well characterized and understood measurements including all the phases of a process and the results. Baselines provide:
  •   A comparison between the ‘as is’ and ‘to be’ states
  •   Used for trending performance over time
  •   Can help set a benchmark (achievement target)
  •   Allows predictive modeling (‘what if?’ scenarios)
  •   A fall back point for change.

Keep terminology and definitions simple and concise to ease the learning curve and to improve communication. As new terms do come up, take a few minutes to stop and gain consensus on the meaning and definition of these terms. Then document them and move on.

Why do we need metrics?

Good, quantifiable metrics provide a wealth of knowledge for support and management of processes and for the delivery of services to the customers. They help us to recognize the value of our processes and services and the benefits gained by the organization when using these services. Benefits of metrics include:

•   Establishing baselines
•   Presenting improvement opportunities
•   Providing an understanding of current performance
•   Eliminating assumptions
•   Decreasing waste
•   Managing costs.

This list can be extended once we find additional benefits in metrics. The bottom line is that metrics should be a constant staple for all decision making.

Reliable metrics give stakeholders confidence in the use and performance of their processes and services. Metrics provide an understanding of current issues, pain points, areas of concern, and areas of success.

Methodology for using metrics

To create consistency across the organization, we recommend developing a framework to generate and implement metrics. Whether it is called a ‘process’ or ‘procedure’ doesn’t really matter, the priority is maintaining consistency in the creation of metrics. Figure 2 illustrates a sample framework to help get started with a methodology that will fit your organization’s requirements. Gather requirements from business owners and from the service provider to understand all aspects of service provision. As part of the requirement, make sure AQLs are also collected. This will provide an understanding of the expectations which will eventually become the performance targets documented in the Service Level Agreement (SLA). Ensure all parties sign off on the requirements before continuing. The following points provide details for each step in the example framework found in Figure 2:

1. Based on the requirements, determine the metrics needed to provide performance information for the service, processes, outcomes, and the infrastructure. At this point, ensure the metric development activities are included in the Service Acceptance Criteria (SAC) which can be used as a quality checklist through the service lifecycle. These first two steps should occur early in your service lifecycle.
2. Once the metrics list is complete, distribute it to the business owners and members of the service provider to validate that the metrics will provide the necessary performance information. Take this through a formalized activity documenting:
   • All individuals who reviewed the metrics
   • All feedback
   • Actions items
   • Official approval (sign-off).
3. As part of design, review all appropriate tools to find if the capabilities to collect the metrics are available with the current toolset. There may be times when the current toolset does not have the functionality required to collect the measurements; if this occurs check if the vendor provides modules or upgrades that include your requirements. This will maintain the current toolset and manage the cost of an acquisition. This review should also consider if the current toolset does not provide the measurement functions required, in this case a tool acquisition may be part of the overall design.
4. Have a third party, usually an engineer, review the metrics and selected tools to verify that the measurements and metrics meet the requirements of the stakeholders. An engineer will provide the technical skills required to ensure the tools can collect the agreed upon metrics. This too should be a formalized activity following similar steps to Activity 3. Once this final review is complete, ensure all requirements for the metrics are included in the Service Design Package (SDP).
5. Implement the metrics through the ‘service transition’ phase to build, test, and deploy the metrics into production (service operations). As the service or process is built, all aspects of metrics collection must be included in the release. This will ensure the metrics are put through the same rigor of testing as the other parts of the release. This is critical if the tool requires agents to be deployed to other servers or through firewalls (based on the information passing through). These transition activities will reduce possible issues that could arise in the operational environment.

These steps provide areas of consideration when creating your metrics activities. As you develop your methodology make sure you follow the service lifecycle phases treating this new process as any other production process or service.

Figure 2 Metrics framework

You may notice that the approach defined in Figure 2 closely follows the Seven-Step Improvement process (found in ITIL^® v3 Continual Service Improvement (CSI)). This process provides a simple methodology for managing metrics and is a good starting point to create your own method. The organization’s vision and mission statements can be used to offer guidance to the activities and help ensure the measurements and metrics continue to provide value. The following is a quick recap of the Seven-Step Improvement process¹:

•   Based on the business goals and objectives, services being delivered, and the methods used for delivery, define what should be measured. What should be measured are process activities and outcomes that are considered important for management decision making.
•   Identify what actually can be measured with the tools employed, and performance data and information available in the knowledge base.
•   Gather the metrics into defined repositories and check their integrity.
•   Compile the metrics and apply them to the service delivery processes.
•   Analyze the measurement results for validity (i.e. verify they measure what was intended to be measured), reliability (i.e. verify the measurements reflect actual performance characteristics) and accuracy (i.e. ensure there are no errors in the data or the calculations).
•   Present the metrics results with summary conclusions indicated by the performance measurements.
•   Implement corrective actions, including revised metrics as necessary to provide valuable decision support to the business.

Purpose for metrics (metrics tree)

The variety of tools offered today provides extensive opportunities for collecting measurements. In fact, it is very easy to become overwhelmed with the amount of information available through these tools. Measurements and metrics require purpose; the organization must have a reason to collect and use measurements. This purpose can be found using the ‘metrics tree’ (Figure 3). The ‘metrics tree’ helps to establish measurement criteria and linkages to achieve success and provide purpose to your metrics. These criteria include:

•   Focus on the vision and mission — ensure the vision and mission statements are consistently addressed throughout all levels of the ‘metrics tree’ as these provide the strategic direction of the organization.
•   Maintain goals and objectives — maintain up-to-date goals and objectives as the organization continues to move forward and achieve successes. As goals and objectives are obtained, create new ones to keep the forward momentum.
•   Develop Critical Success Factors (CSFs) — CSFs are required for an activity or project to accomplish the goals and objectives defined.
•   Determine Key Performance Indicators (KPIs) —KPIs are quantifiable measurements in support of the CSFs.
•   Select quantifiable measurements and metrics (customer requirements, outcomes, operational) — selective measurements and metrics provide the data and information needed to provide feedback and responses to the higher levels of the metrics tree.
•   Identify intangible measurements — choose measurements and metrics that, while not monetary (tangible) by nature, provide insight into the softer aspects of a service or process. These can include stakeholder perception or comments, complaints, and compliments.

Figure 3 The metrics tree

Integrating metrics

By following the guidance above, you will begin to collect useful measurements and metrics from the tools that support your processes and services. While these individual measurements are important and have a purpose, in many cases a measurement doesn’t offer full value by itself. Measurements and metrics should be integrated and combined to provide a complete picture of the situation at hand. Bringing these together creates context for these metrics which helps tell the story of success for our processes and services through the information gained from our efforts.

To fully understand all aspects of service provision, multiple metrics will provide both the purpose and meaning for the activities in support of our deliverables. These metrics will provide the evidence of the achievement of requirements and successful delivery of the expected outcomes. The combination of metrics will also help us understand whether a single metric is delivering good or bad news. For example, if metric ‘A’ shows a percentage increase in service desk calls, how do we know what it means? We need other metrics to support and bring clarity to metric ‘A,’ thus telling a story.

Metrics reporting

The value and benefits of metrics are delivered via a reporting structure. Reports provide the metrics information to stakeholders in multiple formats to address their needs and present the information in a usable manner. Of course, the basic requirements for reports must be understood and are derived from questions such as:

•   Who is the audience?
•   What information do they need?
•   When do they need it?
•   How do they get it?
•   How will the report be used?
•   What format do they prefer?

These questions must be asked for every report to ensure we are delivering the right information to the right audience. Developing a report questionnaire or standard report request will allow consistent information to be collected concerning the above questions. A good reporting structure will have a tremendous impact on customer satisfaction and communication.

For the most part, we use two basic mediums of report delivery: electronic or hard copy (paper). Today’s preferred method is electronic as this provides greater flexibility in style, format, and usability. Electronic reports also offer an increased opportunity to measure report usage. These measurements include attributes such as:

•   Number of hits per site
•   Unique visitors
•   Time spent on the report site
•   Pages viewed per visit
•   Most popular reports
•   Most popular report pages.

The format of the report is also critical based on the diverse audience viewing the report. Again, electronic reporting can include basic reports, dashboards, drill-down reports, linked reports, etc., all of which increase the usage of reports by providing a more intuitive look and feel for the audience. This gives a more personal touch for each individual, allowing them to have basic customization features depending on the reporting tool.

Electronic reports also offer greater levels of information security and control. An issue we have seen is the number of individuals who have access to reporting tools to create ad hoc reports. While this is helpful to many people because of expediency, it also creates issues with security, standardization and communication (e.g., reports created and sent to anyone in and out of the organization). These uncontrolled reports (unknown number and audience) create an extremely large information and communication challenge. The organization can be sending mixed messages to their customers and stakeholders as one report may deliver ‘good’ news while another report on the same topic may deliver the opposite message. What message does that send, other than IT doesn’t know what they are doing? We are quite sure that is not the message we want to deliver! To alleviate or prevent this issue, we recommend the following considerations:

•   Standardize reporting (methodologies and formats)
•   Create a reporting group or team
•   Limit the number of reporting tools
•   Limit access to reporting tools (gain control)
•   Create a reporting service and put it in your service catalog.

These steps will provide high levels of consistency for all reports which will also create an auditable methodology for report management as well as improve aspects of knowledge management. Improvements in reporting communication and security will increase the perception of the service provider with the customers and stakeholders. And once again, this is a measureable approach.

Securing and protecting metrics

Metrics are vital to the organization’s success and therefore, should be handled as highly valued knowledge assets. With that said, metrics provide information, via reports, that can create incredible amounts of knowledge which should be part of the Service Knowledge Management System (SKMS). Ultimately, as a service provider your metrics are part of your intellectual property.

As such, the repositories containing the measurements and metrics should be managed and tracked as Configuration Items (CIs); thus following the same processes and procedures used to manage the important components required to deliver services. This will ensure that these repositories are:

•   Part of the SKMS
•   Secured with limited write and update access
•   Backed up regularly
•   Stored on network storage devices
•   Part of the contingency plan
•   Audited regularly to ensure accuracy.

These steps will promote confidence in the data and information contained in the metrics repositories when making decisions or taking action.

Calculations

Many of the metric calculations found in this book are based on simplified formulas. While workable, these are not intended to be the ‘end-all, be-all’ calculations. As you mature your use of metrics, the formulas will evolve as additional measurements, new calculations, new/updated tools, or changes in technology all impact the methods used to calculate the metrics.

Metrics are normally calculated and presented in various forms such as numbers, ratios, percentages, and averages. These cover the majority of metric calculations in the industry and in this book. Flexibility is a key attribute to metrics with the ongoing changes in both business and technology that will require continued monitoring and review of the metrics for relevance. If you are using tools to collect measurements and calculate the metrics (the better solution in many cases), flexibility will be gained in the updates and upgrades provided by the suppliers. So, again, be creative and find ways to improve your measurements and metrics.

Continual Service Improvement (CSI)

All metrics have a role in CSI as they provide objective evidence of process or service improvement. Therefore, we do not have a CSI metrics chapter in this book. That, of course, doesn’t mean that there are no CSI specific measurements or metrics. If you decide to develop CSI metrics, a good starting point would be to include CSI activities in all processes. As part of a process, a CSI activity will become inherent to the normal process flow providing regular opportunities to consistently review the process execution. This will create CSI measurements every time the process is executed.

¹ Cabinet Office (UK), ITIL^®, Continual Service Improvement (London, England, The Stationery Office (TSO), 2011) 49-64