Chapter 16

Measuring IT Performance

In This Chapter

Appreciating the growing importance of IT and the need to measure it

Metric for measuring IT service delivery

Monitoring IT projects effectively

There are few businesses today that are not dependent on IT in some form or another. Whether from the simple desk top computer that allows employees to do their job through to complex networks and bespoke technology that are paramount to business success.

If your business depends heavily on IT or you need IT systems to deliver your product or service to your customer effectively and efficiently then you would be wise to measure IT performance through the adoption of some of the IT-focused KPIs detailed in this chapter.

Why IT Matters More Than Ever

Technology has significantly assisted business success. Now you can

• Easily and cheaply communicate with suppliers and customers.
• Sell online without a physical presence on the high street.
• Venture into new markets unhindered by geographical location.
• Reduce costs and become much more efficient.

But these advantages do come at a price.

For example most businesses today handle and store personal data. In days gone by, that data would be held in physical ledgers or in some paper filing system. Now everything is stored electronically, and as such it is vulnerable to attack and misappropriation.

There have been many examples of valuable personal data going missing or being carelessly lost by employees. In November 2007 news broke in the UK that two computer discs holding personal information on 25 million British citizens had been lost in the mail. The data included names, addresses, National Insurance numbers and in some cases bank information. These types of IT failures and breaches of security are fairly common and can end up costing the business a lot of money not to mention the reputational damage and loss in customer confidence.

In addition there has also been a marked rise in cyber crime, including remote hackers shutting your system down and demanding a ransom. Hackers often target valuable personal data, including credit card details or bank details which are then sold on to undesirables seeking to exploit the data in some way. Add to that the constant threat of viruses, worms or malware that can cause havoc to an IT system, and you begin to see that the misuse of IT poses a serious threat even while IT itself provides significant opportunities.

One survey by PricewaterhouseCoopers (PwC) found that 92 per cent of firms with more than 250 employees, and 83 per cent of smaller firms, with up to 25 employees, stated they had experienced a security incident in the last year – more than double in just 24 months. According to PwC a security breech in a large firm can cost anywhere between £280,000 and £690,000.

Clearly IT matters more than ever.

Measuring IT service delivery

Obviously, it’s important for customer satisfaction and repeat purchase that you provide good service across your business including IT service.

For example, most IT functions have a help desk that provides both internal employees and external customers with technical support in some form or another. When users call the help desk or send an email they would like their query resolved quickly, so they can get back to whatever it was they were doing before the IT started to go wrong.

The benefits of measuring IT service delivery are clear. If you resolve a customer IT problem at first contact, you are improving operational efficiency, reducing costs, and improving user satisfaction all at the same time. If you don’t resolve the problem, it can impact productivity and cause on-going frustration. These unresolved issues usually lead to repeat calls, which in turn add direct and indirect cost to your help desk operations.

Research has indicated that 34 per cent of customers who don’t get their query or problem resolved quickly are likely to go to your competitor. In addition, 30 per cent of call centre operational costs can be attributed to not being able to resolve the query in the first contact, so it really is important to measure service delivery and get it right.

While it may be easier for you to assess product quality – because it’s often easier to define and measure it – you must also measure the quality of IT service delivery.

Defining and measuring service delivery

Understanding the quality of service you deliver to your customers is important when considering your operational efficiency.

The best way to define and measure service delivery is through the KPI known as SERVQUAL which was developed in the mid-1980s. The methodology is designed to measure quality of service by comparing the service expectations a customer has with the actual service experience.

SERVQUAL breaks service quality into the following dimensions, also often referred to as the acronym RATER:

• Reliability: Are your people able to perform the promised service dependably and accurately?
• Assurance: Do your people convey confidence and trust through their knowledge and courtesy?
• Tangibles: Do the appearance of physical facilities, equipment, personnel, communication materials and so on reassure your customers?
• Empathy: Do your people provide caring, individualised attention to customers?
• Responsiveness: Are your people willing to help and respond to your customers’ needs?

When it comes to measuring the quality of the service you provide SERVQUAL is particularly useful because it is both a measurement and management model.

The authors of the methodology also identified five gaps that can result in a perceived drop in service by customers. Often simply being aware of these gaps can provide valuable insights into service improvement.

The five gaps are:

• Gap between management perception and consumer expectation: This gap arises when people inside the business don’t fully appreciate or understand what the customer actually wants. For example, you may believe that your customers want cheaper product but what they really want is access to a real person when they call you business with an issue or query. This is why measuring your customer KPIs in part 4 are so important so you can be sure you cater to what your customers want and not what you assume they want.
• Gap between management perception and quality of service specification: This gap arises when management has not set an appropriate performance standard. Even if you know what your customers really want, you have to then lay out a benchmark for what constitutes quality service. For example, your customers may tell you that they want to get through to that real person quickly. You need to define what quickly means so that the people delivering the service know what they are aiming at. The performance standard may therefore be to answer the call within five rings and resolve at least 90 per cent of all calls in the first call.
• Gap between service quality specification and service delivery: This gap arises when a shortfall exists between the performance standard and actual performance. For example a gap would arise if your people were consistently answering calls later than five rings and were only able to resolve 10 per cent of the calls in the first conversation. This shortfall in performance may be caused by poor communication or training. If your people are not informed about why the performance standard is in place they may feel that it’s just some unnecessary arbitrary rule to make their life more complicated. If however they understand that it is what the customers have asked for they may be more inclined to get on board and meet those standards.
• Gap between service delivery and external communication: Marketing, advertising and what your people say to customers all have a profound effect on what they expect from your business. This gap arises when the promised level of service doesn’t materialize when it’s expected. For example if an insurance company advertises that in the case of a claim their customers are assigned a personal claim handler who will be their sole point of content and that doesn’t happen then the customer has every right to be frustrated by that gap in service.
• Gap between expected service and experienced service: This gap arises when there is a difference between the expected service and the service experience. Often this gap is caused by a combination of gap 1 and gap 4 and often represents a breakdown in communication.

Understanding the gaps between service expectation and actual service perception will provide companies with an insight into the customer service areas where they are not delivering the service levels customer were expecting.

Measuring service delivery in practice

The IT service delivery KPIs that are most useful are Service Quality (SERVQUAL) and Help Desk First Call Resolution (HD FCR).

KPI: Service Quality (SERVQUAL)

The key performance question SERVQUAL helps to answer is: ‘To what extent are we delivering service quality to our customers?’

SERVQUAL is established using two questionnaires of 22 questions which measure expectation and perception of the service on a scale of 1 to 7 (1 being strongly disagree and 7 being strongly agree). In addition, respondents rank the five dimensions of the RATER scale (Reliability, Assurance, Tangibles, Empathy, and Responsiveness) to identify relative importance. This is achieved by allocating 100 points across the five dimensions.

Finally you calculate the gap score for each dimension by subtracting the Expectation score from the Perception score. A negative gap score indicates that the actual service was less than what was expected. Obviously you are aiming for a neutral or positive gap score!

For example, say ABC Bank is keen to see how their customers perceive their quality of service. First they have to establish what customers expect from a bank. This is achieved through 22 expectation statements which ask customers to rank their expectations around what an excellent bank would look like. Customers are therefore asked to rate each of the statement from 1–7 for statement such as:

1. Excellent banking companies will have modern-looking equipment.
2. The physical facilities at excellent banks will be visually appealing.
3. Employees at excellent banks will be neat in their appearance.
4. Materials associated with the service (brochures or statements) will be visually appealing at an excellent bank.
5. When excellent banks promise to do something by a certain time, they do.
6. When a customer has a problem, excellent banks will show a sincere interest in solving it.
7. Excellent banks will perform the service right the first time.

The customers are then asked to complete a perception statement which relates to their perceptions of ABC bank in particular. Again they are asked to rate the statement from 1–7 for statements such as:

1. The bank has modern looking equipment.
2. The bank’s physical features are visually appealing.
3. The bank’s reception desk employees are neatly dressed.
4. Materials associated with the service (such as pamphlets or statements) are visually appealing at the bank.
5. When the bank promises to do something by a certain time, it does so.
6. When you have a problem, the bank shows a sincere interest in solving it.
7. The bank performs the service right the first time.

Feel free to adapt the SERVQUAL methodology and amend the dimensions and questions slightly to make it a better fit for your business

Domino’s Pizza modified the classic dimensions into the following six outcome items:

1. Domino’s has delicious home-delivery pizza.
2. Domino’s has nutritious home-delivery pizza.
3. Domino’s home-delivery pizza has flavourful sauce.
4. Domino’s provides a generous amount of toppings for its home-delivery pizza.
5. Domino’s home-delivery pizza is made with superior ingredients.
6. Domino’s prepared its home-delivery pizza crust exactly the way I like it.

KPI: First Call Resolution (FCR)

The key performance question FCR helps to answer is: ‘How effectively are we resolving our customer queries at first contact?’ Understanding help desk first call resolution will provide an insight into the effectiveness and efficiency of the IT help-desk operations as well as insights into the costs and user satisfaction levels with the IT support.

There are different ways to collect the necessary data in order to measure FCR. The easiest option is by comparing the number of IT calls (or emails) with the number of cases resolved at first contact. This data can normally be collected from the agent logs – where IT operators detail whether calls have resolved or escalated.

FCR call statistics are calculated as follows:

Total number of calls/Total number of resolved at first call × 100

For example say your IT help desk logged the following statistics over two months:

Based on Call Statistics the FCR for month 1 would be

(3890/4500) × 100 = 86.44 per cent

and for month 2

(5000/5400) × 100 = 92.59 per cent

Over-emphasis on FCR can sometimes distract from the fact that not all contacts add value. For example, if your user manual or technical information that accompanies your product is not clear enough you may get more calls than you should. The easiest way to solve this and avoid the unnecessary cost is to improve the clarity of the supporting documentation.

Measuring IT project performance

It is very common for IT related objectives to be delivered via projects such as new software design projects, hard ware or software implementation projects, or website development projects.

Unfortunately, IT projects have an especially poor track record for successful implementation. In fact, research indicates that up to 70 per cent of IT projects fail to deliver their objectives. According to the British Computer Society (BCS), only 16 per cent of IT projects – at best – can be considered truly successful.

Failure in the context of IT projects is the same as failure in any project. Either the project wasn’t delivered on schedule, wasn’t delivered on budget or didn’t deliver what it was supposed to deliver.

If you want to ensure that your IT projects are successful then you need to measure IT project performance using the classic project KPIs.

Using the classic three project KPIs

There are three classic project KPIs that are particularly useful for keeping your IT projects on track. They are:

• IT Project Schedule Variance (IT PSV)
• IT Project Cost Variance (IT PCV)
• IT Project Earned Value (IT EV)

IT project schedule variance measures actual progress against the agreed schedule to help ensure the IT project is delivered on time.

IT spending can be notoriously high as companies seek to invest in increasingly more sophisticated systems to automate processes as well as collect, store and analyse data. In fact one burgeoning area of IT expense is systems that promise a plethora of KPIs … all at the touch of a button.

Unfortunately the financial track record of IT projects is not much better than schedule with an estimated 1/3 of all IT projects in the private sector running between 10–20 per cent over budget. And it’s not better in the public sector – the implementation of Universal Credit, a single welfare payment to replace the multitude that currently exist in the UK is still on-going but it is already significantly over budget. It is also estimated that website projects typically end up costing 25 per cent more than the agreed budget. IT Project Cost variance allows you to measure whether an IT project has gone over budget and if so by how much.

For larger, longer-running IT projects however you really want to measure progress as it’s happening rather than retrospectively which is what ITPSV and IT PCV do.

IT EV is particularly useful because it looks at the cost of work in progress and allows you to understand how much work has been completed compared to how much was expected to a particular date.

In addition to assessing progress to date, the IT EV metric allows you to project what the likely costs of the complete project will be; assuming performance levels remain unchanged moving forward.

Measuring IT project performance KPIs in practice

The IT project performance KPIs that are most useful are IT Project Schedule Variance, IT Project Cost Variance and IT Project Earned Value.

KPI: IT Project Schedule Variance

The key performance question IT project schedule variance helps to answer is: To what extent are our IT projects delivered on schedule?

IT Project Schedule Variance (IT PSV) = Scheduled Completion Time (SCT) – Actual Completion Time (ACT)

ACT and SCT are measured in time intervals such as days or weeks.

For example say you are running three separate IT projects:

• IT Project A: SCT = 105; ACT = 129
• IT Project B: SCT = 25; ACT = 25
• IT Project C: SCT = 40; ACT = 35

IT PSV for each is as follows:

• IT PSV Project A = 105 – 129 = –24
• IT PSV Project B = 25 – 25 = 0
• IT PSV Project C = 40 – 35 = 5

If you want to calculate the total IT Project Schedule Variance (i.e. the schedule variance of all projects combined) you simply add the individual IT project variances for each separate IT project together for an actual number, or calculate a straight or weighted average variance score.

Total IT PSV = (–24) + (0) + (5) = –19

If IT Project Schedule Variance is zero then the project was completed on time, as promised. If the variance is negative it shows an overrun and if the variance is positive it highlights competition ahead of the planned completion date. Positive numbers may indicate poor planning or the IT project’s owner deliberate attempt to add too many contingency days into the project so they can look good when it’s finished ahead of time.

KPI: IT Project Cost Variance

The key performance question IT project cost variance helps to answer is: ‘To what extent are our IT projects delivered on budget?’

IT Project Cost Variance (IT PCV) = Scheduled Project Cost (SPC) – Actual Project Cost (APC)

For example, going back to the three separate projects:

• IT Project A: SPC = $800,000 APC = $950,000
• IT Project B: SPC = $150,000 APC = $152,000
• IT Project C: SPC = $350,000 APC = $300,000

The IT PCV for each project is

• Project A = $800,000 – $950,000 = -$150,000
• Project B = $150,000 – $152,000 = -$2,000
• Project C = $350,000 – $300,000 = $50,000

To calculate the overall IT Project Cost Variance simple add the individual project variances for every separate IT project together for an actual number or calculate a straight or weighted average variance score.

Total IT PCV = (–$150,000) + (–$2,000) + ($50,000) = –$102,000

If IT Project Cost Variance is zero then the project was completed on budget, as promised. If the variance is negative it shows an over spend and if the variance is positive it shows the project was delivered under budget. Like the IT PSV a positive numbers may indicate poor planning or the IT project’s owner deliberate attempt to add inflate the estimated budget so they can look good when it’s finished under budget.

KPI: IT Project Earned Value

The key performance question IT EV helps to answer is: ‘To what extent are our IT projects making the desired progress?’

IT Earned Value (IT EV) = Budgeted Cost of Work Performed (BCWP) × per cent complete

BCWP is the total budgeted costs for labour and resources for the project.

Performance Level = Actual Cost of Work Scheduled (ACWS)/EV

Actual Cost of Work Scheduled is the total amount in labour and resources that has been spent on the project to date.

For example, say you have initiated an IT Project and the BCWP is $200,000. So far the ACWP or what’s actually been spent so far is $145,000 but the project is only 30 per cent complete.

The IT Earned Value (IT EV) = $200,000 × 30 per cent = $60,000