CHAPTER 1: IT MEASUREMENT IN CONTEXT
I think therefore I am. I measure therefore I act. Anonymous.
Why measure IT
Quite simply, if you want to understand, control or improve your IT, or communicate it to others, you need measurement.
If you want to know how good your IT is, both in terms of quality and value for money, you need to measure it. Then you’ll understand whether you can keep it as it is, or need to do something about it. Or at least you will, if you know what measurement ranges constitute good and not so good practice.
If you want to know if your IT is heading in the right direction, or is set to give you a nasty shock or a disappointed, sinking feeling, you need to find things to measure that will tell you so. That way, you’ll know whether you can relax or, on the other hand, need to take action.
It’s a bit like a car driver and the speedometer: too fast and you need to brake, too slow and you need to accelerate. Like the speedometer, IT measures are context-specific. What is ‘in the right ballpark’ on a dual carriageway may be quite inappropriate for a winding country lane; IT performance as measured in a large steady state operation may be hugely over-ambitious for a not-yet-stable new operation.
Client perspectives
The key stakeholders on the client side will want measures that show if they are getting what they need and what they are paying for. If not, remedial action will be needed.
Let’s start with who pays: the business overseen by the Board. It doesn’t matter if it has in-house IT or if it is externally sourced. At the very least, it’ll want to:
- know it’s getting what it’s asked for;
- check what it’s asked for works for the business and its users;
- be satisfied it’s getting value for money;
- know things not working for it are fixed quickly in line with the needs of the business and to know necessary changes are managed through effectively.
To give it the governance it needs requires measurement.
The users, whose work may be completely dependent on IT, will want to:
- have a service that is ‘just there’, working reliably and as required;
- have problems dealt with quickly and efficiently;
- feel the provider is responsive and communicative;
- have their say on what is and isn’t good about the service – and to be confident something will be done about their feedback.
Provider perspectives
It’s a pretty good idea for the IT provider to understand and work to the business’s success criteria for its IT. If the IT provider is serving a lot of businesses it still needs to have measures that align with how these businesses judge their IT as successfully supporting them.
The IT provider will need much more, though. Here is a selection of essentials, which, however, will not be sufficient for every provider’s needs:
- Are we complying with the contractual or agreed terms?
- What do the users think? Do they have niggles with our performance (which we need to address)?
- How is our productivity? How do our unit costs compare with others’? (If we’re lagging our competitiveness is at risk.)
- How good is our problem and change management performance?
- Do our projects deliver to spec, time and budget? If not do we know why and do we know what to do to put things right?
Contracts and agreements
What the client expects of the provider is normally set out in a contract or service level agreement (SLA). Measurement is used to assess and demonstrate achievement of the required terms of the contract or agreement.
Providers should aim to comply with SLAs. Applied ‘within the family’, they are used as a basis for assessing how things are going and for deciding what and how to improve.
Contracts mean more than that. They apply when the client and provider are in different organisations, albeit sometimes with the same parent organisation. Failure to comply with a contract will have consequences, possibly of a financial nature, so a lot of effort is usually deployed to stay on the right side of compliance!
How often should you measure IT?
Operational services like IT need to be continually managed and that means measurement, otherwise there is an ever-present danger of them falling into disrepair. Measurement provides an objective basis for management action.
Some things need daily attention, some even hourly or by-the-minute. A business-critical incident may need continual management attention, aided by measurements and assessments, just as, for example, a skid needs to be managed from start to hopefully accident-free end, with the aid of the speedometer, the steering wheel and eyes looking out the window.
Other things will need to be re-assessed less frequently, but, nevertheless, regularly. Thus, although a problem causing the service to be unavailable will have to be addressed there and then, a longer-term perspective on availability is needed to make sure required service levels are met and to enable weaknesses to be addressed. So, service availability may be measured on 13 consecutive periods, each of 4 weeks, over the course of the year, or on rolling, overlapping periods, e.g. 13 weeks (quarter of a year) rolled forward every 4 weeks.
Thus, frequency depends on what you’re measuring and why. Frequency, like the measures themselves, is context-specific.
Who should measure? For whom?
Everybody is responsible so nobody is responsible. Anonymous.
The IT provider will invariably be responsible for giving the client business the bulk of measurements it requires about its IT, though it may choose to do spot checks itself and to arrange for its own customer satisfaction surveys to be carried out.
The IT provider’s senior management will need measurements from the teams responsible for particular aspects of the service or from nominated individuals with specific responsibility for obtaining and reporting on the information needed. For example, the service desk manager might be responsible for reporting on incident-handling performance; alternatively, a single manager could be responsible for both incident and problem management, and so be charged with reporting on both.
What do you do with the measurements?
The question is unknown, but the answer is 42: loosely taken from The Hitchhiker’s Guide to the Galaxy (Douglas Adams, 1979).
When you measure, it’s a good idea if you know what to do with the measurement. If your speedometer says 35 mph and the speed limit is 30 mph, you’ll want to slow down. If you’re buying curtains, you’ll need to know some window measurements to help you decide how much material to order.
Measurement is used for control, improvement and communication. In many cases, there will be a threshold beyond which satisfactory turns into unsatisfactory, for example, expenditure up to your income level may be satisfactory, but above your income level, unsatisfactory, with remedial action needed. In other cases, present performance may be used as a starting point for future improvement.
Setting norms or target values is a skill based on experience and also, in many cases, comparison with others or with your own past performance.
Below are two examples of target setting:
- IT service levels in contracts and SLAs are set to make sure the needs of the business are met, but may involve trade-offs as to how much the client organisation is prepared to pay to secure incremental improvements. When SLA targets are comfortably being exceeded, the client might ask for them to be tightened up, especially with an in-house provider.
- Unit costs are often compared with industry norms, with these, in effect, becoming a target to be met or exceeded.
The actual measured values can be used, preferably in colour and graphic form, to show how well we’re doing, whether we’re within target/tolerance and our direction of travel. There is an example later in the book (see Figure 4).
A word about terminology
In this book, when we use the term ‘measure’ as a noun, it is to refer both to things that are measured and to the measured value. By analogy with a tabletop, you can measure height, length and width, so height, length and width are useful measures for any tabletop. Their values for a particular tabletop might be: 75 cm, 140 cm and 70 cm respectively; so for that tabletop, the measures are height 75 cm, length 140 cm and width 70 cm.
‘Measure’ can also be used as a verb to mean performing the act of measuring and we will occasionally use it in that sense.
We will avoid the use of the word ‘measure’ to denote an action (or the performing of an action) that is not specifically related to measurement. So for example, we will use ‘remedial actions’ rather than ‘remedial measures’.
A ‘metric’ is often likened to a measurement scale, so in this case the tabletop metrics are height, length and width measured in centimetres and the value of the metrics for this particular table are 75, 140 and 70. The term is often used synonymously with the use of ‘measure’ as a noun as described above; where we refer to metrics in this book, that is what we mean.
Do measures always have to be ‘numeric’ (quantitative)? In this book, the answer is no. We will use the term ‘measure’ to include assessments that are not necessarily numeric: for example, customer satisfaction may be rated on a five-point scale such as: very satisfied, satisfied, neither satisfied nor dissatisfied, dissatisfied and very dissatisfied, although it can help if they can be presented numerically. So in the example, a measurement system using 5, 4, 3, 2 and 1 may be more useful, especially for graphically presenting and manipulating management information on customer satisfaction.
Inputs, outputs and outcomes
IT measurement applies to inputs, outputs and, to a greater or lesser extent, outcomes.
Any business activity takes up resources, undertakes processes and generates outputs, which may in turn make possible or make easier the achievement of business outcomes. A supermarket:
- employs people and buys products and services (its inputs);
- provides products and customer-oriented services in its stores and online (its outputs);
and thereby:
- makes such-and-such profit and achieves such-and-such market share (its outcomes).
The organisation’s senior management needs to keep its eye on the inputs, outputs and outcomes:
- inputs need to be kept to the minimum necessary to achieve business objectives, taking a longer-term strategic view wherever appropriate;
- outputs need to be good and innovative enough to keep customers happy and to drive achievement of desired outcomes, again taking an appropriately strategic perspective.
A client business will want to ensure that the time, effort and, especially, money it spends on IT are no higher than industry norms, unless there is a staggeringly good reason why, and that the IT that it receives meets business and user needs and contributes to business success.
An IT provider will want to ensure its resourcing is as efficient as possible, its client charges are competitive and its offerings are well received by clients. The provider will want its services to enable the provider’s own success; at the same time, it will want itself to be perceived as contributing effectively to its clients’ business success.
The provider’s task of measuring its IT is more onerous than the clients’: the provider has to have an eye on all its clients’ needs, which means measurement, as well as on day-to-day, medium-term and strategic IT performance. We will come back to these matters later.
Relation to key performance indicators
Key performance indicators (KPIs) are measures used in the governance of organisations and units to provide managers and external stakeholders with essential (key) information on the performance of the organisation.
KPIs are chosen to reflect what is important to the organisation, for example, for reasons of assessment, control, improvement or communications. Each KPI has a target, the achievement of which shows whether performance against that indicator can be deemed successful. For example, the performance of an IT department in successfully resolving incidents could be judged by whether it does so on time for at least 95 per cent of the incidents, so the KPI in this case is 95 per cent on-time closure. What constitutes ‘on time’ would be set to reflect client needs, with incidents of differing seriousness having different time-to-resolve targets against which on-time performance would be judged.
When you measure your IT, you need to measure your KPIs for IT. There will probably be other things you want to measure, too, as you will read later in this book.
KPIs are not the same as organisational goals or objectives, but their values may indicate how to be more successful at meeting these goals either now or in the future. A goal for an insurance company might be a 10 per cent increase in business, to which achievement of its IT-related KPIs (such as 99.8 per cent customer availability) might contribute.
Relation to health checks
If you go to the doctor for a health check, the doctor will use measurements, e.g. temperature, heart rate and blood pressure. The doctor may also check for the existence of certain symptoms and may check your records to see if your body is missing some of the features that you’d expect to have.
Health checks such as benchmarks, IT capability maturity assessments and certifications to quality management standards are used to assess organisations’ achievement against defined performance levels and, in the latter two cases, the maturity and repeatability of their processes.
In other words, health checks are measurements of particular aspects of performance, against definitions of ‘what good looks like’. Are you super-fit and living healthily?
Relation to auditing
Auditing is used to check for compliance with standards and procedures. It is also used to help detect inadequate governance, fraud and corruption.
Auditing involves inspection of records and interviewing of people. It can involve scrutiny of things that would get measured anyway such as expenditure. It can also involve scrutiny of things that are measured specifically for auditing purposes, such as the percentage of software on the system that is correctly licensed.
Unexpected measurements can be an indication of non-compliance or worse. For example, if an inspection of the hardware and software on the system does not match the inventory, it may indicate that there have been unauthorised purchases or some other irregularity.
The trending of audit measurements, such as the percentage of correctly licensed software over time, can be a useful means of showing whether probity and compliance are improving over time. Action can be taken to address areas where improvement is needed.
Compliance and conformance
As mentioned above, you can use measurement to check the extent of compliance with standards and procedures. You can also check compliance with contracts and SLAs.
Don’t be tempted to measure compliance with frameworks like ITIL®, though. Frameworks aren’t generally defined tightly enough to allow you to say you’re complying with them and have measurements to demonstrate compliance. You can set up your procedures in conformity with a framework like ITIL, but it’s the procedures with which you comply, not the framework, and so it’s compliance with the procedures that you can measure; it’s against the procedures that you can audit.
Warning! Measure what is important
Whether you work for the IT provider or the client business, you need to measure (or assess) what is important to your organisation. What is important isn’t always easy to measure; conversely, what is easy to measure isn’t always important. Always bear in mind that measurement is a means to an end – effective or better IT, in this case – and never an end in itself. How comprehensively we can measure doesn’t automatically lead to business success; indeed too much measurement would be a waste of effort.
What you measure really needs the blessing of appropriately senior, responsible management in your organisation. These are the people who will be in a position to act on what the measurements are telling them. They should have enough knowledge and wisdom to steer against measuring the wrong things, or measuring them in isolation, and so optimising the wrong things. For example, an isolated measure on the time to close incidents without dealing with the time it takes to fix the underlying cause may drive perverse behaviour. Thus, an IT department could close an incident stemming from a recurring fault by providing an awkward circumvention and so make the incident closure statistics look good, but annoy its customers by not sorting out the underlying cause. The business and the IT provider should want IT optimised for their respective and common needs; they don’t, as a rule, want isolated aspects of IT provision optimised in their own right.
You will need to involve relevant stakeholders in deciding what to suggest to senior management is measured – unless senior management tells you, in which case it should have consulted the stakeholders! It is recognised good practice to use workshops to decide on an organisation’s measurements. These workshops should be led by managers with enough authority to make decisions that stick and by people competent in steering sensible decisions on measurement.
Beware blips
If you measured how you feel every hour of every day, you might see a lot of ups and downs. You’d be unwise to seek psychiatric help just because you felt miserable for half an hour because you needed a cup of coffee. But if you felt miserable every day for six months, you might want to get some help. So how often you measure depends on what you’re measuring. And it needs judgement to decide how quickly you need to react to what you are seeing. If you’re responsible for buying umbrellas for a department store, you wouldn’t do it on the strength of an isolated light shower, but rather you’d base your decision on historical statistics and, perhaps, longer-range weather forecasts; on the other hand, if you had umbrellas in the storeroom and a shower started, you’d put them out for sale.
Keep it proportionate
It’s a statement of the obvious that you should not spend more on measuring than you’ll realise in benefits as a result of the measurement. In fact, with IT skills at a premium, you probably want to spend a whole lot less on measurement than you’d expect to gain from it. Don’t make an industry out of it! You can always come back and look at things in more detail if the need arises, particularly if you’ve planned your approach to measurement with flexibility in mind.
Figure 1: Keep it proportionate!
Put measurement in safe hands
Measurement is a powerful means of assessing and presenting the truth about things that are measurable. Its reliability depends on the quality of the tools used and of the individuals using them.
Be wary not just of mistakes and competence shortfalls, but also of measurements that are deliberately falsified or embellished. For example:
- a provider could try to suggest it’s complying better with the contract than it really is;
- a health check could be skewed to show the unit concerned is in a better state than it really is.
So don’t entrust measurement to individuals whose work is never checked. If in doubt about a set of measurements, ask for proof.