CHAPTER 20
Peer Reviews and Formal Audits

It is so easy to get caught up in the daily business of keeping your maintenance team busy meeting the customer’s needs. Sometimes you just don’t have time to see the bigger picture. You may have started out the year modifying processes to gain efficiencies, but do you have the time to track the progress and make a comparison to last year’s? A scheduled review can help ensure that you do take the time.

A periodic review can provide powerful information to you. The reviews can be even more powerful if they are required for every maintenance team in your company’s IT department because they would encourage building comparisons and sharing best practices. While there may be a fear that something bad will be pointed out in a comparison, you can also look at this method as a technique to use to improve yourself and your team’s performance.

Yearly review of your maintenance team by a peer maintenance manager is recommended. The results should be provided to you, and you can decide what steps to take to improve. Your direct supervisor can help come up with action steps and follow up on the progress.

The reviewer should take on the perspective of the company’s chief information officer (CIO). For maintenance, CIOs are interested in “better” (improving quality and customer satisfaction) and “cheaper” (spending less this year than last year). The CIO is the one who has to ask if it is better to continue providing maintenance services in-house or to have them outsourced. Reviewers will be looking for ways to improve maintenance so that it can compete with other alternatives such as outsourced support.

What to Review

At a high level, the review is an evaluation of the quality of the service being provided to the customer, the effectiveness of the team, and the appropriateness of the associated costs. The review should assess what level of business value versus cost your team is delivering and probe to see if anything should be removed from maintenance or shut down. The review should point out any activities performed by the team that are of low value to the business, which might do better by eliminating some activities and spending the money more effectively elsewhere.

This review is different from customer feedback obtained from the customer. A peer IT manager should perform this review. The focus is to look for ways to improve the maintenance team’s effectiveness and efficiency.

Figure 20-1 presents a way of looking at what the review should look at. What is the balance between what was promised and what was delivered? Situations in which more was promised than was delivered should be noted in the review. The balance scale assumes there are adequate metrics to measure and track all the key items. If such metrics are unavailable, that should also be noted in the review.

Figure 20-1: Balance

Reviews should have two types of findings: (1) the items known to be deficiencies, and (2) the items that are unknown because they are not documented or there are no metrics with which to track them. The second type of finding may not be as readily discerned as known deficiencies.

The following is a suggested checklist for reviewing a maintenance team and can be adapted for your own specific circumstances.

Maintenance Review Checklist

Scope and Metrics

•   Review the Service Level Agreement (SLA) service deliverables. Were they delivered?

•   Is the SLA still valid or does it need to be updated?

•   Are metrics established to track performance specified in the SLA?

•   Were other commitments made to the customer?

•   Were these other commitments delivered?

•   Are additional systems, interfaces, or reports now included in maintenance but not documented?

Cost

•   What was the original budget?

•   Are the actual costs being tracked to the breakdown of the budget?

•   Are the actual costs in line with the approved budget? If not, is the variance explained?

•   Are there enough labor time “buckets” to determine where effort is being spent?

Team

•   Review the Coverage Matrix or its equivalent. Is there adequate team member coverage of everything being supported?

•   Is there at least one backup person covering the primary team members?

Customer

•   How often is the customer contacted?

•   Is the customer contacted at different levels?

•   Is a formal Customer Survey performed in person?

•   What issues does the customer have with maintenance? How were any customer issues addressed?

Formal Audits

The word “audit” can cause anxiety in some people. Their minds jump to an image of the Internal Revenue Service calling them in for a “friendly review” of their taxes. Internal audits in IT can create the same anxiety. But audits provide the organization with important feedback on compliance with internal procedures and other requirements. Non-compliance with requirements could place the team or the entire organization at risk.

Audits can provide insight that you might otherwise not have obtained. The audit process is more formal than the peer review. Professional auditors, not other IT managers, conduct audits. Their job is to find items that are not consistent, such as following procedures.

The following section presents three types of maintenance team audits.

Types of Audits

•   Financial Audits

A financial audit compares the maintenance team’s actual expenses to the approved budget. The audit also reviews the methods used to track and control expenses.

•   Security Audits

A security audit focuses on the security of the computer systems. This type of audit can help the maintenance team and the company determine if there are vulnerabilities that could be exploited by outside hackers. In many cases, a company specializing in computer security is retained to audit a system or systems. These companies invest their time in learning the latest hacking methods and operating system vulnerabilities. They use all known methods for breaching security to see if an intrusion is possible and if an intrusion is detected and logged. They then recommend appropriate corrective actions if their testing determines that a security problem exists.

•   General/Procedural Audit

A general/procedural audit takes a broad view of the maintenance team. It reviews the quality of existing maintenance team procedures and determines if the team has sufficiently documented that it follows the procedures. Sarbanes-Oxley audits are in this category.

Preparing for an Audit

Preparing for an audit should begin the day you take responsibility for the maintenance team. Auditors are looking for documented procedures, documented compliance with those procedures, and any risks that are not being addressed. In other words, the auditors are looking to see if the manager is doing an effective job. So the best way to prepare for an audit is to put in place many of the tools presented in this book so that you can perform your job effectively.

When an audit is scheduled, determine what kind of audit it will be. Then review the relevant documents and determine if you are in compliance.

If there is anything about your team’s operation that you know already is not quite right? You have a limited amount of time to fix any deficiencies before the audit takes place. Do what you can to remove the deficiencies before the audit.

When an audit is under way, assist with requests for information from the auditors and make sure that you understand what they are doing. It is reasonable for you to challenge the auditor if you have a strong opinion that the auditor is approaching a subject incorrectly.

Internal audit findings are not necessarily a bad thing. They constitute an independent view into the maintenance process. Audit findings are opportunities for improvement, although the decision to implement the improvements may be out of your hands.