CHAPTER 8

Evaluating a Training Course on Performance Appraisal and Coaching

This case study shows what a company can do if it wants to upgrade its performance appraisal process. In my experience, the human resources managers and line managers I have talked with are not happy with their programs. There seems to be a constant revision of philosophy, objectives, forms, and procedures. Most organizations are still using performance appraisal to decide on salary adjustments and provide information for personnel decisions including promotion and termination. This book concentrates on programs designed to improve performance. If you are looking for training to help you improve your program, this case study will be of great interest. You can see what Kemper did and its results. The evaluation process was based on the Kirkpatrick “four-level” model for determining the effectiveness of the training.

Kemper National Insurance Companies

Need and Purpose

Our training program is a success if it accomplishes four objectives: participants like the program, participants gain needed knowledge and skills, participants apply what they learned to their jobs, and participants assist the company in achieving its mission and objectives. The purpose of the program is to improve performance. The purpose of evaluation is to verify and improve the effectiveness of training. The evaluation design includes ways and means of measuring the effectiveness of the program in achieving each of the four objectives just defined.

The Training Course

The program was conducted at the Charlotte branch of Kemper. All supervisors and managers attended the course during a three-month period between December 1989 and March 1990. The program and its evaluation received the complete support of the branch manager. Figure 8-1 describes the program content and objectives.

Evaluation Design

Both quantitative and qualitative data were collected. Data collection techniques included existing tools as well as measurements designed for this evaluation. This section describes the data collection tools that we used for the four levels on which we evaluated the training program.

1. Reaction. How well did the participants like the training? Each participant completed the reaction sheet shown in Figure 8-2 at the end of the course. The results were tabulated and summarized.

2. Learning. What knowledge and skills did participants gain from the program? We collected data by administering the Performance Appraisal Skills Inventory (the Organization Design and Development, available from King of Prussia, Penn.) before and after training. The inventory contains eighteen performance appraisal situations. For each situation, the participant selects the best answer from four possible choices.

3. Behavior. To what extent have participants transferred knowledge and skills learned in the program to their jobs in these four areas: preparing for the performance appraisal, establishing two-way communication with employees, gaining agreement on the appraisal, and documenting the report form?

To collect data, we used results from the performance appraisal report form checklist shown in Figure 8-3, administered before and after training; the performance appraisal questionnaire for managers, shown in Figure 8-4; the performance appraisal questionnaire for employees, shown in Figure 8-5; and unobtrusive data, which included informal observations obtained from many sources, including the human resources manager, the immediate supervisor of those completing forms and conducting interviews, and those who completed forms and conducted interviews.

Figure 8-1. Performance appraisal and coaching seminar.

Objectives

During this course, participants will:

1. Self-assess individual strengths and weaknesses in the skill areas necessary to establish two-way communication and gain agreement in the six steps of the performance appraisal discussion:

• Building trust

• Opening

• Accomplishments and concerns

• Planning

• Evaluating and rating

• Closing

2. Identify individual improvement goals for strengthening skills needed for conducting effective performance appraisal discussions.

3. Practice applying the following coaching process to the six steps of the performance appraisal discussion:

• Identify the situation.

• Clarify information.

• Explore options.

• Agree on actions.

• Follow up.

4. Learn to recognize when specific coaching techniques can be used to establish two-way communication and reduce defensiveness during the performance appraisal discussion.

5. Analyze various ways in which both the employee and the supervisor can prepare for the performance appraisal discussion:

• Reviewing objectives, standards, and reports

• Employee self-appraisal

• Input from next-level manager

6. Define each section of the performance appraisal report form and explain how to use it as a tool in the performance appraisal discussion.

7. Explain how the wording of the performance appraisal report form can enhance the clarity of the completed form and reinforce the interactive tone of the overall process.

8. Demonstrate ability in writing

• Performance improvement needs

• Performance improvement objectives

• Achievement of prior objectives

9. Identify the criteria for a timely, high-quality performance appraisal report form through the use of a checklist.

Figure 8-2. Reaction sheet.

Figure 8-3. Performance appraisal report form checklist.

On the basis of the completed appraisals that you have brought to the course, how would you answer these questions?

Performance Standards	YesNo
•Do standards reflect the current job?
•Are standards attached and evaluated?
Attendance
•Are attendance problems documented according to policy?
•Have you refrained from describing the personal reasons for absences?
Achievement of Prior Objectives
•Are prior objectives restated and evaluated?
•If prior objectives are not met, are clear circumstances or reasons stated?
•If prior objectives are not met, will an outside reader know how this will affect the performance rating?
Attributes
•Are attributes coded properly and supported by job-specific behavioral examples?
•Are attributes used to recognize and reinforce past performance?
Performance Improvement Needs: Immediate Needs
•Do the needs relate to failure to meet standards or achieve objectives?
•Do supporting comments indicate a sense of urgency about the need?
•Are supporting comments job related and specific?
•Do supporting comments reflect input that the employee provided about the need?
Performance Improvement Needs: Other
•Are the needs specific, and do they involve job-related areas that require improvement?
•Are they related to the current position?
•Do supporting comments reflect input that the employee provided about the need?
Objectives
•Are performance improvement objectives listed first and linked to the need in the preceding sections?
•Do they state specifically how well the employee should do or achieve?
•Do they state specifically what the employee should do or achieve to be acceptable?
•Do they state specifically under what conditions (time frame, resources, training) the employee should perform?
•Do supporting comments reflect input that the employee provided about the objectives?
Performance Rating
•Is the rating consistent with the results and narrative of the entire performance analysis?
•Is the rating based on the principle of zero-based appraisal?
Development Objectives (if applicable)
•Is it clear that development objectives are not used in determining the performance indicator?
•Is it clear that they are not requirements or standards of the current job?
•Are these objectives specific in terms of what the employee should do, how well, and under what conditions?
•Do supporting comments reflect input that the employee provided about the objective?
Promotability
•Do identified position(s) fit the employee’s experience and skills?
•Are listed position(s) properly titled and coded?
•If the employee is immediately promotable to another functional area, has the performance appraisal been signed by another department manager?
Relocation
•Did you discuss current relocation preferences with the employee at the time of the appraisal?
Supervisor’s Comments
•Are the comments job related and consistent with the rest of the appraisal?
•Does the Comments section effectively summarize the appraisal?
Other
•Have you completed each section with all the required documentation?
•If the employee is participating in the career development program, is the career development plan properly completed and attached?
•Does the appraisal reflect evidence of two-way communication?
•Does the appraisal language reflect employee input?
•If a third party reviewed the completed appraisal, would the documentation be clear and consistent throughout each section?
•Is the appraisal free of references to personal issues and circumstances of employee’s life?
•Has the appraisal been completed by the due date?

4. Results. To assess the results, we asked this question: What gain has there been in the achievement of the following two human resources objectives? Ninety-five percent of all performance appraisals are completed on schedule, and the quality and accuracy of the appraisals improve in five areas: candidness, completeness, developmental plans, ratings, and feedback.

Data were collected through an analysis by the branch human resources manager of completed performance appraisals.

Evaluation Results

The evaluations were conducted as planned, and the results were communicated to executives and other interested and concerned persons as follows.

The performance appraisal and coaching course was designed to improve the skills of managers and supervisors in coaching effectively during the performance appraisal discussion and in writing the performance appraisal report.

The Charlotte branch was selected as the site for piloting the course and for evaluating its effectiveness. The training received the enthusiastic support of the branch manager, Jim Murphy, and of the branch human resources manager, Peggy Jones, and it was positively received by the Charlotte supervisors and managers. Forty-one branch supervisors and managers completed the course between December and March, a fact that made it possible to study the effectiveness of training with an entire management staff.

Procedure and Findings

Evaluation that verifies and improves the effectiveness of training is conducted at four levels: reaction, learning, behavior, and results. Evidence to determine the effectiveness of training must be gathered at each level. Table 8-1 lists the questions that need to be answered at each level of evaluation and the data collection tools that were used in answering each question.

Figure 8-4. Performance appraisal questionnaire for managers.

Instructions: This survey is designed to describe your experiences in conducting performance appraisals since completing the performance appraisal and coaching course.

Please answer the questions below by circling the number that corresponds to your response.

1.Characterize your preparation for conducting performance appraisals since completing the course.
2.Characterize the actual performance appraisal discussions that you have conducted since completing the course.
3.Characterize your documentation of the performance appraisal report form since completing the course.
4.To what degree have you been successful in reaching agreement with your employees on the main issues of the performance appraisal discussion since completing the course?
5.Which aspects of the performance appraisal process are still the most difficult for you? Check your response(s):
____ Preparing for the performance appraisal
____ Discussing employee strengths
____ Discussing performance problems
____ Developing an improvement plan
____ Overcoming defensiveness
____ Conducting the performance appraisal discussion
____ Reaching agreement on main issues
____ Documenting performance improvement needs
____ Writing objectives
____ Documenting achievement of prior objectives

Please comment on the items that you have checked.

What other comments would you like to make on conducting performance appraisals? (Use the back of this sheet if necessary.)

Please use the enclosed envelope to return the completed questionnaire. Thank you for your cooperation.

The findings at each level of evaluation indicate that the performance appraisal and coaching course makes a difference in increasing both the quality of the coaching that takes place during the discussion and the quality of the performance appraisal report.

Reaction. Level 1 findings indicate that course participants were satisfied customers. The course evaluations received confirmed that participants reacted positively to the course. Positive reactions increase participants’ receptivity to the knowledge and skills presented in the course. The majority of participants felt that the course objectives had been met and that the course was highly relevant to their jobs. The average overall rating on a 5-point scale was 4.37.

Learning. Level 2 findings indicate that course participants made gains in the knowledge and skills needed to conduct and document quality performance appraisals. Data gathered from administration of the quality checklist before and after training indicate that 94 percent of the performance appraisals written by participants after the course were of higher quality than the appraisals that they had written before training. It was also significant that, while appraisal quality was as low as 54 percent before training, the lowest quality observed after training was 78 percent.

Figure 8-5. Performance appraisal questionnaire for employees.

Instructions: Your manager recently completed a course on performance appraisal. In order to better understand the effectiveness of this course, we are interested in your reactions to your most recent performance appraisal.

Since this questionnaire is anonymous, do not sign your name.

Please answer the questions below by circling your response or the number that corresponds to your response.

Please use the enclosed envelope to return the completed questionnaire. Thank you for your cooperation.

Table 8-1. Evaluation questions and data collection tools.

Behavior. Level 3 findings provide evidence that course participants applied the knowledge and skills acquired in the course when they conducted subsequent performance appraisals. Data gathered with the quality checklist before and after training highlighted three areas of particular improvement: two-way discussion, documentation of attributes, and objectives. The course provides practice sessions to enhance skills needed to involve employees in the performance appraisal discussion and to show evidence of discussion and employee input in the performance appraisal report. Appraisals conducted after participation in the course showed nearly four times more two-way discussion after the course than before it.

Performance appraisals are audited by branch human resources staff to identify errors and potential problems. When errors are found, a performance appraisal is returned to the appraising supervisor for improvement. Strong evidence that participants applied knowledge and skills learned in the course on the job was provided by an immediate decline in the number of returns. Before the training program, eight appraisals were returned for improvement in one month. After the program, no more than two appraisals per month were returned.

The Charlotte branch human resources manager reported that, from the conclusion of the study through the fourth quarter, the number of appraisals returned through the audit had remained low in comparison to previous years. Her report shows the percentage of audited appraisals that were of acceptable quality each quarter after the program:

Q3	90%
Q4	96
Q1	95
Q2	80
Q3	95
Q4	96

The lower quality of appraisals during the second quarter of the second year reflects the fact that five appraisals were returned for clarification of objectives and documentation of the achievement of prior objectives and that one appraisal lacked proper documentation of attendance. The Charlotte branch human resources manager is continuing to coach the management staff in these areas, but she states that the narratives now indicate much more two-way discussion than she saw before the training took place.

Another indication that participants applied new knowledge and skills was the results of two questionnaires. One questionnaire was designed for management staff who were trained and the other for the employees who reported to them.

The manager questionnaire showed that 77 percent of the management staff considered handling performance problems within the performance appraisal to be easier after taking the course. Because the skills needed to coach employees with performance improvement needs effectively are practiced during the course, supervisors and managers are likely to find handling performance problems easier because they are more skilled at doing so.

The employee questionnaire was designed to determine how the employees who had been appraised by the trained supervisors felt about how they had been coached. Evidence provided by employees indicated that supervisors were effective in three important areas: employees felt that supervisors listened (83 percent), that supervisors valued their input highly (75 percent), and that agreement was reached on main issues of the performance appraisal (77 percent).

Anecdotal data provided by the human resources manager and the participants themselves confirmed that the course had made a difference in the quality of the performance appraisals being written. By the end of the first quarter after the program, the human resources manager stated that she was seeing a marked difference in the overall quality of performance appraisals. Participants commented that it was much easier to do the appraisal after they had completed the course.

Results. Level 4 findings were drawn from data collected with the quality checklist before and after training. The course increased the quality of performance appraisals in several important areas: objectives, performance feedback, and completeness. Specific objectives that met the quality criteria presented in the course increased by 36 percent after training.

Performance feedback, both in recognizing employee strengths and in coaching for performance improvement, was of higher quality after the course. The number of appraisals containing attributes supported by behavioral examples of how the employee exhibited the attribute on the job increased by 49 percent.

Additional evidence of effective feedback on performance improvement needs was provided by the 36 percent increase found in specific objectives, which clearly state what the employee will do to maintain or improve performance and how measurement will take place. The language used to document objectives indicates that employees are becoming more involved in discussing and finding solutions to performance issues. Data from the checklist after training showed a 35 percent increase in evidence of discussion.

Conclusions and Recommendations

The evidence presented in this report supports the assumption that the performance appraisal and coaching course results in supervisors and managers who are more confident and competent in conducting quality performance appraisals. The evidence also shows an increase in the number of performance appraisal reports documented correctly.

Skills and knowledge gained during training need to be reinforced on the job. Reinforcement is achieved when all staff are trained with support from top management. Managers who have themselves been trained can coach the supervisors who report to them. Human resources staff can also provide ongoing coaching.

Training all supervisors and managers in one location creates a great opportunity for affecting the culture in terms of the overall desired outcome of the course. The evaluation clearly shows that supervisors now use a joint problem-solving approach to encourage employees to assume responsibility for their own performance. It also shows that supervisors provide candid feedback on all aspects of performance. Managers and supervisors who are more comfortable with the authoritarian management style may find this approach uncomfortable, but when it is modeled and reinforced by their own manager and peers, change is likely to occur.

This chapter is based on Donald L. Kirkpatrick, Evaluating Training Programs: The Four Levels, 2nd ed. (San Francisco: Berrett-Koehler, 1998).