D
The Quarterly Project Review
Once you have built a capable software organization, you need to maintain it. If you don’t, this capability will almost certainly deteriorate, and probably very quickly. The quarterly project review is an effective way to maintain a strong software capability. In the review process, you examine the status of the major projects at least once every three months. When your organization is just starting to use the TSP, review the projects every month or so. Once the projects are functioning properly, you can move to quarterly reviews.
This appendix describes why quarterly project reviews are important and how to conduct them. It also provides a review checklist, a review strategy, and some examples of the issues you are likely to encounter.
Why you should Hold Reviews
As an executive, your principal objective is to use business assets to take advantage of strategic opportunities. You will need a capable and effective engineering organization, and for this you will need a capable and effective software operation. To know that your software work is done competently, you must know where the projects stand and how these projects are being run. You also need to assess project and program management. Quarterly reviews will help you to do this.
Another reason to hold periodic reviews is to help the engineers do better work. Quality work requires discipline, and discipline requires coaching and management support. By holding periodic reviews, you maintain management’s focus on the way the teams are performing their work. The reasons that reviews help you do this are as follows:
1. To produce quality products on schedule and for planned costs, the engineers must consistently follow disciplined practices.
2. To consistently follow disciplined practices, the engineers need to know that management cares about the quality of their work.
3. To demonstrate concern about quality, the managers must examine the work regularly and insist that the engineers use disciplined practices.
4. To sustain your emphasis on quality, the managers must know that software quality is important to you.
5. The quarterly project review is an effective way to demonstrate that you care about the quality of the software work.
Review Considerations
In reviewing projects, executives often do more harm than good. The two principal areas of concern are schedule management and team motivation. Executives are prone to setting aggressive schedule goals and insisting that their dates be met. Unless they have engineering experience, they often fail to appreciate the negative impact that unreasonably tight schedules can have on an engineering team.
Schedule Management
When most managers review a project, they only ask about the schedule. This is a mistake for two reasons. First, the actual date when the job will be completed is primarily determined by the scope of the job, the date the work started, the resources applied, and the quality of the engineers’ work. Second, when managers push engineers to complete work quickly, the engineers rush through the tasks, make many mistakes, and spend more time fixing problems than they would have spent doing the job correctly. Excessive pressure extends schedules by months and often years.
By concentrating on the schedule, managers overlook the things they can influence. These are to start jobs promptly, provide adequate staffing, and ensure that the work is done in a disciplined and professional way.
Team Motivation
Getting engineers to follow disciplined practices is a motivational issue. While managers must emphasize the importance of disciplined practices, any attempt to coerce or demand disciplined behavior will not be effective. There are five elements to motivating disciplined engineering behavior.
First, if the job is bigger than expected, reset the goals. Driving teams to meet unreasonable goals builds losing teams. Make a new strategy, reset the plan, and get the team on a winning track. The engineers will feel better about their jobs, and they will do better work.
Second, people most often achieve challenging goals when they have enthusiastic support. When teams are winning and the crowd is cheering, everything seems to work. The performers are energized, enthusiastic, and striving to break new records. So focus on success. Compliment the engineers on what they have accomplished. Emphasize their achievements and concentrate on building a winning team.
Third, consistently disciplined work requires coaching and support. Even when properly trained, and even with strong management support, engineers will not routinely follow disciplined personal practices. It is not that they are uncooperative but that disciplined behavior is difficult. Few people can maintain discipline without guidance and support. That is why sports professionals and performing artists have coaches, conductors, and directors.
Fourth, the engineers must know how to do disciplined work, and they must believe they are more productive when they do. Establishing and maintaining such beliefs takes training, experience, and continued management reinforcement. Provide the engineers with the proper training. If you don’t, they won’t be able to do disciplined or quality work regardless of their motivation and commitment.
And last, the engineers’ managers must recognize the importance of disciplined work. They must view disciplined behavior as a required part of the job. If you and your managers truly believe that it is always faster and cheaper to do the job in a planned and disciplined way, motivating disciplined behavior will not be difficult.
If you concentrate on providing adequate trained resources and motivating disciplined work, your reviews will be rewarding for you and for your people.
The Review Strategy
In assessing the performance of TSP teams, it is relatively easy to determine the status of the projects against their plans. The detail of TSP plans and their earned-value and task-time methods provide precise status measures. With these measures, you will also know the team’s rate of progress and can tell to within days when the engineers will finish the job.
This degree of predictability would be a major step for most software organizations, but it is merely a first step. To achieve an effective software capability, you need to assess the engineers’ work in terms of performance stages:
• The common stage: Teams are performing pretty much as they always have. The first step in moving from the common to the initial stage is to train the engineers in the PSP and to launch their teams under the guidance of trained TSP coaches.
• The initial stage: The engineers have been trained in the PSP, their teams have been launched with the TSP, and they have started to track their time, calculate their earned-value, and report their task-time data. The initial-stage review ensures that teams are starting to use PSP and TSP practices properly and that they are following their defined process and plan.
• The standard stage: Teams are planning, tracking, and reporting on their work, as well as measuring and managing the quality of their products. The principal challenge at the standard stage is for teams to record and use their defect data. Once they do this, benchmarked improvement is possible.
This suggests the following review strategy:
1. Once the engineers have been trained and their teams launched, focus first on the initial stage. Are the engineers following their plans, tracking and recording time, measuring task hours, and reporting earned-value progress? If not, find out why and get these problems fixed before moving to the standard-stage topics. As soon as several team members are doing these things, move on to the standard-stage review.
2. At the standard stage, your initial focus should be on defect data. Are the engineers recording all the defect data? If not, get that problem fixed before asking more questions about quality.
3. In asking about defects, remember that the data are sensitive. Do not say anything that implies criticism of the engineers. This topic is covered in more detail in Appendix E.
4. Once the teams have made a good start on gathering defect data, ask about the quality metrics. Stress the importance of good defect data and ask about yields, defect levels, defect ratios, and time ratios. Ask how these measures compare with the plan, the TSP guidelines, or the benchmarked TSP teams.
5. In subsequent reviews, look at trends. Where is the team improving and where are there problems? Have the team set both short- and long-term improvement goals, and see what management can do to help.
Conclude the review by asking for specific commitments. Where teams are not following plans or gathering time and size data, have them describe the improvements they will make before the next review. Once teams meet the basic planning and tracking criteria, ask about defect data and quality. Then move on to benchmarked improvement. Where do the team’s measures fall short, what are the next improvement steps, and what short-term goals has the team set? Also ask what the team plans to show you at the next review. Have the commitments documented, and their status reported as the first item in the next review. If you conduct such regular quarterly reviews, your teams will soon be measuring and managing the quality of their work.
The rest of this appendix and Appendix E describe how to conduct reviews and suggest questions to ask.
The Review Process
Your principal objective in doing a project review is to determine how the work is being done. Since the TSP provides the data needed to measure project status, the team should be able to make a status presentation with little or no preparation. However, the team leader will almost certainly prepare a special presentation. Your objective in asking questions is to find out how this team is actually performing. That generally will require at least some questions that the team does not expect.
While there are many ways to conduct reviews, it is usually helpful to follow a defined process. This makes the reviews more efficient and helps to ensure that they are complete. The Quarterly Review Checklist in Table D.1 suggests the questions and the order in which to ask them. You should follow this checklist fairly closely at first, but feel free to modify it as you gain experience with reviews.
When to Hold the First Review
Hold the first project review a month or two after the project launch, but give the engineers time to start the job. Too early a review will expose issues that the engineers should solve for themselves. While new TSP teams will have many questions, the coach can handle most of them without management help. After a month or two, the engineers will have answered their initial questions and gathered some early data. The objective of the first review is to make sure the teams are following the process. In later reviews, you can judge whether the team’s plan is realistic or if changes are required.
Table D.1 The Quarterly Review Checklist
The following paragraphs illustrate the questions to ask TSP teams and the answers you will likely get. The examples also illustrate some of the information that is potentially available on TSP projects.
The Initial-Stage Review
To show how an initial-stage review is conducted, we will again use Janet’s project, after the team has completed four weeks of work. In describing project status, she shows the data in Figures D.1, D.2, D.3, and D.4. Janet first explains that the team completed the launch ahead of time so the engineers are slightly ahead of their schedule for the requirements tasks. They are now starting system test planning and the requirements inspections. However, she is concerned about the EV projection in Figure D.2. It indicates that the project will complete eight weeks late.
Janet then shows the weekly report form in Figure D.4 and explains that the team is behind on earned value (EV)* to date with an actual value of 7.3 as opposed to the plan of 8.9. Using the weekly data, she points out two problems: the project hours are behind plan and the engineers are taking longer to complete tasks than planned. From the last line in the weekly data section of Figure D.4, she shows that the completed tasks have taken 421 hours, compared with the plan of 380 hours. This 11% overrun would be serious if it continued for the rest of the project.
* Earned value measures a team’s progress against its plan. Each task’s planned value (PV) equals that task’s planned percentage of the total job. When that task is completed, the project earns that value (EV).
Figure D.1 Project Status—4 Weeks
Figure D.2 Project Earned Value—4-Week Actual
Questions on Planning
In following the Quarterly Review Checklist, first ask about the team’s plan. Is it sufficiently detailed to guide the engineers’ work, and does it reflect what the engineers are actually doing? Does the team plan follow its defined process, does it include inspection and review steps, and do the engineers have the training and skills to handle the tasks they have been assigned? Typical planning problems concern underestimates, unplanned tasks, or skill problems. Also, teams often start with incomplete or inaccurate requirements and must take time to correct them. An early indication of this problem is requirements tasks that take longer than planned. This is the case with Janet’s team.
Figure D.3 Weekly task Hours—4-Week Actual
Figure D.4 The Week form
Questions on Task Time
In explaining the impact of the team’s weekly task-hour problem, Janet points to Figure D.3. The engineers have been below the plan for the last three weeks, but their task-hour rate is improving. Although the team is running about 18% behind on EV to date, Janet believes there is still time to correct the problem. The engineers have also suggested actions to improve their weekly task hours:
1. Establish morning quiet times when the engineers would not answer their phones, there would be no meetings, and the team members would not be disturbed.
2. Work at home two days a week.
3. Get clerical support to handle supplies, make copies, do typing, or do any other miscellaneous chores that will save the engineers time.
4. Work on Saturday and take off Monday.
5. Come in late two mornings and work later in the evening.
Questions on Earned Value
As shown in Table D.2, the relationship between earned value and task hours can tell you a great deal about a project. Depending on the situation, try to understand the team’s problems and what management can do to help. Ask what the engineers see as the problems and use the team’s data to interpret the answers. Janet’s project appears to be case 5 in Table D.2. In discussing the EV problem, Janet points to the completion projection in Figure D.2. This shows that, at the present rate of progress, the project will be completed in 41 weeks instead of the planned 33 weeks.
Completion Projections
Janet explains that the project has an 8-week schedule exposure because the team had earned 7.30 EV in the first three weeks, for an average weekly rate of 2.43 EV.* At this rate, the project would reach 100 EV (and finish the job) at 41 weeks (100/2.43 = 41.15). However, because it is still early and the team’s task-hour rate is improving, she believes the team can recover.
* Note that the team earned no EV in week 1 during the TSP launch, so the 7.3 EV was earned in three weeks.
Table D.2 Interpreting Earned-Value and Task-Time Data
Questions on Load-Balancing
In answer to your questions on workload balancing, Janet explains that the engineers are still following the plan they developed during the launch, but their workload distribution is becoming unbalanced. She hopes to use the current plan until the relaunch in about two months but will watch the situation carefully. If necessary, they will take half a day to update the plan and rebalance the workload.
Questions on Time Recording
In answer to your questions on time recording, Janet believes that the engineers are recording their time properly but has not looked at their time-recording logs. She will do so right away. If any engineers are recording time data once a day or once a week, she agrees that these data would not be accurate enough for managing the project.
Questions on Size Recording
Janet did not have size data at the 4-week review, but she did at the 17-week review three months later. In answer to your questions on size recording, she shows Figure D.5 and explains that the plan line shows the team’s original estimate of 24,730 KLOC, with updates for the completed modules. As each module is unit tested, the actual size measures are substituted for the estimates. The estimated total line shows a linear extrapolation of the growth in the total size estimate as components are completed. She also explains that the lines at the bottom of the figure show the actual sizes of the coded and unit tested modules, together with a linear projection of the actual size data.
Figure D.5 Line-of-code Projections
By projecting the plan and actual size data, she can judge how big the system will be and when they will complete coding and unit testing. This is when the plan and actual size lines inter-sect—at 32 weeks. Here, the size value is about 30,000 LOC—indicating that the finished product size will be about 30,000 LOC instead of the original estimate of 24,730 LOC. It also suggests that the coding and unit testing will be completed at 32 weeks. Since they plan only 3 weeks of work after the last unit test, this indicates about a 2-week schedule exposure. While still a concern, this is a substantial improvement over the 8-week exposure she reported in the first quarterly review.
Standard-Stage Reviews
This team is doing a reasonable job of making and following its plan, although the data-gathering situation is not yet clear. If, at the next review, Janet reports that the engineers are properly gathering the time and size data, you can move on to the standard-stage review questions. These questions are discussed in Appendix E.
Summary and Conclusions
The following eight principal points are made in this appendix:
1. The objective of the quarterly review is to motivate teams to do their best work.
2. Hold the first project review about one to two months after the project has been launched.
3. In conducting the review, use the TSP Quarterly Review Checklist.
4. TSP reviews are conducted in stages, with the initial-stage reviews first concentrating on planning and tracking.
5. If necessary, repeat the initial-stage questions until the results are satisfactory.
6. Once at least some of the engineers have demonstrated competence at the initial stage, move on to the standard-stage questions. These questions are discussed in Appendix E.
7. At the end of the review, ask for a summary of the actions to be taken, and verify that someone is assigned to each action item.
8. Close the meeting by summarizing the principal topics that you will explore in the next review, and thank the team and team leader for their efforts.