CHAPTER 12

Evaluating the Project or Program

Project evaluation is an opportunity not only for project management but also for project knowledge management (PKM). This chapter focuses on the former, contributing to the project management perspective of evaluating projects/programs. The PKM perspective is elaborated in the context of ProwLO, based on a separate guidance (Rosinski 2019). HybridP3M considers project evaluation, the final frontier of project management, not, for example, benefits management. The reason is multifold. It is undervalued, unpopular, and therefore not well developed. HybridP3M aims to change this situation. Second, building on the first reason, one can observe that due to lack of experience (as a result of missing practical application), involved actors are not provided with the opportunity to learn the skills involved, nor the opportunity to develop the right attitudes considering the political dimension of evaluation. To some extent evaluation remains a mystery. Third and finally, following the first reason, without a proper understanding of project evaluation, the current status quo, learning across projects remains elusive. Project evaluation is essential in the context of realizing learning organizations, and thus, optimization, a key HybridP3M objective. Schindler and Eppler (2003) once noticed that there are distinct process and content aspects to project evaluation methods. HybridP3M recognizes that the evaluation process from an activity perspective is subject to details and sensitive to best practices. Accordingly, this acknowledgment may motivate development of corporate formats for project evaluation, tailored according to corporate needs and scalable according to specific project needs. Inspired by Schindler and Eppler, HybridP3M has introduced a meta-level process to adopt evaluation methods, based on unique corporate formats. Second, HybridP3M has defined standard content elements to be derived from evaluation. These elements imply activity for their capture. However, HybridP3M allows flexibility depending on the selected evaluation method and context of evaluation, such as the moment during a lifecycle. In order words, HybridP3M permits to some degree tailoring. Figure 12.1 depicts the meta-process of evaluating projects/programs, whereas Figure 12.2 focuses on evaluation method essentials, content-wise.

image

Figure 12.1 Meta-process of evaluating projects/programs

Adopting Evaluation Methods

At the beginning of the life cycle, a decision needs to be made on what kind of evaluation methods will be used in the project or program. This decision involves the particular formats available and the relevant associated timing. Developing corporate formats, aligned with Hybrid-P3M’s guidance, is basically a common knowledge need. According to HybridP3M, there are three different formats at the fundamental level: (1) gate reviews, which benefit from the stage-gate paradigm, (2) post control (at the end of the lifecycle), and (3) postmortem reviews (at some future point after project closure). It is important that this decision takes place at the beginning because evaluation, taking up time and resources, needs to be planned in advance for mutual agreement. Evaluation in an ad hoc fashion is simply a too chaotic approach.

Performing Gate Review

Gate reviews are very important. One of the benefits is that gate reviews provide a reflection opportunity on decision making. Decision and taken actions take time to take effect. Changes in project behavior can only be noticed thanks to the right amount of time elapse. When decisions are made regarding the next stage, there is an opportunity to reflect on project behavior of the ending stage or earlier stages. This allows corrective measures such as to revoke earlier made decisions or making new decisions to deal with existing problems or new problems as they may evolve. In other words, gate reviews are a perfect tool for iterative decision making, supported by the Cases method.

image

Figure 12.2 Evaluation method essentials

Performing Post Control

Post control complements gate reviews and provides an opportunity for consolidation of newly gained knowledge rooted in experience. Post control as a method was originally introduced by Schindler and Eppler, and they compared it to other approaches like project audit, after action review, and post-project appraisal. In HybridP3M, post control depends on the selected format for project/program evaluation. From an output perspective, however, it is recommended to adopt HybridP3M’s content suggestion. In practice, the extent of evaluation is always a function of the strategic importance of a project or program and also depends on the business pressure to move on to the next project.

Performing Postmortem Review

Postmortem reviews complement post control mainly in terms of evaluating benefits as benefits realization takes time in most cases. Also after the project or program has ended, there may be better time frame to organize evaluation and hold a moment to reflect. The main problem with postmortem reviews, and to a lesser extent post control, is that project amnesia becomes significant. People simply tend to forget, and personal experiences, including a view on historical events, do not always reflect facts.

Analyzing Decisions

The first major element of project evaluation is analysis of decisions. The goal here is not to create a culture of blaming; instead a culture of openness is advocated. But even a culture of openness, if any organization claims one, has its limits due to present authority relations, decision power, confidentiality, and politics. These factors contribute to the condition that team members respect decisions as they are keeping an “appropriate” distance to the decision-making authorities, at the cost of potentially constructive criticism. But if an organization deeply wants to learn, then decision makers should take accountability for their actions and allow a health check. There are two approaches possible, or a combination. Analysis is restricted to introspection for the benefit of individual learning, or analysis takes place in the context of group learning, including dialogue and discussion, and gets reported for the benefit of systematic organizational learning. The advantage of the first approach is the greater respect for confidentiality. The advantage of the second approach is knowledge validation and greater knowledge integration of captured results.

Analyzing Benefits Realization

The second major element of project evaluation is analysis of benefits realization. This type of analysis involves a reflection on the business case. The most relevant question in this context is whether it was a good investment. How good an investment it was really depends on actual benefits realized. As mentioned earlier, it may take time to identify or measure actual benefits. In some cases, benefits will be recurring thanks to exploitation of new capability. Some benefits might be monetary, others of a more strategic nature. Every type of benefit needs to be taken into account based on the adopted benefits management approach.

Analyzing Project Performance

Organizations that take professionalization of project and program management seriously need to analyze project performance systematically. Project performance is a complex concept consisting of four elements: project results (deliverables and outcomes), project management performance (taking into account variables such as time and cost), technical performance (specialist work), and knowledge management performance (e.g., alignment with corporate-level KM strategy), which are no simple concepts either. It takes experience and intelligence to develop an appropriate conception of each of these elements, a conception that enables assessment, ideally supported by metrics. HybridP3M has no easy solution for this and recommends organizations to develop this knowledge independently. However, each element can be assessed informally based on tacit knowledge. That would be the initial situation (an immature approach). Another issue is whether this specific analysis is not better performed in the context of an audit by project external people (e.g., external consultants or internal members of quality assurance). The benefit of the latter approach is increased objectivity and specialization in all relevant aspects. But HybridP3M believes that a project audit can complement but not replace personal evaluation of project performance by the key actors involved. It is namely a part of growing and learning by individuals.

Analyzing Delivery Process

Analysis of the delivery process is arguably an easier activity, partly thanks to the fact that the project management team may rely on customer feedback. It is not confined to an evaluation of internal performance, subject to personal views based on personal stakes. Again intelligent use of metrics can make this activity more objective. It should be stressed that this activity should be performed during the project (in addition to project end, possibly project postmortem) so that things can be adapted based on the lessons learned. Analysis of the delivery process takes into account pure delivery aspects, and there are four identified, namely: (1) agility, (2) customer satisfaction, (3) efficiency, and (4) product quality. The first element requires a common understanding of Agile, possibly supported by a definition. As the customer may benefit from better results thanks to Agile delivery, he or she should express his or her view on the incremental, iterative, and generally flexible character of the delivery process as experienced firsthand (supported by effective change management), or provide main concerns (e.g., raise the fact that the delivery actually lacks/ lacked agility). Assessment of customer satisfaction completely relies on customer feedback. Assessment of efficiency, on the other hand, takes into account cycles of development and requires performance measures. Depending on the industry, the organization should develop its own approach to measuring efficiency. Finally, product quality is a measure partly depending on quality metrics (number of faults, quantified rework, an account of user acceptance, etc.) and partly depending on the subjective perception of the end result by the customer.

Defining Project Success

Project success has no universally accepted definition. In fact, it is a relatively obscure concept. As stated by Rodolfo Siles (2020), “Project success has been historically defined as a project that meets its objectives under budget and under schedule.” This definition, perhaps in line with the major bodies of knowledge (competing with HybridP3M), clearly associates performance with success through the latter part of the statement. But HybridP3M views success independent of performance. Analysis of project performance, as described earlier, stands in its own right (and thus the greater good of performance management). So according to HybridP3M, objectively speaking, success refers to the degree to which objectives are met, which are dynamic by the way in an Agile, realistic context. But success also depends on interpretation and a personal view of the greater meaning of a project or program. In such a context, benefits and outcomes are crucial. Project success becomes something real when benefits and outcomes can be quantified and measured; otherwise it is just an opinion. This stresses the importance of benefits management. A popular interpretation of project success is when the project satisfies customer and/or specific stakeholders. But this is only one perspective. Therefore, HybridP3M recommends to develop common understanding of project success, or at least facilitate a healthy discussion, and usage of a definition of success useful in the process of closing the project and final handover, concluding results and opening a serious evaluation opportunity.

Concluding Project Success

At the end of the project, in preparation of final handover, project success must be concluded based on the previously deliberated and accepted definition. Findings on performance, based on analysis of project performance, are of secondary importance but may compensate any harsh conclusions, such as project failure, placing success in a greater perspective. The conclusion should be recorded and shared with the project management team and project board, so a final report can be prepared for the customer.

Reporting and Explaining Project Success to Customer

HybridP3M thinks that it is in the interest of the supplier to communicate the conclusion on project success with the customer. In case there is success, this can only have a positive effect, a reason to celebrate. And in case success is partial or lacking, reasons for why can be explained. Reporting and explaining can only do good in terms of public relations. It is a gentleman’s approach. Success is not an end result, thanks to new capability benefits may materialize later and learning may prove invaluable. This should always be taken into account.

Identifying Follow-On Actions

In order to make an impact, evaluation should lead to improvement. In this respect, follow-on actions play a key role. As a conceptual knowledge need, follow-on actions are often derived from cases and lessons learned in the context of knowledge management. But from a project management perspective, the roots of follow-on actions are of secondary importance. Essentially, follow-on actions suggest changes. These changes may apply to operations, but sometimes also relate to tactics or strategy. Thanks to definition of actions the suggested changes have practical meaning. In conclusion, follow-on actions are a highly “actionable” knowledge type. If regarded and taken seriously at the corporate level, they may drive institutionalization of change. The quality of follow-on actions highly depends on the quality of the evaluation process and capture of related knowledge types like cases and lessons learned.

Process Aspects

Figure 12.3 captures the knowledge nature of evaluating the project or program.

image

Figure 12.3 Tacit–explicit continuum of integrating knowledge management

In order to become a repeatable process, evaluation should rely on explicit process know-how. The social dimension of evaluation will always maintain the position of tacit knowledge in the process.

Figure 12.4 captures the manageability of evaluating the project or program.

image

Figure 12.4 Step-by-step process versus skilled activity continuum of evaluating the project or program

Evaluation is essentially a step-by-step process. Therefore, it can be promoted based on corporate standards effectively. Proficiency, however, depends on skilled activity.

Figure 12.5 captures the specialization level of evaluating the project or program.

image

Figure 12.5 Management–specialist continuum of evaluating the project or program

Evaluation is a generic management skill but could be considered a specialization as well. It does not require a distinct role however. Project managers, supported by project knowledge managers, require training to perform evaluation well.

Figure 12.6 captures IT support in relation to evaluating the project or program.

image

Figure 12.6 Available IT support for evaluating the project or program

Evaluation itself does not rely on IT for support. The associated documentation, however, should be integrated based on document management solutions with knowledge integration capability.

Figure 12.7 captures the complexity of evaluating the project or program.

image

Figure 12.7 Task complexity scale of evaluating the project or program

Evaluation is a relatively complex process due to the social setting in which it takes place. Also business pressures and life cycle dynamics pose potential barriers to evaluation.

MAIDEO Requirements

Table 12.1 presents MAIDEO requirements related to “evaluating the project/program.”

Table 12.1 MAIDEO requirements related to evaluating the project/program

Requirement

Level

Dimension

Senior management has an explicit policy for project or program evaluation.

1

Strategy and policy

The organization applies some form of post control.

1

Organization and process

The organization applies gate reviews consistently.

2

Organization and process

The organization has developed coherent and holistic evaluation methods and techniques, inspired by existing standards such as HybridP3M.

2

Organization and process

The organization applies project postmortem reviews in order to better evaluate benefits.

3

Organization and process

Various aspects of project performance are evaluated.

3

Organization and process

Various aspects of delivery are evaluated.

3

Organization and process

Project success is defined and evaluated accordingly.

4

Organization and process

Project success as concluded is communicated with the customer and explained.

4

Organization and process

Follow-on actions are defined in the context of evaluation.

5

Organization and process

Follow-on actions are taken seriously by corporate management or portfolio management.

5

People and culture

Key decisions are analyzed including the unique decision-making process behind them.

5

Organization and process

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
34.201.16.34