CHAPTER 5

Discussion

5.1 Summary

Phase 1 helped us identify the current state of the practice of expert judgment within project management. Phases 2 and 3 provided us insight about how to improve the practice of expert judgment within project management using the state of the art/science found outside project management, and thereby narrow the theory-practice gap. From the perspective of Kerzner’s project management maturity model (2011) (shown in Figure 22), this study should help provide a pathway for developing basic knowledge about expert judgment and defining the process of expert judgment in an effort to move from level 1 through level 2 and into 1evel 3, and ultimately beyond with time. In sum, by introducing language of basic knowledge and process definitions about expert judgment, it is expected that the practice of using expert judgment in project management would move from an immature level 1 stage to a more sophisticated level 3 stage of project management maturity.

5.2 Key Findings

The following are some general findings and conclusions that have resulted from this study.

5.2.1 State of the Art/Science Is Established and Growing

Based on the literature review conducted in phase 1 of this study, it is clearly apparent that other disciplines are more advanced than is project management when it comes to expert judgment elicitation methods (particularly the evaluative methods). Even though the scope of the review was limited to a one-year period, there were more than 100 relevant articles on the subject of expert judgment (see Figure 3), of which many have been captured in this report. Upon reviewing Chapter 2 and the scope of the references, it is clear to see there is a robust body of knowledge outside project management in regard to expert judgment elicitation. A more comprehensive review of the literature (spanning a greater period of time) would yield even more opportunities for improvement.

5.2.2 State of the Practice in Project Management Is Informal and Emergent

Based upon the descriptive survey of phase 2 of this study, it is clear that the state of the practice of expert judgment within project management resides within level 1 (or optimistically level 2) of Kerzner’s project management maturity model (seen in Figure 22). It is based on ad hoc (and likely inconsistent) processes for conducting expert judgment, typically conducted without the benefit of written guidance. Presented in contrast to the findings of Chapter 2, there is a clear gap between the practice and the theory, and as a result, a significant opportunity exists to improve the practice.

5.2.3 Expert Judgment Elicitation in Project Management Can Mature

The experiments of phase 3 of this study provide evidence that additional informative research and evidence can be applied to mature and standardize the practice of expert judgment elicitation in project management to eventually become a repeatable, well-defined, and structured tool/technique within the toolkit of project management practitioners everywhere.

In that direction, the next suggestion provides a series of suggested practices to improve the practice of expert judgment in project management.

5.3 Suggested Practices

5.3.1 Use a Generic Process

Because much of the practice of project management employs a variety of process models, such a model should be used when using expert judgment as a tool/technique. Currently, only one in 10 project management practitioners uses written guidance or policy for expert judgment elicitation. The presence of written guidance is strongly correlated to the frequency of practitioner usage of predefined structured processes, which are known to help alleviate judgmental biases and, thus, improve the accuracy of estimates. There is a variety of existing processes for eliciting expert judgment; several have been provided in Chapter 2. Most processes are sufficiently malleable to allow adjustment for individual circumstances. We suggest adopting or creating a generic process and providing policy on how to conduct expert judgment elicitation. The generic seven-step process presented in Chapter 2 will now be used to make further suggestions on how to elicit expert judgment.

5.3.2 Frame the Problem

Clearly identifying the specific information you need to obtain through expert judgment is critical. Solving the wrong problem perfectly is no better than solving the right problem imperfectly. As a start, if you are using the PMBOK^® Guide set of processes, you may be interested in looking up the indicated output(s) for the specific process where you intend to employ expert judgment as tool/technique (see Table 4). Once you have determined the form of the information sought, you will be able to identify whether the expert judgment task will be evaluative or generative in nature. Recall that evaluative methods result in expert judgment that evaluates (or otherwise estimates, forecasts, predicts, or quantifies) desired information. Examples include cost or time estimates and risk probabilities or impacts. Generative methods result in expert judgment that generates lists or descriptions of desired information. Examples include activity lists, risk registers, and stakeholder lists. By naming your desired information as either evaluative or generative, you will be able to select an appropriate method.

5.3.3 Plan the Elicitation

In order to get the best judgments or estimates, you must select the most appropriate method from among the hundreds available. Some methods work best for evaluative tasks and others for generative tasks. In the PMBOK^® Guide, the generative methods (although not classified that way) are grouped under the lists and descriptions of group creativity techniques, alternatives generation, and group decision-making techniques:

• Brainstorming
• Nominal group technique
• Mind mapping
• Affinity diagramming
• Delphi technique
• Lateral thinking
• Analysis of alternative

This report provides a more extensive list of available methods (particularly the ones for evaluative tasks) than does the PMBOK^® Guide. Once you have selected a method that best suits your task, an expert judgment elicitation protocol should be designed. The design of the elicitation should include the following elements:

• Type of information sought (i.e., generative or evaluative)
• Specific information requiring expert judgment
• Method for eliciting expert judgment (e.g., see Tables 5 and 6)
• Mode for eliciting the expert judgment (e.g., interactive group, nominal group, individual)
• Type of expertise required and method of expert selection
• Form of pre-elicitation training (to inform about process and to debias)

Using these elements as a starting point, an elicitation protocol can be developed. It should be field-tested to determine face validity and to identify which areas require improvement or refinement.

5.3.4 Select Experts

Expert selection is paramount for expert judgment to be useful and informative. To determine how best to select experts, having just identified the type of information and method/mode of elicitation, you will now identify the requisite expertise. This might include the following:

• Expertise about scope and activities needed to complete a specific project
• Expertise about costing and estimates for work packages
• Expertise about potential risks posed by various phases of a specific project
• Expertise about stakeholders

Select four to eight experts (from inside the project team and outside of the organization to ensure a diversity of perspectives) who have both the necessary experience and knowledge, as well as the appropriate credentials.

5.3.5 Train Experts

Train the experts before the actual expert judgment elicitation. Share with them what information is being sought and why/how it will be used in managing the project. Provide experts with awareness of various biases, heuristics, and common pitfalls. Also give them practical means of dealing with these issues, often through practice. Demonstrate the elicitation protocol through practice problems.

5.3.6 Elicit Judgments Using Appropriate Methods

Actual elicitation requires familiarity with the expert judgment method. After determining if a generative or evaluative method is required, select a specific elicitation method. Note that although brainstorming is most frequently used, it is not appropriate for many expert judgment elicitation situations, including obtaining evaluative information. Do not over-rely on brainstorming or other ad hoc methods; instead, select the best method and read up on how it works. Then practice the method with normative experts to develop proficiency in administering the elicitation. After that, elicit the judgments.

5.3.7 Analyze Judgments and Combine (if Desired)

If possible, assess expert performance (e.g., through performance on seeded variables) to determine weights and which expert judgments should be included. However, do not arbitrarily omit expert judgment. For generative elicitations, behavioral aggregation of judgments (e.g., consensus) will be sufficient. For evaluative elicitations, simply average expert judgments to get an aggregate judgment unless you have a compelling reason to do otherwise (since the research has shown averaging to outperform other combination techniques in most cases).

5.3.8 Document Results and Communicate

Documentation is necessary for both historic purposes and communication purposes. The documentation of expert judgment should allow any reviewer to reconstruct the logic and outcomes of the expert elicitation. Documentation should occur at all seven steps of the generic protocol, including the following:

1. Problem Statement: A concise statement of the problem and list of information needed
2. Elicitation Plan: An expert judgment plan and a validated elicitation protocol (with instructions)
3. Expert Selection: A list of experts, their affiliations, and their curricula vitae
4. Expert Training: Lesson plan for training, copies of training materials, list of attendees, and questions/lessons learned for future improvement
5. Judgment Elicitation: Compilation of all judgments and rationale for why those judgments were made (and notes about the elicitation process)
6. Judgment Aggregation: A description of the method, documentation of any interaction, and the aggregated judgment
7. Elicitation Documentation: Using the above information, a comprehensive report should be developed to communicate the expert judgments and how they were developed