Executive Summary

Problem: In the face of unknown futures, project managers often turn to expert judgment as a source of key information in an effort to ensure that projects are completed on time, on budget, and in accordance with stakeholder expectations. As a project management tool/technique, expert judgment is ubiquitous in the body of knowledge. However, most organizations do not have written guidance on how to elicit expert judgment, and most project managers rely on ad hoc methods for gathering expert judgment, which is known to result in flawed information that can adversely impact project success. Therefore, this study sets out to identify ways in which expert judgment might be improved for project management.

Background: This study was primarily sponsored by the Project Management Institute. The study comprised three phases. The first phase investigated the state of the art/science of expert judgment elicitation broadly (e.g., across a variety of disciplinary areas, such as engineering, environmental management, medicine, political science, and space exploration). The second phase identified the state of the practice of expert judgment in project management. Together, the first two phases identified several theory-practice gaps. The third phase examined methods for closing key gaps. A complete description of the study (including the problem, the background, and the methods) is provided in Chapter 1 of this report.

Process: Spanning 15 months, this study employed a mixed-methods approach. A comprehensive review of the literature was conducted to identify the state of the art/science. Using a 10-step process, more than a thousand relevant articles and studies were found for the review period, of which dozens were deemed applicable to the problem at hand. Chapter 2 of the report contains a detailed description of phase 1. A descriptive survey was conducted to determine the state of the practice. Surveys were sent to all of the regional chapters of the Project Management Institute, a professional organization with nearly half a million members. There were more than 400 responses from a representative sample of the organization. The sample was subjected to descriptive statistics and multivariate analysis. Chapter 3 of the report provides a detailed description of phase 2. Finally, to close a key gap, experiments were conducted. Undergraduate students from two universities provided expert judgments through tested elicitation protocols. The details of phase 3 are described in Chapter 4 of the report.

Findings and Conclusions: There were many findings and conclusions. In general, the state of the art/science of expert judgment outside of project management is established and continually evolving. The state of the practice of expert judgment in project management is informal and emergent. There is considerable opportunity to inform and mature the practice of expert judgment in project management by adopting best practices from other professions and disciplines. Further specific findings and the rationale for them are provided throughout the report and in Chapter 5 specifically.

Recommendations: To advance the practice of using expert judgment in project management, the following general recommendations are provided:

• Organizations should provide written guidance on how to conduct expert judgment using a standard framework that includes the following seven steps: frame the problem, plan the elicitation, select the experts, train the experts, elicit the judgments, analyze and combine the judgments, and document and communicate the results.
• In order to best leverage the judgment of experts, clearly frame the problem to be considered and identify the exact nature of the information to be sought.
• Determine if the expert judgment is generative (i.e., creating lists, risks, options, etc.) or evaluative in nature (i.e., estimating cost, duration, quantifying a phenomenon of interest, etc.). Select an appropriate expert judgment elicitation method. Create an elicitation protocol using that method and test it with normative experts.
• Using appropriate criteria, select a diverse pool of four to eight experts from both inside the project management organization and outside it, including key stakeholders. Fluency (numeracy) tests will help in selecting experts for generative (evaluative) tasks.
• Train experts prior to the elicitation on the reasons judgments are needed, the process of the protocol, and means to mitigate known biases through practice.
• Elicit expert judgments using previously selected, established methods. Do not over-rely on brainstorming or ad hoc processes.
• Where possible, evaluate expert performance in order to weight or select judgments accordingly. Unless there is a compelling reason to do otherwise, use simple averaging to combine evaluative judgments. Use an interactive consensus process to combine generative judgments.
• Document the entire elicitation process, including recording the method used, expert information, and resulting judgments. This will serve as an organizational process asset and lessons learned for future projects that involve the elicitation of expert judgment.

Further specific recommendations are provided in Chapter 5 of the report, and the References list is a good source for self-study to improve proficiency in conducting expert judgment elicitations in project management.