Decision Analysis and Resolution: A Support Process Area at Maturity Level 3
Purpose
The purpose of Decision Analysis and Resolution (DAR) is to analyze possible decisions using a formal evaluation process that evaluates identified alternatives against established criteria.
Tip
DAR provides organizations with a criteria-based approach to making important decisions objectively.
Introductory Notes
The Decision Analysis and Resolution process area involves establishing guidelines to determine which issues should be subject to a formal evaluation process and applying formal evaluation processes to these issues.
A formal evaluation process is a structured approach to evaluating alternative solutions against established criteria to determine a recommended solution.
Hint
Many key acquisition decisions are made early in a project’s lifecycle and have a major impact on project outcomes. Start by using formal evaluation methods on these key decisions before applying them to selected day-to-day decisions. An effective source selection process and well-thought-out lifecycle milestone decisions are results of classic DAR activities.
A formal evaluation process involves the following actions:
• Establishing the criteria for evaluating alternatives
• Identifying alternative solutions
• Selecting methods for evaluating alternatives
• Evaluating alternative solutions using established criteria and methods
• Selecting recommended solutions from alternatives based on evaluation criteria
Rather than using the phrase “alternative solutions to address issues” each time, in this process area, one of two shorter phrases are used: “alternative solutions” or “alternatives.”
Tip
DAR takes the blame out of decision making. A bad decision is made when all of the necessary information that might influence the decision was not considered and relevant stakeholders were not consulted.
A repeatable criteria based decision-making process is especially important, both for making critical decisions that define and guide the acquisition process and later for critical decisions made with the selected supplier. The establishment of a formal process for decision making provides the acquirer with documentation of decision rationale. Such documentation allows criteria for critical decisions to be revisited when changes or technology insertion decisions that affect requirements or other critical project parameters are considered. A formal process also supports the communication of decisions between the acquirer and supplier.
Hint
Maintain the rationale for key decisions that drive the acquisition strategy to make early project thinking available to team members and stakeholders who join the project later.
A formal evaluation process reduces the subjective nature of a decision and provides a higher probability of selecting a solution that meets multiple demands of relevant stakeholders.
X-Ref
For information on how to create an acquisition strategy, see “Techniques for Developing an Acquisition Strategy by Profiling Software Risks” at www.sei.cmu.edu/library/abstracts/reports/06tr002.cfm.
While the primary application of this process area is to technical concerns, formal evaluation processes can be applied to many nontechnical issues, particularly when a project is being planned. Issues that have multiple alternative solutions and evaluation criteria lend themselves to a formal evaluation process.
Guidelines are created for deciding when to use formal evaluation processes to address unplanned issues. Guidelines often suggest using formal evaluation processes when issues are associated with medium-to-high-impact risks or when issues affect the ability to achieve project objectives.
Tip
Examples of nontechnical issues include staffing profiles for the acquisition project team over time and the mix of skills required to ensure project success.
Defining an issue well helps to define the scope of alternatives to be considered. The right scope (i.e., not too broad, not too narrow) will aid in making an appropriate decision for resolving the defined issue.
Formal evaluation processes can vary in formality, type of criteria, and methods employed. Less formal decisions can be analyzed in a few hours, use few criteria (e.g., effectiveness, cost to implement), and result in a one- or two-page report. More formal decisions can require separate plans, months of effort, meetings to develop and approve criteria, simulations, prototypes, piloting, and extensive documentation.
Both numeric and non-numeric criteria can be used in a formal evaluation process. Numeric criteria use weights to reflect the relative importance of criteria. Non-numeric criteria use a subjective ranking scale (e.g., high, medium, low). More formal decisions can require a full trade study.
Tip
Tools such as the Analytic Hierarchy Process (AHP), Quality Function Deployment (QFD), Pugh Method, Delphi Method, prioritization matrices, cause-and-effect diagrams, decision trees, weighted-criteria spreadsheets, and simulations can be used to incorporate weights in decision making.
A formal evaluation process identifies and evaluates alternative solutions. The eventual selection of a solution can involve iterative activities of identification and evaluation. Portions of identified alternatives can be combined, emerging technologies can change alternatives, and the business situation of suppliers can change during the evaluation period.
A recommended alternative is accompanied by documentation of selected methods, criteria, alternatives, and rationale for the recommendation. The documentation is distributed to relevant stakeholders; it provides a record of the formal evaluation process and rationale, which are useful to other projects that encounter a similar issue.
While some of the decisions made throughout the life of the project involve the use of a formal evaluation process, others do not. As mentioned earlier, guidelines should be established to determine which issues should be subject to a formal evaluation process.
Related Process Areas
Refer to the Integrated Project Management process area for more information about establishing the project’s defined process.
Refer to the Risk Management process area for more information about identifying and analyzing risks and mitigating risks.
Specific Practices by Goal
SG 1 Evaluate Alternatives
Decisions are based on an evaluation of alternatives using established criteria.
Issues requiring a formal evaluation process can be identified at any time. The objective should be to identify issues as early as possible to maximize the time available to resolve them.
SP 1.1 Establish Guidelines for Decision Analysis
Establish and maintain guidelines to determine which issues are subject to a formal evaluation process.
Not every decision is significant enough to require a formal evaluation process. The choice between the trivial and the truly important is unclear without explicit guidance. Whether a decision is significant or not is dependent on the project and circumstances and is determined by established guidelines.
Refer to the Risk Management process area for more information about evaluating, categorizing, and prioritizing risks.
Example Work Products
1. Guidelines for when to apply a formal evaluation process
1. Establish guidelines for when to use a formal evaluation process.
2. Incorporate the use of guidelines into the defined process as appropriate.
Refer to the Integrated Project Management process area for more information about establishing the project’s defined process.
SP 1.2 Establish Evaluation Criteria
Establish and maintain criteria for evaluating alternatives and the relative ranking of these criteria.
Evaluation criteria provide the basis for evaluating alternative solutions. Criteria are ranked so that the highest ranked criteria exert the most influence on the evaluation.
This process area is referenced by many other process areas in the model, and many contexts in which a formal evaluation process can be used. Therefore, in some situations you may find that criteria have already been defined as part of another process. This specific practice does not suggest that a second development of criteria be conducted.
A well-defined statement of the issue to be addressed and the decision to be made focuses the analysis to be performed. Such a statement also aids in defining evaluation criteria that minimize the possibility that decisions will be second guessed or that the reason for making the decision will be forgotten. Decisions based on criteria that are explicitly defined and established remove barriers to stakeholder buy-in.
Example Work Products
1. Documented evaluation criteria
2. Rankings of criteria importance
Subpractices
1. Define the criteria for evaluating alternative solutions.
Criteria should be traceable to requirements, scenarios, business case assumptions, business objectives, or other documented sources.
2. Define the range and scale for ranking the evaluation criteria.
Scales of relative importance for evaluation criteria can be established with non-numeric values or with formulas that relate the evaluation parameter to a numeric weight.
3. Rank the criteria.
The criteria are ranked according to the defined range and scale to reflect the needs, objectives, and priorities of the relevant stakeholders.
4. Assess the criteria and their relative importance.
5. Evolve the evaluation criteria to improve their validity.
6. Document the rationale for the selection and rejection of evaluation criteria.
Documentation of selection criteria and rationale may be needed to justify solutions or for future reference and use.
SP 1.3 Identify Alternative Solutions
Identify alternative solutions to address issues.
A wider range of alternatives can surface by soliciting as many stakeholders as practical for input. Input from stakeholders with diverse skills and backgrounds can help teams identify and address assumptions, constraints, and biases. Brainstorming sessions can stimulate innovative alternatives through rapid interaction and feedback.
Sufficient candidate solutions may not be furnished for analysis. As the analysis proceeds, other alternatives should be added to the list of potential candidate solutions. The generation and consideration of multiple alternatives early in a decision analysis and resolution process increases the likelihood that an acceptable decision will be made and that consequences of the decision will be understood.
Hint
When you are evaluating competing designs proposed by your suppliers, make sure you solicit alternatives that may be more innovative and less obvious. Consider commissioning a special study by a team with a lower stake in the selection outcome to help ensure that a wide range of alternatives are identified.
Example Work Products
1. Identified alternatives
Example Supplier Deliverables
1. Supplier identified alternatives, if any
Subpractices
1. Perform a literature search.
A literature search can uncover what others have done both inside and outside the organization. Such a search can provide a deeper understanding of the problem, alternatives to consider, barriers to implementation, existing trade studies, and lessons learned from similar decisions.
2. Identify alternatives for consideration in addition to the alternatives that may be provided with the issue.
Evaluation criteria are an effective starting point for identifying alternatives. Evaluation criteria identify priorities of relevant stakeholders and the importance of technical, logistical, or other challenges.
Combining key attributes of existing alternatives can generate additional and sometimes stronger alternatives.
Solicit alternatives from relevant stakeholders. Brainstorming sessions, interviews, and working groups can be used effectively to uncover alternatives.
3. Document proposed alternatives.
SP 1.4 Select Evaluation Methods
Select evaluation methods.
Methods for evaluating alternative solutions against established criteria can range from simulations to the use of probabilistic models and decision theory. These methods should be carefully selected. The level of detail of a method should be commensurate with cost, schedule, performance, and risk impacts.
Hint
Make sure the method chosen is appropriate for the decision. A complex and time-consuming approach may be overkill for a simple binary decision.
While many problems may require only one evaluation method, some problems may require multiple methods. For example, simulations may augment a trade study to determine which design alternative best meets a given criterion.
Suppliers competing to develop a technical solution for the acquirer may be directly evaluated in a final competition that involves a performance or functional demonstration of proposed solutions.
Example Work Products
1. Selected evaluation methods
Subpractices
1. Select methods based on the purpose for analyzing a decision and on the availability of the information used to support the method.
2. Select evaluation methods based on their ability to focus on the issues at hand without being overly influenced by side issues.
Results of simulations can be skewed by random activities in the solution that are not directly related to the issues at hand.
3. Determine the measures needed to support the evaluation method.
Consider the impact on cost, schedule, performance, and risks.
SP 1.5 Evaluate Alternative Solutions
Evaluate alternative solutions using established criteria and methods.
Evaluating alternative solutions involves analysis, discussion, and review. Iterative cycles of analysis are sometimes necessary. Supporting analyses, experimentation, prototyping, piloting, or simulations may be needed to substantiate scoring and conclusions.
Often, the relative importance of criteria is imprecise and the total effect on a solution is not apparent until after the analysis is performed. In cases where the resulting scores differ by relatively small amounts, the best selection among alternative solutions may not be clear. Challenges to criteria and assumptions should be encouraged.
Tip
Evaluation criteria for a competitive source selection should be part of the solicitation package so that all stakeholders understand the relative importance of solicitation requirements.
Example Work Products
1. Evaluation results
X-Ref
Refer to SP 1.2 of SSAD for more information on establishing evaluation criteria for use in selecting suppliers.
Subpractices
1. Evaluate proposed alternative solutions using the established evaluation criteria and selected methods.
2. Evaluate assumptions related to the evaluation criteria and the evidence that supports the assumptions.
3. Evaluate whether uncertainty in the values for alternative solutions affects the evaluation and address these uncertainties as appropriate.
For instance, if the score varies between two values, is the difference significant enough to make a difference in the final solution set? Does the variation in score represent a high-impact risk? To address these concerns, simulations may be run, further studies may be performed, or evaluation criteria may be modified, among other things.
4. Perform simulations, modeling, prototypes, and pilots as necessary to exercise the evaluation criteria, methods, and alternative solutions.
Untested criteria, their relative importance, and supporting data or functions can cause the validity of solutions to be questioned. Criteria and their relative priorities and scales can be tested with trial runs against a set of alternatives. These trial runs of a select set of criteria allow for the evaluation of the cumulative impact of criteria on a solution. If trials reveal problems, different criteria or alternatives might be considered to avoid biases.
5. Consider new alternative solutions, criteria, or methods if proposed alternatives do not test well; repeat evaluations until alternatives do test well.
Document the rationale for the addition of new alternatives or methods and changes to criteria, as well as the results of interim evaluations. Determine the scores for each alternative based on criteria evaluations and scoring methods previously determined.
6. Document the results of the evaluation.
Document the rationale for the addition of new alternatives or methods and changes to criteria, as well as the results of interim evaluations.
SP 1.6 Select Solutions
Select solutions from alternatives based on evaluation criteria.
Selecting solutions involves weighing results from the evaluation of alternatives. Risks associated with the implementation of solutions should be assessed.
Tip
Selecting a solution always requires humans to make decisions. These decisions are informed by the results of evaluations. Other considerations (such as risk or external constraints) may cause a project to select solutions that score lower based on the chosen criteria. If this is the case, the rationale for the selection becomes even more critical to maintain.
Example Work Products
1. Recommended solutions to address significant issues
Subpractices
1. Assess the risks associated with implementing the recommended solution.
Refer to the Risk Management process area for more information about identifying and analyzing risks and mitigating risks.
Decisions must often be made with incomplete information. There can be substantial risk associated with the decision because of having incomplete information.
When decisions must be made according to a specific schedule, time and resources may not be available for gathering complete information. Consequently, risky decisions made with incomplete information can require re-analysis later. Identified risks should be monitored.
2. Document and communicate to relevant stakeholders the results and rationale for the recommended solution.
It is important to record both why a solution is selected and why another solution was rejected.