The approach we took is called decision matrix analysis; see http://www.asq.org/learn-about-quality/decision-making-tools/overview/decision-matrix.html for a brief description of this technique. Our prioritization approach posits that a software group should preferentially tackle those projects that most strongly align with the group’s business and technical success drivers. Therefore, the first step in building your own model is to

Begin by listing the factors that feed into your group’s thought process when you decide whether a specific project proposal makes good sense. Identify only 10 to 15 project prioritization drivers as more will become unwieldy. Consider including key customer representatives in this process. Involving them as partners in deciding how to make resource decisions can be a procedural and political plus.

Table 3-1 lists the factors the Contoso Internet development team considered incorporating into their model. Some of these might not apply to your situation, and you may well have other powerful factors that influence your decisions. It’s important to maintain a balance between business and technical drivers. Always chasing the next sale (a business-driven perspective) is as dysfunctional as always chasing the next shiny object (an engineering-driven perspective).

Next, to determine how each candidate project stacks up with respect to each prioritization driver, you must

After some experimentation, we selected a four-level scale, with point values of 1, 3, 5, and 7. A scale with an even number of choices forces the raters to make a decision, rather than assigning "medium" ratings to many of the drivers. After all, the objective of the prioritization is to spread the candidate projects across a spectrum so you can identify the high-impact opportunities. That’s the same rationale behind using a scale from 1 to 7, rather than using the narrower range of 1 to 4.

Note that several of the priority drivers in Table 3-1 actually contribute in a negative way to a project’s desirability. Examples include the risk from the necessary technologies, impact on other Web sites, user interface complexity, and the amount of maintenance and support required. We defined each factor’s rating scale so that higher scores correspond to a more favorable indicator for undertaking a project. Using this scheme, even the negative drivers contribute points to the total score. As an alternative approach, you could assign negative weights to those drivers instead of reversing the rating scale. This would amplify the impact the negative drivers have on the project scores.

You also need to provide some cues to help model users interpret the scale consistently. Table 3-1 presents cues that define what is meant by a rating of 1, 3, 5, or 7 for each prioritization driver. Modify these as appropriate for your situation. Quantifying the cues when possible will reduce the subjective bias that goes into assigning ratings and helps to achieve consistent use of the scale over time and between people.

Some of these drivers probably play a greater part in your decision making than others, so step three in creating your prioritization model is to

Allocate 100 points among your project prioritization drivers such that those that are most important to you receive the greatest weight. This is not a high-precision exercise, so use multiples of five for the weights. Assigning fewer than five points to a driver suggests that it plays an insignificant role in your thought process.

To apply the model, you rate each candidate project with respect to each prioritization driver. Formulas in the spreadsheet will multiply each driver rating by the corresponding weighting factor. The sum of these weighted scores for all drivers yields a priority score for each project. The 1 to 7 rating scale yields a maximum possible priority score of 700 points. All other things being equal, those projects that have the highest scores should most closely align with your organization’s value drivers or strategic objectives, and hence they should be your top priorities.

Table 3-2 illustrates a sample application of the prioritization model. It shows the drivers the Contoso Internet development team chose from the master list in Table 3-1, how they allocated their 100 weighting points, and how they rated four candidate projects. This weight allocation indicates that the opportunity the project poses for generating revenue most strongly influences the decision to undertake a proposed project. That factor accounts for one-fifth of what makes a project attractive. The model suggests that Project C should have the top priority, whereas Project D can wait for a future time, perhaps forever.

However, even the most compelling theoretical model is worthless unless you’re confident that it provides some meaningful predictive capability. Therefore, the final model development step is to

The Contoso team calibrated its model by analyzing several recently completed projects. We compared the model-derived scores with our subjective, after-the-fact evaluation of how smart it really was to have undertaken each of those projects. Initially we saw some disconnects. If the model yields approximately the same score for two projects, but you believe that one was a great idea and you should have avoided the other, you need to adjust the model. The weighting factors might not be right, the scale definitions might need tweaking, perhaps you included unnecessary drivers that skewed the results, or you could be missing some important prioritization drivers.

We modified our initial weighting factors until the model scores roughly agreed with our subjective evaluation of how favorable it really was to have undertaken each past project. In one case, two projects that had nearly identical scores were both favorable but for different reasons. The fact that the model yielded high scores for them both gave us confidence that our model had a useful predictive capability.

Make sure you understand your model’s shortcomings, as some stakeholders might attempt to exploit any loopholes. For example, in the Contoso Pharmaceuticals model, tight deadlines ("immediacy of the need") contributed to higher priority. We needed to ensure that this factor was used appropriately and not abused. We didn’t want users to wait until a crisis erupted to bring a project to our attention, in an attempt to gain top priority. However, we did need to favor those projects that truly were an immediate need.