14.8. Project Prioritization

An important set of decisions with which a pharmaceutical sponsor is faced is that of the choice of drug development projects and more generally of the prioritization of the projects chosen. This topic has been considered by a number of authors, in particular by Bergman and Gittins (1985) but also by Senn (1996, 1997, 1998), Senn and Rosati (2002), Zipfel (2003), and Burman and Senn (2003).

It is instructive to consider the simpler problem of deciding which drugs to develop, since assessing the value of potential projects prior to inclusion in a portfolio may also suggest how, if at all, those projects that are included might be prioritised. A simple analogy, which is frequently made, is that of packing a number of objects of different values and volumes into a suitcase so that the resulting package is as valuable as possible. The suitcase is a metaphor for the constraints facing drug developers, the volumes of the objects the extent to which the projects contribute individually to reaching or exceeding the constraints and the values of the objects as the expected returns the objects will bring to the sponsor. Put like this, the problem is one of optimisation subject to constraints, and it might be thought that linear programming, more specifically integer programming, would be the appropriate tool for solving the problem. In fact, there are various aspects of the problem that suggest that these techniques may be less useful than might at first sight be supposed (this point will be taken up later) and in any case it is useful to examine individual projects in terms of return on investment. We will review this aspect first before returning to optimization considerations.

The simplest useful index that may be used to evaluate projects in development for the purpose of portfolio management is the so-called Pearson Index (Pearson, 1972), which is, effectively, a return on investment index. This calculates for a given project the expected reward as a ratio of the expected cost. It may be defined as follows. First, we assume that the project consists of n stages that have been optimally sequenced. Sunk costs are to be ignored, so that at any stage we can redefine a project, for the purpose of making decisions, as consisting of future stages only. The discounted cost of a given stage i is c_i and the conditional probability of success in that stage given that the project has reached that stage is p_i, with p₀ = 1 by convention. The expected current value of the future revenue of a successful project is r. Then the Pearson Index may be written as

Here the denominator is the expected future cost of a project and this reflects the fact that costs will only be paid for a given stage if it is reached and not if the project fails. The first term in the numerator is the unconditional expected future revenue. That is, it is the expected future revenue, r, of a successful project multiplied by the probability of success, Π_i=1ⁿ p_i. Subtracting the denominator from this term produces the expected return net of expected costs, which is then the numerator of the Pearson Index.

The index has the necessary property that other things being equal it will rank projects with a favourable cost and probability architecture (Senn, 1996, 1998; Zipfel, 2003) more highly than those with an unfavourable architecture. Other things being equal, projects that will fail early (if they fail) are to be preferred to those that will fail late and projects with later costs are to be preferred to those with earlier ones. An example is given by Senn (1996).

Consider four projects, A, B, C and D in three stages, details for which (including costs and success probabilities) are given in Table 14.2. The value of r (reward) in all four cases is 28. There is nothing to choose among these projects in terms of cost to see to completion (1 + 2 + 3 = 6), nor in terms of return if successful (28 − 6 = 22), nor in terms of probability of success (0.8 × 0.6 × 0.4 = 0.192). These figures are the same in all four cases. However, Projects C and D have a favourable cost architecture and Projects B and D have a favourable probability architecture. In fact, as shown in Table 14.2, the Pearson Index values for the four projects are 0.058 for Project A, 0.331 for Projects B and C and 1.133 for Project D. The index indicates a ratio of 1.133/0.058 = 19.5 between Projects D and A and thus the most attractive project (Project D) is nearly 20 times as attractive as the least attractive one (Project A).

The three-stage example of the Pearson index is simple enough to be solved with a hand calculator. In a more complex situation, PROC DTREE may, of course, be useful. Program 14.17 calculates the expected reward, i.e., the numerator in the Pearson index, for two of the scenarios. The original PEARSON data set gives the cost architecture for Project A (and B). It shows that the costs of the three succesive trials are in order 1, 2 and 3, and that the tree reaches the end as soon as a trial failure is encountered. It also gives the terminal gain 28 in case all the chance events turn out to be OK trial results.

The MODIFY statement in PROC DTREE can be used interactively to modify the costs for the different stages. In the program, the cost of the first trial is changed from 1 to 3 and for the third trial from 3 to 1. The resulting PEARSON data set contains the appropriate costs for trial C (and D). The PROB data set states that the probabilities of failure is 0.2, 0.4 and 0.6 for Trials 1, 2 and 3, respectively. This is the probability architecture for Projects A and C.

Example 14-17. Calculaton of the expecting reward in the Pearson model

data pearson;
    input _stname_ $ 1-8 _sttype_ $ 9-12 _outcome_ $ 13-22
        _reward_ 23-26 _success_ $ 27-36;
    datalines;
Start   D   No go      .  .
        .   Go        −1  Stage1
Stage1  C   Failure1   .  .
        .   OK1       −2  Stage2
Stage2  C   Failure2   .  .
        .   OK2       −3  Stage3
Stage3  C   Failure3   .  .
        .   OK3       28  .
;
data prob;
   input _event1_ $10. _prob1_ _event2_ $5. _prob2_;
   datalines;
Failure1  0.2  OK1  0.8
Failure2  0.4  OK2  0.6
Failure3  0.6  OK3  0.4
    ;
proc dtree stagein=pearson probin=prob;
    evaluate/summary;
    save;
    modify Go reward −3;
    modify OK2 reward −1;
    evaluate/summary;
    run;
    quit;

Example. Output from Program 14.17

Decision Parameters

                        Decision Criterion:    Maximize Expected Value (MAXEV)
                   Optimal Decision Yields:    1.34

                                    Optimal Decision Policy

                                       Up to Stage Start

                            Alternatives    Cumulative    Evaluating
                            or Outcomes         Reward         Value
                            ----------------------------------------
                            No go                    0         0.00
                            Go                      −1         2.34*


                                     Decision Parameters

                        Decision Criterion:    Maximize Expected Value (MAXEV)
                   Optimal Decision Yields:    0.296

Optimal Decision Policy

                                       Up to Stage Start

                            Alternatives    Cumulative    Evaluating
                            or Outcomes         Reward         Value
                            ----------------------------------------
                            No go                    0        0.000
                            Go                      −3        3.296*

The way that the pearson index could be used in practice would be to rank projects by their ratios and to select them starting with the highest and proceeding as far down the list as constraints permit. this is strongly analogous to the neyman-pearson lemma which indicates that selecting points in the sample space to construct a critical region such that a desideratum of power is maximised subject to a constraint of size (type i error rate) should be done by including points with the highest likelihood ratio. ^[]. For the project selection problem, the overall research budget is analogous to the tolerated Type I error rate, the expected return is analogous to power and the Pearson Index to the likelihood ratio. This brings us back to the issue of integer programming discussed previously. As is well known in the context of hypothesis testing, for problems in which the sample space is discrete, improved power can be delivered for a given size under suitable circumstances by going beyond the likelihood ratio criterion. The apparently unending discussion of "superior" alternatives to Fisher's exact test bears witness to this. The analogy for the portfolio selection problem would be that by going beyond the Pearson Index formulation we might, by sacrificing a large project with a high index but replacing it by two smaller projects with less favourable indices, end up with a more valuable portfolio. This is, however, probably not a good idea for similar reasons to those that suggest that "improvements" to Fisher's exact test should not be employed. It is better to look upon the resources available in a drug development project as having some flexibility so that by trading projects or raising capital one could avoid such index-violating decisions.

^[] It is perhaps worth pointing out that Alan Pearson of the Pearson Index is not to be confused with Egon Pearson of the Neyman-Pearson lemma!

Nevertheless, for the sake of completeness and in order to illustrate further features of SAS, we will demonstrate the application of integer programming in a simple example below. This should not be taken as an indication that this is necessarily the way we believe that the portfolio management problem should be approached.

Let us look at a simple portfolio planning example using integer programming (Program 14.18). The costs and rewards for ten project candidates are specified in the PROJECTS data set. The budget restriction is given by the _RHS_ variable and set to 100. Finally, all optimisation variables are binary, that is, a project is either run (ACTIVE=1) or not run (ACTIVE=0).

Example 14-18. Integer programming to optimise portfolio

data projects;
   input _id_ $6. project1-project10 _type_ $ _rhs_;
   datalines;
cost   47 35 22 22 18 16 15 10  9  6 le    100
reward 95 75 40 25 22 20 46 15 30 32 max     .
binary  1  1  1  1  1  1  1  1  1  1 binary  .
;

proc lp data=projects activeout=results;
proc transpose data=results out=results;
proc sort data=results;
    by _name_;
proc transpose data=projects out=projects;
    id _id_;
proc sort data=projects;
    by _name_;
data together (rename=(_name_=Projects));
    merge projects results;
    by _name_;
    activity=col1;
    index=reward/cost;
    if substr(_name_,1,7)="project";
    keep _name_ cost reward index activity;
proc sort data=together;
    by descending index;
proc print data=together;
    run;

Example. Output from Program 14.18

The LP Procedure

                               Constraint Summary

                Constraint            S/S                          Dual
            Row Name       Type       Col       Rhs  Activity  Activity

              1 cost       LE          11       100        99         0
              2 reward     OBJECTVE     .         0       243         .

           Obs    Projects     cost    reward    activity     index

             1    project10      6       32          1       5.33333
             2    project9       9       30          1       3.33333
             3    project7      15       46          1       3.06667
             4    project2      35       75          0       2.14286
             5    project1      47       95          1       2.02128
             6    project3      22       40          1       1.81818
             7    project8      10       15          0       1.50000
             8    project6      16       20          0       1.25000
             9    project5      18       22          0       1.22222
            10    project4      22       25          0       1.13636

Output 14.18 displays some of the output from the LP procedure as well as a list of the projects, ordered by the value of the reward/cost-index, and indicating the optimal set of projects. The output of PROC LP states that the optimal solution gives a reward of 243 at a cost of 99. Note, from the project list, that the fourth most profitable project (project2) according to the index is not chosen while two projects with lower index values are included.

We may try to change the budget restriction and study how the reward is increasing and the optimal set of projects is changing. In our example, the optimal reward is constant, 243, for budget restriction in the interval [99,103). It is 248 for budgets in [103,109). Considering risks and cost of capital, it may not be worth an additional investment of 103 – 99 = 4 to increase expected rewards by 248 – 243 = 5. However, increasing the budget to 109 increases the reward to 258 and, with a budget of 112, a reward of 278 is possible. The last option corresponds to running the five projects with highest index.

Although the Pearson index is useful, it is not perfect. Its adequacy depends on the extent to which it captures the options facing a drug developer as regards projects in general. The index captures the obvious option on any project as regards costs, that is to say, to abandon projects that no longer have any chance of being successful. Other options that it does not capture, however, include the option to revise decisions in the light of developing market information and the option to revise investment decisions in the light of changes in the rate of interest (Senn, 1998; Senn and Rosati, 2002). For a general introduction to this field, see Dixit and Pyndick (1994).