14.2. Introductory example: stop or go?
Decision trees are very useful for analysing decision problems. The DTREE procedure can be used to construct, plot and evaluate such decision trees. In some cases there is an enormous number, or even a continuum, of decision alternatives available. Such problems can often be attacked with optimisation routines and/or simulations (see Sections 14.5 and 14.7). In this section, we will introduce a very simple decision tree example, which will be expanded in Sections 14.4 and 14.5.
A classic decision problem is whether to invest or not. In a drug development setting, the question might be if a large and expensive Phase III programme should be launched or if the development of the drug should be discontinued. The consequences of a discontinuation are fairly easy to assess. There is, however, considerable uncertainty in the results of a Phase III programme, comparing a new drug candidate with the current standard therapy. To simplify things, the result might be that the new drug is regarded to be either superior, non-inferior (but not superior), or inferior to the active control. The new drug cannot be marketed in case of inferiority as regulatory approval will then not be given. Superior or non-inferior efficacy results will both lead to marketing. However, the regulatory labeling will be different and the market appreciation and sales are likely to be widely different.
Program 14.1 uses PROC DTREE of the SAS/OR to create a decision tree for the described go/no go problem. The STAGE1 data set describes the structure of the tree. Each of the five rows in this data set corresponds to a branch and the four variables define tree characterisics such as node types and branch labels. For example, the _STTYPE_ variable assumes two values, D and C, that identify the decision and chance nodes, respectively. The decision node is a decision to conduct the study or not and the chance node is a (random) outcome of the trial (superiority, non-inferiority or inferiority). The _OUTCOME_ variable specifies labels for the five branches in this decision tree.
Example 14-1. Simple go/no go problem with an efficacy outcome variable
data stage1;
length _outcome_ $6.;
input _stname_ $ _sttype_ $ _outcome_ $ _success_ $;
datalines;
Decision D No_go .
. . Go Develop
Develop C Super .
. . Noninf .
. . Infer .
;
* Trial's outcome;
symbol1 value=triangle height=10 color=black width=3 line=1;
* Decision point;
symbol2 value=square height=10 color=black width=3 line=1;
* End nodes;
symbol3 value=none height=10 color=black width=3 line=1;
proc dtree stagein=stage1;
treeplot/graphics norc nolegend
linka=1 linkb=2 symbold=2 symbolc=1 symbole=3;
run;
quit; |
The decision tree generated by Program 14.1 is displayed in Figure 14.1. This decision tree is almost the simplest one that is meaningful. In general, decision trees that describe the clinical development of a drug involve multiple decisions and have a lot more decision and/or chance nodes. We will consider decision trees arising in clinical trials further in Section 14.4. Here we will refine the tree somewhat on the consequence side.
Figure 14-1. Decision tree in the simple go/no go problem. The square node is the decision to conduct the trial or not and the triangle node is the trial's efficacy outcome (superiority, non-inferiority or inferiority).
Efficacy of a drug is not everything. Safety is also important. Assume, for example, that the competitor drug is connected to a specific type of adverse events (AE). Our new drug may or may not have the advantage of a considerably lower rate of these AEs, depending on whether the same biological pathway leading to the safety problem is triggered. We will ignore the possibility of getting a higher rate of the AE, as it is assumed that the safety problem is zero-one. It is, however, straightforward to include a possibility of a safety disadvantage in the analysis.
Program 14.2 produces a decision tree (Figure 14.2) which also reflects the outcome of the adverse event comparison in the Phase III trial. The STAGE2 data set defines seven branches and three nodes:
DECISION (decision node): A decision to conduct the trial or not.
DEVELOP (chance node): Outcome of the efficacy analysis (superiority, non-inferiority or inferiority).
AE (chance node): Outcome of the safety analysis (superiority or equivalence).
The tree does not consider the AE comparison when the experimental drug is inferior to the competitor drug in terms of efficacy because the inferiority outcome is assumed to imply that regulatory approval will not be achieved.
Example 14-2. Simple go/no go problem with multiple outcome variables (efficacy and safety)
data stage2;
length _outcome_ $15.;
input _stname_ $ _sttype_ $ _outcome_ $ _success_ $;
datalines;
Decision D No_go .
. . Go Develop
Develop C Eff_super AE
. . Eff_noninf AE
. . Eff_infer .
AE C AE_super .
. . AE_equal .
;
* Trial's outcome;
symbol1 value=triangle height=10 color=black width=3 line=1;
* Decision point;
symbol2 value=square height=10 color=black width=3 line=1;
* End nodes;
symbol3 value=none height=10 color=black width=3 line=1;
proc dtree stagein=stage2;
treeplot/graphics norc nolegend
linka=1 linkb=2 symbold=2 symbolc=1 symbole=3;
run;
quit; |
Figure 14-2. Decision tree in the simple go/no go problem with two outcome variables. The square node is the decision to conduct the trial or not and the triangle nodes represent the trial's efficacy outcome (superiority, non-inferiority or inferiority) and safety outcome (superiority or equivalence).
14.2.1. Evaluating a Decision Tree
Structuring a problem is an essential part in solving it. In the above example, this step was rather trivial. For a complex problem, however, it can be quite difficult to find a good description which is as simple as possible and yet captures the essence of the problem. Given the tree structure of a problem, the optimal decision will sometimes be evident. However, it is often necessary to assess the consequences and uncertainties quantitatively.
In the Phase III investment decision, let us say that we can estimate the probabilities of superiority, non-inferiority and inferiority outcomes in a Phase III trial. This estimation of probabilities is, of course, related to power calculations and depends on a number of factors, such as the precise definition of non-inferiority, the assumed effect size and variability, etc. We will come back to such topics later and for now take the computation of these probabilities for granted. Uncertainties in a regulatory response and sales can also be included in a decision analysis model but we will ignore these aspects for now. We will also ignore any other potential safety issues apart from the specific kind of AE under consideration.
Let E+, E= and E− denote the events of superiority, non-inferiority and inferiority, respectively. Similarly, denote the event of proven safety benefit of the new drug with respect to the specified AE by S+ and the complement event (no proven safety improvement) by S=. For the following example, let
We assume that the outcome for the specified AE is independent of the effect outcome and that
The independence may not, of course, be the case in practice and the joint probability of efficacy and safety may have to be considered instead. (Compare the conditioning in a similar situation in Program 14.4.)
Note that the probabilities given are for the different conclusions from the Phase III programme. It is quite possible that superiority cannot be concluded even if the new drug in fact produces a better effect than the competitor. The probabilities are included in the PROB3 data set in Program 14.3.
In order to compare the alternatives we also need to quantify the value of the possible outcomes. Immediate discontinuation of the project is likely to mean ignorable costs and incomes from the drug (we will not consider the possibilities of out-licensing or limited development). We will therefore regard the discontinuation possibility to have value 0. Previous costs or future unavoidable costs are of no interest in the decision problem. What matters is, of course, only the costs and gains which can be influenced by the decisions made now or in the future.
Expected net profit is taken as the optimality criterion. A more sophisticated analysis should consider some additional factors:
A small expected gain from an investment may not be worth the risk of losing the investment. The degree of risk aversion is likely to be larger in smaller companies, for which a failure in Phase III may jeopardise the company's survival.
The resources (e.g., money or personnel) may be limited, forcing the company to choose between several promising projects (see Section 14.8).
Future cash-flows should be discounted by an appropriate rate. The discount rate should primarily reflect the cost of capital, e.g., the interest rate paid. Some organisations assume a higher discount rate in order to also account for the investment risk and/or limited personnel resources. Although such high discount rates might be used as a simplifying tool, we do not recommend them for important decisions. It is better to explicitely model resource restrictions and risk aversion.
One way of dealing with risk is to define the utility as an appropriate concave function of money (Raiffa, 1968, Chapter 4) and use the expected utility as the optimality criterion. For the sake of simplicity, however, we will assume that cash-flows are discounted and that risk aversion and resource restrictions can be ignored.
Suppose that the cost of running the Phase III programme is 250 (the monetary unit in this example can be a million of US dollars, or MUSD). An inferior drug will not be possible to sell and therefore produces no income. Non-inferiority is enough to get some sales but the income will be considerably higher in the case of superiority. Ignoring the development costs, we assume the net profit, as a function of the Phase III programme outcome, to be G(·). The net profit here denotes the sales minus promotional and manufacturing costs over the drug's life-cycle. We will let
and, as stated earlier, G(E−,·) = 0.
Program 14.3 creates a decision tree for the introduced decision analysis problem. The development costs are specified in the STAGE3 data set and the gains are included in the PAYOFF3 data set.
Example 14-3. Evaluated decision tree in the simple go/no go problem with multiple outcome variables (efficacy and safety)
data stage3;
length _outcome_ $10.;
input _stname_ $ _sttype_ $ _outcome_ $ _reward_ _success_ $;
datalines;
Phase3 D No_go . .
. . Go −250 Develop
Develop C Eff_super . AE
. . Eff_noninf . AE
. . Eff_inf . .
AE C AE_super . .
. . AE_equal . .
;data prob3;
length _event1_ _event2_ _event3_ $10.;
input _event1_ $ _prob1_ _event2_ $ _prob2_ _event3_ $ _prob3_ ;
datalines;
Eff_super 0.2 Eff_noninf 0.5 Eff_inf 0.3
AE_super 0.3 AE_equal 0.7 . .
;
data payoff3;
length _state1_ _state2_ $10.;
input _state1_ $ _state2_ $ _value_;
datalines;
Eff_super AE_super 1200
Eff_super AE_equal 550
Eff_noninf AE_super 450
Eff_noninf AE_equal 100
Eff_inf . 0
;
* Trial's outcome;
symbol1 value=triangle height=10 color=black width=3 line=1;
* Decision point;
symbol2 value=square height=10 color=black width=3 line=1;
* End nodes;
symbol3 value=none height=10 color=black width=3 line=1;
proc dtree stagein=stage3 probin=prob3 payoffs=payoff3;
ods select parameters policy;
treeplot/graphics norc nolegend compress
linka=1 linkb=2 symbold=2 symbolc=1 symbole=3;
evaluate/summary;
run;
quit; |
Example. Output from Program 14.3
Decision Parameters
Decision Criterion: Maximize Expected Value (MAXEV)
Optimal Decision Yields: 1.5
Optimal Decision Policy
Up to Stage Phase3
Alternatives Cumulative Evaluating
or Outcomes Reward Value
----------------------------------------
No_go 0 0.0
Go −250 251.5* |
Figure 14.3 displays the decision tree generated by Program 14.3 and cumulative rewards (CR) and expected values (EV) for each branch of the tree. Output 14.3 shows that the optimal value for the go option is +1.5 MUSD. This value is the result of a cumulative cost (negative reward) of 250 MUSD for running the trial and an evaluated expected value at the end node of 251.5 MUSD. The stop option has a zero value.
Figure 14-3. Evaluated decision tree in the simple go/no go problem with two outcome variables. The square node is the decision to conduct the trial or not and the triangle nodes represent the trial's efficacy outcome (superiority, non-inferiority or inferiority) and safety outcome (superiority or equivalence).
It should be noted that there are many possible descriptions of the same problem. Instead of using separate chance nodes for efficacy and safety outcomes, one can, of course, consider only one chance node with an outcome space consisting of five combined events: {E+,S+}, {E+,S=}, {E=,S+}, {E=,S=} and {E–}. Different levels of refinement are possible, and more or less suitable according to the situation. For example, one can average over all possible safety outcomes and just use the probabilities for the efficacy outcomes together with the expected gains given these outcomes, i.e.,
and
This problem formulation would be adequate at the moment. However, the simple decision problem that we have analysed in this section will be expanded in Section 14.4 and the separation between efficacy and safety outcomes will then be crucial. The previous assumption about independence of efficacy and safety results was made for convenience. Dependence is easily treated by providing conditional probabilities in the data set specified in the PROBIN option of PROC DTREE. This is shown in Program 14.4 (see Section 14.4) in another situation of dependence.
Before expanding the drug development problem, we will give a short general description of decision analysis.