19.2.1 Non-stationary Markov Chains

The transitions in multi-state models are normally age-dependent. To handle this, we need to extend the concepts and notation that we introduced in Section 18.4.1 so that they apply to the non-stationary case.

For a non-stationary Markov Chain, in place of the single transition matrix P, we need for each nonnegative integer n, a matrix P_n whose ijth entry is p_ij(n, n + 1)

The main quantities of interest are the probabilities p_ij(k, n) of being in state j at time n conditional upon being in state i at time k. In deriving (18.18), we do not need the fact that the matrices are the same, and the argument given is easily adapted to show that

(19.1)

which reduces to (18.5) in the stationary case. Equation (18.6) now takes the form

For example, in our simple life-death chain the transition matrices are given by

In the multiple decrement example, the matrix P_n will be similar to that above, except the first row will be (p^(τ)_{x + n}, q_{x + n}⁽¹⁾, q⁽²⁾_{x + n}, …, q_{x + n}^(m)), and each other row will have 1 on the main diagonal and 0s elsewhere.

We leave is as an exercise for the reader to write down the matrix P_n in the joint two life case.

We now introduce a multi-state model which is not covered by our previous cases of multiple lives or multiple-decrement theory, and to which we will refer to frequently in the rest of the chapter. It allows for a two-way transition between certain states, which is really where the generality of the multi-state model is the most helpful. This is the chain as depicted in Figure 19.2 where a life can be healthy or unhealthy, but we now allow for recovery from the unhealthy state.

There are various applications, The unhealthy state could refer to being disabled, or sick, or a number of other possibilities for a state which we wish to distinguish somehow from the main group of lives, but which can transfer back to the main group.

Example 19.1 Consider the model of Figure 19.2 and suppose that transition matrices for times 0, 1, 2 are given as follows;

What is the probability that a person who is healthy at time 0 will be unhealthy at time 2 and deceased at time 3?

Solution.

Using formula (19.19), the required probability equals

19.2.2 Discrete-time multi-state insurances

To model the general contract of this type, we will take a Markov chain with states numbered from 0 to N. Suppose the process begins in state a at time 0. Fix any two states i and j, and consider a contract which pays a benefit of b_k at time k + 1, provided a transfer from state i to state j occurs between time k and time k + 1. That is, the process was in state i at time k and state j at time k + 1.

Notation If we have several different transfer benefits we will distinguish by writing b_k as . In general, for quantities like benefits, premiums, reserves and expenses, we will let superscripts refer to states, and maintain our previous usage of subscripts to refer to time. This differs from our convention for probabilities where we use subscripts to refer to states, and write times in brackets.

A convenient method of discussing multi-state benefits in the context of the stochastic model is to make use of indicator random variables. For each nonnegative integer k, let I_k be the random variable that takes the values of 0 or 1 respectively, accordingly as a transfer from i to j has not or has occurred between time k and k + 1. The present value of the benefits on the contract is given by

(19.2)

Since

(19.3)

the actuarial present value is

(19.4)

We again have our familiar pattern with a sum of three-termed factors. In fact, we can consider each summand as a four-termed factor where the probability of receiving a payment is itself the product of two factors, the probability of transition to state i by time k and the probability of transition from state i to state j in the next period.

Example 19.2 Consider the chain given in Example 19.1. A 3-year contract written on a healthy life provides for benefits at the end of the year of death. The death benefit is 3 if the person dies while healthy and 2 if the person dies while unhealthy. If interest is a constant 5%, find the actuarial present value.

Solution. We can view this as two separate contracts, one paying 3 for transfer from state 0 to state 2, and the second paying 2 for transfer from state 1 to state 2. We use the matrices in Example 19.1 and formula (19.19). For the first contract,

so that

For the second contract,

so that

Therefore, the total APV is 1.2912.

Example 19.3 Take the same chain as in the previous example. Consider a contract that pays 1 at the end of any year of transfer from being healthy to being unhealthy, provided this occurs within the next 3 years. Find the expected value and variance of the benefits.

Solution. We now have

so that

The variance calculation can not be done conveniently by adapting a version of formula (15.15). The situation here is different from the multiple decrement model where benefits are paid upon transition to an absorbing state. In this case, there can be benefits paid at more than one time. We must instead use the approach taken in the current payment formula for annuity variances. Note however that the covariance calculations will be more complicated than that given by (15.15). Letting I_k take the value 1 if transfer from the healthy state to the unhealthy state occurs between time k and k + 1, we have

while

which is the probability of becoming unhealthy, recovering, and then becoming unhealthy again, thereby being paid at both time 1 and time 3. We can then calculate the covariances as

leading to the calculation of the variance as

Suppose we wished to modify the contract of the previous example so that it paid off only on the first occurrence of becoming unhealthy. In our simple example, this could be done readily enough by changing the probability of a benefit payment at time 3 to 0.105, subtracting 0.006, which we have calculated above as the probability of the second occurrence. To handle the problem in general, we need a more systematic approach. Consider a contract paying benefits upon the first transition from a state i₀ to a state j₀. A method that works here ( as well for many other problems ) is to add additional states.

The easiest way is to add a single new absorbing state z, and all transfers from state i₀ to state j₀ are redirected to z, and the contract is modified to pay upon a transfer from i₀ to z. To state this precisely, let p* denote probabilities in the augmented chain. To simplify the notation, fix a time n and write p_ij and p*_ij for p_ij(n, n + 1) and p*_ij(n, n + 1) respectively. Then p* is the same as p except that

There is a more involved method which has the advantage that it preserves relationships between other states that might be needed in more complex problems. This is to add N + 1 new states i*, i = 0, …, N, where state i* can be considered as a sort of ‘clone’ of state i. Transitions between un-starred states are as before except that transitions from i₀ to j₀ are directed to j*₀ in place of j₀. Except for that, there are no transitions between the two types of states, starred and un-starred. Transitions between starred states are as given by the corresponding transitions between their clones. In this case, and for all ordered pairs (i, j) not equal to (i₀, j₀),

(19.5)

For example, if our original matrix for a two state model is given by

and we take i₀ = 0, j₀ = 1, the the matrix of the augmented chain, using the order 0, 1, 0*, 1* is

In addition to this, the provisions of the insurance are modified to pay upon a transfer from state i₀ to state j*₀. After the first occurrence of such an event, there can never be a return to state i₀, so subsequent occurrences are ruled out.

Remark Solving multi-state problems with large matrices can involve a great deal of calculation. There are a number of computing packages that can assist with this. One convenient example is the MMULT function of Excel®.

19.2.3 Multi-state annuities

We now look at annuity contracts associated with a chain. Suppose again that the individual begins in state a at time 0. For a fixed state i, we can consider a contract that pays c_k at time k provided the person is in state i at that time. Let I_k now be the indicator random variable that takes the value of 1 or 0 according as the process is or is not in state i at time k. Then the present value of the benefits is the random variable

from which we calculate immediately

(19.6)

To derive variances, we calculate for all m < n

From this, we calculate the covariances and then proceed as in Example 19.3.

Example 19.4 Consider again the chain given in Example 19.1. A contract on a healthy life provides for four yearly payments beginning at time 0. The amount of the payment is 1 if the person is healthy, or 2 if the person is unhealthy. If interest is a constant 5%, find the actuarial present value.

Solution. We view this as two separate annuities. Contract 1 is for state 1 and contract 2 is for state 2. In addition to the matrix P₀P₁ calculated in Example 19.1, we need

The APV of contract 1 is

and that of contract 2 is

giving a total APV of 3.6150.

Example 19.5 Suppose the contract in Example 19.2 is to be purchased by three level annual premiums of π, which are payable only if the person is healthy. Calculate π.

Solution. Equating values of premiums and benefits,

which gives π = 0.6449.

There are other types of annuity contracts that can be handled by the trick of adding states. Suppose for example, a contract provides periodic annuity benefits starting when the process enters state i₀ (or starting at time 0 if the process begins in state i₀), but which stop completely when the process leaves state i₀, even if there is a subsequent return. The easiest approach here is to add an absorbing state z and all transitions from state i₀ to other states are directed to state z, ensuring the process never returns to i₀. If we want to preserve information about other transitions, the method is to add a clone for each state as we did in the insurance case. Transitions from state i₀ to any state j ≠ i₀ are redirected to the clone j*, also ensuring that the process will never return to state i₀.

19.3.1 Forces of transition

With the continuous time framework, we are able to define analogues of the force of mortality in our original life-death model, and the forces of decrement in the multiple-decrement model. As we have seen in earlier examples, the process is often specified by giving these forces, and then the central problem is to use these to calculate the desired probabilities.

Definition 19.2 We define the force of transition from state i to state j at time t as follows. If i ≠ j

while

Note that μ_ii is always nonpostive, which may seem a bit strange at first, but it is easily explained if we consider, for example, the multiple-decrement case where state 0 refers to an active life. The negativity of μ₀₀ reflects the action of the other forces of decrement, which cause transfer out of state 0.

An important fact is the following.

Proof. Fix any i and t. Refer to the little o notation introduced in Section 18.3. From the definitions above, for all states j and h > 0,

(19.7)

and

(19.8)

Summing over all j gives

Subtract 1 from each side, divide by h and take a limit as h approaches 0 to establish the conclusion.

We now want to relate the forces of transition to the transition probabilities. This is much more complicated in our general setting than in the simple cases we looked at previously.

(19.9)

Proof. We start by deriving the discrete version of the above formula. Following the reasoning used in Equation (18.18), we can deduce that for states i, j times s < t and h > 0

(19.10)

reflecting the fact that in order to reach state j from state i in the time interval s, t + h we must first reach some state k by time t and then go from state k to state j in the next h time units. The system of equations given by (19.10) is known as the Chapman-Kolmogorov equations.

Substituting from (19.7) and (19.8) into the second factor of the summand,

Subtracting p_ij(s, t) from both sides, dividing by h and taking a limit gives (19.19).

We can give an intuitive explanation of (19.9) paralleling that given in the discrete case. View each side as a type of density function for reaching state j at time t when starting from state i at time s. The right side reflects the fact that in order to accomplish this, we must first reach some state k at time t and then at that instant of time, transfer from state k to state j (or in the case that k = j, not transfer back into some other state).

We view the statement of the theorem as a system of differential equations in the variable t for a fixed value of s. The word ‘forward’ arises since our transition is moving forward in time. There is a related system, the corresponding backwards equations, which will be developed in Exercise 19.6.

There will normally be many solutions to a system of differential equations of this type. There will however often be a unique solution if we specify appropriate initial conditions. In our context, these conditions are the obvious requirements that for all t,

We have of course encountered versions of the Kolmogorov equations before. Take s = 0. The simplest case with N = 1 gives Equation (8.8). In the multiple decrement model, the Kolmogorov equations produce Equation (11.11). These statements may not be completely obvious since the notation is somewhat different. It will be instructive for the reader to verify these claims.

There is a nice matrix formulation of (19.19). Fixing s, t we let P denote the transition matrix for the time interval (s, t). This is the matrix with an entry in the ith row and jth column of p_ij(s, t), as we had in the discrete case. We also have a corresponding matrix M in which the entry in the ith row and jth column is μ_ij(t). M is often referred to as the intensity matrix. We let P′ denote the matrix in which each entry is the partial derivative with respect to t of the corresponding entry in P. The Kolmogorov forward equations together with the initial conditions can then be expressed as

(19.11)

and

(19.12)

The following is a simple but useful consequence of the forward equations. It generalizes a result from our life-death model, when we have a state, such as that of being alive, which cannot be entered from any other state. That is, for all j ≠ i and s < t, we have p_ji(s, t) = 0. We will refer to such a state as being anti-absorbing.

In the following theorem, we adopt a formally weaker hypothesis involving forces rather than probabilities, which by the second statement turns out to be equivalent.

and

Proof. From Theorem 19.2 we get

(19.13)

a familiar differential equation which we first encountered in (8.15) and which is easily solved to give

for some function K(s) independent of t, Since p_ii(s, s) = 1, and p_ji(s, s) = 0 for j ≠ i, we must have K(s) = 1 if j = i or 0 if j ≠ i and the conclusion follows.

Another quantity of interest is the so-called sojourn probability for a state. We define

Note that the sojourn probability will in general differ from p_ii(s, t), as the corresponding event requires the process to remain in state i for an entire interval of time, rather than just at the endpoints. The two probabilities will however be the same for an anti-absorbing state.

Since sojourn probabilities are featured prominently in the next section, we will attempt to find a way to evaluate them. Let

This is the probability that starting in state i at time s, the process is back in state i at time t, having left state i sometime during the interval. Let

Now we employ once again the technique of adding states. Given a process and a state i, we add a single new state i* which is a clone of state i. Transitions out of state i* follow the same pattern as transitions out of state i, while transitions that originally came into state i will be diverted to the cloned state i*. Now to say that the new process is in state i at a time s and also at a later time t means that it must have stayed there throughout as it cannot re-enter. In addition, the forces of transition out of state i remain the same in both processes. So we have that

We do have to consider the quantity . Now

since the new process will move from state i to state i* precisely when the original process moves from state i back to state i having left state i at some point. From Definition 19.2,

Applying Theorem 19.3 to state i in the new process then leads to the formula

This formula does not however allow one to calculate the term on the left exactly since ν_i also involves sojourn probabilities. All we can say without further assumptions is that

Suppose, however, that we have a process such that, for all states i and t, h > 0

(19.14)

This assumption says that it is highly unlikely that in a small time interval there is a move out of a state and back into it. It is the same idea that underlies a Poisson process as discussed in Section 18.3, where it was highly unlikely to have two occurrences of an event in a small time interval. When this assumption holds, we have that v_i(t) = 0 for all t and we can now state the following theorem.

Remark We can view the first statement in Theorem 19.3 as the special case of Theorem 19.4, when q_i is actually equal to 0.

An alternative direct proof of Theorem 19.4 is outlined in Exercise 19.18.

To summarize our conclusions in this section we have shown that given the forces of transition, we can in theory, solve a system of differential equations to arrive at the required probabilities that we are interested in for our applications. However, in practice, the solution can be difficult or impossible to obtain exactly. In the following subsections we look at various possibilities for obtaining or approximating the probabilities from the forces.

19.3.2 Path-by-path analysis

Assume then that we are given μ_ij(t) for all i, j and t, and we want to determine the probability that starting in state i at any time s, we will be in state j at time t. What we can do in a relatively straightforward manner is the following. Given any path going from state i to state j, we can derive a formula for the probability that starting in state i at time s, the process will be in state j at time t, having following the prescribed path exactly. Precisely, we are given a path π which is a sequence of states, i₀, i₁, …, i_M where i₀ = i and i_M = j. We then want the probability that the process will make a transition from state i to i₁ followed by a transition from i₁ to i₂, followed by a transition from i₂ to i₃ and so on, continuing in such a fashion, without visiting any other states, and finally reaching state j at or before time t, and remaining in state j until time t. Let us denote such a probability by p^π_{i, j}(s, t). These can be calculated by integration, using only the forces and sojourn probabilities. So assuming that (19.14) holds, we have reduced the problem to one of evaluating integrals.

To illustrate, consider the simplest possible case of a path of length 1, that is π = (i, j). Then arguing much the same as we did in the single life and multiple decrement models we can write

The above formula parallels formula (11.13) except that in that case all non-active states are absorbing, so one cannot get out of them, which means that the third factor in the integrand does not appear as it is equal to 1. To recap, the formula above shows that in order to go from i to j along the one-step path, three events have to occur. First, the process must remain in state i until some time r between time s and time t. This event has probability . It cannot first go to another state and return, since that would constitute a different path from the given one. Secondly, there must be a transition to state j at time r. We can think of the probability of this as being μ_{i, j}(r)dr. Finally, the process must remain in state j from time r to time t. It cannot leave state j and return as this would similarly constitute a different path. We get the required probability by integrating the product of these probabilities over all times r between s and t.

As the path length increases the formula gets more complicated. Consider a two-step path, π = (i, k, j). Now we need a double integral. The formula is

This looks extremely complex, but it simply records what are now five steps to meet the required conditions. The process must remain in state i until some time r₁, then transfer to state k, then remain in state k until some time r₂, then transfer to state j, and then finally remain in state j until time t.

A similar calculation that arises for a given path Π, is to find the probability that starting in state i at time s, the process will transfer to state j before time t having followed precisely the prescribed path. We get the same multiple integral as described above, except the final sojourn probability is omitted, since the process need only enter, but not necessarily remain in the final state on the path. Of course, when the final state is absorbing, the two problems are identical.

Example 19.6 Consider a heathy-unhealthy-deceased model as introduced in Figure 19.2. The forces of transition are constant with μ₀₁ = 2, μ₀₂ = 1, μ₁₀ = 1, μ₁₂ = 3. Find the probability that a healthy life will become unhealthy, and then die without recovering before time 1.

Solution. We have μ₀₀ = −3, μ₁₁ = −4. The path in question is π = (0, 1, 2) and the required probability is

The double integral evaluates to

In general, a path of length n will require an n-dimensional integral with an integrand consisting of 2n + 1 factors, so this can quickly become computationally infeasible. Aside from that there is another difficulty. Usually, what we really want is p_ij(s, t). In some cases this might be obtained by adding up p^π_{i, j}(s, t) for all possible paths π from i to j. However, even in the simplest of models, there could well be an infinite number of such paths and this procedure will not work. This will normally occur whenever there is a positive probability of two-way transitions, as in the healthy-unhealthy-deceased model where healthy people can recover. A person could go from a healthy state to the deceased state, by getting sick and dying, or getting sick, then recovering, then getting sick again and dying, or recovering twice before death, or any of a number of infinite possibilities. We need other approaches to calculating transition probabilities, which we discuss in the next two subsections.

19.3.3 Numerical approximation

There are various ways to find a numerical approximation to the solution of the Kolmogorov equations. We will describe one approach that is fairly simple to implement. It is similar to Euler’s method, described in Section 8.9, for Thiele’s equation. For a fixed time s, the goal is to approximate the probabilities p_ij(s, t). Pick a small interval of time h, and change our unit of time, so that each original time unit is now 1/h units. (So, for example, if our original time unit was a year, and h = 1/12, this changes the time unit to months.). We then essentially define a discrete time multi-state model, which will in generally be nonstationary. We will define one step transition probabilities for this model by simply looking at formulas (19.7) and (19.19), which formed the basis of the Kolmogorov equations, only omitting the o(h) terms in each case. Since these terms becomes very small as h does, we can hope that for small enough h, we obtain a good approximation. We also switch the origin of time so that the original time s is now time 0 and the original time s + mh for an integer m is now time m. We then take our approximating discrete non-stationary Markov chain to be that in which the transition probabilities, denoted by , are given by

We then obtain transition matrices which have for an (i, j)th entry, the quantity

Another way of looking at this is that we simply get the matrix by taking the intensity matrix M(s + mh), multiplying all non-diagonal entries by h and adjusting the diagonal entries so that the rows add to 1.

When t = s + mh for some integer m we can then obtain approximations by

Example 19.7 In a chain with three states, you are given that

Find an approximation to p₀₁(1, 2) using the method described above with h = 1/3.

Solution. We have

so that

and similarly

Then the approximation to p₀₁(1, 2) is the entry in row 0 and column 1 of , which is 0.15131.

19.3.4 Stationary continuous time processes

One case where we can make some progress in solving the Kolmogorov equations directly is that of a stationary process as defined in Chapter 18. We then have that each μ_ij(t) is a constant, denoted by μ_ij. Moreover, the process will be determined by the quantities p_ij(t) = p_ij(0, t) since the stationary condition implies that p_ij(s, t) = p_ij(t − s).

As an example, we can write down completely the solution for the general two-state stationary process. Suppose the intensity matrix is

Let ρ(t) = e^{− t(μ + ν)}. Then the transition matrix P(t), which has the entry of p_ij(t) in the ith row and jth column, is given by

(19.15)

We can verify that P satisfies equations (19.11) and ( 19.12) by direct calculation, noting that dρ(t)/dt) = −(μ + ν)ρ(t), and that ρ(0) = 1.

For the general stationary process, solutions to P′ = PM can be generated from eigenvectors of M. These are non-zero vectors for which

for some constant λ known as an eigenvalue of M. (The left hand side above is a matrix multiplication in which we view a as a 1 × N + 1 matrix. According to the convention we have used for the transition probabilities, the vector appears on the left.) To obtain eigenvectors, we first find the eigenvalues as those constants for which the determinant of (λI − M) = 0. This involves solving a polynomial equation in λ. We can then solve for the resulting eigenvectors. Examples will follow.

Note now that given any such eigenvector a and any fixed i, the matrix P that has all zero entries except for an ith row of (a₀e^λt, a₁e^λt…a_Ne^λt) is a solution to P′ = PM. This follows since the ijth entry of PM will be λa_je^λt which is the derivative of the corresponding entry in .

Note next that given any finite number of solutions to P′ = PM, the linearity implies that any linear combination of these solutions will also be a solution. We therefore seek a linear combination that will satisfy the initial conditions. This can always be done in the case that we can find N linearly independent eigenvectors. We illustrate with a simple example.

Example 19.8 For the data as given in Example 19.6, find the probabilities of the following events.

(a) A life now disabled will be active at time 0.5.

(b) A life now active will be deceased at time 1.

Solution. The intensity matrix is

We can see immediately that 0 is an eigenvalue with an eigenvector of (0, 0, 1).

We can also note that since the rows of P add up to 1, the rows of add to 0, and the last row of P is (0, 0, 1), it is sufficient to solve the reduced system in which the superscript r indicates a matrix with the row N and column N removed. So we can consider eigenvectors of

The determinant of λI − M^r is λ² + 7λ + 10 giving eigenvalues of −5 and −2. Solving, we find respective eigenvectors of (1, −2) and (1, 1). (Note that eigenvectors are not unique as they can be multiplied by any non-zero constant.).

To satisfy the initial conditions, we want the first row of P^r to be the particular linear combination of the vectors a = e^{− 5t}(1, −2) and b = e^{− 2t}(1, 1) which gives the vector (1, 0) when t = 0. We solve this to get a first row vector of (1/3)a + (2/3)b. For the second row, we want a linear combination that gives the vector (0,1) when t = 0. We solve this to get a second row vector equal to ( − 1/3)a + (1/3)b. Transition probabilities are then given by

The answers to the particular questions asked are

1. p₁₀(0, 0.5) = 0.0953.
2. 1 − p₀₀(0, 1) − p₀₁(0, 1) = 0.8218.

   (The answer to part (b) is of course larger than that of Example 19.6 since here we allow for an arbitrary number of recoveries from being sick, as well as death from a healthy state.

19.3.5 Some methods for non-stationary processes

The above procedure can also be used solve the problem of determining probabilities from the forces, when these are ‘piecewise constant’. We can find transition matrices for each time interval on which the forces are constant, and then use the discrete time technique. The following simple example illustrates the procedure. It involves only two time intervals but the method can easily be extended.

Example 19.9 For a two-state model, we have forces of transition given by

Find the probability that the process will be in state 1 at time 2.5 given that it is is state 0 at time 0.

Solution.

From the matrix (19.19), we have

Substituting, the required probability is 0.39446.

One method for handling a perfectly general chain is to approximate it by a piecewise continuous one, and then use the method outlined above. To do so, one can choose a small time unit, and approximate the forces of transition by replacing them with those that are constant on each time interval, possibly using the midpoint value.

19.3.6 Extension of the common shock model

As mentioned in the introduction, multiple life theory, multiple decrement theory, or the more general model discussed in Chapter 17 can all be looked at in a multi-state framework. In the standard cases where failure times are independent, this does not normally provide any advantage over the conventional treatments that we have described previously. The multi-state approach, however, can be useful in modelling certain types of dependence. As an illustration, we will revisit the common shock model of Section 17.5 and discuss some possible refinements. We confine the analysis to the case of two types of failure, the first with failure time equal to min (T*₁, Z) and the second with failure time equal to min (T*₂, Z). Failure for the two types is dependent due to the common shock but there will be additional sources of dependence when T*₁ and T*₂ are themselves not independent. An example of this is the ‘broken-heart’ syndrome mentioned in Section 10.2 where one type of failure can hasten failure of the other type.

We consider a multi-state model with four states, as given in Figure 19.1, but adapted to cover general failure times. State 0 means that neither type of failure has occurred, state 1 means that only the second cause of failure has occurred, state 2 means that only the first type of failure has occurred, and state 3 means that both types of failure have occurred. (The states 1 and 2 then are labelled by the number of the surviving type.) Let μ_i(t) denote the hazard function for T*_i, and ρ(t) denote the hazard function for Z.

Suppose that forces of failure are of the form.

for some functions ε₁(t) and ε₂(t). When these are 0, we have precisely the common shock model. In this more general setting, we use ε_i(t) to build in some dependence between T*₁ and T*₂ by letting the distributions change after the first failure. The ‘broken-heart syndrome’ would call for positive values of ε_i(t). On the other hand, there may be cases where we assume that the prospects for failure type i improves after the first failure, and we would reflect this with a negative ε_i(t).

Probabilities in this model can all be calculated by the procedure of Section 19.3.2 since there are at most three paths between any two states. Moreover, since no state can be re-entered once the process leaves it, we can use Theorem 19.4 for the sojourn probabilities.

For the following examples we assume that all forces are constant.

Example 19.10 Calculate the probability that at time n, the first type of failure will have occurred but not the second.

Solution. From Theorem 19.1,

We want

The integral is easily evaluated to give

(19.16)

As a check, suppose that ε₁ = ε₂ = ρ = 0, so we simply have two independent exponential failure times. The answer reduces to

which is just the probability that the first type of failure occurred before time n multiplied by the probability that the second type did not occur before time n.

Example 19.11 Find the probability that the second cause of failure occurs before time n and after the first cause of failure.

Solution. If Π is the path (0,2,3) we want,

(a special case of formula (19.17) in the next section.) Substituting from (19.16) and integrating, the answer is

When ε₂ = 0 we recover the answer of Example 17.12 (a).

19.3.7 Insurance and annuity applications in continuous time

Consider an insurance contract that provides for two fixed states j and k, payments at the moment of transfer whenever a transfer occurs from state j to state k. The amount paid for a transfer at time t will be b_t (denoted by b^(jk)_t if we wish to distinguish between several transfer benefits). Suppose the process is in state i at time 0. Then the actuarial present value parallels the result we saw in the multiple decrement case and is given by

If benefits are to be paid on only the first transfer from state j to state k, then this can be handled by adding new states, exactly as we did in the discrete case.

Similarly, for a contract that provides payments made continuously at the periodic rate of c_t (written as c^(j)_t if needed to distinguish states) at time t, provided the process is in state j the present value is given

Some annuities may make use of the sojourn probabilities. Suppose, for example, that at time 0, the process is in state i, and payments at the periodic rate of c_t are paid as long as the process remains in state i. All payments stop upon the first exit from state i. The present value is given by

It is possible to fit the continuous time insurances and annuities into a stochastic model, and calculate variances as well as expected values, as we did in the discrete case. We will not pursue this here. In general, this will require integration of functions whose values are random variables rather than definite numbers. This presents both theoretical and computational difficulties.

More general probabilities can be deduced from the insurance formulas by the standard method we described in Section 10.8.4 of taking zero interest and benefit functions that take the value 1 or 0. For example suppose the process is in state i. The probability that within n years it will at some point make a transfer from state j to state k is given by

(19.17)

Example 19.12 An insurance contract based on the model in Example 19.8 provides for a benefit at time t of e^.04t provided that a person now healthy dies while unhealthy. The force of interest is a constant 5%. Find the actuarial present value.

Solution. This is given directly by

The following is an example which can be thought of as a generalization of contingent insurances. Suppose we have a path of states Π = i₀i₁…i_M as described in Section 19.3.2. At time 0, the process is in state i₀ and an insurance contract provides a payment of 1 at the moment that the process enters state i_M, having previously followed precisely the path Π. For example, consider the chain covering the two lives (x) and (y) as given in the introduction and the path Π = {0, 1, 3}. The contract based on this path is the second death-contingent insurance paying upon the death of (x) if this occurs after the death of (y).

Example 19.13 Suppose that the force of interest and all forces of transition are constant. Find a formula for , the actuarial present value of the insurance based on the path Π.

Solution. To simplify notation, let and Then, following the explanation for the calculation of probabilities in section 19.3.2, we can set up the following multiple integral in which r_j denotes the time of transfer from state i_{j − 1} to state i_j.

The integration can be carried out in a straightforward manner to give

The formula is quite easy to remember and to compute from. The numerator is the product of all the forces of transition along the path. The denominator is a product of factors, one for each state on the path, except for the final one. Each such factor is the sum of all outgoing forces from that state plus δ. We can take δ = 0 to give the probability that the path Π will be followed.

Note that Formula (8.8), Example 10.5 and Formula (10.25) with substitution from these first two are special cases of the above.