2.2.1 Criteria for internal validity and the role of economic theory

We argue that in an ex post evaluation of a program, the goal is to make causal inferences with a high degree of “internal validity”: the aim is to make credible inferences and qualify them as precisely as possible. In such a descriptive exercise, the degree of “external validity” is irrelevant. On the other hand, “external validity” will be of paramount importance when one wants to make predictive statements about the impact of the same program on a different population, or when one wants to use the inferences to make guesses about the possible effects of a slightly different program. That is, we view “external validity” to be the central issue in an attempt to use the results of an ex post evaluation for an ex ante program evaluation; we further discuss this in the next section.

What constitutes an inference with high “internal validity”? ¹⁰ Throughout this chapter we will consider three criteria. The first is the extent to which there is a tight correspondence between what we know about the assignment-to-treatment mechanism and our statistical model of the process. In some cases, the assignment mechanism might leave very little room as to how it is to be formally translated into a statistical assumption. In other cases, little might be known about the process leading to treatment status, leaving much more discretion in the hands of the analyst to model the process. We view this discretion as potentially expanding the set of “plausible” (yet different) inferences that can be made, and hence generating doubt as to which one is correct.

The second criterion is the broadness of the class of models with which the causal inferences are consistent. Ideally, one would like to make a causal inference that is consistent with any conceivable behavioral model. By this criterion, it would be undesirable to make a causal inference that is only valid if a very specific behavioral model is true, and it is unknown how the inferences would change under plausible deviations from the model in question.

The last criterion we will consider is the extent to which the research design is testable; that is, the extent to which we can treat the proposed treatment assignment mechanism as a null hypothesis that could, in principle, be falsified with data (e.g. probabilistically, via a formal statistical test).

Overall, if one were to adopt these three criteria, then a research design would have low “internal validity” when (1) the statistical model is not based on what is actually known about the treatment assignment mechanism, but based entirely on speculation, (2) inferences are known only to be valid for one specific behavioral model amongst many other plausible alternatives and (3) there is no way to test the key assumption that achieves identification.

What is the role of economic (for that matter, any other) theory in the ex post evaluation problem? First of all, economic theories motivate what outcomes we wish to examine, and what causal relationships we wish to explore. For example, our models of job search (see McCall and McCall (2008) for example) may motivate us to examine the impact of a change in benefit levels on unemployment duration. Or if we were interested in the likely impacts of the “program” of a hike in the minimum wage, economists are likely to be most interested in the impact on employment, either for the purposes of measuring demand elasticities, or perhaps assessing the empirical relevance of a perfectly competitive labor market against that of a market in which firms face upward-sloping labor supply curves (Card and Krueger, 1995; Manning, 2003).

Second, when our institutional knowledge does not put enough structure on the problem to identify any causal effects, then assumptions about individuals’ behavior must be made to make any causal statement, however conditional and qualified. In this way, structural assumptions motivated by economic theory can help “fill in the gaps” in the knowledge of the treatment assignment process.

Overall, in an ex post evaluation, the imposition of structural assumptions motivated by economic theory is done out of necessity. The ideal is to conjecture as little as possible about individuals’ behavior so as to make the causal inferences valid under the broadest class of all possible models. For example, one could imagine beginning with a simple Rosen-type model of schooling with wealth maximization (Rosen, 1987) as a basis for empirically estimating the impact of a college subsidy program on educational attainment and lifetime earnings. The problem with such an approach is that this would raise the question as to whether the causal inferences entirely depend on that particular Rosen-type model. What if one added consumption decisions to the model? What about saving and borrowing? What if there are credit constraints? What if there are unpredictable shocks to non-labor income? What if agents maximize present discounted utility rather than discounted lifetime wealth? The possible permutations go on and on.

It is tempting to reason that we have no choice but to adopt a specific model of economic behavior and to admit that causal inferences are conditional only on the model being true; that the only alternative is to make causal inferences that depend on assumptions that we do not even know we are making. ¹¹ But this reasoning equates the specificity of a model with its completeness, which we believe to be very different notions.

Suppose, for example—in the context of evaluating the impact of our hypothetical job search assistance program—that the type of a randomly drawn individual from the population is given by the random variable (with a cdf ), that represents all the constraints and actions the individual takes prior to, and in anticipation of, the determination of participating in the program , and that outcomes are determined by the system given by Eqs (1)-(3). While there is no discussion of utility functions, production functions, information sets, or discount rates, the fact is that this is a complete model of the data generating process; that is, we have enough information to derive expressions for the joint distribution of the observables from the primitives of and (1)-(3). At the same time it is not a very specific (or economic) model, but in fact, quite the opposite: it is perhaps the most general formulation that one could consider. It is difficult to imagine any economic model—including a standard job search model—being inconsistent with this framework.

Another example of this can be seen in the context of the impact of a job training program on earnings. One of the many different economic structures consistent with Roy, 1951; Heckman and Honore, 1990) into training. ¹² The Roy-type model is certainly specific, assuming perfect foresight on earnings in both the “training” or “no-training” regimes, as well as income maximization behavior. If one obtains causal inferences in the Roy model framework, an open question would be how the inferences change under different theoretical frameworks (e.g. a job search-type model, where training shifts the wage offer distribution upward). But if we can show that the causal inferences are valid within the more general—but nonetheless complete—formulation of (1)-(3), then we know the inferences will still hold under both the Roy-type model, a job search model, or any number of plausible alternative economic theories.

2.3.1 Using ex post evaluations for ex ante predictions

In this chapter, we focus most of our attention on the goals of the ex post evaluation problem, that of achieving a high degree of internal validity. We recognize that the weighted average effects that are often identified in ex post evaluation research designs may not correspond to a potentially more intuitive “parameter of interest”, raising the issue of “external validity”. Accordingly—using well-known results in the econometric and evaluation literature—we sketch out a few approaches for extrapolating from the average effects obtained from the ex post analysis to effects that might be the focus of an ex ante evaluation.

Throughout the chapter, we limit ourselves to contexts in which a potential instrument is binary, because the real-world examples where potential instruments have been explicitly or “naturally” randomized, the instrument is invariably binary. As is well-understood in the evaluation literature, this creates a gap between what causal effects we can estimate and the potentially more “general” average effects of interest. It is intuitive that such a gap would diminish if one had access to an instrumental variable that is continuously distributed. Indeed, as Heckman and Vytlacil (2005) show, when the instrument is essentially randomized (and excluded from the outcome equation) and continuously distributed in such a way that is continuously distributed on the unit interval, then the full set of what they define as Marginal Treatment Effects (MTE) can be used construct various policy parameters of interest.

3.1.1 Simple random assignment

We start by considering simple random assignment with perfect compliance. “Perfect compliance” refers to the case that individuals who are assigned a particular treatment, do indeed receive the treatment. For example, consider a re-employment program for unemployment insurance claimants, where the “program” is being contacted (via telephone and/or personal visit) by a career counselor, who provides information that facilitates the job search process. Here, participation in this “program” is not voluntary, and it is easy to imagine a public agency randomly choosing a subset of the population of UI claimants to receive this treatment. The outcome might be time to re-employment or total earnings in a period following the treatment.

In terms of the framework defined by Eqs (1)-(3), this situation can be formally represented as

• D1: (Simple Random Assignment):

That is, for the entire population being studied, every individual has the same probability of being assigned to the program.

It is immediately clear that the distribution of becomes degenerate, with a single mass point at , and so the difference in the means in Eq. (5) becomes

where the ATE is the “average treatment effect”. The weights from Eq. (4) are in this case. A key problem posed in Eq. (5) is the potential relationship between the latent propensity and (the functions and ). Pure random assignment “solves” the problem by eliminating all variation in .

Internal validity: pan-theoretic causal inference

Let us now assess this research design on the basis of the three criteria described in Section 2.2.1. First, given the general formulation of the problem in Eqs (1)-(3), Condition D1 is much less an assumption, but rather a literal description of the assignment process—the “D” denotes a descriptive element of the data generating process. Indeed, it is not clear how else one would formally describe the randomized experiment.

Second, the causal inference is apparently valid for any model that is consistent with the structure given in Eqs (1)-(3). As discussed in Section 2.2.1, it is difficult to conceive of a model of behavior that would not be consistent with (1), (2), and (3). So even though we are not explicitly laying out the elements of a specific model of behavior (e.g. a job search model), it should be clear that given the distribution , Eqs (1)-(3), and Condition D1 constitutes a complete model of the data generating process, and that causal inference is far from being “atheoretic”. Indeed, the causal inference is best described as “pan-theoretic”, consistent with a broad—arguably the broadest—class of possible behavioral models.

Finally, and perhaps most crucially, even though one could consider D1 to be a descriptive statement, we could alternatively treat it as a hypothesis, one with testable implications. Specifically, D1 implies

     (6)

and similarly, . That is, the distribution of unobserved “types” is identical in the treatment and control groups. Since is unobservable, this itself is not testable. But a direct consequence of the result is that the pre-determined characteristics/actions must be identical between the two groups as well,

which is a testable implication (as long as there are some observable elements of ).

The implication that the entire joint distribution of all pre-determined characteristics be identical in both the treatment and control states is indeed quite a stringent test, and also independent of any model of the determination of . It is difficult to imagine a more stringent test.

Although it may be tempting to conclude that “even random assignment must assume that the unobservables are uncorrelated with treatment”, on the contrary, the key point here is that the balance of unobservable types between the treatment and control groups is not a primitive assumption; instead, it is a direct consequence of the assignment mechanism, which is described by D1. Furthermore, balance in the observable elements of is not an additional assumption, but a natural implication of balance in the unobservable type .

One might also find D1 “unappealing” since mathematically it seems like a strong condition. But from an ex post evaluation perspective, whether D1 is a “strong” or “weak” condition is not as important as the fact that D1 is beyond realistic: it is practically a literal description of the randomizing process.

3.1.2 Stratified/block randomization

Now, suppose there is a subset of elements in —call this vector —that are observed by the experimenter. A minor variant on the above mechanism is when the probability of assignment to treatment is different for different groups defined by , but the probability of treatment is identical for all individuals within each group defined by . ¹⁴ In our hypothetical job search assistance experiment, we could imagine initially stratifying the study population by their previous unemployment spell history: “short”, “medium”, and “long”-(predicted) spell UI claimants. This assignment procedure can be described as

• D2: (Random Assignment Conditional on )

In this case, where there may be substantial variation in the unobservable type for a given , the probability of receiving treatment is identical for everyone with the same .

The results from simple random assignment naturally follow,

essentially an average treatment effect, conditional on .

We mention this case not because D2 is a weaker, and hence more palatable assumption, but rather, it is useful to know that the statement in D2—like the mechanism described by D1—is one that typically occurs when randomized experiments are implemented. For example, in the Negative Income Tax Experiments (Robins, 1985; Ashenfelter and Plant, 1990), were the pre-experimental incomes, and families were randomized into the various treatment groups with varying probabilities, but those probabilities were identical for every unit with the same . Another example is the Moving to Opportunity Experiment (Orr et al., 2003), which investigated the impact of individuals moving to a more economically advantaged neighborhood. The experiment was done in 5 different cities (Baltimore, Boston, Chicago, Los Angeles, and New York) over the period 1994-1998. Unanticipated variation in the rate at which people found eligible leases led them to change the fraction of individuals randomly assigned to the treatments two different times during the experiment (Orr et al., 2003, page 232). In this case, families were divided into different “blocks” or “strata” by location time and there was a different randomization ratio for each of these blocks.

This design—being very similar to the simple randomization case—would have a similar level of internal validity, according to two of our three criteria. Whether this design is testable (the third criterion we are considering) depends on the available data. By the same argument as in the simple random assignment case, we have

So if the conditional randomization scheme is based on all of the s that are observed by the analyst, then there are no testable implications. On the other hand, if there are additional elements in that are observed (but not used in the stratification), then once again, one can treat D2 as a hypothesis, and test that hypothesis by examining whether the distribution of those extra variables are the same in the treated and control groups (conditional on ).

3.1.3 The randomized experiment: pre-specified research design and a chance setup

We have focused so far on the role that randomization (as described by D1 or D2) plays in ensuring a balance of the unobservable types in the treated and control groups, and have argued that in principle, this can deliver causal inferences with a high degree of internal validity.

Another characteristic of the randomized experiment is that it can be described as “pre-specified” research design. In principle, before the experiment is carried out, the researcher is able to dictate in advance what analyses are to be performed. Indeed, in medical research conducted in the US, prior to conducting an medical experiment, investigators will frequently post a complete description of the experiment in advance at a web site such as clincaltrials.gov. This posting includes how the randomization will be performed, the rules for selecting subjects, the outcomes that will be investigated, and what statistical tests will be performed. Among other things, such pre-announcement prevents the possibility of “selective reporting”—reporting the results only from those trials that achieve the “desired” result. The underlying notion motivating such procedure has been described as providing a “severe test”—a test which “provides an overwhelmingly good chance of revealing the presence of a specific error, if it exists—but not otherwise” (Mayo, 1996, page 7). This notion conveys the idea that convincing statistical evidence does not rely only on the “fit” of the data to a particular hypothesis but on the procedure used to arrive at the result. Good procedures are ones that make fewer “errors.”

It should be recognized, of course, that this “ideal” of pre-specification is rarely implemented in social experiments in economics. In the empirical analysis of randomized evaluations, analysts often cannot help but be interested in the effects for different sub-groups (in which they were not initially interested), and the analysis can soon resemble a data-mining exercise. ¹⁵ That said, the problem of data-mining is not specific to randomized experiments, and a researcher armed with a lot of explanatory variables in a non-experimental setting can easily find many “significant” results even among purely randomly generated “data” (see Freedman (1983) for one illustration). It is probably constructive to consider that there is a spectrum of pre-specification, with the pre-announcement procedure described above on one extreme, and specification searching and “significance hunting” with non-experimental data on the other. In our discussion below, we make the case that detailed knowledge of the assignment-to-treatment process can serve much the same role as a pre-specified research design in “planned” experiments—as a kind of “straight jacket” which largely dictates the nature of statistical analysis.

Another noteworthy consequence of this particular data generating process is that it is essentially a “statistical machine” or a “chance set up” (Hacking, 1965) whose “operating characteristics” or statistical properties are well-understood, such as a coin flip. Indeed, after a randomizer assigns n individuals to the (well-defined) treatment, and n individuals to the control, for a total of individuals, one can conduct a non-parametric exact test of a sharp null hypothesis that does not require any particular distributional assumptions.

Consider the sharp null hypothesis that there is no treatment effect for any individuals (which implies that the two samples are drawn from the same distribution). In this case the assignment of the label “treatment” or “control” is arbitrary. In this example there are different ways the labels “treatment” and “control” could have been assigned. Now consider the following procedure:

1. Compute the difference in means (or any other interesting test statistic). Call this .

2. Permute the label treatment or control and compute the test statistic under this assignment of labels. This will generate different values of the test statistic for . These collection of these observations yield an exact distribution of the test statistic.

3. One can compute the -value such that the probability that a draw from this distribution would exceed .

This particular “randomization” or “permutation” test was originally proposed by Fisher (1935) for its utility to “supply confirmation whenever, rightly or, more often wrongly, it is suspected that the simpler tests have been appreciably injured by departures from normality.” (Fisher, 1966, page 48) (see Lehmann (1959, pages 183–192) for a detailed discussion). Our purpose in introducing it here is not to advocate for randomization inference as an “all purpose” solution for hypothesis testing; rather our purpose is to show just how powerful detailed institutional knowledge of the DGP can be.

3.1.4 Ex ante evaluation: predicting the effects of an expansion in the program

Up to this point, with our focus on an ex post evaluation we have considered the question, “For the individuals exposed to the randomized evaluation, what was the impact of the program?” We now consider a particular ex ante evaluation question, “What would be the impact of a full-scale implementation of the program?”, in a context when that full-scale implementation has not occurred. It is not difficult to imagine that the individuals who participate in a small-scale randomized evaluation may differ from those who would receive treatment under full-scale implementation. One could take the perspective that this therefore makes the highly credible/internally valid causal inferences from the randomized evaluation irrelevant, and hence that there is no choice but to pursue non-experimental methods, such as a structural modeling approach to evaluation, to answer the “real” question of interest.

Here we present an alternative view that this ex ante evaluation question is an extrapolation problem. And far from being irrelevant, estimates from a randomized evaluation can form the basis for such an extrapolation. And rather than viewing structural modeling and estimation as an alternative or substitute for experimental methods, we consider the two approaches to be potentially quite complementary in carrying out this extrapolation. That is, one can adopt certain assumptions about behavior and the structure of the economy to make precise the linkage between highly credible impact estimates from a small-scale experiment and the impact of a hypothetical full-scale implementation.

We illustrate this with the following example. Suppose one conducted a small-scale randomized evaluation of a job training program where participation in the experimental study was voluntary, while actual receipt of training was randomized. The question is, what would be the impact on earnings if we opened up the program so that participation in the program was voluntary?

First, let us define the parameter of interest as

where and are the earnings of a randomly drawn individual under two regimes: full-scale implementation of the program (), or no program at all (). This corresponds to the parameter of interest that motivates the Policy Relevant Treatment Effect (PRTE) of Heckman and Vytlacil (2001b). We might like to know the average earnings gain for everyone in the population. We can also express this as

     (7)

where the is the treatment status indicator in the regime, and the subscripts denote the potential outcomes.

Make the following assumptions:

• S1 (Linear Production Technology): , where is the amount of output, is the total amount of labor supplied by workers with units of human capital, and are technological parameters with .

• S2 (Job Training as Human Capital): the random variable is the individual’s endowment of human capital, and is the gain in human capital due to training, so that in the implementation regime, human capital is .

• S3 (Inelastic Labor Supply): each individual inelastically supplies units of labor.

• S4 (Profit maximizing price-taking): .

This setup will imply that

Thus, S1 through S4 are simply a set of economic assumptions that says potential outcomes are unaffected by the implementation of the program; this corresponds to what Heckman and Vytlacil (2005) call policy invariance. This policy invariance comes about because of the linear production technology, which implies that wages are determined by the technological parameters, and not the supply of labor for each level of human capital.

With this invariance, we may suppress the superscript for the potential outcomes; Eq. (7) will become

¹⁶ Note that the key causal parameter will in general be different from , where is the indicator for having participated in the smaller scale randomized experiment (bearing the risk of not being selected for treatment). That is, the concern is that those who participate in the experimental study may not be representative of the population that would eventually participate in a full-scale implementation.

How could they be linked? Consider the additional assumptions

• S5 (Income Maximization; Perfect Information; Selection on Gains): iff , with is the “fixed” cost to applying for the program, and is the monetary cost to the individual from receiving the treatment.

Together, S5 and S6 imply that we could characterize the selection into the program in the experimental regime as

where is the probability of being randomized into receiving the treatment (conditional on participating in the experimental study). Note that this presumes that in the experimental regime, all individuals in the population have the option of signing up for the experimental evaluation.

Finally, assume a functional form for the distribution of training effects in the population:

• S7 (Functional Form: Normality): is normally distributed with variance .

Applying assumption S7 yields the following expressions

     (8)

where and are the standard normal pdf and cdf, respectively.

The probability of assignment to treatment in the experimental regime, , characterizes the scale of the program. The smaller is, the smaller the expected gain to participating in the experimental study, and hence the average effect of the study participants will be more positively selected. On the other hand, as approaches 1, the experimental estimate approaches the policy parameter of interest because the experiment becomes the program of interest. Although we are considering the problem of predicting a “scaling up” of the program, this is an interesting case to case to consider because it implies that for an already existing program, one can potentially conduct a randomized evaluation, where a small fraction of individuals are denied the program ( close to 1), and the resulting experimentally identified effect can be directly used to predict the aggregate impact of completely shutting down the program. ¹⁷

The left-hand side of the first equation is the “parameter of interest” (i.e. what we want to know) in an ex ante evaluation problem. The left-hand side of the second equation is “what can be identified” (i.e. what we do know) from the experimental data in the ex post evaluation problem. The latter may not be “economically interesting” per se, but at the same time it is far from being unrelated to the former.

Indeed, the average treatment effect identified from the randomized experiment is the starting point or “leading term”, when we combine the above two expressions to yield

with the only unknown parameters in this expression being , the predicted take-up in a full-scale implementation, and , the degree of heterogeneity of the potential training effects in the entire population. It is intuitive that any ex ante evaluation of the full-scale implementation that has not yet occurred will, at a minimum, need these two quantities.

In presenting this example, we do not mean to assert that the economic assumptions S1 through S7 are particularly realistic. Nor do we assert they are minimally sufficient to lead to an extrapolative expression. There are as many different ways to model the economy as there are economists (and probably more!). Instead, we are simply illustrating that an ex ante evaluation attempt can directly use the results of an ex post evaluation, and in this way the description of the data generating process in an ex post evaluation (D1 or D2) can be quite complementary to the structural economic assumptions (S1 through S7). D1 is the key assumption that helps you identify whether there is credible evidence—arguably the most credible that is possible—of a causal phenomenon, while S1 through S7 provides a precise framework to think about making educated guesses about the effects of a program that has yet to be implemented. Although may not be of direct interest, obtaining credible estimates of this quantity would seem helpful for making a prediction about .

3.2.1 Assessment

In terms of our three criteria to assess internal validity, how does this research design fare—particularly in comparison to the randomized experiment with perfect compliance? First, only part of the data generating process given by D3, S8, and S9 is a literal description of the assignment process: if is truly randomly assigned, then D3 is not so much an assumption, but a description. On the other hand, S8 and S9 will typically be conjectures about behavior rather than being an implication of our institutional knowledge. ²³ This is an example where there are “gaps” in our understanding of the assignment process, and structural assumptions work together with experimental variation to achieve identification.

As for our second criterion, with the addition of the structure imposed by S8 and S9, it is clear that the class of all behavioral models for which the causal inference in Eq. (9) is valid is smaller. It is helpful to consider, for our hypothetical instrument, the kinds of economic models that would or would not be consistent with S8 and S9. If individuals are choosing the best job search activity amongst all known available feasible options, then the instrument of “providing information about the existence of a career counseling program” could be viewed as adding one more known alternative. A standard revealed preference argument would dictate that if an individual already chose to participate under , then it would still be optimal if : this would satisfy S9. Furthermore, it is arguably true that most attempts at modeling this process would not specify a direct impact of this added information on human capital; this would be an argument for S8. On the other hand, what if the information received about the program carried a signal of some other factor? It could indicate to the individual, that the state agency is monitoring their job search behavior more closely. This might induce the individual to search more intensively, independently of participating in the career counseling program; this would be violation of the exclusion restriction S8. Or perhaps the information provided sends a positive signal about the state of the job market and induces the individual to pursue other job search activities instead of the program; this might lead to a violation of S9.

For our third criterion, we can see that some aspects of D3, S8, and S9 are potentially testable. Suppose the elements of can be categorized into the vector (the variables determined prior to ) and (after ). And suppose we can observe a subset of elements from each of these vectors as variables and , respectively. Then the randomization in D3 has the direct implication that the distributions of for and should be identical:

Furthermore, since the exclusion restriction S8 dictates that all factors that determine are not influenced by , then D3 and S8 jointly imply that the distribution of are identical for the two groups:

The practical limitation here is that this test pre-supposes the researcher’s really do reflect elements of that influence . If are not a subset of , then even if there is imbalance in , S8 could still hold. Contrast this with the implication of D3 (and D1 and D2) that any variable determined prior to the random assignment should have a distribution that is identical between the two randomly assigned groups. Also, there seems no obvious way to test the proposition that does not directly impact , which is another condition required by S8.

Finally, if S9 holds, it must also be true that

That is, if probabilistic monotonicity holds for all , then it must also hold for groups of individuals, defined by the value of (which is a function of ). This inequality also holds for any variables determined prior to .

In summary, we conclude (unsurprisingly) that for programs where there is random assignment in the “encouragement” of individuals to participate, causal inferences will be of strictly lower internal validity, relative to the perfect compliance case. Nevertheless, the design does seem to satisfy—even if to a lesser degree—the three criteria that we are considering. Our knowledge of the assignment process does dictate an important aspect of the statistical model (and other aspects need to be modeled with structural assumptions), the causal inferences using the Wald estimand appear valid within a reasonably broad class of models (even it is not as broad as that for the perfect compliance case), and there are certain aspects of the design that generate testable implications.

3.2.2 Ex ante evaluation: extrapolating from LATE to ATE

Perhaps the most common criticism leveled at the LATE parameter is that it may not be the “parameter of interest”. ²⁴ By the same token, one may have little reason to be satisfied with the particular weights in the average effect expressed in (9). Returning to our hypothetical example in which the instrument “provide information on career counseling program” is randomized, the researcher may not be interested in an average effect that over-samples those who are more influenced by the instrument. For example, a researcher might be interested in predicting the average impact of individuals of a mandatory job counseling program, like the hypothetical example in the case of the randomized experiment with perfect compliance. That is, it may be of interest to predict what would happen if people were required to participate (i.e. every UI claimant will receive a call/visit from a job counselor). Moreover, it has been suggested that LATE is an “instrument-specific” parameter and a singular focus on LATE risks conflating “definition of parameters with issues of identification” (Heckman and Vytlacil, 2005): different instruments can be expected to yield different “LATEs”.

Our view is that these are valid criticisms from the perspective of an ex ante evaluation standpoint, where causal “parameters of interest” are defined by a theoretical framework describing the policy problem. But from an ex post evaluation perspective, within which internal validity is the primary goal, these issues are, by definition, unimportant. When the goal is to describe whatever one can about the causal effects of a program that was actually implemented (e.g. randomization of “information about the job counseling program”, ), a rigorous analysis will lead to precise statements about the causal phenomena that are possible to credibly identify. Sometimes what one can credibly identify may correspond to a desired parameter from a well-defined ex ante evaluation; sometimes it will not.

Although there has been considerable emphasis in the applied literature on the fact that LATE may differ from ATE, as well as discussion in the theoretical literature about the “merits” of LATE as a parameter, far less effort has been spent in actually using estimates of LATE to learn about ATE. Even though standard “textbook” selection models can lead to simple ways to extrapolate from LATE to ATE (see Heckman and Vytlacil (2001a) and Heckman et al. (2001, 2003)), our survey of the applied literature revealed very few other attempts to actually produce these extrapolations.

Although we have argued that the ATE from a randomized experiment may not directly correspond to the parameter of interest, for the following derivations, let us stipulate that ATE is useful, either for extrapolation (as illustrated in Section 3.1.4), or as an “instrument-invariant” benchmark that can be compared to other ATEs extrapolated from other instruments or alternative identification strategies.

Consider re-writing the structure given by Eqs (1)-(3), and D3, S8, and S9 as

where are constants, is a binary instrument, and characterize both the individual’s type and all other factors that determine and selection, and is independent of by D3 and S8. and are constants in the selection equation: S9 is satisfied. , , can be normalized to be mean zero error terms. The ATE is by construction equal to .

Let us adopt the following functional form assumption:

• S10 (Normality of Errors): and are both bivariate normals with covariance matrices and , respectively, where we are—without loss of generality—normalizing to have a variance of 1.

This is simply the standard dummy endogenous variable system (as in Heckman (1976, 1978) and Maddala (1983)), with the special case of a dummy variable instrument, and is a case that is considered in recent work by Angrist (2004) and Oreopoulos (2006). With this one functional form assumption, we obtain a relationship between LATE and ATE.

In particular we know that

     (10)

where and are the pdf and cdf of the standard normal, respectively. ²⁵ This is a standard result that directly follows from the early work on selection models (Heckman (1976, 1978). See also Heckman et al. (2001, 2003)). This framework has been used to discuss the relationship between ATE and LATE (see Heckman et al. (2001), Angrist (2004), and Oreopoulos (2006)). With a few exceptions (such as Heckman et al. (2001), for example), other applied researchers typically do not make use of the fact that even with information on just the three variables , , and , the “selection correction” term in Eq. (10) can be computed. In particular

implies that

     (11)

and analogously that

     (12)

Having identified and , we have the expression

     (13)

This expression is quite similar to that given in Section 3.1.4. Once again, the result of an ex post evaluation can be viewed as the leading term in the extrapolative goal of an ex ante evaluation: to obtain the effects of a program that was not implemented (i.e. random assignment with perfect compliance) or the ATE. This expression also shows how the goals of the ex post and ex ante evaluation problems can be complementary. Ex post evaluations aim to get the best estimate of the first term in the above equation, whereas ex ante evaluations are concerned with the assumptions necessary to extrapolate from LATE to ATE as in the above expression.

LATE versus ATE in practice

If S10 were adopted, how much might estimates of LATE differ from those of ATE in practice? To investigate this, we obtained data from a select group of empirical studies and computed both the estimates of LATE and the “selection correction” term in (13), using (11) and (12). ²⁶

The results are summarized in Table 1. For each of the studies, we present the simple difference in the means , the Wald estimate of LATE, the implied selection error correlations, , , a selection correction term, and the implied ATE. For comparison, we also give the average value of Y. Standard deviations are in brackets. The next to last row of the table gives the value of the second term in (13), and its significance (calculated using the delta method).

Table 1 Extrapolating from LATE to ATE

The quantity is the implied covariance between the gains and the selection error . A “selection on gains” phenomenon would imply . With the study of Abadie et al. (2002) in the first column, we see a substantial negative selection term, where the resulting LATE is significantly less than either the simple difference, or the implied ATE. If indeed the normality assumption is correct, this would imply that for the purposes of obtaining ATE, which might be the target of an ex ante evaluation, simply using LATE would be misleading, and ultimately even worse than using the simple difference in means.

On the other hand, in the analysis of Angrist and Evans (1998), the estimated LATE is actually quite similar to the implied ATE. So while a skeptic might consider it “uninteresting” to know the impact of having more than 2 children for the “compliers” (those whose family size was impacted by the gender mix of the first two children), it turns out in this context—if one accepts the functional form assumption—LATE and ATE do not differ very much. ²⁷ The other studies are examples of intermediate cases: LATE may not be equal to ATE, but it is closer to ATE than the simple difference .

Our point here is neither to recommend nor to discourage the use of this normal selection model for extrapolation. Rather, it is to illustrate and emphasize that even if LATE is identifying an average effect for a conceptually different population from the ATE, this does not necessarily mean that the two quantities in an actual application are very different. Our other point is that any inference that uses LATE to make any statement about the causal phenomena outside the context from which a LATE is generated, must necessarily rely on a structural assumption, whether implicitly or explicitly. In this discussion of extrapolating from LATE to ATE, we are being explicit that we are able to do this through a bivariate normal assumption. While such an assumption may seem unpalatable to some, it is clear that to insist on making no extrapolative assumptions is to abandon the ex ante evaluation goal entirely.

3.3.1 Assessment: Valid or invalid RD?

We now assess this design on the basis of the three criteria discussed in Section 2.2.1. First, some of the conditions for identification are indeed literally descriptions of the assignment process: D4 and D5. Others, like S11 through S13, are conjectures about the assignment process, and the underlying determinants of . S11 requires that there is positive density of at the threshold. S12 allows —which can be viewed as capturing “all other factors” that determine —to have its own structural effect on . It is therefore not as restrictive as a standard exclusion restriction, but S12 does require that the impact of is continuous. S13 is the most important condition for identification, and we discuss it further below.

Our second criterion is the extent to which inferences could be consistent with many competing behavioral models. The question is to what extent does S12 and S13 restrict the class of models for which the RD causal inference remains valid? When program status is determined solely on the basis of a score , and is used for nothing else but the determination of , we expect most economic models to predict that the only reason why would have a discontinuous impact on would be because an individual’s status switches from non-treated to treated. So, from the perspective of modeling economic behavior, S12 does not seem to be particularly restrictive.

By contrast, S13 is potentially restrictive, ruling out some plausible economic behavior. Are individuals able to influence the assignment variable, and if so, what is the nature of this control? This is probably the most important question to ask when assessing whether a particular application should be analyzed as an RD design. If individuals have a great deal of control over the assignment variable and if there is a perceived benefit to a treatment, one would certainly expect individuals on one side of the threshold to be systematically different from those on the other side.

Consider the test-taking example from Thistlethwaite and Campbell (1960). Suppose there are two types of students: and . Suppose type students are more able than types, and that types are also keenly aware that passing the relevant threshold (50 percent) will give them a scholarship benefit, while types are completely ignorant of the scholarship and the rule. Now suppose that 50 percent of the questions are trivial to answer correctly, but due to random chance, students will sometimes make careless errors when they initially answer the test questions, but would certainly correct the errors if they checked their work. In this scenario, only type students will make sure to check their answers before turning in the exam, thereby assuring themselves of a passing score. The density of their score is depicted in the truncated density in Fig. 1. Thus, while we would expect those who barely passed the exam to be a mixture of type and type students, those who barely failed would exclusively be type students. In this example, it is clear that the marginal failing students do not represent a valid counterfactual for the marginal passing students. Analyzing this scenario within an RD framework would be inappropriate.

Figure 1 Density of Assignment Variable Conditional on , .

On the other hand, consider the same scenario, except assume that questions on the exam are not trivial; there are no guaranteed passes, no matter how many times the students check their answers before turning in the exam. In this case, it seems more plausible that among those scoring near the threshold, it is a matter of “luck” as to which side of the threshold they land. Type students can exert more effort—because they know a scholarship is at stake—but they do not know the exact score they will obtain. This can be depicted by the untruncated density in Fig. 1. In this scenario, it would be reasonable to argue that those who marginally failed and passed would be otherwise comparable, and that an RD analysis would be appropriate and would yield credible estimates of the impact of the scholarship.

These two examples make it clear that one must have some knowledge about the mechanism generating the assignment variable, beyond knowing that if it crosses the threshold, the treatment is “turned on”. The“folk wisdom” in the literature is to judge whether the RD is appropriate based on whether individuals could manipulate the assignment variable and precisely “sort” around the discontinuity threshold. The key word here should be “precise”, rather than “manipulate”. After all, in both examples above, individuals do exert some control over the test score. And indeed in virtually every known application of the RD design, it is easy to tell a plausible story that the assignment variable is to some degree influenced by someone. But individuals will not always have precise control over the assignment variable. It should, perhaps, seem obvious that it is necessary to rule out precise sorting to justify the use of an RD design. After all, individual self-selection into treatment or control regimes is exactly why simple comparison of means is unlikely to yield valid causal inferences. Precise sorting around the threshold is self-selection.

Finally, the data generating process given by D4, D5, S11, S12, and S13 has many testable implications. D4 and D5 are directly verifiable, and S11 can be checked since the marginal density can be observed from the data. S12 appears fundamentally unverifiable, but S13, which generates the local randomization result, is testable in two ways. First, as McCrary (2008) points out, the continuity of each type’s density (S13) implies that the observed marginal density is continuous, leading to a natural test: examining the data for a potential discontinuity in the density of at the threshold; McCrary (2008) proposes an estimator for this test. Second, the local randomization result implies that

because is by definition determined prior to (Eq. (1)). This is analogous to the test of randomization where treated and control observations are tested for balance in observable elements of . ³²

In summary, based on the above three criteria, the RD design does appear to have potential to deliver highly credible causal inferences because some important aspects of the model are literal descriptions of the assignment process and because the conditions for identification are consistent with a broad class of behavioral models (e.g. can be endogenous—as it is influenced by actions in —as long there is “imprecise” manipulation of ). Perhaps most importantly, the key condition that is not derived from our institutional knowledge (and is the key restriction on economic behavior (S13)), has a testable implication that is as strong as that given by the randomized experiment.

3.3.2 Ex ante evaluation: extrapolating from the RD to treatment on the treated

Consider again our hypothetical job search assistance program, where individuals were assigned to the treatment group if their score (based on earnings and employment information and the state agency’s model of who would most benefit from the program) exceeded the threshold . From an ex ante evaluation perspective, one could potentially be interested in predicting the policy impact of “shutting down the program” Heckman and Vytlacil (2005). As pointed out in Heckman and Vytlacil (2005), the treatment on the treated parameter (TOT) is an important ingredient for such a prediction. But as we made precise in the ex post evaluation discussion, the RD estimand identifies a particular weighted average treatment effect that in general will be different from the TOT. Is there a way to extrapolate from the average effect in (14) to TOT?

We now sketch out one such proposal for doing this, recognizing this is a relatively uncharted area of research (and that TOT, while an ingredient in computing the impact of the policy of “shutting down the program”, may not be of interest for a different proposed policy). Using D4, D5, S11, S12, and S13, we can see that the TOT is

This reveals that the key missing ingredient is the second term of this difference.

Note that we have

where the second line again follows from Bayes’ rule.

Suppose has bounded support, and also assume

• S14: (Differentiable Density) Let have continuous th derivative on , for all .

• S15: (Differentiable Outcome Function) Let have continuous th derivative on , for all .

With the addition of S14 and S15, this will imply that we can use the Taylor approximation

In principle, once this function can be (approximately) identified, one can average the effects over the treated population using the conditional density .

Once again, we see that the leading term of an extrapolation for an ex ante evaluation, is related to the results of an ex post evaluation: it is precisely the counterfactual average in Eq. (14). There are a number of ways of estimating derivatives nonparametrically (Fan and Gijbels, 1996; Pagan and Ullah, 1999), or one could alternatively attempt to approximate the function for with a low-order global polynomial. We recognize that, in practical empirical applications, estimates of higher order derivatives may be quite imprecise.

Nevertheless, we provide this simple method to illustrate the separate role of conditions for credible causal inference (D4, D5, S11, S12, and S13), and the additional structure needed (S14 and S15) to identify a policy-relevant parameter such as the TOT. What alternative additional structure could be imposed on Eqs (1)-(3) to identify parameters such as the TOT seems to be an open question, and likely to be somewhat context-dependent.

3.3.3 Estimation issues for the RD design

There has been a recent explosion of applied research utilizing the RD design (see Lee and Lemieux (2009)). From this applied literature, a certain “folk wisdom” has emerged about sensible approaches to implementing the RD design in practice. The key challenge with the RD design is how to use a finite sample of data on and to estimate the conditional expectations in the discontinuity . Lee and Lemieux (2009) discuss these common practices and their justification in greater detail. Here we simply highlight some of the key points of that review, and then conclude with the recommended “checklist” suggested by Lee and Lemieux (2009). ³³

A Recommended “checklist” for implementation

Below we summarize the recommendations given by Lee and Lemieux (2009) for the analysis, presentation, and estimation of RD designs.

1. To assess the possibility of manipulation of the assignment variable, show its distribution. The most straightforward thing to do is to present a histogram of the assignment variable, after partitioning the support of into intervals; in practice, may have a natural discreteness to it. The bin widths should be as small as possible, without compromising the ability to visually see the overall shape of the distribution. The bin-to-bin jumps in the frequencies can provide a sense in which any jump at the threshold is “unusual”. For this reason, we recommend against plotting a smooth function comprised of kernel density estimates. A more formal test of a discontinuity in the density can be found in McCrary (2008).

6. Explore the sensitivity of the results to the inclusion of baseline covariates. As discussed above, in a neighborhood of the discontinuity threshold, pre-determined covariates will have the same distribution on either side of the threshold, implying that inclusion of those covariates in a local regression should not affect the estimated discontinuity. If the estimates do change in an important way, it may indicate a potential sorting of the assignment variable that may be reflected in a discontinuity in one or more of the baseline covariates. Lee and Lemieux (2009) show how the assumption that the covariates can be approximated by the same order of polynomial as as a function of can be used to justify including the covariates linearly in a polynomial regression.

Although it is impractical for researchers to present every permutation of presentation (e.g. points 2-4 for every one of 20 baseline covariates), probing the sensitivity of the results to this array of tests and alternative specifications—even if they only appear in online appendices—is an important component of a thorough RD analysis.