More on external (and construct) validity

The Papers and Hot Beverages (PHB) blog had a nice discussion (link) of some of the points I raised in my previous post about “pursuing external validity by letting treatments vary” (link). PHB starts by proposing that we can rewrite a simple treatment effects model along the lines of the following (modified from PHB’s expression to make things clearer):

$latex Y = \mu + \rho T + \epsilon$
$latex \hspace{2em} = \mu+\left(\sum_k \alpha_k x_k + \sum_j \sum_k \kappa_{jk} z_j x_k\right) T + \epsilon$.

The idea is that the treatment may bundle various components, captured by the $latex x$ terms, each of which has its own effect. Moreover, each of these components may interact with features of the context, captured by the $latex z$ terms.

The proposal to explore external validity by “letting treatments vary” amounts to trying to identify the effect of one of the $latex x$ components by generating variation in that component that is independent of the other $latex x$ components. Of course, in doing so, one does not resolve the problem of covariation with the $latex z$ components. So in this way, I understand why PHB was not “convinced” about the strategy of letting treatments vary as being sufficient for testing a parsimonious proposition that focuses on the effect of a particular component of a treatment bundle in a manner that does not incorporate contextual conditions. Of course, I was not trying to propose that such a strategy is sufficient in this way. Just that it is another way to think about accumulating knowledge across studies.

We can also go further and provide a more complete characterization of the problem of interpreting a treatment effect. Indeed, PHB’s characterization imposes some restrictions relative to the following:

$latex Y = \mu + \rho T + \epsilon$

$latex = \mu + \left( \sum_{k} \alpha_k x_k + \sum_{k}\sum_{k’\ne k} \beta_{kk’}x_kx_{k’} \right.$
$latex \left. + \sum_{j} \sum_{k} \kappa_{jk}z_jx_k + \sum_{j} \sum_{k} \sum_{k’\ne k} \delta_{jkk’}z_jx_kx_{k’} \right)T + \epsilon $

The $latex \alpha$s are effects of elements in $latex x$ that depend on neither other elements of the treatment bundle $latex x$ nor the context $latex z$. The $latex \beta$s are the ways that elements of the treatment bundle modify each others’ effects regardless of context. The $latex \kappa$s are ways that the context modifies the effects of elements of $latex x$ separately. Finally, the $latex \delta$s are ways that context modifies the ways that elements of $latex x$ modify the effects of each other.

When using causal estimates to develop theories, we typically want to interpret manipulations of $latex T$ in parsimonious terms. The upshot is that in trying to be parsimonious we may ignore elements of $latex x$ or $latex z$. Even if the effect of $latex T$ is well identified, our parsimonious interpretation may not be valid.

This is a mess of an expression. But I find it strangely mesmerizing. It gives some indication of how complicated is the work of interpreting causal effects.