The Papers and Hot Beverages (PHB) blog had a nice discussion (link) of some of the points I raised in my previous post about “pursuing external validity by letting treatments vary” (link). PHB starts by proposing that we can rewrite a simple treatment effects model along the lines of the following (modified from PHB’s expression to make things clearer):
The idea is that the treatment may bundle various components, captured by the terms, each of which has its own effect. Moreover, each of these components may interact with features of the context, captured by the terms.
The proposal to explore external validity by “letting treatments vary” amounts to trying to identify the effect of one of the components by generating variation in that component that is independent of the other components. Of course, in doing so, one does not resolve the problem of covariation with the components. So in this way, I understand why PHB was not “convinced” about the strategy of letting treatments vary as being sufficient for testing a parsimonious proposition that focuses on the effect of a particular component of a treatment bundle in a manner that does not incorporate contextual conditions. Of course, I was not trying to propose that such a strategy is sufficient in this way. Just that it is another way to think about accumulating knowledge across studies.
We can also go further and provide a more complete characterization of the problem of interpreting a treatment effect. Indeed, PHB’s characterization imposes some restrictions relative to the following:
The s are effects of elements in that depend on neither other elements of the treatment bundle nor the context . The s are the ways that elements of the treatment bundle modify each others’ effects regardless of context. The s are ways that the context modifies the effects of elements of separately. Finally, the s are ways that context modifies the ways that elements of modify the effects of each other.
When using causal estimates to develop theories, we typically want to interpret manipulations of in parsimonious terms. The upshot is that in trying to be parsimonious we may ignore elements of or . Even if the effect of is well identified, our parsimonious interpretation may not be valid.
This is a mess of an expression. But I find it strangely mesmerizing. It gives some indication of how complicated is the work of interpreting causal effects.