More on external (and construct) validity

The Papers and Hot Beverages (PHB) blog had a nice discussion (link) of some of the points I raised in my previous post about “pursuing external validity by letting treatments vary” (link). PHB starts by proposing that we can rewrite a simple treatment effects model along the lines of the following (modified from PHB’s expression to make things clearer):

$latex Y = \mu + \rho T + \epsilon$
$latex \hspace{2em} = \mu+\left(\sum_k \alpha_k x_k + \sum_j \sum_k \kappa_{jk} z_j x_k\right) T + \epsilon$.

The idea is that the treatment may bundle various components, captured by the $latex x$ terms, each of which has its own effect. Moreover, each of these components may interact with features of the context, captured by the $latex z$ terms.

The proposal to explore external validity by “letting treatments vary” amounts to trying to identify the effect of one of the $latex x$ components by generating variation in that component that is independent of the other $latex x$ components. Of course, in doing so, one does not resolve the problem of covariation with the $latex z$ components. So in this way, I understand why PHB was not “convinced” about the strategy of letting treatments vary as being sufficient for testing a parsimonious proposition that focuses on the effect of a particular component of a treatment bundle in a manner that does not incorporate contextual conditions. Of course, I was not trying to propose that such a strategy is sufficient in this way. Just that it is another way to think about accumulating knowledge across studies.

We can also go further and provide a more complete characterization of the problem of interpreting a treatment effect. Indeed, PHB’s characterization imposes some restrictions relative to the following:

$latex Y = \mu + \rho T + \epsilon$

$latex = \mu + \left( \sum_{k} \alpha_k x_k + \sum_{k}\sum_{k’\ne k} \beta_{kk’}x_kx_{k’} \right.$
$latex \left. + \sum_{j} \sum_{k} \kappa_{jk}z_jx_k + \sum_{j} \sum_{k} \sum_{k’\ne k} \delta_{jkk’}z_jx_kx_{k’} \right)T + \epsilon$

The $latex \alpha$s are effects of elements in $latex x$ that depend on neither other elements of the treatment bundle $latex x$ nor the context $latex z$. The $latex \beta$s are the ways that elements of the treatment bundle modify each others’ effects regardless of context. The $latex \kappa$s are ways that the context modifies the effects of elements of $latex x$ separately. Finally, the $latex \delta$s are ways that context modifies the ways that elements of $latex x$ modify the effects of each other.

When using causal estimates to develop theories, we typically want to interpret manipulations of $latex T$ in parsimonious terms. The upshot is that in trying to be parsimonious we may ignore elements of $latex x$ or $latex z$. Even if the effect of $latex T$ is well identified, our parsimonious interpretation may not be valid.

This is a mess of an expression. But I find it strangely mesmerizing. It gives some indication of how complicated is the work of interpreting causal effects.

4 thoughts on “More on external (and construct) validity”

1. Fernando

Why not a zz term? Today’s moderator is tomorrow’s treatment.

And while at it, why not a zzxx term?

But then you might as well use a causal diagram, where heterogeneity is pervasive.

Put differently, parametric specifications only make sense to me in the context of testing severe simplifications.

2. lmguirola

I initially thought of formulating the model in a non-parametric form, but thought this way made it easier to see the analogy with the standard linear model; obviously one can make it it as messy as one wishes.

Luis

3. Fernando

Cyrus:

Would love to put the graph in the comment but our present means if communication have too low a bandwidth. Is like we all use vernacular privately yet communicate in Latin.

Ok, I could put a link to a picture but that is like putting lipstick on a pig ðŸ™‚ But if you insist I’ll email you a graph with a molar treatment, to use the terminology of Shading, Cook, and Campbell that is completely saturated saturated by the environment. The more interesting question here is how you define and measure compliance.

So

This site uses Akismet to reduce spam. Learn how your comment data is processed.