Cyrus Samii

The Papers and Hot Beverages (PHB) blog had a nice discussion (link) of some of the points I raised in my previous post about “pursuing external validity by letting treatments vary” (link). PHB starts by proposing that we can rewrite a simple treatment effects model along the lines of the following (modified from PHB’s expression to make things clearer):

$latex Y = \mu + \rho T + \epsilon$
$latex \hspace{2em} = \mu+\left(\sum_k \alpha_k x_k + \sum_j \sum_k \kappa_{jk} z_j x_k\right) T + \epsilon$.

The idea is that the treatment may bundle various components, captured by the $latex x$ terms, each of which has its own effect. Moreover, each of these components may interact with features of the context, captured by the $latex z$ terms.

The proposal to explore external validity by “letting treatments vary” amounts to trying to identify the effect of one of the $latex x$ components by generating variation in that component that is independent of the other $latex x$ components. Of course, in doing so, one does not resolve the problem of covariation with the $latex z$ components. So in this way, I understand why PHB was not “convinced” about the strategy of letting treatments vary as being sufficient for testing a parsimonious proposition that focuses on the effect of a particular component of a treatment bundle in a manner that does not incorporate contextual conditions. Of course, I was not trying to propose that such a strategy is sufficient in this way. Just that it is another way to think about accumulating knowledge across studies.

We can also go further and provide a more complete characterization of the problem of interpreting a treatment effect. Indeed, PHB’s characterization imposes some restrictions relative to the following:

$latex Y = \mu + \rho T + \epsilon$

$latex = \mu + \left( \sum_{k} \alpha_k x_k + \sum_{k}\sum_{k’\ne k} \beta_{kk’}x_kx_{k’} \right.$
$latex \left. + \sum_{j} \sum_{k} \kappa_{jk}z_jx_k + \sum_{j} \sum_{k} \sum_{k’\ne k} \delta_{jkk’}z_jx_kx_{k’} \right)T + \epsilon $

The $latex \alpha$s are effects of elements in $latex x$ that depend on neither other elements of the treatment bundle $latex x$ nor the context $latex z$. The $latex \beta$s are the ways that elements of the treatment bundle modify each others’ effects regardless of context. The $latex \kappa$s are ways that the context modifies the effects of elements of $latex x$ separately. Finally, the $latex \delta$s are ways that context modifies the ways that elements of $latex x$ modify the effects of each other.

When using causal estimates to develop theories, we typically want to interpret manipulations of $latex T$ in parsimonious terms. The upshot is that in trying to be parsimonious we may ignore elements of $latex x$ or $latex z$. Even if the effect of $latex T$ is well identified, our parsimonious interpretation may not be valid.

This is a mess of an expression. But I find it strangely mesmerizing. It gives some indication of how complicated is the work of interpreting causal effects.

The essence of the design-based approach is that one establishes optimality principles in terms of research design parameters, while making minimal assumptions about the distributional properties of the variables in the analysis. The motivation for a design-based perspective is three-fold.

First, I do a lot of field research. Most field research projects seek to obtain data on a variety of outcome variables. Each of these outcome variables might differ in its distributional properties. In such cases, one wants optimal design principles that are robust to such variety. The design- based approach achieves precisely this robustness goal. This is the line of thinking one associates with Leslie Kish, William Cochran, and the literature on “official statistics.”

Second, the design-based approach aims to minimize errors of inference (for example, inaccurate confidence intervals) that arise when one uses methods that rely on distributional assumptions that are inaccurate. The design based approach achieves such error minimization by defining inferential properties primarily in terms of the design parameters that are directly controlled, and therefore known, by the researcher. This is the line of thinking that one associates with David Freedman.

Third, the design-based approach minimizes reliance on data parameters that are only revealed after the data are collected. This allows for the pre-specification and ex ante agreement among scholars on the inferential leverage that a design offers. For example, it allows for ex ante specification and agreement on what hypotheses can be tested and with what power. A design document and pre-analysis plan can guard against ex post manipulations and “results fishing,” arguably allowing for more credible accumulation of scientific knowledge. This is the line of thinking that one associates with current proponents of pre-analysis plans like Edward Miguel and Macartan Humphreys.

The Development Impact Blog (link) has an outstanding post by Ken Leonard discussing the differences between prosocial and intrinsic motivations for public service workers:

When we hear of a fireman who works for money, we immediately think about the wage, its relationship to performance and the way incentives are organized within the system. If you hope to be rescued by someone who works only for the money, you want to know how and under what circumstances he earns more. The term intrinsic motivation, however, often leads to policy paralysis: if they love to do their job, then let them continue to do their job. Once we recognize, however, that public sector workers might be motivated by the gratitude or admiration of others (not the act of doing their job), we might be more likely to ask about wages, incentives and organizations—the same questions we would automatically ask about money. For example, does seeking the gratitude of patients or students increase or decrease useful effort? Can organizations increase exposure to positive incentives and decrease exposure to negative incentives?

(Link to post.).

So intrinsic: “I like doing this work.” Prosocial (or, perhaps better termed, Other-regarding): “I like what this work does for others” or “I like how others view my doing this work.”

If you find such a discussion of intrinsic/extrinsic, other/self-regarding motivations interesting, you should really read Jon Elster: link.

This also reminds me of a conversation I had with Leonard Wantchekon last year:

“I am hoping to organize a conference on organization theory—you know, as a way to gain insights on why state institutions sometimes work and sometimes don’t,” he said.

“Sounds great!” I replied.

“Yeah, and you know what?” he continued. “If the key result of your model or field experiment is ‘higher wages lead to better performance’, you’re not invited!”