Job with SI on leading impact evaluations

An interesting job opportunity announcement landed in my inbox from our friends at Social Impact (link):

Social Impact’s impact evaluation work has grown quite rapidly, with 15 IEs now in the planning or implementation stage. Given the rapid growth, we are looking for additional technical staff. Essentially, recent post-docs who have IE/field research experience and who could technically lead a portfolio of IEs. The specific qualifications and sector of expertise are somewhat flexible if we find a good candidate, and I think this is a great opportunity for someone who wants to get more hands-on development research experience with regular opportunities for publication.


The full recruitment notice with instructions on how to apply is linked here: Posting_Impact-evaluation-advisor.

Share

“, robust” for experimenters

Forthcoming in Statistics and Probability Letters are results by Peter Aronow and myself on how heteroskedasticity-robust and homoskedasticity variance estimators for regression coefficients relate to the exact randomization variance of the difference-in-means estimator in a randomized experiment.

The gated link is here (link). For those without access to the journal, contact me for a copy.

The main results are that the ol’ White heteroskedasticity-robust variance estimator (aka, “, robust”) yields a conservative (in expectation) approximation to the exact randomization variance of the difference-in-means for a given sample, and estimates precisely the randomization variance of the difference-in-means when we assume that the experimental sample is a random sample from some larger population (the “super-population” interpretation). There are two slight tweaks though: (1) the exact equivalency is for the “leverage corrected” version of White’s estimator, but the difference between this version and White’s original version is negligible in all but very small samples; (2) because of Jensen’s inequality, these nice results for the variance don’t necessarily translate to their square root — a.k.a., your standard errors — but the consequences shouldn’t too horrible. The take-away, then, is that experimenters can feel okay about using “, robust” to obtain standard errors for their average treatment effect estimates in randomized experiments (assuming no cluster randomization, which would require other kinds of adjustment).

We also show that in the special case of a randomized experiment with a balanced design (equal numbers of treated and control), all sorts of estimators — including the heteroskedasticity-robust estimator, homoskedasticity estimator, and “constant effects” permutation estimator — are actually algebraically equivalent! So balance between treatment and control group sizes is a nice feature because of the way that it eases variance estimation.

Share

Causal identification in social network experiments

The always illuminating Development Impact Blog (link) posts a synopsis of a nice paper by Jing Cai on social network effects in the take up of weather insurance in China (link). Social network effects are identified via a nice experimental design, which I’ll paraphrase as follows: financial education and insurance offers are given to a random subset of individuals in year 1. Then in year 2, insurance offers are given to the rest of the individuals. Social network effects on year 2 targets are measured in terms of their take-up rate as a function of the fraction of their friends that had been targeted in year 1. Positive social network effects correspond to a positive relation between year 2 take-up and the fraction of one’s friends having been targeted in year 1. The icing on the cake is that experiment also randomized the price of insurance offers, and could therefore translate the social network effect into a price equivalent, findings that the “effect is equivalent to decreasing the average insurance premium by 12%.”

One concern that I am not sure is addressed in the paper is a potential selection bias that can arise due to differences in individuals’ network size. To see how this works, consider a toy example. A community consists of 4 people labeled 1, 2, 3, and 4. Graphs of the friendship networks between these people are as follows: Person 1 is friends with 2 and 3, person 2 is friends with 1, 3, and 4, and so forth.

Suppose that in year 1, 2 out of these 4 are randomly assigned to receive an intervention. That yields 6 possible year 1 treatment groups, each equally likely: {1,2}, {1,3}, {1,4}, {2,3}, {2,4}, or {3,4}. As in the paper discussed above, suppose what we are interested in is the effect of some fraction of your friends receiving the year 1 treatment. We can compute what these fractions would be under each of the year 1 treatment assignments. With these, we can compute the propensity that a given fraction of a person’s friends received the year 1 treatment. These calculations are shown in the following table:

What this shows is that simple randomization of year 1 assignment does not result in simple random assignment of fractions for the year 2 analysis. Take person 1, for example. This person has a 2/3 chance of having half of her friends targeted in year 1, and a 1/3 chance of having all of her friends targeted. This is in contrast to person 2, who has no chance of having either half or all friends targeted. The design does not result in representative samples of people receiving the different fraction values. Examining differences conditional on fraction values does not necessarily recover causal effects because the populations being compared are not exchangeable. To justify a causal interpretation, one would have to assume that network structure is ignorable or that such ignorability could be achieved through covariates, etc. The problem arises because fraction value is a function of both a person’s friendship network and the year 1 assignment. Only one of those two elements is randomized.

Peter Aronow and I have generalized this issue in a paper on “Estimating Causal Effects Under General Interference.” A recent working draft that we presented at the NYU Development Economics seminar is here: link. The basic idea behind the methods that we propose are to (1) define a way to measure direct and indirect exposures, (2) determine how for each unit of analysis the design induces probabilities of different kinds of exposures, and (3) use principles from unequal probability sampling to estimate average causal effects of the various direct and indirect exposures. Another point that arises from the analysis is that you can use the propensities of different exposures to assess the causal leverage of different designs. For example, consider again the toy example above. Suppose that these four units constitute one cluster among many possible clusters. Conceivably, a design that randomized not only which people were targeted in the first round within a cluster but also how many were targeted from cluster to cluster could produce fraction value propensities that were equal or, short of that, at least ensured that everyone had some probability having each of a variety of fraction values.

Share