Forthcoming in Statistics and Probability Letters are results by Peter Aronow and myself on how heteroskedasticity-robust and homoskedasticity variance estimators for regression coefficients relate to the exact randomization variance of the difference-in-means estimator in a randomized experiment.
The gated link is here (link). For those without access to the journal, contact me for a copy.
The main results are that the ol’ White heteroskedasticity-robust variance estimator (aka, “, robust”) yields a conservative (in expectation) approximation to the exact randomization variance of the difference-in-means for a given sample, and estimates precisely the randomization variance of the difference-in-means when we assume that the experimental sample is a random sample from some larger population (the “super-population” interpretation). There are two slight tweaks though: (1) the exact equivalency is for the “leverage corrected” version of White’s estimator, but the difference between this version and White’s original version is negligible in all but very small samples; (2) because of Jensen’s inequality, these nice results for the variance don’t necessarily translate to their square root — a.k.a., your standard errors — but the consequences shouldn’t too horrible. The take-away, then, is that experimenters can feel okay about using “, robust” to obtain standard errors for their average treatment effect estimates in randomized experiments (assuming no cluster randomization, which would require other kinds of adjustment).
We also show that in the special case of a randomized experiment with a balanced design (equal numbers of treated and control), all sorts of estimators — including the heteroskedasticity-robust estimator, homoskedasticity estimator, and “constant effects” permutation estimator — are actually algebraically equivalent! So balance between treatment and control group sizes is a nice feature because of the way that it eases variance estimation.
