Here (link) is a note working through some algebra for comparing the following:
- The Neyman “conservative” estimator for the variance of the difference-in-means estimator for the average treatment effect. This estimator is derived by applying sampling theory to the case of a randomized experiment on a fixed population or sample. Hardcore experimentalists might insist on using this estimator to derive the standard errors of a treatment effect estimate from a randomized experiment. This is also known as the conservative “randomization inference” based variance estimator.
- The Huber-White heteroskedasticity robust variance estimator for the coefficient from a regression of an outcome on a binary treatment variable. This is a standard-use estimator for obtaining standard errors in contemporary econometrics. Taking from Freedman’s famous words, though, “randomization does not justify” this estimator.
If you work through the algebra some more, you will see that they are equivalent in balanced experiments, but not quite equivalent otherwise.
This post is part of a series of little explorations I’ve been doing into variance estimators for treatment effects. See also here, here, and here.
UPDATE 1 (4/8/11): A friend notes also that under a balanced design, the homoskedastic OLS variance estimator is also algebraically equivalent. When the design is not balanced, the homoskedastic and heteroskedastic robust estimators can differ quite a bit, with the latter being closer to the Neyman estimator, but still not equivalent to the Neyman estimator due to the manner in which treated versus control group residuals are weighted.
UPDATE 2 (4/12/11): The attached note is updated to carry through the algebra showing that the difference between the two estimators is very slight.
UPDATE 3 (4/12/11): A reader pointed out via email that this version of the heteroskedasticity robust estimator is known as “HC1”, and that Angrist and Pischke (2009) have a discussion of alternative forms (see Ch. 8, especially p. 304). From Angrist and Pischke’s presentation, we see that HC2 is exactly equivalent to the Neyman conservative estimator, and this estimator is indeed available in, e.g., Stata.
UPDATE 4 (4/12/11): Another colleague pointed out (don’t you love these offline comments?) that the Neyman conservative estimator typically carries an N/(N-1) finite sample correction premultiplying the expression shown in the note, in which case even in a balanced design, the estimators differ on the order of 1/(N-1). Later discovered that this was not true.
