Some thoughts on blinding and permutation methods

Here is an interesting twitter thread on blinding and permutation methods:

What Adam is proposing here is related to the “mock analysis” that Humphreys et al. discuss in their 2013 paper on fishing: link to preprint

I have also had discussions recently with Pieter Serneels and Andrew Zeitlin about this idea, who are writing on this topic (look out for their work, I will update with a link to it when it is available).

Generally I think simulation and “mock” analysis are great for checking power and other inferential characteristics. The DeclareDesign project is an attempt to systematize this approach: link.

That said, I think the statistics behind the blinding + permutation approach are a bit more subtle than what Adam’s post suggests. My concerns can be expressed using a toy example. Suppose the following toy research design:

  • We have only two units.
  • We run an experiment that randomly (fair coin flip) assigns one unit to treatment and the other to control.
  • Control potential outcomes are (0, 5) (that is, for the first unit the outcome is 0 under control and for the second unit the outcome is 5 under control).
  • There are two possible treatments that could be assigned. Treatment A has no effect, and so the potential outcomes under treatment A are (0, 5). Treatment B generates an effect such that under treatment B, potential outcomes are (5, 7) (so for the first unit, the effect is 5, and for the second it is 2).
  • That being the case, if treatment A were being applied, the experiment would always generate data (0, 5). If treatment B were being applied, then the experiment would generate either (0, 7) as data, or (5, 5) as data.
  • Now, let us suppose that we, the analysts, do not know which of treatment A or B was applied, nor do we know all of the potential outcomes. Rather, all we know are the fact that there are two units, one was assigned to treatment, and then the outcome data. We blind ourselves to which of the two units was assigned to treatment. Ultimately we are interested to learn whether treatment A or B was applied, but at the moment we want to operate in a manner that is blind to treatment assignment so as to figure a good way to test.

This toy example captures the situation that is relevant in Adam’s illustration of the blinding + permute method. However, it is straightforward to see the problem in this case. If it is indeed the case that treatment B was applied, then the resulting data will not allow us to characterize the null distribution (that is, the distribution that would have arisen had A been applied). Moreover, the resulting data could either over- or under-state the variance of outcomes under the null. That being the case, it seems problematic to adhere too closely to what one learns under blinding + permutation. Rather, I would propose that one use it only to get “ballpark” ideas of how different estimation strategies perform, but then for more refinement, I think you’d have to either use analytical results or try simulating data under different assumptions on the potential outcomes.


Notes on matrix completion methods

(Note: some typos in the notes corrected now.)

Below, I have posted some notes on matrix completion, inspired by this great Twitter thread by Scott Cunningham:

Have a look at Scott’s thread first. Also, have a look at the material that he posted. Then, the following may be helpful for further deciphering that methods (in formats friendly for online and offline reading):

Update: I had a very useful twitter discussion with @analisereal on the identification conditions behind matrix completion for estimating the ATT. Here is the thread and then I am updating the notes to incorporate these points:


Descriptive quantitative work in political science

Here is a roundup of replies to a question I posted on Twitter regarding descriptive quantitative research in political science:

Outside political science, I can think of a number of examples, although I was interested in political examples per se, and particularly ones that are published as papers:

One thing that distinguishes poli sci from, say, econ is that poli sci has lots of books, many of which contain important descriptive work, as in this:

Nonetheless, I was mostly interested in work published in paper form.

An important class of measurement contributions in poli sci include dimension reduction, scaling, and latent variable estimation methods. This includes things like ideal point estimation as well as analyses of text:

  • Example 1:
  • Example 2:
  • Example 3:
  • Example 4:

(Chris’s last name is spelled Fariss, by the way.)

Poli sci scholars have also done a lot to elaborate small area estimation techniques and use them in analyzing survey data, as with the “MRP” papers, e.g.:

Taxonomy, that is, organizing cases on the basis of conceptual categories, is another class of measurement-related work:

Sometimes descriptive work can indirectly inform causal questions:

What I was most interested in were creative contributions that don’t apply especially new statistical methods, but are the result of shoe-leather effort that allows us to view important dynamics more clearly. Examples:

Here’s a “hard copy” of this post (which I will update again after all edits are in), for archival sake, in anticipation of potential Twitter link instability: [PDF]


R and Stata code for inverse covariance weighting

A previous post had discussed differences between dimension reduction through principal components and factor analysis on the one hand and inverse covariance weighting (ICW) on the other: [link].

Here is a link to a Stata .ado GitHub repository with the code for ICW index construction, including both an R example as well as a Stata .do file that loads a program to construct indices: [.git]. The .do file itself contains instructions on using the function “make_index_gr”, which generates an ICW index that can include weights and can be set to standardize with respect to any subset of the data (e.g., against the control group).

Please give them a try and if you find any bugs, please let me know. Also, if anyone wants to do a more professional job with the coding, and even integrate them into broader packages, please be my guest.


  • The make_index_gr Stata program was modified on 2018-05-03 so that the resulting index indeed centers on the standardization group.
  • Post was edited on 2018-05-04 to link to the GitHub repository.

EGAP platform

I am grateful for your consideration as candidate for Executive Director. I have been active within EGAP since 2009, and it has been singularly important intellectually and professionally. I think we can increase the value that EGAP offers to its members, the broader social science community, and policy makers. The organization needs to balance many priorities. If I were elected as Executive Director, I would aim to promote the following to the extent that we can, taking into considering resource constraints and the need for balance:

1. Methodological training

EGAP events offer unique opportunities for increasing our methodological sophistication as a research community. We can devote more to this, including lectures and initiatives on the use of statistical and substantive theory to inform research design and analysis plans. EGAP can be the hub for methodological excellence in field-experimental and otherwise quantitative fieldwork-driven social science.

2. Policy engagement

We can be more systematic in promoting policy engagement. For example, we could host our meetings in national capitals and then hold expert sessions with local policy makers as separate events alongside the regular meeting.

3. New venues for scholarly publication

We can use the EGAP network to establish new venues for scholarly publication to overcome the fact that conventional journals are too slow and unreliable. A modest goal would be a working papers series (along the lines of NBER or BREAD), an ambitious one would be EGAP “proceedings” journals that operate in a manner similar to proceedings outlets in other disciplines like in computer science.

4. Geographic diversity

I think that it is important for us to continue broadening the geographic reach of the network in terms of membership, sites for events, and education activities such as our “learning days.” This includes doing more to engage scholars in Latin America, the Middle East, Africa, and Asia.

5. Deconcentrating leadership

I would like to distribute leadership positions over a broader set of members. My sense is that, at present, administration of meetings, member selection, and research programs is concentrated among too few individuals, and that this has only worked to date because of the exceptional commitment and energy of these few individuals. But this is unsustainable. I will look into different possibilities for delegation of tasks such as member selection, event organization, and management of research initiatives on the basis of region or thematic groups.