Cyrus Samii

Here are some notes I put together for my Quant II class on this topic: PDF. My personal take-away from this is that OLS is fine for these kinds of applications, and carries with it the benefits of simplicity, well-understood operating characteristics, and consistency when we have, e.g., lots of dummy variable to soak up “fixed effects.” But reasonable people may disagree, although with little consequence I think. I emphasize “aggregation bias” in these notes—something that is still not part of conventional training in statistics or econometrics but is of central concern in the contemporary causal inference literature. In the simulations here, the aggregation bias is not such a big deal. To see an example where it is a big deal, overwhelming omitted variable bias, see this simulation code: R code. Acknowledgement of aggregation bias renders many conventional practices illegitimate for estimating causal effects. For example, the practice of interpreting MLE estimates by “setting all other variables to their mean” and then looking at predicted values doesn’t work. Rather, one needs to look at within-sample predictions that incorporate all of the heterogeneity present in the sample. The simulations in the notes here demonstrate.

Here is a link to slides from my talk at University of British Columbia (UBC) political science department yesterday: link. The talk was on a randomized field experiment that Eric Mvukiyehe and I recently completed in Liberia. The field experiment tested the effectiveness of curriculum-based and security-institution-based strategies for preventing intimidation and violence during the 2011 elections. The endline data are still coming in, so the presentation focused on the theoretical motivation and design, with only a light discussion of preliminary results. We are hoping to have a project report in the coming weeks, and then papers over the coming year or so. Updates will be posted when those are out.

The talk was part of an excellent series that the UBC political science department is hosting on “Experiments in Development.” Here is the full roster of speakers: link. The series will host the semi-annual Experiments in Governance and Politics (EGAP) meeting in a few weeks as well: link.

Here is a link to the slides that Peter presented yesterday at the NYU-CESS conference on experiments in political science: PDF. (The link to the conference site is here: link.) Here is a very slightly updated version of the paper that includes a minor correction: PDF. Comments or additional corrections welcome.

A few interesting issues came up during the Q&A that are worth some more discussion and consideration:

• Uncertainty over exposure models: The exposure model is formalized as . As discussed, the properties of the ‘s are known by design. This leaves the potential for uncertainty about either or . Uncertainty about is basically a measurement problem, and so we can put a probability distribution on values of based all the available data and then integrate over it. E.g., if is a row in a network adjacency matrix, then we could use available data to create a bunch of imputed adjacency matrices, and then integrate over those imputations. Uncertainty over is different. When you change you are changing the set of causal estimands. In this case, the issue is one of model selection. Because our framework allows for arbitrarily complex exposure models, one could apply the usual model selection principles to work from a complex model to a more parsimonious nested model.
• Reciprocal effects and dynamics in indirect exposure: A question came up as to how we would handle the possibility that effects could initially transmit from A to B, but then transmit back from B to A, and so forth in a dynamic and reciprocal way. A thought that comes immediately to mind is that this too may be best characterized as a measurement problem — e.g., are you measuring outcomes after all such dynamics have led to steady state or are you measuring outcomes at some point mid-way toward steady state? Causal effects could reasonably be defined in either terms and estimated using the methods proposed in the paper; one should simply be careful that one is measuring the outcomes that they think they are measuring.

Happy to hear more questions or comments. Upcoming presentations of the work will be at the Princeton methodology seminar and the EGAP conference in Vancouver.

A call for research designs that came in over the wires. This is an exceptional opportunity for researchers at the early phase of a project to strengthen their design:

Call for Research Designs

Conference on Field Experiments in International Relations

Experiments provide the best method for identifying causal effects in social science, but they have been undersupplied in international relations. To help fill this gap, we are convening a conference of interested scholars to discuss research designs for field experiments in transnational affairs. The conference will be held in Park City, Utah September 21-22, 2012. After the conference, scholars will revise the designs, execute the experiments, and then present their findings and receive feedback at a follow up meeting to be held at Princeton University in the fall of 2013.

Rationale

Individuals, firms, non-governmental organizations, and international bureaucrats play vital roles in the modern world. But unlike nation states – which are difficult to manipulate either practically or ethically – scholars can employ experiments with these non-state actors as subjects.

Experiments’ strong internal validity makes them an especially attractive research method for causal inference. Randomization permits the precise estimation of causal effects because in expectation it balances not only the observable factors that might confound results, but it also neutralizes all unobservable confounds. This is a significant advantage over observational research, which can never establish with certainty that the model employed is properly specified.

Field experiments add an additional improvement by addressing concerns over external validity. The day-to-day international actions of non-state actors and the effects of their behavior on global outcomes are worthy of close study. Field experiments in IR – where non-state actors as subjects represent the actual units of interest – likely can be better defended as externally valid while retaining many of the internal-validity advantages of lab experiments. Convening a significant group of scholars focused on brainstorming research designs, refining plans, and analyzing results should help international relations take an important step toward uncovering causal effects in global affairs.

Details

Helen Milner of Princeton University, along with Michael Findley and Daniel Nielson of Brigham Young University, are organizing the conference. Princeton’s Robert Keohane, Columbia’s Donald Green, Stanford’s Michael Tomz and Jeremy Weinstein, Yale’s Susan Hyde, and Harvard’s Dustin Tingley are also planning to attend. Interested scholars should submit an abstract no longer than 500 words by March 15, 2012 to BYU’s Political Economy and Development Lab at pedl@byu.edu.

The initial conference will focus on research designs, not finished papers. The abstracts should therefore articulate the research question, hypotheses, and causal mechanisms along with the anticipated subject pool, experimental conditions, outcome measures, and data analysis strategy. Both collaborative and sole-authored projects are encouraged.

Topics covering the full range of international and transnational relations – including but not limited to political economy, security, environment, and human rights – are welcome. Abstracts should address some aspect of transboundary interactions and should be field experiments rather than survey or laboratory studies, meaning that the subjects are the actual objects of inquiry rather than proxies, the outcome of interest is behavioral (not attitudinal), and the subjects’ actions are observed in a natural setting.

Princeton and BYU have made funds available to sponsor some – but not all – of the conference participants. Please provide contact information with your abstract, and also indicate whether or not your home institution can pay for your airfare and lodging. Meals and ground transportation are provided. Examples of field experiment designs in international relations are available upon request.

For NYU grad and undergrads, this came in via email today:

New York University is one of eight universities to offer the Dalai Lama Fellowship, an international program of the Dalai Lama Center. One fellow will be chosen from each university to receive up to $10,000 toward a year-long project of his of her design in one of four focus areas:

1. diminishing violence;
2. mitigating income and wealth inequities;
3. encouraging cross-cultural and inter-religious communications and understanding;
4. promoting environmental sustainability.

All full-time students – undergraduate, graduate and professional students – who will be enrolled at NYU in Academic Year 2012 – 2013, are invited to apply. Details here: link.

Application Deadline: February 21, 2012

Please direct any inquiries to civic.engagement@nyu.edu or 212 998-2329.