Welcome to Fall 2022 Field Methods. This class will teach you about designing social science RCTs. This involves the following steps:
- Posing a normatively-defined problem out in the real world. That is, tell us about “something that is wrong in the world” that deserves attention and resources to try to rectify.
- Using evidence and theory to identify a mechanism that is an important driver of this problem.
- Designing an intervention that can disrupt the mechanism.
- Designing an experimental test of the intervention.
The class will be hands-on and driven by research interests of people in the class. We will have two introductory sessions, and then you will go through 3 cycles of an RCT design cycle:
- 1st week of cycle (intervention concept): Use theory and data to argue for the importance of a particular mechanism, use it to define an intervention strategy, and present an intervention concept.
- 2nd week of cycle (outcomes measurement strategy): Define primary and secondary outcomes of interest and propose a strategy for measuring these outcomes. Defend the reliability and validity of these measures.
- 3rd week of cycle (experimental test): Define the causal effects of interest, a randomization strategy for identifying them, and an estimation and testing strategy for assessing. Defend your experimental design on the basis of unbiasedness and statistical power.
After the first two weeks, all of your work will be done in groups of 3 or 4 people. (I will assign the groups, and we may change the group assignments over the course of the semester.) Each week will involve making progress in successive steps of RCT design ass outlined above. During class time, the groups will present material to each other for discussion.
- We will meet in person in Room 217 Fridays 11-noon. Because much of the class involves group discussion I am not planning a Zoom hybrid option.
- No class on the following dates:
- Sept 16 (APSA)
- Oct 28 (EGAP meeting at Cornell)
- If you need to miss a class, coordinate with your group so that your contributions are incorporated in that week’s group presentation.
- I will add readings in the weekly plan entries below.
- If you want to meet to discuss your class work or to discuss accommodations, please sign up for office hours using Calendly: https://calendly.com/cdsamii
- Pre-requisites are Quant I and II or equivalent. Please discuss if you are unsure.
On Sept 9, we will be meeting in class with the goal of selecting the 3 “problems” that will serve as the basis of the group projects over the course of the semester. To do this, each of you will do a 5-minute “lightening talk” to try to persuade us that the problem you have selected is one of the problems we should select as a class. I want you to prepare at most three slides (really, no more than three) in which you do the following:
- State the problem (along the lines of what we did in class today) and defend your stance that it is indeed a problem, on normative grounds.
- Show us some compelling statistics that demonstrate that the problem is serious. This can be shown through indication of how extensive the problem is, in terms of really strong correlations with other problems, etc.
Your goal is to take 5 minutes to convince your skeptical peers that this problem matters.
By Thursday evening the day before class, email your three slides to me in PDF form, so that I can compile them into one slide deck.
Then, we will deliberate and ultimately vote to select three problems that will serve as the basis of the group projects for the semester.
Given what we heard today, there does appear to be substantial overlap in research interests, so it is possible that we will combine some people’s problems together into one.
Now, you are about a month away from when you will start having to design your first RCT. You should start doing some reading in preparation for that. Here are readings you should start working through, and continue to work through over the coming weeks:
- Gerber, Alan S., and Donald P. Green. Field experiments: Design, analysis, and interpretation. WW Norton, 2012. Ch 1-4. Available from Library or order a hard copy. Materials for replicating their examples is here: https://isps.yale.edu/FEDAI
- Duflo, Esther, Rachel Glennerster, and Michael Kremer. “Using randomization in development economics research: A toolkit.” Handbook of development economics 4 (2007): 3895-3962. link
- Athey, Susan, and Guido W. Imbens. “The econometrics of randomized experiments.” Handbook of economic field experiments. Vol. 1. North-Holland, 2017. 73-140. Skim this alongside Gerber and Green Chapter 3. link
- Learn about the estimatr R package, which will be our workhorse package when we get to analyzing experimental data: link
- Evidence in Governance and Politics (EGAP) has an online resource book that has various resources for field experimental design. Start perusing the material there: link
- Coppock, A. 2020. “10 Things to Know About Statistical Power.” EGAP Resource. link
I obviously don’t expect you to read all of this by Sept 9. Rather, make a plan for yourself to make your way through this material over the coming month.
First week slides: link
Note that we do not have class on September 16 (APSA). Nonetheless, in addition to continuing with the readings listed above, I want you to work through the code in this design effects exercise: link
I know it’s sort of silly to just copy and paste code, but I want you to understand what the code is doing exactly. It shows steps that you could use to test out experimental designs that include clustering and stratification. Send to me a PDF, ideally produced using RMarkdown, in which you replicate the exercise and comment on what you are doing in each code chunk.
In class on Sept 9 we selected the “problems” that will motivate our RCT design group projects for this semester:
- Participation in extremist movements
- Failure of integration for survival migrants
- Non-material performance motivation for local bureaucrats and public servants
We will proceed through these in the order listed above. So, we will spend the next three weeks on “Participation in extremist movements,” working through the three-part cycle listed above, and then we will move onto “Failure of integration for survival migrants” after that, and so.
First thing is the create the groups. We will have three groups of 4 working together. I will be sending an email with a proposed grouping, and then we can discuss.
First, please be sure that you have completed the design exercise. If you haven’t done so already, please send me a PDF of your RMarkdown output so that I can check that you are implementing and understanding things correctly. Please send those to me by Wednesday. We will take the first 15 minutes or so to discuss.
Then, we will turn to our first group project, focusing on the problem of “Participation in extremist movements.” Here are the group assignments that I am proposing:
Group 1 Avi Daniel Jimmy Dylan
Group 2 Mengfan Alper Henry Julieta
Group 3 Sangyong Lanie Sylvan Jona
I based the groups on overlap in the interests and contexts that I saw in each of the problems that you presented last week.
As discussed last week, each group can interpret and apply the theme of “Participation in extremist movements” in different ways. So, group 1 could stick with Jimmy’s original emphasis on Islamic extremism in Mozambique (that’s what I would recommend for them), but other groups could consider manifestations in another region/country and on the basis of other ideologies.
For Friday, each group should prepare to do a 15-20 minute presentation. What I would like each group to present is their version of the problem (2-4 slides). Then, I would like 2-4 slides that review possible causal processes or mechanisms that lead to participation in extremist movements within the context they are studying. For this part, you should consult the current literature and give appropriate references in the slides. Any intervention to prevent participation in extremist movements should seek to intervene on a mechanism that existing theory and evidence suggests is an important explanation for why people join extremist movements in the context that you are studying. Your assignment is to consult existing literature and evidence to see what some of these mechanisms might be. It’s possible that the mechanisms that are important for participation in Islamic extremism in Mozambique differ from, say, participation in rightwing extremist groups in the US, for example. It will be interesting to see the relationships between what different groups find.
Given the short amount of time, I don’t think you will be at the point where you can design your intervention yet. That will have to wait until the following week. (There is enough flexibility in the schedule to accommodate taking an extra week to do so.)
For this coming Friday, each group will be doing presentations in which they define their intervention. This will involve the following:
- 1 slide reminding us of your framing of the problem in your context,
- 1 slide telling us the precise population and outcomes (behaviors, attitudes, or beliefs) that are implicated in your framing of the problem and that will be the focus of your approach to intervention,
- 1 slide telling us the mechanism that you have selected, explaining why you have selected it (such as evidence for its demonstrated relevance, how compelling it is in terms of theory, whether it lends itself to intervention both in terms of feasibility and ethics, etc.) and how the mechanism explains the pattern of outcomes in the population that are your focus,
- 1 slide describing the intervention, including what kinds of tasks or incentives are being introduced and who exactly will be doing the intervening (e.g., will there be a partnership with an organization or agency?),
- 1 slide explaining how you propose that the intervention could work to change outcomes among your population of interest, based on your theoretical framing (mechanism) and existing evidence or examples of interventions already in use, and
- 1 slide explaining any potential political obstacles or ethical issues that you will be navigating.
At least one person in your group should have the task of searching for examples of existing interventions, to help to motivate and distinguish what your group is proposing to do. Others can focus on developing the content of your group’s intervention.
We will be discussing each group’s proposed interventions in terms of how well motivated they are on the basis of their theoretical framing and then in terms of feasibility, politics, and ethics.
Looking ahead to the following two weeks, we will next be turning to outcome measurement strategies and then experimental design strategies in the two weeks after this Friday.