Introduction

Welcome to Fall 2022 Field Methods. This class will teach you about designing social science RCTs. This involves the following steps:

Posing a normatively-defined problem out in the real world. That is, tell us about “something that is wrong in the world” that deserves attention and resources to try to rectify.
Using evidence and theory to identify a mechanism that is an important driver of this problem.
Designing an intervention that can disrupt the mechanism.
Designing an experimental test of the intervention.

The class will be hands-on and driven by research interests of people in the class.

Logistics

We will meet in person in Room 217 Tuesdays 2-4pm.
We will not meet as a class on the following dates:
- Oct 3
- Oct 10
- Oct 24
- Nov 21
If you need to miss a class, please be sure to give me advance notice so that we an adjust the course schedule as needed.
I will add instructions, including any recommended readings, in the weekly plan entries below.
If you want to meet to discuss your class work or to discuss accommodations, please sign up for office hours using Calendly: https://calendly.com/cdsamii
Pre-requisites are Quant I and II or equivalent. Please discuss if you are unsure.

Weekly Plan

Sep 5: Class overview and introduction

Readings:

Samii, Cyrus. 2023. “Methodologies for ‘Political Science as Problem Solving.'” Forthcoming in the Oxford Handbook of Methodological Pluralism, edited by Janet Box- Steffensmeier, Valeria Sinclair-Chapman, and Dino Christenson, Oxford University Press. [link]
Healy, Kieran. 2018. “Making Slides.” Blogpost: https://kieranhealy.org/blog/archives/2018/03/24/making-slides/

Additional background:

Riecken, Henry W. 1968. “Social Sciences and Social Problems.” Social Science Information. 8(1):101-129. [link]
Davis, Justine, and Kristin Michelitch. 2022. “Introduction to Field Experiments: Thinking Through Identity and Positionality.” PS: Political Science and Politics. 55(4):735-740. [link] [link to entire issue]
Blumer, Herbert. 1971. “Social Problems as Collective Behavior.” Social Problems. 18(3):298-306. [link]

Sep 12: Problem definition 1

One on one meetings.

Please prepare bullet points in which you describe your “societal problem” and then some potential data sources that you could consult to demonstrate the real-world extent of the problem. Send them to me next Monday so that we can hit the ground running when we meet to discuss on Tuesday.

We will meet in my office (Office 424). Here is the meeting schedule:

2:00-2:20pm Matias
2:20-2:40pm Kyle
2:40-3:00pm Yinxuan
3:00-3:20pm Munroe
3:20-3:40pm Meghna
3:40-4:00pm Cristina

Sep 19: Problem definition 2

Presentations and discussion of problem definitions.

Presentation schedule:

2:00-2:15pm Matias
2:15-2:30pm Kyle
2:30-2:45pm Yinxuan
2:45-3:00pm Munroe
3:00-3:15pm Meghna
3:15-3:30pm Cristina
3:30-3:45pm Ivan

Prepare 4-5 slides so that you can speak for about 5-10 minutes. The slides should cover:

Problem definition
Description of context in which you are measuring the extent of the problem
Measurement approach, explaining how you operationalize the outcome variable that characterizes the problem and how you account for any confounding factors in your descriptive analysis.
Results of your descriptive data analysis.

Sep 26: Characterizing mechanisms 1

One on one meetings.

2:00-2:15pm Matias
2:15-2:30pm Kyle
2:30-2:45pm Yinxuan
2:45-3:00pm Munroe
3:00-3:15pm Meghna
3:15-3:30pm Cristina
3:30-3:45pm Ivan

Readings:

Humphreys, M, and Jacobs, A. 2017. “Qualitative Inference from Causal Models.” Working paper. [link]
Humphreys, M, and Jacobs, A. 2023. Integrated Inferences, Ch. 6. “Theories as Causal Models.” Book manuscript. [link]
Rodrigues, Daniela, et al. 2022. “Reflection on modern methods: constructing directed acyclic graphs (DAGs) with domain experts for health services research.” International Journal of Epidemiology. 51(4):1339–1348. [link]

Oct 3: Characterizing mechanisms 2

(No class meeting.) Circulate memos of DAGs and empirical strategies, and provide comments on others’ memos.

Feedback groups:

Ivan, Kyle, Matias, and Yinxuan
Cristina, Meghna, Munroe

Dropbox with memos: [link]

Within each group, exchange memos and write a 1 page “reviewer’s reports” on the others’ memos, with the following elements:

Provide your one-sentence summary interpretation of what the analysis is trying to accomplish. What are the key concepts and theoretical mechanisms.
Assess whether the proposed mechanisms well-founded on the basis of existing theory. Are there psychological, behavioral, or strategic considerations that are missing?
Assess whether the proposed empirical analyses in terms of causal identification and measurement. Do the proposed analyses allow for clear inferences about the mechanisms? Do the proposed outcome measures (if they are defined) offer clear interpretation with respect to the proposed mechanisms?

Please have your response memos completed by Wednesday Oct 11 and then email them to your entire group as well as me.

Oct 10: Characterizing mechanisms 3

(No class meeting.) Continue with implementation of empirical analysis of mechanisms.

Oct 17: Characterizing mechanisms 4

Presentations and discussion of results of empirical analysis of mechanisms.

2:00-2:15pm Matias
2:15-2:30pm Kyle
2:30-2:45pm Yinxuan
2:45-3:00pm Munroe
3:00-3:15pm Meghna
3:15-3:30pm Cristina
3:30-3:45pm Ivan

Oct 24: Intervention concept 1

(No class meeting.) Circulate memo on intervention concepts.

Your memo should address the following components:

Problem statement: Restate the problem motivating the intervention.
Population of interest: Based on the descriptive work that you have done, for whom is the problem that you are particularly addressing relevant, and for whom do you want to know the effects an the intervention?
Mechanism: Based on existing theoretical, observational-causal work, or even experimental work that either you have done or that you have consulted, what mechanism do you find to be (1) relevant to sustaining your problem and (2) amenable to intervention?
Intervention concept: A general explanation for what kind of intervention could be introduced to bring about desired change via the mechanism. Is the intervention just information, or are there material aspects? Is the intervention at the level of individuals or groups of people?
Intervention operationalization: Proposals for how the intervention concept could be organized. Who could be engaged to administer the intervention? What is the timeframe?

Please prepare a memo and then submit it via email to me and your peer feedback group by Friday at noon. We will maintain the same feedback groups:

Ivan, Kyle, Matias, and Yinxuan
Cristina, Meghna, Munroe

Please transmit your peer feedback by noon on the following Tuesday. We will be meeting for one on one meetings on Oct 31.

Oct 31: Intervention concept 2

One on one meetings.

We will reverse the order for the next two weeks:

2:00-2:15pm Ivan
2:15-2:30pm Cristina
2:30-2:45pm Meghna
2:45-3:00pm Munroe
3:00-3:15pm Yinxuan
3:15-3:30pm Kyle
3:30-3:45pm Matias

Nov 7: Intervention concept 3

Presentations and discussion of intervention concepts.

2:00-2:15pm Ivan
2:15-2:30pm Cristina
2:30-2:45pm Meghna
2:45-3:00pm Munroe
3:00-3:15pm Yinxuan
3:15-3:30pm Kyle
3:30-3:45pm Matias

Nov 14: Defining effects and outcome measurement

One on one meetings.

2:00-2:15pm Ivan
2:15-2:30pm Cristina
2:30-2:45pm Meghna
2:45-3:00pm Munroe
3:00-3:15pm Yinxuan
3:15-3:30pm Kyle
3:30-3:45pm Matias

Nov 21

(No class meeting.) Continue working on defining effects and outcome measurement.

Nov 28: Defining effects and outcome measurement

Presentations and discussion of target effects and outcome measurement plan.

For the presentations, here are points that you should be sure that you cover:

The problem you have been studying all semester has been defined in terms of outcomes (for example, political participation, policy effort, etc.). For your study, you should define 1 or 2 “primary outcomes” of interest.
Note that RCTs usually have a relatively short timeline — often over the course of just one year, and possibly over the course of 2-3, but typically not longer than that. So, the primary outcomes in your RCT may have to be shorter term than the ultimate outcomes in society that you are trying to affect.
Given these parameters, what will be your primary outcomes? How will you measure them? If you can use standardized measurement protocols that have been used by other scholars, that is always nice. Or, you could define a measure based on administrative records. Or, if need be, you could come up with your own measure that you obtain by surveying people in a study sample.
Define a benchmark for “success.” Your intervention will not entirely “solve” the problem that motivated the intervention. That being the case, what do you think is a reasonable standard for success? What effects are you trying to obtain and what would be a minimum effect size that you think would be meaningful? This will be important when we turn to power calculations next. Perhaps it could be improvement in terms your primary outcome measure of 5 percentage points, or 10 percentage points, etc. Explain what you think would be a reasonable standard for success.
Insofar as your RCT will try to speak to theories about your problem of interest, what moderator or mediator effects would you want to estimate?
Finally, are there any outcomes or effects that you want to study to assess secondary issues that might affect ones interpretation of the results — for example, spillover effects, or effects on outcomes that track whether there are any unintended perverse outcomes?

You could organize your presentation to address each of these questions.

For some guidance, you can have a look at outcomes are specified on some RCT registries:

We will use a new ordering for the presentation times, as follows:

2:00-2:20pm Munroe
2:20-2:40pm Kyle
2:40-3:00pm Yinxuan
3:00-3:20pm Ivan
3:20-3:40pm Meghna
3:40-4:00pm Matias

(NB: Cristina indicated that she will have to miss tomorrow.)

Dec 5: Randomization, analysis plan and power analysis

One on one meetings:

2:00-2:20pm Cristina
2:20-2:40pm Kyle
2:40-3:00pm Yinxuan
3:00-3:20pm Ivan
3:20-3:40pm Meghna
3:40-4:00pm Munroe

Exercise

Design exercise: [link]
You can modify this code to do your own power analysis, or you can use the Declare Design package (see below).

Resources

Duflo, Esther, Rachel Glennerster, and Michael Kremer. “Using randomization in development economics research: A toolkit.” Handbook of development economics 4 (2007): 3895-3962. [link]
Quant 2 lectures on experimental design and power: [lecture I] [lecture II]
Declare Design toolkit for R: [link]
- See especially this section of the Declare Design book: [link]
On three-arm trials over full factorial designs: Muralidharan et al. (2023)

Dec 12: Randomization, analysis plan and power analysis

Presentations and discussion of randomization, analysis plan and power analysis

Here are the elements to include in your presentation:

The primary effects that you are targeting, and then if you had time to work on these also, any secondary outcome or moderator/mediator effects that you are targeting in your analysis.
The randomization plan, in which you explain any clustering, stratification, or other restrictions on the randomization.
The regression specification(s) that you are using to estimate your effects of interest. Indicate which coefficients or combinations of coefficients capture your effects of interest. Explain what hypothesis tests you are using for them. Include details on covariate adjustment, adjustment for any block fixed effects, weighting to account for non-uniform assignment probabilities, and whether and how you are clustering your standard errors.
Power analysis that shows either the sample size necessary to achieve the “minimally meaningful effect” that you are targeting, or the minimum detectable effects sizes (which can be expressed in terms of control group standard deviations) that you are able to recover for different potential sample sizes. For any key assumptions that you are making (e.g., the explanatory power of covariates or intra-cluster correlations), examine sensitivity of your results to variations in those assumptions.