(See the bottom for a bit of a plot twist.)
Tale 1
A researcher has some intuitions about how to test a theoretical proposition using what appears to be a nice natural experiment. Before discussing things with other colleagues and without putting much more thought into it, the researcher pulls together some data and does some analyses. The idea is just to see whether there are any interesting patterns to pursue further. The researcher tries out different outcome measures. The researcher also tries out different specifications that come to mind based on deeper, iterative reflection on the problem. All results come back noisy, with no clear patterns—it’s a null result. The researcher declares that the either the proposition or the natural experiment has flaws that were not apparent at the outset.
Tale 2
A researcher has some intuitions about how to test a theoretical proposition using what appears to be a nice natural experiment. The researcher discusses the idea with some colleagues who agree that this has potential as a natural experiment and also addresses a question in a research program that draws a lot of interest. The research thinks harder about the theory and the natural experiment, finding some subtleties that make the analysis of the natural experiment more informative about the theoretical proposition. The researcher writes up the theoretical analysis giving rise to the proposition, the various tests that the proposition implies, and the data analysis plan and presents them for comment at workshops and conferences. The researcher receives suggestions and generally positive feedback about the value of pursuing the project. This is somewhat unusual — in the past, ideas have been shot down either at the more inchoate stage or even at workshops based on pre-analysis plans. So the researcher is excited to carry out the analysis. All results come back noisy, with no clear patterns—it’s a null result. The researcher declares that the either the proposition or the natural experiment has flaws that were not apparent at the outset.
Reflections
How do these tales end?
Probably for tale 1, the researcher just abandons the project. Should there be a record available of this? I suppose, insofar as the absence of a record is “file drawering.” But the project was exploratory and informal, so I do not think it is reasonable that a full-on journal publication would be warranted. Rather, what would be nice would be a place to deposit logs of such exploratory analyses along with an abstract and keywords. Then, others interested in the pursuing similar ideas would have a place to search and learn from others attempts. The researcher could add a line to the CV in a section called “research logs.”
Tale 2 is different. It is the process leading up to the null that makes it different. Given the “peer review” in the pre-analysis stage, the findings should be of interest to that community of researchers. The null finding merits a full-on journal publication.
Plot twist
Nothing about the results being “null” in the above affects the logic of the story. Indeed, I focused only on null results here because I wanted a story that shows how we can feel good about a null result. That’s the second tale. However, if we made the results “significant” instead, it does not follow that one should, as a result of that change, now “feel good” in the first tale.
