In the current PS, Luke Keele (link) makes a great point about the need for journals to let researchers take existing theories and subject them to multiple rounds of empirical scrutiny:
Of course, there is nothing wrong with new theories; however, I believe that an overemphasis on theory can impede the ability of the discipline to establish causal relationships. One point of emphasis in the identification revolution is that causal inference is difficult. A series of regression models is hardly the last word in whether a causal theory holds. Within causal inference, it is generally understood that only a series of different studies using disparate research designs can provide solid evidence for a causal relationship. This understanding implies that, as opposed to a single test, a theory requires a number of tests from different research designs. We might assert that whereas theory is critically necessary, too many theories might be harmful. It is probably the case that the gain from an incomplete test of a new theory is less than the gain from a new test of an existing theory.
Economics provides a useful example. The question of whether attending college increases income is one of long-standing interest to economists. The theory behind the question is relatively simple. Although I hesitate to state that the causal hypothesis that college leads to higher incomes is definitively settled, a review of the many different research designs used makes a convincing case for a causal effect. If all of the researchers conducting these studies had been told by reviewers that new theory was needed, little progress would have been made.
When we overvalue novel theories, we tend to dismiss attempts to answer old questions with new research designs. We can value papers with new theories and little or no empirics. We also can value papers with little in the way of new theory but that present novel research designs that provide new empirical evidence about causal relationships. We could argue convincingly that both topics require such attention that it may be difficult to present both novel theory and empirics in the same paper. Currently, I would state that, in general, a paper that does not engage in new theory development will have a difficult time being published in a top journal. I don’t think that is healthy. If we really want researchers to clearly establish causal relationships that is often worth doing alone in a single paper.
In my (and Luke’s) home discipline of political science, I have frequently seen reviewers at top journals suggesting rejection for papers because they don’t think the theoretical contribution is novel enough, even if the research design is compelling. (I am not talking about reviews for my own work either, but that of my much more capable colleagues who have discussed their reviews with me.) But journals seem to be okay with papers that propose new theories with no empirical tests whatsoever. Weird, isn’t it?
I am not saying pure theory papers shouldn’t be published—quite the contrary in fact. What makes sense is some division of labor in the discipline. I recognize the importance of pure theory papers. I also recognize the importance of compelling empirical work that offers a new and credible test of an existing theoretical claim. The usual refrain that I hear when I say this is that “top economics journals don’t seem to have this problem,” and with that I agree. I wonder why there is such a difference.
I agree with pretty much everything else in Luke’s paper too. It is part of a symposium on whether “big data,” “causal inference,” and “formal theory” are conflicting trends, an obviously ridiculous proposition, but one that triggered some nice essays by the contributors.
