What a solid theoretical framework does for you

Paul Krugman had a superb paragraph in a column last week [link], explaining how we can’t get away from theoretical models:

[W]henever somebody claims to have a deeper understanding of economics (or actually anything) that transcends the insights of simple models, my reaction is that this is self-delusion. Any time you make any kind of causal statement about economics, you are at least implicitly using a model of how the economy works. And when you refuse to be explicit about that model, you almost always end up – whether you know it or not – de facto using models that are much more simplistic than the crossing curves or whatever your intellectual opponents are using.

This came to mind today as I was commenting on some student work, and felt the need to explain how important it was to have a strong theoretical foundation even if you are working with a well-identified experiment or natural experiment. Here’s what I wrote (with specific references to the paper removed):

Developing a theoretical framework is important for lots of reasons. First, it provides a basis for both deriving hypotheses coherently and also setting us up to draw out implications from the results of the empirical analysis. Right now the hypotheses sort of come from nowhere, based on some intuitions. But this is inadequate motivation. How would evidence in favor of (or against) these hypotheses affect the implicit or explicit models that we rely on to form expectations about the phenomenon you are studying? Second, it helps people who do not care about the specific application you are studying to take interest nonetheless in your research. Identifying the relevant theoretical framework is a way of addressing the all important question, “what more general thing is this a case of”? (Sorry for the hanging preposition.) We want to reduce the specific, applied problem to something that can be analyzed in a general way such that the results of this particular study have implications for other types of actors in other types of situations.

Well that’s how I see it at least.


“Assumptions are self destructive in their honesty”

Here’s a great quote from Pearl and Bareinboim (2014, p.2) [link] in their analysis of “external validity” and conditions that allow for one to transport the results of a causal analysis from one context to another:

[The literature on external validity] consists primarily of threats, namely, explanations of what may go wrong when we try to transport results from one study to another while ignoring their differences. Rarely do we find an analysis of “licensing assumptions,” namely, formal conditions under which the transport of results across differing environments or populations is licensed from first principles.

The reasons for this asymmetry are several. First, threats are safer to cite than assumptions. He who cites “threats” appears prudent, cautious and thoughtful, whereas he who seeks licensing assumptions risks suspicions of attempting to endorse those assumptions.

Second, assumptions are self destructive in their honesty. The more explicit the assumption, the more criticism it invites, for it tends to trigger a richer space of alternative scenarios in which the assumption may fail. Researchers prefer therefore to declare threats in public and make assumptions in private.

Third, whereas threats can be communicated in plain English, supported by anecdotal pointers to familiar experiences, assumptions require a formal language within which the notion “environment” (or “population”) is given precise characterization, and differences among environments can be encoded and analyzed.

There are so many truths in there that extend beyond research on external validity.