I’ve had a few discussions recently about how to think about substantive theory. What should we be looking for?
A proposal I like comes from a passing remark by Dixit in Lawlessness and Economics (2004, p. 22; link):
The aim of theory should be to construct a collection of models that is sufficiently small to be remembered and used, and covers a sufficiently large portion of the spectrum of facts.
This is not so different than Clark and Primo’s proposal of theory as map-like working approximations that we use for guidance in addressing particular problems (link). I like their view and it’s one that I endorse when discussing how theory and empirics interact in the recent JOP piece (link; ungated).
Personally, I don’t use the word “model” lightly, and I suspect that Dixit doesn’t either. When I use it I do in fact mean a formal model. An important benefit of a formal model, to me, is its low semantic ambiguity, at least when compared to verbally stated theories. There is nothing more frustrating than debating the internal consistency of a theory when everyone has a different interpretation of the terms. Of course, formalization does not solve the problem of relating the theory back to reality, but then this issue of operationalization is separate.
EGAP has just announced a new round of funding for research on taxation and publicly financed goods. You can view the call for expressions of interest here: link. Expressions of interest are due by September 15 (!).
The funding round will be an EGAP “metaketa.” This means that the projects will be aligned in terms of the interventions and outcomes that they study so as to allow for meta-analysis. A recent issue of the American Economic Journal:Applied featured studies from a similar initiative on microcredit: link. Here is a link to EGAP’s explanation of the metaketa approach: link.
Having been involved in the drafting of the request for proposals (RFP), I want to emphasize a few points. The “Focus” section of the RFP indicates,
We aim to fund research on strategies to move citizen-government relations toward responsiveness on the part of government and corresponding tax compliance on the part of citizens. Interventions of particular interest are: the provision of government-funded public goods; the empowerment of citizens vis a vis predatory tax collectors; and/or the strengthening of civil society initiatives that help citizens to comply with tax regulations, while demanding effective and responsive public action. Projects implemented in collaboration with governments and/or civil society organizations are strongly encouraged to apply.
In considering whether to apply, it is okay to use a broad definition of “taxation.” That is, it does not necessarily have to be a study about property or income taxes, say. Usage fees for publicly provided services, for example, could fall within the parameters of the RFP, so long as the proposed research looks into the reciprocal exchange between citizens, who have fee obligations, and public agencies, who have service obligations. The primary interest is in strategies to nudge society-state relations in the virtuous direction of reciprocal exchange on the basis of such obligations.
The RFP also emphasizes research in developing countries, meaning essentially countries that are not high-income by World Bank standards, although this is not a formally specified parameter.
The timeline is rather tight on this, so those applying should have a clear idea of exactly which government agencies or civil society organizations they would be able to work.
Vox recently posted an article on “problems facing science” (link). A panel of 270 scientists from across a range of disciplines chimed in. A major theme, and arguably the biggest problem identified after issues related to accessing grants, was that “bad incentives” undermine scientific integrity. Specifically, these bad incentives arise because publication and grant decisions tend overwhelmingly to be based on assessments of whether research results are “exciting.” Vox also reported that the “fix” for this problem, as suggested by many of the panelists, was for editors and reviewers to “put a greater emphasis on rigorous methods and processes rather than splashy results.”
Recently, Comparative Political Studies hosted a special issue dedicated to applying a results-free review process (link). The editors of this special issue concluded that the process promoted attention to “theoretical consistency and substantive importance.” It introduced some complications too, such as questions about how to handle statistically insignificant results and how to accommodate research designs other than experiments or certain types of observational templates. But generally, they concluded that the process “exceeded our expectations.”
These two articles reference other detailed arguments promoting the idea of review based on whether hypotheses are well motivated and methods rigorously applied. I have also elaborated on why I think this kind of “ex ante science” is a good idea (link1 link2). The principles of “ex ante science” are to evaluate the value of applied empirical research contributions on the basis of whether the empirical analyses are well motivated in substantive or theoretical terms, whether the empirical methods are tightly derived from the substantive motivation, and whether the proposed empirical methods are robust. One avoids referencing results in judging the value of the contribution.
Here I want to suggest something that we can start doing immediately to promote this goal: voluntary commitment by journal reviewers to evaluate manuscripts on the basis of principles of ex ante science. Journal editors give reviewers discretion to apply their judgment in evaluating a manuscript. This grants a license to those interested in promoting the principles of ex ante science to do just that.
Here are some operational guidelines. As a reviewer you could begin by masking results prior to starting to read a manuscript. Then, you could structure your review so that it addresses the questions pertaining to the principles stated above.
Let’s take it even further, in the interest of promoting a norm of reviews based on principles of ex ante science: To resolve any ambiguity about one’s commitments to these principles, as a reviewer make it explicit. Reviews could begin with a declaration along the lines of “This review is based on assessments of whether or not the empirical analyses are well motivated and the empirical methods robust. Results were masked in judging the merits of the manuscript.”
Lecture slides [PDF]
– HT example [R]
– Blocks [R]
– Sarndal et al. (1992) [amazon]
– Thompson (1997) [amazon]
– Lin (2013) [pdf]
– Peter’s website: [link]
– Joel’s website: [link]
These are two ways to take a bunch of variables that are supposed to measure common latent factors and reduce them to a single or a few indices. What is the difference? I get the question fairly often, so I thought I’d put this post up.
The two approaches do different things. Inverse covariance weighting applies an assumption that there is one latent trait of interest, and constructs an optimal weighted average on the basis of that assumption. Factor analysis tries to partial out an array of orthogonal latent factors.
An intuitive way to think of it is like this:
Suppose you have data that consists of three variables: College Math Grade, Math GRE, and Verbal GRE. The two math variables will be highly correlated, and the verbal variable will be somewhat correlated with the math scores.
The inverse covariance weighted average of these three variables would result in an index that gives about 25% weight to each math score and then 50% weight to the verbal score. It “rewards” the verbal score for providing new information that the math scores don’t. The resulting index could be interpreted as a “general scholastic aptitude” index.
A factor analysis of these three variables would yield two orthogonal factors, the first factor of which would give almost 50% weight to each math variable and almost zero weight to the verbal variable, and the second would give almost zero weight to each math variable and almost 100% weight to the verbal variable. So you would get a “pure math” factor and a “pure verbal” factor.
Which one is better? It depends on the goals of your analysis.
I discuss this a bit more in my lecture on “measurement” in the quant field methods class (see links at top right). There is some R code there to play around with these concepts too.