{"id":247,"date":"2010-11-12T07:56:48","date_gmt":"2010-11-12T12:56:48","guid":{"rendered":"https:\/\/cyrussamii.com\/?p=247"},"modified":"2010-11-12T07:58:35","modified_gmt":"2010-11-12T12:58:35","slug":"risks-incentives-and-shady-research","status":"publish","type":"post","link":"https:\/\/cyrussamii.com\/?p=247","title":{"rendered":"Risks, incentives, and shady research"},"content":{"rendered":"

Over at the Social Science Statistics blog, Richard Nielsen riffs (link<\/a>) on what is probably the biggest threat to valid inference in political science (I can’t speak for other social sciences, but wouldn’t be surprised if it were similar): the need to demonstrate that something, anything<\/em> in your empirical analysis “significantly” departs from some null hypothesis. \u00a0 The recent article by Gerber et al (2010; linked on Richard’s post) is remarkable in revealing how this insidious norm manifests itself in the discipline’s publications, affecting the “most influential and highly cited outlets” the most.<\/p>\n

The fact is, having “stars in your regression table” is still pretty much a sine qua non<\/em> for publication, meaning that coefficients or effect estimates published in political science articles are systematically biased away from the null (an amusing discussion by Gelman and Weakliem on this:\u00a0link<\/a>).<\/p>\n

Of course, there must be some standard by which quantitative papers are judged worthy for publication. \u00a0Right now an almost necessary condition is whether the paper found anything that departed “significantly” from the null. It’s easy to see why this would be used as a criterion: it would seem to ensure that only “remarkable” research is published. \u00a0But this view fails to appreciate the strategic incentives that such a criterion creates. \u00a0It leads people to manipulate their findings or search for said significance in order to get published. In that case, “significance” loses much of its value.<\/p>\n

Is statistical significance the best criterion among feasible alternatives? \u00a0In my view, no. \u00a0Papers should rather be judged on whether the research question is important (though that’s a hard one to judge) and well posed (slightly easier), \u00a0and then, whether the design and methods are rigorous (a bit easier, since there are technical criteria). \u00a0The Experiments in Governance and Politics (EGAP) working group is trying to make a push in this direction by providing a CONSORT-style (link<\/a>) registry for new field experiments (link<\/a>).<\/p>\n

So, how about a publication review protocol that had reviewers only view the research question and motivating material and then the design and methods, without providing them access to any of the findings? \u00a0Then, reviewers would decide on publication from this. \u00a0Once this decision was made, the full paper would be submitted and revisions could be called-for. \u00a0I know in the age of working papers on the internet word about findings will often have already percolated, but this would at least establish a norm that such things should be ignored.<\/p>\n","protected":false},"excerpt":{"rendered":"

Over at the Social Science Statistics blog, Richard Nielsen riffs (link) on what is probably the biggest threat to valid inference in political science (I can’t speak for other social sciences, but wouldn’t be surprised if it were similar): the need to demonstrate that something, anything in your empirical analysis “significantly” departs from some null … <\/p>\n

Continue reading “Risks, incentives, and shady research”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-247","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/posts\/247","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=247"}],"version-history":[{"count":10,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/posts\/247\/revisions"}],"predecessor-version":[{"id":257,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=\/wp\/v2\/posts\/247\/revisions\/257"}],"wp:attachment":[{"href":"https:\/\/cyrussamii.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=247"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=247"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cyrussamii.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=247"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}