Philosophical foundations for design-based inference

“[Here are] two questions about ravens:

• The general raven question: What is the proportion of blackness among ravens?
• The specific raven question: Is it the case that 100 percent of ravens are black?

Consider a particular observation of a white shoe. Does it tell us anything about the raven color? It depends on what procedure the observation was part of. If the white shoe was encountered as part of a random sample of nonblack things, then it is evidence. It is just one data point, but it is a nonblack thing that turned out not to be a raven. It is part of a sample that we can use to answer the specific question (though not the general question), and work out whether there are nonblack ravens. But if the very same white shoe is encountered in a sample of nonravens, it tells us nothing. The observation is now part of a procedure that cannot answer either question.

The same is true with observations of black ravens. If we see a black raven in a random sample of ravens, it is informative. It is just one data point, but it is part of a sample that can answer our questions. But the same black raven tells us nothing about our two raven questions if it is encountered in a sample of black things; there is no way to use such a sample to answer either question. The role of procedures is fundamental; an observation is only evidence if it is embedded in the right kind of procedure.”
(Godfrey-Smith 2003, pp. 215-216).

This passage on the “procedural naturalism” view of science is from Peter Godfrey-Smith’s book-length survey of current debates in the philosophy of science: amazon.

When you write a pre-analysis plan, this is how you should be thinking. And you should be relating it to theoretical propositions (what the two questions about ravens are standing in for).