The Rules of Replication
Recently I traveled to Vienna for APS’s International Convention on Psychological Science, where I gave a talk on “The Rules of Replication.” Thanks to the other great talks in the session, it was well attended. But as anyone who goes to academic conferences knows, “well attended” typically means that at best, there may have been a couple hundred people in the room. And it seems like kind of a waste to prepare a talk—one that I will probably only give once—for such a limited audience. So after reading Daniel Lakens’s convincing argument why blogs are often better than scientific articles, along with a suggestion by Yihui Xie1 that if one doesn’t have a blog, one doesn’t exist2, I thought I’d write up my talk in a series of blog posts.
So the gist of my talk was this:
People’s reaction to replication studies can be surprising. It often seems that no matter how careful replicators3 are in the design of their study, or how thoughtful and restrained they are in the interpretation of their results, when these studies are eventually published, replication authors are criticized for violating some set of rules that they never knew existed. In short, there seems to be a unique set of rules–rules that are often made up by people who don’t actually do replication studies–that are meant to apply selectively to replications.
And in case you think that I’m a overly sensitive about this issue and that nobody has ever really suggested such a set of rules, there’s this:
The purpose of this note is to propose rules for the interaction of replicators and authors, which should eventually be enforced by reviewers of proposals and reports of replication research (Kahneman 2014).
But do we really need a separate set of rules governing how replications are conducted? Shouldn’t broader rules that we have about ideal scientific practices be good enough? In these first few posts, I’ll discuss some of the reasons why I think distinct rules for replication studies are problematic, starting with the topic of this post:
Author of many of the packages people use to do reproducible science, including blogdown, which I used to create this blog. Yup, I wrote the post using RMarkdown, Blogdown, Hugo, hosted it on github, and wrote it on my linux computer. And not Ubuntu or anything cheesy like that, but ArchLinux. So, pretty hardcore. I hope this increases my openness credibility enough to make up for those nine years I spent working for Elsevier.↩
Okay, the quote was really about having a website, but in the context of documentation of a package called “blogdown,” I extrapolated.↩
And like most people who conduct replications, I hate this term. But “people who conduct replications” is too unwieldy, so I’ll use it.↩
I don’t know a lot about philosophy of science, and I know that there is lots of disagreement about how science should proceed, but these always sound like the ad hoc modifications that pretty much all philosophers of science agree are problematic if those who propose them are unwilling to follow them up with empirical tests.↩
Wouldn’t it be great if a side effect of preregistration would be to reduce concerns about replication studies? Such a shift would emphasize that studies (and hence, the authors who conduct them) should be evaluated based on their design and methods, not the results that are obtained.↩
Including sample size, preregistration, etc.↩
Kahneman actually brings up this last issue with the following quote about the timing or original and replication results:
The relationship is also radically asymmetric: the replicator is in the offense, the author plays defense. The threat is one-sided because of the strong presumption in scientific discourse that more recent news is more believable.
I will leave it to others to document whether Kahneman’s view about the role of timing is correct. My own impression (and one that I believe is supported by evidence) is that the finding that comes first is weighted more heavily, and that it is extremely difficult to overturn findings in the literature even if the original evidence was extremely weak.↩
Simple, right? Any Registered Replication Report editors want to comment on how easy this is?↩
You see what I did there? Once original authors buy into the idea that preregistration is essential, interactions with replicators become simple. All a replicator needs to do is follow the preregistration plan.↩