The tumbrels have rolled, the guillotines have dropped, and I’m hoping that the publication of the Stapel report represents the end of Revolution 1.0. I also hope that all the heads that have fallen have been guilty of the accusations against them. And now I hope that we can move on to the next phase of the revolution.
Why do I call what’s happening in psychological (and other) sciences now a “revolution”? Indeed, there are many similarities between our situation and various historical political revolutions. For one thing, most of the cries for reform are not new. We have been hearing about problems of replication and analysis for a long time with calls for various types of changes. For another, the current situation is not due simply to one factor; rather, currently there is a confluence of events that have lead to the louder cries for reform – issues of fraud, replication, researcher degrees of freedom, post hoc analyses and theorizing, etc.
And, I think, that like early in many revolutions, the calls to revolution, are overly extreme: calls to have people post all methods and data online, register all hypotheses and experiments, provide an analysis plan, do direct replications of all studies. These ideas all have some merits and some demerits. But we shouldn’t use them in the service of chopping off any more heads. As much as anyone, I believe in the value of science and the goodwill of scientists. So, let’s stop the witch-hunt and move on.
Three New R’s of the Research Revolution
There are three new R’s I think we need to keep in mind as we move toward a more balanced Revolution 2.0.
One is R-E-S-P-E-C-T. None of us thought, as we became psychological scientists, “Hey, I’m going into to make the big bucks.” And it was not for the nice offices or the adoring fans. We do it because we find it interesting, we want to uncover truth, or we believe it’s the only thing we are competent to do. Therefore, I think it’s time to pull back and stop being accusatory. And there should be no ex post facto laws. (See Article I of the United States Constitution.) That means that if someone has been adhering to past standards for running, analyzing, and reporting research, it is not fair to critique them for it now. We can create new norms for the future. It is time for us all to move on.
Another R is refutation. Science proceeds not only by coming up with new theories, finding support for them, and connecting them to other theories, but also by discarding old theories that have lost support. A longer analysis of this topic is for a later time, but it is very related to the third R.
The R that many people are focusing on now is Replication. “Let’s call for replications,” “let’s publish replications and failures to replicate”, etc. Whether and how this should be done has become a divisive issue in the field of psychological science.
This piece in The Guardian notes that editors of the journal Psychological Science thought about, and then rejected, the idea of publishing replication attempts. The idea is not gone; rather, it has moved from Psychological Science to its more flexible sister journal Perspectives on Psychological Science. The goal is to combine much information from many researchers as we do in meta-analyses. We are not witch-hunting for fraud or for bad research practices; instead we are looking for the generalizability and limitations of the research. Many replications (and failures to replicate) tell us something important. In fact, I believe that we need to listen to all of our data including pilot studies and replication successes and failures. Your data might not be perfect for your theory but it’s relevant to other theories. As Nelson Goodman said (more or less):
Every piece of data is consistent with an infinite number of hypotheses and inconsistent with an infinite number of hypotheses.
Or as I put: One scientist’s p < .25 is another scientist’s p < .05.
As psychological scientists we are uniquely situated to not only study psychology content, but also to study the reasoning processes of scientists themselves. Thus, what we find should be useful both for our science and for all people involved in the process of science. So, yes, there should be replication attempts and there should be “science as usual”. And if someone wants to replicate your findings you should be flattered that they (and others) are interested and motivated enough to do so. But there should be no more witch-hunts or fears of witch-hunts. Let’s move on.
(N.B. Many of these issues are addressed in the various articles in the Perspectives Special Issue on Replicability and Research Practices.)