Research Revolution 1.0 – Let’s Move On

The tumbrels have rolled, the guillotines have dropped, and I’m hoping that the publication of the Stapel report represents the end of Revolution 1.0.  I also hope that all the heads that have fallen have been guilty of the accusations against them.  And now I hope that we can move on to the next phase of the revolution.

Why do I call what’s happening in psychological (and other) sciences now a “revolution”?  Indeed, there are many similarities between our situation and various historical political revolutions.  For one thing, most of the cries for reform are not new.  We have been hearing about problems of replication and analysis for a long time with calls for various types of changes.  For another, the current situation is not due simply to one factor; rather, currently there is a confluence of events that have lead to the louder cries for reform – issues of fraud, replication, researcher degrees of freedom, post hoc analyses and theorizing, etc.

And, I think, that like early in many revolutions, the calls to revolution, are overly extreme: calls to have people post all methods and data online, register all hypotheses and experiments, provide an analysis plan, do direct replications of all studies.  These ideas all have some merits and some demerits.  But we shouldn’t use them in the service of chopping off any more heads.  As much as anyone, I believe in the value of science and the goodwill of scientists.  So, let’s stop the witch-hunt and move on.

Three New R’s of the Research Revolution

There are three new R’s I think we need to keep in mind as we move toward a more balanced Revolution 2.0.

One is R-E-S-P-E-C-T.  None of us thought, as we became psychological scientists, “Hey, I’m going into to make the big bucks.”  And it was not for the nice offices or the adoring fans.  We do it because we find it interesting, we want to uncover truth, or we believe it’s the only thing we are competent to do.   Therefore, I think it’s time to pull back and stop being accusatory.  And there should be no ex post facto laws.  (See Article I of the United States Constitution.)  That means that if someone has been adhering to past standards for running, analyzing, and reporting research, it is not fair to critique them for it now.  We can create new norms for the future.  It is time for us all to move on.

Another R is refutation.  Science proceeds not only by coming up with new theories, finding support for them, and connecting them to other theories, but also by discarding old theories that have lost support.  A longer analysis of this topic is for a later time, but it is very related to the third R.

The R that many people are focusing on now is Replication. “Let’s call for replications,” “let’s publish replications and failures to replicate”, etc.  Whether and how this should be done has become a divisive issue in the field of psychological science.

This piece in The Guardian notes that editors of the journal Psychological Science thought about, and then rejected, the idea of publishing replication attempts.  The idea is not gone; rather, it has moved from Psychological Science to its more flexible sister journal Perspectives on Psychological Science.  The goal is to combine much information from many researchers as we do in meta-analyses.  We are not witch-hunting for fraud or for bad research practices; instead we are looking for the generalizability and limitations of the research.  Many replications (and failures to replicate) tell us something important.  In fact, I believe that we need to listen to all of our data including pilot studies and replication successes and failures. Your data might not be perfect for your theory but it’s relevant to other theories.  As Nelson Goodman said (more or less):

Every piece of data is consistent with an infinite number of hypotheses and inconsistent with an infinite number of hypotheses.

Or as I put:  One scientist’s p < .25 is another scientist’s p < .05.

As psychological scientists we are uniquely situated to not only study psychology content, but also to study the reasoning processes of scientists themselves.  Thus, what we find should be useful both for our science and for all people involved in the process of science.  So, yes, there should be replication attempts and there should be “science as usual”.  And if someone wants to replicate your findings you should be flattered that they (and others) are interested and motivated enough to do so.  But there should be no more witch-hunts or fears of witch-hunts.  Let’s move on.

(N.B. Many of these issues are addressed in the various articles in the Perspectives Special Issue on Replicability and Research Practices.)

About these ads
This entry was posted in Research Revolution. Bookmark the permalink.

4 Responses to Research Revolution 1.0 – Let’s Move On

  1. Greg Francis says:

    Regarding the comment, “That means that if someone has been adhering to past standards for running, analyzing, and reporting research, it is not fair to critique them for it now. We can create new norms for the future. It is time for us all to move on.”

    This would be a reasonable attitude if scientific discussions were about the characteristics of the researchers, but science is really about the phenomena. If someone has been adhering to past standards, and those past standards were invalid, then we have to discuss whether the reported effects are valid. Without such a discussion we cannot move forward as a field. To just “move on” implies that we do not think the previous decades of research were important enough to deserve careful scrutiny. Respect is always called for, but we also have a responsibility to ask and answer tough questions.

    I believe psychological science investigates important topics. I believe the vast majority of research psychologists do their very best to explore phenomena honestly and as accurately as they can. I simultaneously believe that many research psychologists fundamentally misunderstand how to properly apply scientific practice. In any given situation the impact of these misunderstandings may be enormous or minuscule, but we cannot know unless we scrutinize and double check scientific work.

    I have always liked this quote from Carl Sagan (1997), “At the heart of science is an essential balance between two seemingly contradictory attitudes–an openness to new ideas, no matter how bizarre or counterintuitive they may be, and the most ruthless skeptical scrutiny of all ideas, old and new. This is how deep truths are winnowed from deep nonsense.” As a field, psychology has been pretty good at the first part, but not very active on the second part.

    • bobbiepops says:

      Greg,
      Of course science is about the phenomena not the researchers — or at least it should (mostly) be. I’m just saying — critique the methods, critique the science, but do not critique the scientists. In your comment above, you sometimes seem to explicitly agree with that view and sometimes not. Maybe you thought by “move on” I meant drop the whole thing? No, I just meant stop picking out and blaming individual scientists who were adhering to the norm in good faith. Both the “reform movement” in general, and you personally, are making lots of good-faith scientists nervous. That’s not a good omen for making voluntary widespread changes to community practices in the future.
      Bobbie

  2. Fritz Strack says:

    The idea of „replication“ has changed its meaning. From an attempt to deepen one’s understanding of a given set of results along with the conditions under which they do (or do not) uphold, replication has now assumed the role of a purgatory for potential sinners (experimenters in the domains of priming, social priming, social psychology,…..) to be cleansed. A failure of these exercises does not inform about the generalizability and the limitations of the research in question but gives license to suspect fraud, dishonesty or, at least, sloppiness.

  3. bobbiepops says:

    Agreed. We should be finding out about generalization and limiting conditions, not obsessing over whether one attempt at replicating one study (whether success or failure) tells us much at all.
    Great paper making those points coming out in PoPS in May or July.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s