The Replication Crisis

Since the 2010s the open question has been how many of psychology’s established findings survive repetition. In 2015 the Open Science Collaboration published an attempt to replicate 100 studies from three leading journals; in around a third of cases the originally significant result could be shown again, and the effect sizes came out considerably smaller on average.

For an archive of psychology this is not a side issue but an organising principle: every file in this collection carries a replication status. The three social-psychology cases show the full range — from a robustly repeated paradigm through an arrangement only partly secured to a study whose foundation the archival work has taken apart.

The crisis is not a blow from outside but the result of a self-examination. It has a clearly nameable trigger: Daryl Bem’s 2011 paper, which concluded in a respected journal and with the usual methods that people can anticipate future events. The methods were standard, the result scientifically highly implausible — which raised the question of what else those methods can produce.

The diagnoses that followed concern incentives rather than the integrity of individual researchers. Underpowered samples, analyses with many open decisions, a publication practice that prints only significant results: each on its own is a small negligence, together they produce a literature in which too many findings look too good.

What has come of it belongs to the crisis as well. Preregistration, shared data, multi-lab studies with thousands of participants: Many Labs 2 tested twenty-eight classic findings across dozens of countries in 2018 and separated cleanly what holds from what does not. Kahneman’s Prospect Theory passed; a chapter of his own popular book did not, which he acknowledged publicly in 2017.

For this archive a rule follows from that, not a mood. A finding that has not been replicated is not thereby false, and one that has been replicated is not thereby true — it is merely better supported. The replication status on every file is a statement about the evidence, not a verdict on the person who produced the finding.

The Clinic and Therapy folder shows the same movement in a field rarely connected with it. Cognitive behavioural therapy is the best-studied talking treatment there is — and for that very reason it shows how strongly publication bias and study quality shift the balance: the effects of the early work were too large, the corrected ones are smaller and hold. Positive psychology, which comes from the same corner, has lost more heavily. And the reversal of learned helplessness by its own authors belongs in this chapter, because it shows what a self-correction looks like that was not forced by a crisis but by better methods.