The results that vanished

In 2011 a respected social psychologist published a paper in one of his field's leading journals claiming to show that people could, in a small but measurable way, sense the future. The experiments followed the standard methods of the discipline to the letter. That, as it turned out, was precisely the problem. If accepted techniques could produce evidence for seeing the future, critics asked, what else might they have been producing all along? The question set off a long period of soul-searching that came to be known as the replication crisis, and it has yet to be fully resolved.

The core of the problem is simple enough to state. A scientific finding is supposed to be reproducible: if another team repeats the experiment carefully, they should get broadly the same result. When a large collaboration attempted to repeat a hundred published psychology studies in 2015, however, fewer than half of the original effects held up. Similar exercises in cancer biology and economics have produced figures that are, depending on how generously one counts, either sobering or alarming. Some celebrated findings, from the claim that standing in a confident 'power pose' changes your hormones to the idea that willpower is a resource that runs out, have been called into question or quietly scaled back.

It is tempting to conclude that scientists have been cooking the books, but outright fraud accounts for only a small fraction of the trouble. The deeper causes are structural. Journals have long preferred striking, positive results to dull or negative ones, and careers depend on publication. A study that fails to reach the magic threshold of statistical significance often never sees the light of day. Researchers sitting on messy data therefore face a quiet temptation to analyse it one more way, to drop an awkward outlier, or to stop collecting once the numbers look promising. None of these choices is obviously dishonest in isolation. Taken together, though, they make it remarkably easy to find a pattern in what is really noise, and the system rewards exactly that.

What is striking is how quickly parts of the scientific community have responded. Many journals now encourage or require researchers to pre-register their studies, stating in advance what they will measure and how, so that the analysis cannot be bent to fit the results after the fact. What is more, data and code are increasingly published alongside papers, allowing others to check the arithmetic for themselves. Large replication projects, once regarded as unglamorous drudgery, have become a respected line of work, and some funders now pay for them directly, something that would have been unthinkable twenty years ago.

Not everyone welcomes these changes. Some researchers complain that a climate of suspicion discourages bold ideas, and that the most interesting science is by its very nature uncertain. They are not entirely wrong. A field in which every finding must be confirmed three times before anyone takes it seriously risks becoming cautious to the point of irrelevance. Then again, a field whose headline results evaporate on contact with a second laboratory is hardly in a strong position to claim relevance either.

Perhaps the most important lesson concerns the rest of us. News reports tend to present a single study as if it settled a question once and for all, when in reality it is one piece of evidence that may or may not survive further scrutiny. Readers who learned to ask whether a result has been repeated would be far harder to mislead. The replication crisis, uncomfortable as it has been, is in a sense a sign that science is doing what it claims to do: correcting itself, slowly and in public. The real danger lies not in discovering that some results were wrong, but in the temptation to stop checking.