An experiment in the DARPA-funded SCORE megaproyecto gave the same data to 457 analysts to reanalyze 100 published studies, revealing that only 34% reproduced the original statistical result precisely.
The SCORE megaproyecto aims to assess the credibility of social and behavioral sciences. In one experiment, presented this Wednesday in Nature, identical datasets and research questions were given to independent researchers. The findings underscore “analytical variability,” stemming from methodological choices such as data cleaning or statistical models.
Among the 504 reanalyses, 74% reached the same general conclusion as the originals, 24% were inconclusive, and 2% opposite. “Each analysis involves methodological decisions that can influence results,” said Guiomar Niso from the Centro de Neurociencias Cajal (CNC-CSIC). Lead researcher Balázs Aczél noted that the findings do not undermine prior research but highlight the need to acknowledge empirical uncertainty.
SCORE also examined reproducibility and replicability. Only 24% of articles shared data publicly; when available, 74% reproduced approximately and 54% precisely. In replications with new data, just 49% succeeded, with smaller effect sizes.
“Research is difficult, and the hard work starts after a discovery,” summed up Tim Errington from the Center for Open Science (COS). The authors advocate for greater transparency to bolster trust in science.
