Non reproducible results as negative results

The cancer researchers Glenn Begley and Lee Ellis made a rather remarkable claim last year. In acommentary that analyzed the dearth of efficacious novel cancer therapies, they revealed that scientists at the biotechnology company Amgen were able to replicate a dismal success rate of 11% of published pre-clinical research studies. They had deliberately chosen highly innovative cancer research papers, hoping that these would form the scientific basis for future cancer therapies that they could develop. However, if the pre-clinical scientific experiments cannot be replicated, it would be folly to expect that clinical treatments based on these questionable scientific concepts would succeed.

If so much published cancer research cannot be replicated, how can the field progress? Even worse, if only 11% of published landmark papers in cancer research are reproducible, it raises questions about how published papers in other areas of biological research fare.

Lee Ellis has now co-authored another paper to delve further into the question. In the study Ellis teamed up with colleagues at the renowned University of Texas MD Anderson Cancer Center to survey faculty members and trainees (PhD students and postdoctoral fellows). Only 15-17% of their colleagues responded to the anonymous survey, but the responses confirmed that reproducibility of papers in peer-reviewed scientific journals is a major problem.

The list of scientific journals in which some of the irreproducible papers were published includes the “elite” of scientific publications: The prestigious Nature tops the list with ten mentions, but one can also find Cancer Research (nine mentions), Cell (six mentions), PNAS (six mentions) and Science (three mentions). However it doesn’t necessarily mean that these high-profile journals are the ones most likely to publish irreproducible results.

The lack of data on successful replications is a major limitation (among others) of this survey so cannot provide definitive answers about the magnitude of the reproducibility problem. It only confirms that lack of reproducibility is a potentially important problem in pre-clinical cancer research, and that high-impact peer-reviewed journals are not immune.

While Begley and Ellis have focused on questioning the reproducibility of cancer research, it is likely that other areas of biological and medical research are also struggling with the problem of reproducibility.

Scientists involved in biological and medical research need to foster a culture that encourages the evaluation of reproducibility and develops the necessary infrastructure. When scientists are unable to replicate results of published papers and contact the authors, the latter need to treat their colleagues with respect and work together to resolve the issue.

New work can only succeed if it is built on solid, reproducible scientific data.

Reference
Original source

Written by Dr. Belén Suárez for The All Results Journals.