http://jamanetwork.com/journals/jama/fullarticle/2603417

Acknowledging and Overcoming Nonreproducibility in Basic and Preclinical Research
John P. A. Ioannidis, MD, DSc
JAMA. Published online February 13, 2017.

“Empirical efforts of reproducibility checks performed by industry investigators on a number of top-cited publications from leading academic institutions have shown reproducibility rates of 11% to 25%….

“When results disagree, it is impossible to be 100% certain whether the original experiments, the subsequent experiment, both, or none are correct or wrong. However, the recurrent nonreproducibility and the large diversity in results are concerning. The reproducibility efforts have generally followed high standards, with full transparency and meticulous attention to detail. If those efforts could not reproduce the original findings, it is unlikely that the average laboratory investigator (who probably spends less effort to so meticulously repeat experiments by other scientists) will be able to do this. Furthermore, the reproducibility effort demonstrated that unanticipated outcomes (eg, unforeseen spontaneous regression of tumors) further complicate experiments. Outcomes diverge even with minor modifications in the experimental conditions.”

There are significant issues about reproducibility and irreproducibility in science, particularly in medical fields.

For example, nobody would want to or be allowed to reproduce the famous “tamping pole” accident in which a large fraction of a man’s brain was destroyed with only a limited effect on functionality.

For moral reasons, you can’t reproduce illnesses or diseases in humans.

For a different reason, irreproducibility also occurs in astronomy. You can’t reproduce a star in a laboratory, there are problems of scale, temperature, direction of gravity etc.

As a result, science has largely split into laboratory branches where reproducibility can be achieved, and observational branches where all you can really do is observe what nature throws at you.

But the lack of reproducibility can lead to a social culture in which a single unconfirmed instance leads to not just massive press coverage but a huge diversion of funds into what was probably an error of observation. I can think offhand of at least three instances where this has happened in a claim of a cure for cancer. A person under treatment recovers from a supposedly incurable cancer – the cure hits the news media – but in each case they probably would have recovered anyway without the treatment regimen.

So, “Empirical efforts of reproducibility checks performed by industry investigators on a number of top-cited publications from leading academic institutions have shown reproducibility rates of 11% to 25%” doesn’t seem all that unlikely.

However …

… there are an infinite number of different ways to define “reproducibility”. Let me take an example. Suppose in a set of 10 patients in a treatment trial 4 are cured. Then someone else goes back and repeats the experiment. Second time, with a set of 10 patients 3 are cured. It could be claimed that results weren’t reproducible.

Ian said:

>“tamping pole” accident

My brian hurts!!

Phineas P. Gage (1823 – May 21, 1860) was an American railroad construction foreman remembered for his improbable survival of an accident in which a large iron rod was driven completely through his head, destroying much of his brain’s left frontal lobe, and for that injury’s reported effects on his personality and behavior over the remaining twelve years of his life—​​effects sufficiently profound (for a time at least) that friends saw him as “no longer Gage.”

Long known as the “American Crowbar Case“—​​once termed “the case which more than all others is calculated to excite our wonder, impair the value of prognosis, and even to subvert our physiological doctrines” —​​Phineas Gage influenced nineteenth-century discussion about the mind and brain, particularly debate on cerebral localization,​​ and was perhaps the first case to suggest the brain’s role in determining personality, and that damage to specific parts of the brain might induce specific personality changes.

Yes, that one. It’s not the sort of medical experiment that you can reproduce.

mollwollfumble said:

Ian said:

>“tamping pole” accident

My brian hurts!!

Phineas P. Gage (1823 – May 21, 1860) was an American railroad construction foreman remembered for his improbable survival of an accident in which a large iron rod was driven completely through his head, destroying much of his brain’s left frontal lobe, and for that injury’s reported effects on his personality and behavior over the remaining twelve years of his life—​​effects sufficiently profound (for a time at least) that friends saw him as “no longer Gage.”

Long known as the “American Crowbar Case“—​​once termed “the case which more than all others is calculated to excite our wonder, impair the value of prognosis, and even to subvert our physiological doctrines” —​​Phineas Gage influenced nineteenth-century discussion about the mind and brain, particularly debate on cerebral localization,​​ and was perhaps the first case to suggest the brain’s role in determining personality, and that damage to specific parts of the brain might induce specific personality changes.

Yes, that one. It’s not the sort of medical experiment that you can reproduce.

Surely it was an accident, not a medical experiment, although of medical interest.