https://www.smh.com.au/national/cold-water-poured-on-scientific-studies-based-on-statistical-cult-20191108-p538t6.html?fbclid=IwAR1nED55BSvdgMUHnYrdPW7P2cwQD5JQer8GtgIC3Qnsdf0oq8yq6hXhjDI

Cold water poured on scientific studies based on ‘statistical cult’

People turn to these treatments because they are supported by a scientific method which, we are led to believe, relies on rigorous statistical analysis of masses of data to prove a point.

But in some branches of science that appears to not have been the case lately.

Some leading statisticians are now asking serious questions about the accuracy of dozens of studies carried out by Australian sports scientists using a controversial method they say is unreliable, deeply flawed and “moves the goalposts”.

Random chance
All experiments on humans involve an element of randomness.

Say you are testing whether a foam roller helps soothe someone’s aching muscles.

At the end of the experiment, five of your subjects say their muscles really do hurt less. This could be because of the foam roller – but it could also be random chance.

To avoid this problem most scientists try to study as many people as possible to work out the variability and average out any effects due to chance.

But sports science tends to study very small groups of people.

That makes it very difficult to tell if the results are real – or just random chance.

To address this, a pair of researchers developed a statistical method called magnitude-based inference in 2006.

In traditional statistics findings are considered to be accurate when there is only a 5 per cent (or less) chance of them being a “false positive”.

The MBI method allows results to have up to a 50 per cent chance of being a false positive and lets scientists make big claims based on studies of just a few people – usually about 10.
—-

The claim that foam rollers could help with sore muscles, made by a team in 2015 that included a Charles Sturt University researcher, was based on a study of just eight people.

The AIS tested curcumin – a component of turmeric – on just 17 men playing social football and basketball but were still able to report in 2015 the first empirical evidence that it too helped with muscle soreness.

—-