A new reform is sweeping across the statistical World recently. The American Statistical Association (ASA) is planning to release a special issue this July to alert researchers on the risks of excessive dependence on p-value and statistical significance. This is a follow-up on a statement delivered back in 2016.
Primarily, ASA asserted the conclusion of a research study should not stop at p-value larger or smaller than a .05 threshold. When there is no significant difference, it does not prove the null hypothesis and vice versa. The same mentality applies to two studies of similar nature, but one leads to a significant difference while the other does not. This is especially so in the case of the same observed effect, as an article published on Nature noted.
The impact of statistical significance is more prominently felt in areas where manufacturing processes are repeatedly tested to prove that they meet certain quality control standards. The reform does not advocate to put a stop on p-value. It encourages researchers to think beyond it and those that are in the grey areas. It remains unclear this has anything to do with the recent “reproducibility crisis”.
More challenges for artificial intelligence
In January, a new analysis led by health-policy researcher John Ioannidis at Stanford University found no correlation between the value of a startup and its publication record. Of all the 47 companies mentioned in the analysis, less than half had published papers that are being cited over 50 times (a quantifiable marker which Ioannidis and his colleagues had chosen to define “high impact”). Of which, eight companies had never published any peer-reviewed paper.
Ioannidis warned of “stealth research” and overpromise of “technology which disrupts medicine” as in the case of the Theranos scandal. Although only two companies mentioned in the analysis use artificial intelligence (AI), an examination of such nature may reveal a gradual loss of faith in some. In the near future, perhaps AI will have to be scrutinized, by both healthcare professionals and academia.
The fact that researchers and journal editors pay more attention to studies that show significant difference, may have unknowingly created a bias. This indirectly encourages researchers to focus on data or statistical methods that will produce significant differences, while ignoring the rest which also matter. Like the way developers will focus on the effectiveness of a new drug but disregards its side effects.
Assessments and regulations should change together with AI
It is too immature to determine if the statistical reform will work. However, if our adherence towards p-value remains and the need to publish more peer-reviewed papers is also on the way. These may deter companies from venturing into innovations that will truly create an impact on healthcare and medicine. On the other hand, since internal data used for training the AI or machine learning algorithm is no longer enough to test its true capability, some companies may risk to push their product out early, just to accumulate data and making sure the technology works. Rather than spending time on developing something that really works.
US Food and Drug Administration (FDA) is piloting a pre-certification program dedicated for software based medical devices. The program was in place because FDA is aware of how quick and adaptable new technology is. So, they wish to have a new regulation which is equally flexible. Both statistical significance and peer-reviewed are not the ideal methods to access research, let alone creations that some of us have never seen before. We should begin to question their versatilities as a benchmark for 21stcentury science and technology.
The Theranos scandal appears like a mutated Milgram experiment. The moral is we ought to educate dreamers that innovations occur everywhere, they do not only exist in Silicon Valley. Over-idolizing a company, chasing after quick success and copying a successful figure will not lead one to success. All these faults in integrity and responsibility, are what rules and regulations will never pick up.
A science writer with data background and an interest in current affair, culture and arts; a no-med from an (almost) all-med family. Follow on Twitter.