“We can only see a short distance ahead, but we can see plenty there that needs to be done.”
Alan Turing, British mathematician
In this report from Intensive Care Medicine published this month, van de Sande and colleagues present results of a systematic review on the utilization and maturity of AI in the ICU setting.
While few or none of the findings are entirely surprising, it is nevertheless a useful review and a sober reminder of just how much more work we have to do to have AI make meaningful impact in this clinical domain. This study is particularly timely given the focus on the ICU sector during the COVID-19 pandemic.
The authors preformed a systematic search with a myriad of databases to identify the eligible studies that used AI to analyze ICU data. As usual, one would need to consider publication bias. This review focused on study design, study aim, dataset size, level of validation, level of readiness, and the outcome of clinical trials as well as the risk of bias evaluated by the prediction model risk of bias assessment tool (PROBAST).
A total of 494 studies were included with 6,455 studies initially identified so AI perhaps is not ready for this specific task of selecting the appropriate studies. Almost all studies were retrospective with very few being prospective observational or clinical trials. Interestingly, the majority of the retrospective studies were thought to have a high risk of bias.
No study reported on the outcome evaluation of an AI model that was integrated into ICU clinical practice. Figure 3 in the manuscript is very enlightening: the ideal goal of clinical outcome evaluation and workflow integration with AI is not the current trend, which remains model prototyping and development with ICU data.
The authors concluded that the vast majority of ICU-AI models remain in the testing and prototyping milieu and are not yet at the bedside. The barriers for progress of AI in the ICU setting include: high bias, small datasets, unclear reporting standards, and patient safety (I would add probably insufficient AI education for clinicians, young and senior). It should be noted here that no pediatric or neonatal ICU studies were included in this review.
Alan Turing was perhaps prescient is forecasting how much work needs to be done (with machine intelligence), and his terse aphorism could not be more true in the ICU setting. It is, of course, not only quantity of work, but the quality of the projects with real-time models that can generalize and yield clinical relevance as well as impact.
The full report can be read here