“Measure what you value instead of valuing only what you can measure.”

Andy Hargreaves, British educator

This is a very timely manuscript from BMJ Health and Care Informatics on the schism between AI projects and systems and actual adoption in health systems. The authors, including our colleague and friend Enrico Coeira from Sydney, propose a need for a practical and comprehensive instrument for assessing these instruments for their translational aspects.

This group developed an evaluation framework called Translational Evaluation of Healthcare AI (TEHAI) to address three main components: capability, utility, and adoption, with a focus on translational and ethical features and therefore real world generalizability. While the authors are correct in that some of the other published tools are too narrowly focused, this tool is very comprehensive and perhaps would be relatively tedious to implement.

This effort is worthy of praise and support. The main issue with this well-intentioned proposal, as well as other such tools, is that, even with excellent assessment tools, clinicians will need to 1) want to use these tools (what is the incentive for this burden?), and also 2) want to adopt these AI systems with a positive assessment. In other words, simply assessing the AI system does not guarantee its adoption. For good AI assessment tools such as this to truly be impactful, we also need a large cohort of clinician champions to serve as intermediaries for the AI and clinical worlds. This will then lead to adoption of these AI systems and prove their value proposition to improve patient outcomes and/or alleviate clinician burden.

Read the full paper here.