Dr. Constance Lehman, Director of Breast Imaging at Massachusetts General Hospital (MGH) called to shift artificial intelligence (AI) driven mammography from “simulated research trials to robust clinical evaluations” in a recent JAMA Oncology commentary. She wrote high quality and affordable mammography remains limited because there is not enough radiologists specialized in breast imaging who can take up the responsibilities of interpreting each examination.
It is time to enter the critical phase of rigorous clinical evaluation
Even if advanced screening programs are in place, they are often prone to variations and errors since diagnostic accuracy is generally absent. The Breast Cancer Surveillance Consortium, a US network which rigorously assess and improve the delivery of breast cancer screening, has once studied over 1.6 millions of modern, all-digital screening mammograms and revealed a wide range of interpretive performance.
They also found that up to 40% of certified, specialized radiologists were not able to meet recommended recall rates (i.e., sensitivity towards abnormalities). Although there has been an increased effort to rely on AI, particularly deep learning models, to assist human in mammographic interpretations, these algorithms tended to be trained using a wide variation of data and with different methods. Coupled with insufficient validation of new discoveries, AI’s real clinical impact remains questionable.
As such, Dr. Lehman feels “in the continued evolution of AI applied to improving human health, it is time to move beyond simulation and reader studies and enter the critical phase of rigorous, prospective clinical evaluation”. She believes “work in both development and validation is needed in larger data bases, including tomosynthesis examinations and diverse (commercial AI) vendors and patient populations. But most importantly, rigorous studies to assess whether results from simulations studies will translate to success in routine clinical practice are now essential”.
Dr. Lehman also credited a study which involved the evaluations and performance comparisons of three commercially available algorithms on a large, all-digital screening mammography database. She wrote the methodology used by the researchers uncovered one of the three AI models achieved a 81.9% sensitivity and 96.6% specificity, as compared to the benchmarks of 86.9% sensitivity and 88.9% specificity set by the Breast Cancer Surveillance Consortium.
A call larger and better AI studies
In essence, the study provided “insights that challenge existing assumptions in the field… the volume of cases may be more important than the diversity of vendors or patient populations in the database used to develop the algorithm”. Dr. Lehman called for larger studies, pointing out some of the highest performance algorithms were developed using more than 70,000 cancer images and 680,000 non-cancer images, in contrast to the lowest performing models which were trained based on 6000 cancer images and 106,000 non-cancer images.
On top of which, Dr. Lehman also noted clinicians should learnt from previous failures with computer-aided detection (CAD) programs. “Although early reader and simulation studies of traditional CD were encouraging, in the end, improved outcomes for patients receiving mammogram interpretations supported by CAD were not shown. Many studies have confirmed that humans respond differently to CAD assistance, and the same may be true for AI-assisted readings”.
In conclusion, Dr. Lehman wrote, “If AI models can be developed that cane reliably detect on mammograms women with cancer from those without it, then quality, affordable screening mammography may finally become available to a large population of women globally who currently have no access to its life-saving potential”.
If you are interested in how technology, particularly, AI, robotics, virtual/augmented/mixed realities, and many others that are influencing Radiology and strategies to deploy them in the clinical setting, do not miss the upcoming AIMed Radiology virtual event, organized in association with the American College of Radiology, taking place on 5 November. Register your interest or get a copy of the agenda here today!