“Lying and dishonesty never work- and it is a great human tragedy that people think dishonesty can work “for a good motive.” Ayn Rand, Russian-American writer and philosopher
Sepsis is one of the top causes of mortality in hospitals and often the focus of models in prediction in healthcare. The reasons for lack of adoption of models such as those for sepsis in clinical practice are multifactorial, including a fundamental lack of trust in algorithms that are not transparent and explainable. There is, however, some degree of acceptance of early warning systems in EHR due partly to convenience and availability (and perhaps some worry about not utilizing such available tools).
These authors from University of Michigan medical School’s Department of Learning Health Sciences published this insightful report this past week on a retrospective cohort study on 27,697 patients admitted to their health system (38,455 hospitalizations).
The focus on the study was an external validation on the Epic Sepsis Model (ESM) score (calculated every 15 minutes) and its capability for prediction of sepsis based on a logistic regression model. This model is used widely as an early warning system in hundreds of health systems in the United States with medical records for over half of the US population. The clinical benefit was assessed by evaluating the added benefit of the ESM score compared to contemporary clinical practice based on timely antibiotics administration.
In this study, sepsis occurred in 2,552 (7%) of the patients. The ESM had a hospitalization area under the curve operating characteristic curve of only 0.63 (poor, as compared to an internal documentation with Epic Systems with AUC of an acceptable to good range of 0.73-0.83). The ESM also failed to identify 1,709 (67%) of patients with sepsis despite generating alerts for 6,971 (18%) of all patient admissions.
In short, this external validation cohort study showed that the ESM had a poor discrimination and calibration in the prediction of the onset of sepsis.
One headline from the Epic magazine claimed “Reducing Sepsis Mortality by One-Fifth with Epic” with the sepsis model that is only available to members of the Epic community. Epic also stated in its information sheet that its sepsis warning system can correctly distinguish two patients with or without sepsis at least 76% of the time.
This present JAMA Internal Medicine study is one of the first validation studies of an EHR prediction model performance. Building a model simply based on billing codes cannot be adequately accurate and over reliance on these tools will have adverse consequences. We all need to be very vigilant of these proprietary prediction tools that are being adopted as part of EHR, especially since these models are not transparent; this effort is absolutely part of our fulfilment of the Hippocratic Oath as clinicians and Quadruple Aim as caretakers.
The full study can be read here