The role of clinical decision support systems (CDSSs) powered by artificial intelligence in clinical practice is rapidly expanding, but the downstream cognitive effects of CDSSs on clinicians remain poorly understood. Because clinicians often process insights generated by CDSSs in the setting of complex clinical environments and existing cognitive heuristics, CDSSs may have unanticipated effects on decision making and patient outcomes. For example, a CDSS alert may result in a laboratory test that otherwise would not have been ordered, which then prompts downstream clinical decisions that are outside the intended scope of the CDSS. Potentially negative unintended consequences of CDSS could also be mediated by clinician cognitive errors such as automation bias and complacency. Studies evaluating CDSSs, however, have mostly been limited to identifying anticipated effects of CDSSs on decision making processes because the process metrics studied typically are selected based on their perceived relevance to the CDSS in question. As a result, unanticipated effects of CDSSs are often missed or identified sporadically via anecdotes or incident reports and poorly incorporated into CDSS design and implementation strategy.

We propose to develop a process to systematically identify clinical decision making patterns that emerge after implementation of CDSSs by mining organic time series data generated from orders clinicians place in the electronic health record. We hypothesize that distinct sequences of orders representing both anticipated and unanticipated clinical decision making patterns associated with CDSS implementation can be identified using an unsupervised learning sequential pattern mining (SPM) approach. To test this method, we plan to perform a post hoc analysis of a randomized control trial of an early sepsis detection CDSS that was performed on all patients admitted to the inpatient medicine service at Stanford Hospital over eight months. For patients randomized to the CDSS group, alerts for severe sepsis were automatically delivered to physicians via paging and triggered a standardized triage process that involved bedside evaluation and additional laboratory testing. In the control group, silent alerts were generated but not delivered. Using the alert time as time zero, we plan to use SPM to generate ranked lists of high frequency order sequences that occur after time zero in the CDSS and control groups. Sequences with strong statistical associations with the early sepsis detection CDSS will be reviewed by domain experts and used to construct Markov chains that represent distinct decision making patterns. Subgroup analyses of different patient populations will also be performed.

Understanding the unanticipated cognitive effects of CDSSs is increasingly important as clinicians are becoming more reliant on CDSSs. A systematic method for identifying these unanticipated effects may enhance our ability to design and implement CDSSs that safely and effectively deliver patient care.



Author: Ron Li

Coauthor(s): Jonathan H. Chen, MD PhD

Status: Work In Progress

Funding Acknowledgment: JHC is supported in part by NIH Big Data 2 Knowledge Award Number K01ES026837 through the National Institute of Environmental Health Sciences.