Deep learning (DL) is the top performer in data-intensive tasks [1, 2] and thus has the potential to improve personalized, predictive, precise, and preventive health enabled by multiple state-of-the-art measurement technologies and big data.

For instance, DL has been shown to surpass human performance in pathology [3] and Electrocardiogram (ECG) [4]. However, in contrast to shallow models where only one coefficient is associated with each input feature, DL models map raw data to desired output via complex nonlinear transformations, and it is challenging for the users to validate the underlying patterns [5] based on the domain knowledge.

Such “black box” nature is of major concern for medical doctors, patients, and regulatory bodies in adopting DL in making daily health and clinical decisions. Thus, there has been increasing demand to map the underlying patterns learned by DL models to an interpretable domain through post-hoc interpretation techniques, so the features extracted by DL models are easy to understand.

“Black box” nature is of major concern for medical doctors, patients, and regulatory bodies in adopting DL in making daily health and clinical decisions.

Figure 1 illustrates how explainable DLs can be developed for health informatics applications. For example, to build a clinical decision support system (CDSS) to assist cardiac intensive care unit (CICU) fellows in making decisions based on continuous bedside bedmaster data, a traditional shallow model (logistic regression etc.) does not take advantage of huge volumes of continuous data. Thus, DL models can be used to achieve higher accuracy.

To assist the clinician in trusting the results of “black box” DL models, it is important to build a post-hoc interpretation module in the CDSS. There are two strategies in developing this module: interpretation by feature scoring and by data synthesis.

The first is to generate importance scores for each individual feature generated by DL models to track the feature’s contribution to the final predictions [6-11].

The validity of feature importance can be evaluated by comparing whether destroying input features following the order of importance score will result in a greater decrease in accuracy compared to randomly destroying input features [7].

black box ai medicine artificial intelligence

Figure 1 Schematic diagram of a clinical decision support system (CDSS) for ECG classification.

These importance scores are typically illustrated as heatmaps for images or line charts for time series data. They will assist clinicians in understanding the CDSS decision process so that when disagreement arises, clinicians can pay attention to the potential ambiguity in data.

The second is “interpretation by data synthesis” that synthesizes an input to maximize the

prediction score of a label of interest [6]. If the synthesized input looks reasonable, then clinicians gain trust in the CDSS; if not, they could detect potential bias in the dataset and proactively prevent the implementation of immature CDSS.

Although interpretation by data synthesis is still immature in biomedical applications, this strategy will have great potential for discovering new knowledge, given our insufficient understanding on biological processes.

This explainable DL is critical for taking the mystique out of the black box and understanding advanced data analytics that ultimately can accelerate the translation of advanced data-driven analytics solutions to increase care quality and to reduce cost of care for humanity.


[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional

neural networks,” in Advances in neural information processing systems, 2012, pp. 1097‐1105.

[2] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real‐time object

detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition,

2016, pp. 779‐788.

[3] D. C. Cireşan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Mitosis detection in breast

cancer histology images with deep neural networks,” in International Conference on Medical

Image Computing and Computer‐assisted Intervention, 2013, pp. 411‐418: Springer.

[4] P. Rajpurkar, A. Y. Hannun, M. Haghpanahi, C. Bourn, and A. Y. Ng, “Cardiologist‐level

arrhythmia detection with convolutional neural networks,” arXiv preprint arXiv:1707.01836,


[5] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015.

[6] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising

image classification models and saliency maps,” arXiv preprint arXiv:1312.6034, 2013.

[7] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European

conference on computer vision, 2014, pp. 818‐833: Springer.

[8] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving for simplicity: The all

convolutional net,” arXiv preprint arXiv:1412.6806, 2014.

[9] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.‐R. Müller, and W. Samek, “On pixel‐wise

explanations for non‐linear classifier decisions by layer‐wise relevance propagation,” PloS one,

vol. 10, no. 7, p. e0130140, 2015.

[10] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” arXiv preprint

arXiv:1703.01365, 2017.

[11] D. Smilkov, N. Thorat, B. Kim, F. Viégas, and M. Wattenberg, “Smoothgrad: removing noise by

adding noise,” arXiv preprint arXiv:1706.03825, 2017.



black box ai medicine artificial intelligenceMay D. Wang, PhD ([email protected]): Director of Biomedical Data Initiative, Kavli Fellow,

Petit Institute Fellow, Fellow of AIMBE, IEEE Senior Member, Professor of Departments of

Biomedical Engineering, Computational Science and Engineering, Electrical and Computer

Engineering, Winship Cancer Institute, IBB, IPaT, Georgia Institute of Technology and Emory University





black box ai medicine artificial intelligence

Ying Sha ([email protected]): Ph.D Student, Department of Biology, Georgia Institute of Technology









black box ai medicine artificial intelligenceJohnny L. Chen ([email protected]): Ph.D Student, Department of Computational Science and Engineering, Georgia Institute of Technology