A Practical Guide Towards Interpreting Time-Series Deep Clinical Predictive Models: A Reproducibility Study

Abstract

Deep learning models for clinical time-series prediction have demonstrated strong performance on tasks such as mortality prediction and sepsis onset detection, but their adoption in clinical practice is hindered by a lack of interpretability. Existing explainability methods designed for images or tabular data often transfer poorly to temporal clinical signals. This paper provides a practical guide for applying and interpreting explanation methods — including gradient-based, attention-based, and perturbation-based approaches — on time-series deep clinical predictive models. Through a systematic reproducibility study, we evaluate the stability, faithfulness, and clinical plausibility of these explanations, offering concrete recommendations for practitioners seeking to deploy interpretable models in real-world healthcare settings.