دورية أكاديمية
Machine learning clinical prediction models for acute kidney injury: the impact of baseline creatinine on prediction efficacy.
| العنوان: | Machine learning clinical prediction models for acute kidney injury: the impact of baseline creatinine on prediction efficacy. |
|---|---|
| المؤلفون: | Kamel Rahimi A; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia. amir.kamel@uq.edu.au.; Digital Health Cooperative Research Centre, Australian Government, Sydney, NSW, Australia. amir.kamel@uq.edu.au., Ghadimi M; The School of Mathematics and Physics, The University of Queensland, St Lucia, Brisbane, 4072, Australia., van der Vegt AH; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia., Canfell OJ; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Digital Health Cooperative Research Centre, Australian Government, Sydney, NSW, Australia.; UQ Business School, The University of Queensland, St Lucia, Brisbane, 4072, Australia., Pole JD; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Dalla Lana School of Public Health, The University of Toronto, Toronto, Canada.; ICES, Toronto, Canada., Sullivan C; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; Metro North Hospital and Health Service, Department of Health, Queensland Government, Herston, Brisbane, 4006, Australia., Shrapnel S; Queensland Digital Health Centre, Faculty of Medicine, The University of Queensland, Herston, Brisbane, 4006, Australia.; The School of Mathematics and Physics, The University of Queensland, St Lucia, Brisbane, 4072, Australia. |
| المصدر: | BMC medical informatics and decision making [BMC Med Inform Decis Mak] 2023 Oct 09; Vol. 23 (1), pp. 207. Date of Electronic Publication: 2023 Oct 09. |
| نوع المنشور: | Journal Article; Research Support, Non-U.S. Gov't |
| اللغة: | English |
| بيانات الدورية: | Publisher: BioMed Central Country of Publication: England NLM ID: 101088682 Publication Model: Electronic Cited Medium: Internet ISSN: 1472-6947 (Electronic) Linking ISSN: 14726947 NLM ISO Abbreviation: BMC Med Inform Decis Mak Subsets: MEDLINE |
| أسماء مطبوعة: | Original Publication: London : BioMed Central, [2001- |
| مواضيع طبية MeSH: | Models, Statistical* , Acute Kidney Injury*, Humans ; Creatinine ; Retrospective Studies ; Prognosis |
| مستخلص: | Background: There are many Machine Learning (ML) models which predict acute kidney injury (AKI) for hospitalised patients. While a primary goal of these models is to support clinical decision-making, the adoption of inconsistent methods of estimating baseline serum creatinine (sCr) may result in a poor understanding of these models' effectiveness in clinical practice. Until now, the performance of such models with different baselines has not been compared on a single dataset. Additionally, AKI prediction models are known to have a high rate of false positive (FP) events regardless of baseline methods. This warrants further exploration of FP events to provide insight into potential underlying reasons. Objective: The first aim of this study was to assess the variance in performance of ML models using three methods of baseline sCr on a retrospective dataset. The second aim was to conduct an error analysis to gain insight into the underlying factors contributing to FP events. Materials and Methods: The Intensive Care Unit (ICU) patients of the Medical Information Mart for Intensive Care (MIMIC)-IV dataset was used with the KDIGO (Kidney Disease Improving Global Outcome) definition to identify AKI episodes. Three different methods of estimating baseline sCr were defined as (1) the minimum sCr, (2) the Modification of Diet in Renal Disease (MDRD) equation and the minimum sCr and (3) the MDRD equation and the mean of preadmission sCr. For the first aim of this study, a suite of ML models was developed for each baseline and the performance of the models was assessed. An analysis of variance was performed to assess the significant difference between eXtreme Gradient Boosting (XGB) models across all baselines. To address the second aim, Explainable AI (XAI) methods were used to analyse the XGB errors with Baseline 3. Results: Regarding the first aim, we observed variances in discriminative metrics and calibration errors of ML models when different baseline methods were adopted. Using Baseline 1 resulted in a 14% reduction in the f1 score for both Baseline 2 and Baseline 3. There was no significant difference observed in the results between Baseline 2 and Baseline 3. For the second aim, the FP cohort was analysed using the XAI methods which led to relabelling data with the mean of sCr in 180 to 0 days pre-ICU as the preferred sCr baseline method. The XGB model using this relabelled data achieved an AUC of 0.85, recall of 0.63, precision of 0.54 and f1 score of 0.58. The cohort size was 31,586 admissions, of which 5,473 (17.32%) had AKI. Conclusion: In the absence of a widely accepted method of baseline sCr, AKI prediction studies need to consider the impact of different baseline methods on the effectiveness of ML models and their potential implications in real-world implementations. The utilisation of XAI methods can be effective in providing insight into the occurrence of prediction errors. This can potentially augment the success rate of ML implementation in routine care. (© 2023. BioMed Central Ltd., part of Springer Nature.) |
| References: | Srisawat N, Hoste EE, Kellum JA. Modern classification of acute kidney injury. Blood Purif. 2010;29(3):300–7. (PMID: 10.1159/00028009920130395) Siew ED, Matheny ME. Choice of reference serum creatinine in defining acute kidney injury. Nephron. 2015;131(2):107–12. (PMID: 10.1159/00043914426332325) Li PKT, Burdmann EA, Mehta RL. Acute kidney injury: global health alert. Arab J Nephrol Transplant. 2013;6(2):75–81. (PMID: 23795368) Coca SG, Singanamala S, Parikh CR. Chronic kidney disease after acute kidney injury: a systematic review and meta-analysis. Kidney Int. 2012;81(5):442–8. (PMID: 10.1038/ki.2011.37922113526) Coca SG, Yusuf B, Shlipak MG, Garg AX, Parikh CR. Long-term risk of mortality and other adverse outcomes after acute kidney injury: a systematic review and meta-analysis. Am J Kidney Dis. 2009;53(6):961–73. (PMID: 10.1053/j.ajkd.2008.11.034193460422726041) See EJ, Jayasinghe K, Glassford N, et al. Long-term risk of adverse outcomes after acute kidney injury: a systematic review and meta-analysis of cohort studies using consensus definitions of exposure. Kidney Int. 2019;95(1):160–72. (PMID: 10.1016/j.kint.2018.08.03630473140) Makris K, Spanou L. Acute kidney injury: definition, pathophysiology and clinical phenotypes. Clin Biochem Rev. 2016;37(2):85. (PMID: 283030735198510) Biswas A, Parikh CR, Feldman HI, et al. Identification of patients expected to benefit from electronic alerts for acute kidney injury. Clin J Am Soc Nephrol. 2018;13(6):842–9. https://doi.org/10.2215/CJN.13351217 . (PMID: 10.2215/CJN.13351217295992995989673) Section 2: AKI Definition. Kidney Int Suppl. 2012;2(1):19–36. https://doi.org/10.1038/kisup.2011.32 . Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, the Aw. Acute renal failure – definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Critical Care. 2004;8(4):R204. https://doi.org/10.1186/cc2872 . Mehta RL, Kellum JA, Shah SV, et al. Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Critical Care. 2007;11(2):R31. https://doi.org/10.1186/cc5713 . (PMID: 10.1186/cc5713173312452206446) Kellum JA, Lameire N, Aspelin P, et al. Kidney disease: improving global outcomes (KDIGO) acute kidney injury work group. KDIGO clinical practice guideline for acute kidney injury. Kidney Int Suppl. 2012;2(1):1–138. https://doi.org/10.1038/kisup.2012.1 . (PMID: 10.1038/kisup.2012.1) Siew ED, Ikizler TA, Matheny ME, et al. Estimating baseline kidney function in hospitalized patients with impaired kidney function. Clin J Am Soc Nephrol. 2012;7(5):712–9. https://doi.org/10.2215/CJN.10821011 . (PMID: 10.2215/CJN.10821011224225363338282) Siew ED, Matheny ME, Ikizler TA, et al. Commonly used surrogates for baseline renal function affect the classification and prognosis of acute kidney injury. Kidney Int. 2010;77(6):536–42. (PMID: 10.1038/ki.2009.47920042998) Bellomo R. Acute Dialysis Quality Initiative workgroup. Acute renal failure-definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit care. 2004;8:R204-R212. https://doi.org/10.1186/cc2872. Pickering JW, Endre ZH. Back-Calculating Baseline Creatinine with MDRD Misclassifies Acute Kidney Injury in the Intensive Care Unit. Clin J Am Soc Nephrol. 2010;5(7):1165–73. Lee TH, Chen J-J, Cheng C-T, Chang C-H. Does artificial intelligence make clinical decision better? A review of artificial intelligence and machine learning in acute kidney injury prediction. Healthcare (Basel). 2021;9(12):1662. https://doi.org/10.3390/healthcare9121662 . (PMID: 10.3390/healthcare912166234946388) Vagliano I, Chesnaye NC, Leopold JH, Jager KJ, Abu-Hanna A, Schut MC. Machine learning models for predicting acute kidney injury: a systematic review and critical appraisal. Clin Kidney J. 2022;15(12):2266–80. https://doi.org/10.1093/ckj/sfac181 . (PMID: 10.1093/ckj/sfac181363813759664575) Chromik J, Klopfenstein SAI, Pfitzner B, et al. Computational approaches to alleviate alarm fatigue in intensive care medicine: a systematic literature review. Front Digit Health. 2022;4:843747. (PMID: 10.3389/fdgth.2022.843747360523159424650) Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical Decision Support Systems Could Be Modified To Reduce “Alert Fatigue” While Still Minimizing The Risk Of Litigation. Health Affairs. 2011;30(12):2310–7. https://doi.org/10.1377/hlthaff.2010.1111 . (PMID: 10.1377/hlthaff.2010.111122147858) Sendelbach S, Funk M. Alarm fatigue: a patient safety concern. AACN Adv Crit Care. 2013;24(4):378–86. https://doi.org/10.4037/NCI.0b013e3182a903f9 . (PMID: 10.4037/NCI.0b013e3182a903f924153215) Cvach M. Monitor alarm fatigue: an integrative review. Biomed Instrum Technol. 2012;46(4):268–77. (PMID: 10.2345/0899-8205-46.4.26822839984) Parreco JMD, Soe-Lin HMD, Parks JJMD, et al. Comparing machine learning algorithms for predicting acute kidney injury. Am Surg. 2019;85(7):725–9. (PMID: 10.1177/00031348190850073131405416) He J, Hu Y, Zhang X, Wu L, Waitman LR, Liu M. Multi-perspective predictive modeling for acute kidney injury in general hospital populations using electronic medical records. JAMIA Open. 2019;2(1):115–22. https://doi.org/10.1093/jamiaopen/ooy043 . (PMID: 10.1093/jamiaopen/ooy04330976758) Xu Z, Chou J, Zhang XS, et al. Identifying sub-phenotypes of acute kidney injury using structured and unstructured electronic health record data with memory networks. J Biomed Informatics. 2020;102:103361. https://doi.org/10.1016/j.jbi.2019.103361 . (PMID: 10.1016/j.jbi.2019.103361) Wang Y, Wei Y, Yang H, Li J, Zhou Y, Wu Q. Utilizing imbalanced electronic health records to predict acute kidney injury by ensemble learning and time series model. BMC Med Inform Decis Mak. 2020;20(1):238. https://doi.org/10.1186/s12911-020-01245-4 . (PMID: 10.1186/s12911-020-01245-4329579777507620) Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594. https://doi.org/10.1136/bmj.g7594 . (PMID: 10.1136/bmj.g759425569120) Johnson A, Bulgarelli, L., Pollard, T., Horng, S., Celi, L. A., & Mark, R. MIMIC-IV. PhysioNet. https://doi.org/10.13026/s6n6-xd98. Zimmerman LP, Reyfman PA, Smith AD, et al. Early prediction of acute kidney injury following ICU admission using a multivariate panel of physiological measurements. BMC Med Inform Decis Mak. 2019;19(1):1–12. Thongprayoon C, Cheungpasitporn W, Harrison AM, et al. The comparison of the commonly used surrogates for baseline renal function in acute kidney injury diagnosis and staging. BMC Nephrol. 2016;17(1):6. https://doi.org/10.1186/s12882-016-0220-z . (PMID: 10.1186/s12882-016-0220-z267489094707008) Shawwa K, Ghosh E, Lanius S, Schwager E, Eshelman L, Kashani KB. Predicting acute kidney injury in critically ill patients using comorbid conditions utilizing machine learning. Clin Kidney J. 2021;14(5):1428–35. (PMID: 10.1093/ckj/sfaa14533959271) Wei C, Zhang L, Feng Y, Ma A, Kang Y. Machine learning model for predicting acute kidney injury progression in critically ill patients. BMC Med Inform Decis Mak. 2022;22(1):17. https://doi.org/10.1186/s12911-021-01740-2 . (PMID: 10.1186/s12911-021-01740-2350458408772216) Morid MA, Sheng ORL, Del Fiol G, Facelli JC, Bray BE, Abdelrahman S. Temporal pattern detection to predict adverse events in critical care: Case study with acute kidney injury. JMIR Med Inform. 2020;8(3):e14272. (PMID: 10.2196/14272321817537109618) Zhang Z, Ho KM, Hong Y. Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care. Crit Care (London, England). 2019;23(1):112–112. https://doi.org/10.1186/s13054-019-2411-z . (PMID: 10.1186/s13054-019-2411-z) Levey AS, Stevens LA, Schmid CH, et al. A new equation to estimate glomerular filtration rate. Ann Intern Med. 2009;150(9):604–12. https://doi.org/10.7326/0003-4819-150-9-200905050-00006 . (PMID: 10.7326/0003-4819-150-9-200905050-00006194148392763564) Sterne JA, White IR, Carlin JB, et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. https://doi.org/10.1136/bmj.b2393 . Abdi H, Williams LJ. Tukey’s honestly significant difference (HSD) test. Encyclopedia Res Design. 2010;3(1):1–5. Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine learning in Python. J Machine Learn Res. 2011;12:2825–30. Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Med. 2019;17(1):230. https://doi.org/10.1186/s12916-019-1466-7 . (PMID: 10.1186/s12916-019-1466-7318428786912996) Lundberg SM, Erion G, Chen H, et al. From local explanations to global understanding with explainable AI for trees. Nat Machine Intell. 2020;2(1):56–67. https://doi.org/10.1038/s42256-019-0138-9 . (PMID: 10.1038/s42256-019-0138-9) Baniecki H, Kretowicz W, Piatyszek P, Wisniewski J, Biecek P. Dalex: responsible machine learning with interactive explainability and fairness in python. J Machine Learn Res. 2021;22(1):9759–65. Toft EL, Kaae SE, Malmqvist J, Brodersen J. Psychosocial consequences of receiving false-positive colorectal cancer screening results: a qualitative study. Scand J Prim Health Care. 2019;37(2):145–54. https://doi.org/10.1080/02813432.2019.1608040 . (PMID: 10.1080/02813432.2019.1608040310795206566584) Ostermann M, Wu V, Sokolov D, Lumlertgul N. Definitions of acute renal dysfunction: An evolving clinical and biomarker paradigm. Curr Opin Crit Care. 2021;27(6):553–9. (PMID: 10.1097/MCC.000000000000088634535002) Makris K, Spanou L. Acute kidney injury: diagnostic approaches and controversies. Clin Biochem Rev. 2016;37(4):153. (PMID: 281678455242479) Bouchard J. Estimating baseline serum creatinine for assessing acute kidney injury: not a one size fits all approach. Kidney Int Rep. 2021;6(3):562. https://doi.org/10.1016/j.ekir.2021.01.030 . (PMID: 10.1016/j.ekir.2021.01.030337353297938179) Beydeda S, Book M, Gruhn V. Model-driven software development. vol 15. Heidelberg: Springer; 2005. Olson RS, Cava WL, Mustahsan Z, Varik A, Moore JH. Data-driven advice for applying machine learning to bioinformatics problems. World Sci. 2018;23:192–203. |
| فهرسة مساهمة: | Keywords: Acute kidney injury; Alert fatigue; Artificial intelligence; Decision Support System, Clinical; Health personnel; Machine learning |
| المشرفين على المادة: | AYI8EX34EU (Creatinine) |
| تواريخ الأحداث: | Date Created: 20231009 Date Completed: 20231101 Latest Revision: 20231119 |
| رمز التحديث: | 20231119 |
| مُعرف محوري في PubMed: | PMC10563357 |
| DOI: | 10.1186/s12911-023-02306-0 |
| PMID: | 37814311 |
| قاعدة البيانات: | MEDLINE |
Find this article in full text from Springer Verlag. 
Full Text Finder | تدمد: | 1472-6947 |
|---|---|
| DOI: | 10.1186/s12911-023-02306-0 |