دورية أكاديمية
Reproducibility of Deep Learning Algorithms Developed for Medical Imaging Analysis: A Systematic Review.
| العنوان: | Reproducibility of Deep Learning Algorithms Developed for Medical Imaging Analysis: A Systematic Review. |
|---|---|
| المؤلفون: | Moassefi M; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA. Moassefi.mana@Mayo.edu., Rouzrokh P; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA.; Orthopedic Surgery Artificial Intelligence Laboratory (OSAIL), Department of Orthopedic Surgery, Mayo Clinic, Rochester, MN, USA., Conte GM; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA., Vahdati S; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA., Fu T; Department of Radiology, University Hospitals Cleveland, Cleveland, OH, USA., Tahmasebi A; Department of Radiology, Thomas Jefferson University, Philadelphia, PA, USA., Younis M; Cleveland Clinic Children's, Cleveland, OH, USA., Farahani K; National Cancer Institute, National Institutes of Health, Bethesda, MA, USA., Gentili A; Department of Radiology, University of California, San Diego, CA, USA., Kline T; Department of Radiology, Mayo Clinic, Rochester, MN, USA., Kitamura FC; DasaInova, Diagnósticos da América S.A, São Paulo, Brazil., Huo Y; Department of Electrical Engineering & Computer Science, Vanderbilt University, Nashville, TN, USA., Kuanar S; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA., Younis K; Phillips Research North America, Cambridge, MD, USA., Erickson BJ; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA., Faghani S; Artificial Intelligence Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA. Faghani.shahriar@mayo.edu. |
| المصدر: | Journal of digital imaging [J Digit Imaging] 2023 Oct; Vol. 36 (5), pp. 2306-2312. Date of Electronic Publication: 2023 Jul 05. |
| نوع المنشور: | Systematic Review; Journal Article; Review |
| اللغة: | English |
| بيانات الدورية: | Publisher: Springer Country of Publication: United States NLM ID: 9100529 Publication Model: Print-Electronic Cited Medium: Internet ISSN: 1618-727X (Electronic) Linking ISSN: 08971889 NLM ISO Abbreviation: J Digit Imaging Subsets: MEDLINE |
| أسماء مطبوعة: | Publication: <2008-2023>: New York : Springer Original Publication: Philadelphia, PA : W.B. Saunders, c1988- |
| مواضيع طبية MeSH: | Artificial Intelligence* , Diagnostic Imaging*, Humans ; Cross-Sectional Studies ; Reproducibility of Results ; Algorithms |
| مستخلص: | Since 2000, there have been more than 8000 publications on radiology artificial intelligence (AI). AI breakthroughs allow complex tasks to be automated and even performed beyond human capabilities. However, the lack of details on the methods and algorithm code undercuts its scientific value. Many science subfields have recently faced a reproducibility crisis, eroding trust in processes and results, and influencing the rise in retractions of scientific papers. For the same reasons, conducting research in deep learning (DL) also requires reproducibility. Although several valuable manuscript checklists for AI in medical imaging exist, they are not focused specifically on reproducibility. In this study, we conducted a systematic review of recently published papers in the field of DL to evaluate if the description of their methodology could allow the reproducibility of their findings. We focused on the Journal of Digital Imaging (JDI), a specialized journal that publishes papers on AI and medical imaging. We used the keyword "Deep Learning" and collected the articles published between January 2020 and January 2022. We screened all the articles and included the ones which reported the development of a DL tool in medical imaging. We extracted the reported details about the dataset, data handling steps, data splitting, model details, and performance metrics of each included article. We found 148 articles. Eighty were included after screening for articles that reported developing a DL model for medical image analysis. Five studies have made their code publicly available, and 35 studies have utilized publicly available datasets. We provided figures to show the ratio and absolute count of reported items from included studies. According to our cross-sectional study, in JDI publications on DL in medical imaging, authors infrequently report the key elements of their study to make it reproducible. (© 2023. The Author(s) under exclusive licence to Society for Imaging Informatics in Medicine.) |
| References: | West E, Mutasa S, Zhu Z, Ha R. Global Trend in Artificial Intelligence-Based Publications in Radiology From 2000 to 2018. AJR Am J Roentgenol. 2019;213: 1204–1206. (PMID: 10.2214/AJR.19.2134631414886) Kelly BS, Judge C, Bollard SM, Clifford SM, Healy GM, Aziz A, et al. Radiology artificial intelligence: a systematic review and evaluation of methods (RAISE). Eur Radiol. 2022. https://doi.org/10.1007/s00330-022-08784-6. (PMID: 10.1007/s00330-022-08784-6966893435593961) Oppenheimer J, Lüken S, Hamm B, Niehues SM. A Prospective Approach to Integration of AI Fracture Detection Software in Radiographs into Clinical Workflow. Life. 2023;13. https://doi.org/10.3390/life13010223. Arbabshirani MR, Fornwalt BK, Mongelluzzo GJ, Suever JD, Geise BD, Patel AA, et al. Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. NPJ Digit Med. 2018;1: 9. (PMID: 10.1038/s41746-017-0015-z655014431304294) Akkus Z, Cai J, Boonrod A, Zeinoddini A, Weston AD, Philbrick KA, et al. A Survey of Deep-Learning Applications in Ultrasound: Artificial Intelligence-Powered Ultrasound for Improving Clinical Workflow. J Am Coll Radiol. 2019;16: 1318–1328. (PMID: 10.1016/j.jacr.2019.06.00431492410) Yu K-H, Lee T-LM, Yen M-H, Kou SC, Rosen B, Chiang J-H, et al. Reproducible Machine Learning Methods for Lung Cancer Detection Using Computed Tomography Images: Algorithm Development and Validation. J Med Internet Res. 2020;22: e16709. Gundersen OE, Kjensmo S. State of the Art: Reproducibility in Artificial Intelligence. AAAI. 2018;32. https://doi.org/10.1609/aaai.v32i1.11503. Stupple A, Singerman D, Celi LA. The reproducibility crisis in the age of digital medicine. NPJ Digit Med. 2019;2: 2. (PMID: 10.1038/s41746-019-0079-z655026231304352) McDermott MBA, Wang S, Marinsek N, Ranganath R, Ghassemi M, Foschini L. Reproducibility in Machine Learning for Health. arXiv [cs.LG]. 2019. Available: http://arxiv.org/abs/1907.01463. McDermott MBA, Wang S, Marinsek N, Ranganath R, Foschini L, Ghassemi M. Reproducibility in machine learning for health research: Still a ways to go. Sci Transl Med. 2021;13. https://doi.org/10.1126/scitranslmed.abb1655. Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med. 2016;8: 341ps12. Campbell DT. Relabeling internal and external validity for applied social scientists. New Dir Prog Eval. 1986;1986: 67–77. Moher D. Reporting guidelines: doing better for readers. BMC Med. 2018;16: 233. (PMID: 10.1186/s12916-018-1226-0629354230545364) Shelmerdine SC, Arthurs OJ, Denniston A, Sebire NJ. Review of study reporting guidelines for clinical studies using artificial intelligence in healthcare. BMJ Health Care Inform. 2021;28. https://doi.org/10.1136/bmjhci-2021-100385. Mongan J, Moy L, Kahn CE Jr. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol Artif Intell. 2020;2: e200029. (PMID: 10.1148/ryai.2020200029801741433937821) Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27: 2011–2015. (PMID: 10.1093/jamia/ocaa088772733332594179) Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169: 467–473. (PMID: 10.7326/M18-085030178033) Kitamura FC, Pan I, Kline TL. Reproducible Artificial Intelligence Research Requires Open Communication of Complete Source Code. Radiol Artif Intell. 2020;2: e200060. (PMID: 10.1148/ryai.2020200060808234733937836) Venkatesh K, Santomartino SM, Sulam J, Yi PH. Code and Data Sharing Practices in the Radiology AI Literature: A Meta-Research Study. Radiology: Artificial Intelligence. 2022; e220081. Simko A, Garpebring A, Jonsson J, Nyholm T, Löfstedt T. Reproducibility of the Methods in Medical Imaging with Deep Learning. arXiv [cs.LG]. 2022. Available: http://arxiv.org/abs/2210.11146. Beam AL, Manrai AK, Ghassemi M. Challenges to the Reproducibility of Machine Learning Models in Health Care. JAMA. 2020;323: 305–306. (PMID: 10.1001/jama.2019.20866733567731904799) Colliot O, Thibeau-Sutre E, Burgos N. Reproducibility in machine learning for medical imaging. arXiv [cs.CV]. 2022. Available: http://arxiv.org/abs/2209.05097. Wright BD, Vo N, Nolan J, Johnson AL, Braaten T, Tritz D, et al. An analysis of key indicators of reproducibility in radiology. Insights Imaging. 2020;11: 65. (PMID: 10.1186/s13244-020-00870-x721458532394098) |
| فهرسة مساهمة: | Keywords: Artificial intelligence; Deep learning; Machine learning; Medical imaging; Reproducibility |
| تواريخ الأحداث: | Date Created: 20230705 Date Completed: 20230918 Latest Revision: 20230922 |
| رمز التحديث: | 20230922 |
| مُعرف محوري في PubMed: | PMC10501962 |
| DOI: | 10.1007/s10278-023-00870-5 |
| PMID: | 37407841 |
| قاعدة البيانات: | MEDLINE |
Find this article in full text from Springer Verlag. 
Full Text Finder | تدمد: | 1618-727X |
|---|---|
| DOI: | 10.1007/s10278-023-00870-5 |