دورية أكاديمية
أعرض في EDS

Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis.

التفاصيل البيبلوغرافية
العنوان: Clinically focused multi-cohort benchmarking as a tool for external validation of artificial intelligence algorithm performance in basic chest radiography analysis.
المؤلفون: Rudolph J; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany. jan.rudolph@med.uni-muenchen.de., Schachtner B; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany.; Comprehensive Pneumology Center, German Center for Lung Research, Munich, Germany., Fink N; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany.; Comprehensive Pneumology Center, German Center for Lung Research, Munich, Germany., Koliogiannis V; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Schwarze V; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Goller S; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Trappmann L; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Hoppe BF; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Mansour N; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Fischer M; Department of Medicine I, University Hospital, LMU Munich, Munich, Germany., Ben Khaled N; Department of Medicine II, University Hospital, LMU Munich, Munich, Germany., Jörgens M; Department of Orthopaedics and Trauma Surgery, Musculoskeletal University Center Munich (MUM), University Hospital, LMU Munich, Munich, Germany., Dinkel J; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany.; Comprehensive Pneumology Center, German Center for Lung Research, Munich, Germany.; Department of Radiology, Asklepios Fachklinik München, Gauting, Germany., Kunz WG; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Ricke J; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Ingrisch M; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Sabel BO; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany., Rueckel J; Department of Radiology, University Hospital, LMU Munich, Marchioninistr. 15, 81377, Munich, Germany.; Institute of Neuroradiology, University Hospital, LMU Munich, Munich, Germany.
المصدر: Scientific reports [Sci Rep] 2022 Jul 27; Vol. 12 (1), pp. 12764. Date of Electronic Publication: 2022 Jul 27.
نوع المنشور: Journal Article; Research Support, Non-U.S. Gov't
اللغة: English
بيانات الدورية: Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101563288 Publication Model: Electronic Cited Medium: Internet ISSN: 2045-2322 (Electronic) Linking ISSN: 20452322 NLM ISO Abbreviation: Sci Rep Subsets: MEDLINE
أسماء مطبوعة: Original Publication: London : Nature Publishing Group, copyright 2011-
مواضيع طبية MeSH: Artificial Intelligence* , Pneumothorax*/etiology, Algorithms ; Benchmarking ; Humans ; Radiography, Thoracic/methods ; Retrospective Studies
مستخلص: Artificial intelligence (AI) algorithms evaluating [supine] chest radiographs ([S]CXRs) have remarkably increased in number recently. Since training and validation are often performed on subsets of the same overall dataset, external validation is mandatory to reproduce results and reveal potential training errors. We applied a multicohort benchmarking to the publicly accessible (S)CXR analyzing AI algorithm CheXNet, comprising three clinically relevant study cohorts which differ in patient positioning ([S]CXRs), the applied reference standards (CT-/[S]CXR-based) and the possibility to also compare algorithm classification with different medical experts' reading performance. The study cohorts include [1] a cohort, characterized by 563 CXRs acquired in the emergency unit that were evaluated by 9 readers (radiologists and non-radiologists) in terms of 4 common pathologies, [2] a collection of 6,248 SCXRs annotated by radiologists in terms of pneumothorax presence, its size and presence of inserted thoracic tube material which allowed for subgroup and confounding bias analysis and [3] a cohort consisting of 166 patients with SCXRs that were evaluated by radiologists for underlying causes of basal lung opacities, all of those cases having been correlated to a timely acquired computed tomography scan (SCXR and CT within < 90 min). CheXNet non-significantly exceeded the radiology resident (RR) consensus in the detection of suspicious lung nodules (cohort [1], AUC AI/RR: 0.851/0.839, p = 0.793) and the radiological readers in the detection of basal pneumonia (cohort [3], AUC AI/reader consensus: 0.825/0.782, p = 0.390) and basal pleural effusion (cohort [3], AUC AI/reader consensus: 0.762/0.710, p = 0.336) in SCXR, partly with AUC values higher than originally published ("Nodule": 0.780, "Infiltration": 0.735, "Effusion": 0.864). The classifier "Infiltration" turned out to be very dependent on patient positioning (best in CXR, worst in SCXR). The pneumothorax SCXR cohort [2] revealed poor algorithm performance in CXRs without inserted thoracic material and in the detection of small pneumothoraces, which can be explained by a known systematic confounding error in the algorithm training process. The benefit of clinically relevant external validation is demonstrated by the differences in algorithm performance as compared to the original publication. Our multi-cohort benchmarking finally enables the consideration of confounders, different reference standards and patient positioning as well as the AI performance comparison with differentially qualified medical readers.
(© 2022. The Author(s).)
References: JAMA. 2016 Dec 13;316(22):2402-2410. (PMID: 27898976)
Eur Radiol. 2021 Oct;31(10):7888-7900. (PMID: 33774722)
Invest Radiol. 2020 Dec;55(12):792-798. (PMID: 32694453)
Chest. 1994 Mar;105(3):885-7. (PMID: 8131557)
Lancet Respir Med. 2019 Aug;7(8):710-720. (PMID: 31182406)
Nature. 2020 Jan;577(7788):89-94. (PMID: 31894144)
Radiology. 2008 Sep;248(3):1083. (PMID: 18710998)
Chest. 1992 Feb;101(2):458-63. (PMID: 1735272)
J Clin Med. 2019 Jun 13;8(6):. (PMID: 31200458)
Radiology. 2017 Aug;284(2):574-582. (PMID: 28436741)
Eur J Radiol. 2020 Feb;123:108774. (PMID: 31841881)
Thorax. 1999 Oct;54(10):867-73. (PMID: 10491448)
Nature. 2017 Feb 2;542(7639):115-118. (PMID: 28117445)
Ann Am Thorac Soc. 2014 Mar;11(3):404-6. (PMID: 24673696)
Ann Thorac Med. 2019 Oct-Dec;14(4):226-238. (PMID: 31620206)
Thorax. 2015 Jul;70(7):653-8. (PMID: 25918121)
Br J Gen Pract. 2006 Aug;56(529):574-8. (PMID: 16882374)
Chest. 2012 Feb;141(2):545-558. (PMID: 22315122)
Chest. 1991 Apr;99(4):928-33. (PMID: 2009797)
Acta Paediatr. 2013 Jul;102(7):e310-4. (PMID: 23565882)
PLoS Med. 2018 Nov 20;15(11):e1002686. (PMID: 30457988)
Acad Emerg Med. 2016 Mar;23(3):223-42. (PMID: 26910112)
Eur J Radiol. 2012 Dec;81(12):3669-74. (PMID: 21466934)
Invest Radiol. 2019 Oct;54(10):627-632. (PMID: 31483764)
Radiology. 1995 Apr;195(1):245-6. (PMID: 7892479)
Radiology. 2008 Mar;246(3):697-722. (PMID: 18195376)
Invest Radiol. 2021 Jun 1;56(6):348-356. (PMID: 33259441)
Invest Radiol. 2022 Feb 1;57(2):90-98. (PMID: 34352804)
Rofo. 1999 Oct;171(4):294-301. (PMID: 10598165)
Respir Care. 2012 Mar;57(3):427-43. (PMID: 22391269)
Radiology. 2000 Nov;217(2):456-9. (PMID: 11058645)
Crit Care Med. 2017 Apr;45(4):715-724. (PMID: 27922877)
Chest. 1993 Nov;104(5):1486-9. (PMID: 8222812)
Diagnostics (Basel). 2021 Oct 11;11(10):. (PMID: 34679566)
Pathog Glob Health. 2019 Jun;113(4):191-205. (PMID: 31215326)
Crit Care Med. 2020 Jul;48(7):e574-e583. (PMID: 32433121)
JAAPA. 2019 Oct;32(10):18-23. (PMID: 31513034)
Acad Radiol. 2018 Oct;25(10):1252-1256. (PMID: 29506819)
تواريخ الأحداث: Date Created: 20220727 Date Completed: 20220729 Latest Revision: 20221006
رمز التحديث: 20231215
مُعرف محوري في PubMed: PMC9329327
DOI: 10.1038/s41598-022-16514-7
PMID: 35896763
قاعدة البيانات: MEDLINE
الوصف
تدمد:2045-2322
DOI:10.1038/s41598-022-16514-7