دورية أكاديمية
أعرض في EDS

Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling.

التفاصيل البيبلوغرافية
العنوان: Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling.
المؤلفون: Holste G; Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA.; Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA., Lin M; Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA.; Department of Surgery, University of Minnesota, Minneapolis, MN, USA., Zhou R; Center for Biostatistics and Data Science, Washington University School of Medicine, St. Louis, MO, USA., Wang F; Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA., Liu L; Center for Biostatistics and Data Science, Washington University School of Medicine, St. Louis, MO, USA., Yan Q; Department of Obstetrics & Gynecology, Columbia University Irving Medical Center, New York, NY, USA., Van Tassel SH; Israel Englander Department of Ophthalmology, Weill Cornell Medicine, New York, NY, USA., Kovacs K; Israel Englander Department of Ophthalmology, Weill Cornell Medicine, New York, NY, USA., Chew EY; Division of Epidemiology and Clinical Applications, National Eye Institute, National Institutes of Health (NIH), Bethesda, MD, USA., Lu Z; National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD, USA., Wang Z; Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA. atlaswang@utexas.edu., Peng Y; Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA. yip4002@med.cornell.edu.
المصدر: NPJ digital medicine [NPJ Digit Med] 2024 Aug 16; Vol. 7 (1), pp. 216. Date of Electronic Publication: 2024 Aug 16.
نوع المنشور: Journal Article
اللغة: English
بيانات الدورية: Publisher: Nature Publishing Group Country of Publication: England NLM ID: 101731738 Publication Model: Electronic Cited Medium: Internet ISSN: 2398-6352 (Electronic) Linking ISSN: 23986352 NLM ISO Abbreviation: NPJ Digit Med Subsets: PubMed not MEDLINE
أسماء مطبوعة: Original Publication: [London] ; New York : Nature Publishing Group, [2018]-
مستخلص: Deep learning has enabled breakthroughs in automated diagnosis from medical imaging, with many successful applications in ophthalmology. However, standard medical image classification approaches only assess disease presence at the time of acquisition, neglecting the common clinical setting of longitudinal imaging. For slow, progressive eye diseases like age-related macular degeneration (AMD) and primary open-angle glaucoma (POAG), patients undergo repeated imaging over time to track disease progression and forecasting the future risk of developing a disease is critical to properly plan treatment. Our proposed Longitudinal Transformer for Survival Analysis (LTSA) enables dynamic disease prognosis from longitudinal medical imaging, modeling the time to disease from sequences of fundus photography images captured over long, irregular time periods. Using longitudinal imaging data from the Age-Related Eye Disease Study (AREDS) and Ocular Hypertension Treatment Study (OHTS), LTSA significantly outperformed a single-image baseline in 19/20 head-to-head comparisons on late AMD prognosis and 18/20 comparisons on POAG prognosis. A temporal attention analysis also suggested that, while the most recent image is typically the most influential, prior imaging still provides additional prognostic value.
(© 2024. The Author(s).)
التعليقات: Erratum in: NPJ Digit Med. 2024 Sep 9;7(1):240. doi: 10.1038/s41746-024-01243-0. (PMID: 39251870)
References: Esteva, A. et al. Deep learning-enabled medical computer vision. NPJ Digit Med. 4, 5 (2021). (PMID: 33420381779455810.1038/s41746-020-00376-2)
Chen, X. et al. Recent advances and clinical applications of deep learning in medical image analysis. Med. Image Anal. 79, 102444 (2022). (PMID: 35472844915657810.1016/j.media.2022.102444)
Lin, M. et al. Improving model fairness in image-based computer-aided diagnosis. Nat. Commun. 14, 6261 (2023). (PMID: 378030091055849810.1038/s41467-023-41974-4)
Chen, Q. et al. Multimodal, multitask, multiattention (M3) deep learning detection of reticular pseudodrusen: Toward automated and accessible classification of age-related macular degeneration. J. Am. Med. Inform. Assoc. 28, 1135–1148 (2021). (PMID: 33792724820027310.1093/jamia/ocaa302)
Peng, Y. et al. DeepSeeNet: A Deep Learning Model for Automated Classification of Patient-based Age-related Macular Degeneration Severity from Color Fundus Photographs. Ophthalmology 126, 565–575 (2019). (PMID: 3047131910.1016/j.ophtha.2018.11.015)
Chen, Q. et al. A multi-task deep learning model for the classification of Age-related Macular Degeneration. AMIA Jt Summits Transl. Sci. Proc. 2019, 505–514 (2019). (PMID: 312590056568069)
Peng, Y. et al. Predicting risk of late age-related macular degeneration using deep learning. NPJ Digit Med. 3, 111 (2020). (PMID: 32904246745300710.1038/s41746-020-00317-z)
Lee, J. et al. Predicting Age-related Macular Degeneration Progression with Longitudinal Fundus Images Using Deep Learning. Mach. Learn Med. Imaging 13583, 11–20 (2022). (PMID: 36656604984243210.1007/978-3-031-21014-3_2)
Ghahramani, G. et al. Multi-task deep learning-based survival analysis on the prognosis of late AMD using the longitudinal data in AREDS. AMIA Annu. Symp. Proc. 2021, 506–515 (2021). (PMID: 35308963)
Yan, Q. et al. Deep-learning-based Prediction of Late Age-Related Macular Degeneration Progression. Nat. Mach. Intell. 2, 141–150 (2020). (PMID: 32285025715373910.1038/s42256-020-0154-9)
Cascarano, A. et al. Machine and deep learning for longitudinal biomedical data: a review of methods and applications. Artif. Intell. Rev. 56, 1711–1771 (2023). (PMID: 10.1007/s10462-023-10561-w)
Ederer, F., Gaasterland, D. E. & Sullivan, E. K. & AGIS Investigators. The Advanced Glaucoma Intervention Study (AGIS): 1. Study design and methods and baseline characteristics of study patients. Control. Clin. Trials 15, 299–325 (1994). (PMID: 795627010.1016/0197-2456(94)90046-9)
Miglior, S. et al. Results of the European Glaucoma Prevention Study. Ophthalmology 112, 366–375 (2005). (PMID: 1574576110.1016/j.ophtha.2005.06.020)
Li, F. et al. A deep-learning system predicts glaucoma incidence and progression using retinal photographs. J. Clin. Investig. 132, e157968 (2022). (PMID: 35642636915169410.1172/JCI157968)
Lin, M. et al. Multi-scale Multi-structure Siamese Network (MMSNet) for Primary Open-Angle Glaucoma Prediction. Mach. Learn Med Imaging 13583, 436–445 (2022). (PMID: 36656619984466810.1007/978-3-031-21014-3_45)
Vaswani, A. et al. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS) 5998–6008 (2017).
Wiegrebe, S., Kopper, P., Sonabend, R., Bischl, B. & Bender, A. Deep Learning for Survival Analysis: A Review. arXiv [stat.ML] (2023).
Wolf, T. N., Pölsterl, S. & Wachinger, C. DAFT: A universal module to interweave tabular data and 3D images in CNNs. Neuroimage 260, 119505 (2022). (PMID: 3587872210.1016/j.neuroimage.2022.119505)
Zhang, Y. et al. CNN-based survival model for pancreatic ductal adenocarcinoma in medical imaging. BMC Med. Imaging 20, 11 (2020). (PMID: 32013871699824910.1186/s12880-020-0418-1)
Chen, R. J. et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) 4015–4025 https://doi.org/10.1109/iccv48922.2021.00398 (IEEE, 2021).
Agarwal, S., Eltigani Osman Abaker, M. & Daescu, O. Survival Prediction Based on Histopathology Imaging and Clinical Data: A Novel, Whole Slide CNN Approach. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 762–771 (Springer International Publishing, https://doi.org/10.1007/978-3-030-87240-3_73 . 2021).
Katzman, J. L. et al. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med. Res. Methodol. 18, 24 (2018). (PMID: 29482517582843310.1186/s12874-018-0482-1)
Ching, T., Zhu, X. & Garmire, L. X. Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data. PLoS Comput. Biol. 14, e1006076 (2018). (PMID: 29634719590992410.1371/journal.pcbi.1006076)
Kawasaki, R. et al. The prevalence of age-related macular degeneration in Asians: a systematic review and meta-analysis. Ophthalmology 117, 921–927 (2010). (PMID: 2011012710.1016/j.ophtha.2009.10.007)
Eye Disease Study Research Group, A.-R. Risk factors associated with age-related macular degeneration: A case-control study in the age-related eye disease study: age-related eye disease study report number 3. Ophthalmology 107, 2224–2232 (2000). (PMID: 10.1016/S0161-6420(00)00409-7)
Wong, W. L. et al. Global prevalence of age-related macular degeneration and disease burden projection for 2020 and 2040: a systematic review and meta-analysis. Lancet Glob. Health 2, e106–e116 (2014). (PMID: 2510465110.1016/S2214-109X(13)70145-1)
Ferris, F. L. et al. Clinical classification of age-related macular degeneration. Ophthalmology 120, 844–851 (2013). (PMID: 2333259010.1016/j.ophtha.2012.10.036)
Bourne, R. R. A. et al. Causes of vision loss worldwide, 1990-2010: a systematic analysis. Lancet Glob. Health 1, e339–e349 (2013). (PMID: 2510459910.1016/S2214-109X(13)70113-X)
Sommer, A. et al. Racial differences in the cause-specific prevalence of blindness in east Baltimore. N. Engl. J. Med. 325, 1412–1417 (1991). (PMID: 192225210.1056/NEJM199111143252004)
Jiang, X., Torres, M. & Varma, R. & Los Angeles Latino Eye Study Group. Variation in Intraocular Pressure and the Risk of Developing Open-Angle Glaucoma: The Los Angeles Latino Eye Study. Am. J. Ophthalmol. 188, 51–59 (2018). (PMID: 29360458587156410.1016/j.ajo.2018.01.013)
Tham, Y.-C. et al. Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis. Ophthalmology 121, 2081–2090 (2014). (PMID: 2497481510.1016/j.ophtha.2014.05.013)
Tatham, A. J., Medeiros, F. A., Zangwill, L. M. & Weinreb, R. N. Strategies to improve early diagnosis in glaucoma. Prog. Brain Res. 221, 103–133 (2015). (PMID: 2651807510.1016/bs.pbr.2015.03.001)
Lee, C., Yoon, J. & Schaar, M. V. Dynamic-DeepHit: A Deep Learning Approach for Dynamic Survival Analysis With Competing Risks Based on Longitudinal Data. IEEE Trans. Biomed. Eng. 67, 122–133 (2020). (PMID: 3095146010.1109/TBME.2019.2909027)
Lin, J. & Luo, S. Deep learning for the dynamic prediction of multivariate longitudinal and survival data. Stat. Med. 41, 2894–2907 (2022). (PMID: 35347750923297810.1002/sim.9392)
Gupta, G., Sunder, V., Prasad, R. & Shroff, G. CRESA: A Deep Learning Approach to Competing Risks, Recurrent Event Survival Analysis. In Advances in Knowledge Discovery and Data Mining 108–122 (Springer International Publishing. https://doi.org/10.1007/978-3-030-16145-3_9 , 2019).
Shu, M. et al. Deep survival analysis with longitudinal X-rays for COVID-19. arXiv [eess.IV] 4046–4055 (2021).
Kang, K. & Song, X. Y. Joint Modeling of Longitudinal Imaging and Survival Data. J. Comput. Graph. Stat. 32, 402–412 (2023). (PMID: 10.1080/10618600.2022.2102027)
Mirabnahrazam, G. et al. Predicting time-to-conversion for dementia of Alzheimer’s type using multi-modal deep survival analysis. Neurobiol. Aging 121, 139–156 (2023). (PMID: 3644241610.1016/j.neurobiolaging.2022.10.005)
Nakagawa, T. et al. Prediction of conversion to Alzheimer’s disease using deep survival analysis of MRI images. Brain Commun. 2, fcaa057 (2020). (PMID: 32954307742552810.1093/braincomms/fcaa057)
Ul Haq, N., Tahir, B., Firdous, S. & Amir Mehmood, M. Towards survival prediction of cancer patients using medical images. PeerJ Comput Sci. 8, e1090 (2022). (PMID: 36426251968089010.7717/peerj-cs.1090)
Liu, H. & Kurc, T. Deep learning for survival analysis in breast cancer with whole slide image data. Bioinformatics 38, 3629–3637 (2022). (PMID: 35674341927279710.1093/bioinformatics/btac381)
Ferris, F. L. et al. A simplified severity scale for age-related macular degeneration: AREDS Report. Arch. Ophthalmol. 123, 1570–1574 (2005). (PMID: 1628662010.1001/archopht.123.11.1570)
Davis, M. D. et al. The Age-Related Eye Disease Study severity scale for age-related macular degeneration: AREDS Report No. 17. Arch. Ophthalmol. 123, 1484–1498 (2005). (PMID: 1628661010.1001/archopht.123.11.1484)
Kass, M. A. et al. The Ocular Hypertension Treatment Study: a randomized trial determines that topical ocular hypotensive medication delays or prevents the onset of primary open-angle glaucoma. Arch. Ophthalmol. 120, 701–713 (2002). (PMID: 1204957410.1001/archopht.120.6.701)
Gordon, M. O. & Kass, M. A. The Ocular Hypertension Treatment Study: design and baseline description of the participants. Arch. Ophthalmol. 117, 573–583 (1999). (PMID: 1032695310.1001/archopht.117.5.573)
Sriram, A. et al. COVID-19 Prognosis via Self-Supervised Representation Learning and Multi-Image Prediction. arXiv [cs.CV] (2021).
He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. arXiv [cs.CV] 770–778 (2015).
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv [cs.CL] (2018).
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res 15, 1929–1958 (2014).
Harrell, F. E. Jr., Califf, R. M., Pryor, D. B., Lee, K. L. & Rosati, R. A. Evaluating the yield of medical tests. JAMA 247, 2543–2546 (1982). (PMID: 706992010.1001/jama.1982.03320430047030)
Vitale, S. et al. Association of 2-Year Progression Along the AREDS AMD Scale and Development of Late Age-Related Macular Degeneration or Loss of Visual Acuity: AREDS Report 41. JAMA Ophthalmol. 138, 610–617 (2020). (PMID: 3227135810.1001/jamaophthalmol.2020.0824)
Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8026–8037 (2019).
Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization. arXiv [cs.LG] (2014).
Bland, J. M. & Altman, D. G. Multiple significance tests: the Bonferroni method. BMJ 310, 170 (1995). (PMID: 7833759254856110.1136/bmj.310.6973.170)
Hofmann, H., Wickham, H. & Kafadar, K. Letter-value plots: Boxplots for large data. J. Comput. Graph. Stat. 26, 469–477 (2017). (PMID: 10.1080/10618600.2017.1305277)
معلومات مُعتمدة: R21 EY035296 United States EY NEI NIH HHS; R21EY035296 U.S. Department of Health & Human Services | NIH | National Eye Institute (NEI)
تواريخ الأحداث: Date Created: 20240816 Latest Revision: 20240909
رمز التحديث: 20240910
مُعرف محوري في PubMed: PMC11329720
DOI: 10.1038/s41746-024-01207-4
PMID: 39152209
قاعدة البيانات: MEDLINE
الوصف
تدمد:2398-6352
DOI:10.1038/s41746-024-01207-4