دورية أكاديمية
Weighted metrics are required when evaluating the performance of prediction models in nested case-control studies.
| العنوان: | Weighted metrics are required when evaluating the performance of prediction models in nested case-control studies. |
|---|---|
| المؤلفون: | Rentroia-Pacheco B; Department of Dermatology, Erasmus Medical Center Cancer Institute, Erasmus University Medical Center, Dr. Molewaterplein 40, Rotterdam, 3015 GD, The Netherlands. b.rentroiapacheco@erasmusmc.nl., Bellomo D; SkylineDx B.V, Rotterdam, The Netherlands., Lakeman IMM; Department of Human Genetics, Leiden University Medical Center, Leiden, The Netherlands.; Department of Clinical Genetics, Leiden University Medical Center, Leiden, The Netherlands., Wakkee M; Department of Dermatology, Erasmus Medical Center Cancer Institute, Erasmus University Medical Center, Dr. Molewaterplein 40, Rotterdam, 3015 GD, The Netherlands., Hollestein LM; Department of Dermatology, Erasmus Medical Center Cancer Institute, Erasmus University Medical Center, Dr. Molewaterplein 40, Rotterdam, 3015 GD, The Netherlands.; Department of Research, Netherlands Comprehensive Cancer Organization (IKNL), Utrecht, The Netherlands., van Klaveren D; Department of Public Health, Center for Medical Decision Making, Erasmus University Medical Center, Rotterdam, The Netherlands. |
| المصدر: | BMC medical research methodology [BMC Med Res Methodol] 2024 May 17; Vol. 24 (1), pp. 115. Date of Electronic Publication: 2024 May 17. |
| نوع المنشور: | Journal Article |
| اللغة: | English |
| بيانات الدورية: | Publisher: BioMed Central Country of Publication: England NLM ID: 100968545 Publication Model: Electronic Cited Medium: Internet ISSN: 1471-2288 (Electronic) Linking ISSN: 14712288 NLM ISO Abbreviation: BMC Med Res Methodol Subsets: MEDLINE |
| أسماء مطبوعة: | Original Publication: London : BioMed Central, [2001- |
| مواضيع طبية MeSH: | Algorithms*, Humans ; Case-Control Studies ; Female ; Models, Statistical ; Breast Neoplasms ; Ovarian Neoplasms ; Middle Aged ; Aged |
| مستخلص: | Background: Nested case-control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data. Methods: We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design. Results: Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58-0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62-0.69) and 0.65 (0.61-0.69), respectively. The unweighted O/E ratio was 18.38 (17.67-19.06) in the NCC datasets, while it was 1.69 (1.42-1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53-1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment. Conclusions: Nested case-control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure. (© 2024. The Author(s).) |
| References: | McCarthy CE, Bonnet LJ, Marcus MW, Field JK. Development and validation of a multivariable risk prediction model for head and neck cancer using the UK Biobank. Int J Oncol. 2020;57(5):1192–202. (PMID: 33491742) Cederholm J, Eeg-Olofsson K, Eliasson B, Zethelius B, Nilsson PM, Gudbjörnsdottir S. Risk prediction of cardiovascular disease in type 2 diabetes: a risk equation from the Swedish National Diabetes Register. Diabetes Care. 2008;31(10):2038–43. (PMID: 10.2337/dc08-0662185914032551651) Moons KGM, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, et al. Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart. 2012;98(9):683–90. (PMID: 10.1136/heartjnl-2011-30124622397945) Ganna A, Reilly M, De Faire U, Pedersen N, Magnusson P, Ingelsson E. Risk prediction measures for case-cohort and nested case-control designs: An application to cardiovascular disease. Am J Epidemiol. 2012;175(7):715–24. (PMID: 10.1093/aje/kwr374223963883324433) Salim A, Delcoigne B, Villaflores K, Koh WP, Yuan JM, van Dam RM, et al. Comparisons of risk prediction methods using nested case-control data. Stat Med. 2017;36(3):455–65. (PMID: 10.1002/sim.714327734520) Biesheuvel CJ, Vergouwe Y, Oudega R, Hoes AW, Grobbee DE, Moons KGM. Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol. 2008;8(1):1–7. (PMID: 10.1186/1471-2288-8-48) Moons KGM, Van Klei W, Kalkman CJ. Preoperative risk factors of intraoperative hypothermia in major surgery under general anesthesia. Anesth Analg. 2003;96(6):1843–4. (PMID: 10.1213/01.ANE.0000063178.15467.D812761031) Choudhury PP, Maas P, Wilcox A, Wheeler W, Brook M, Check D, et al. iCARE: An R package to build, validate and apply absolute risk models. PLoS ONE. 2020;15(2):e0228198. (PMID: 10.1371/journal.pone.0228198) Kim RS. Analysis of Nested Case-Control Study Designs: Revisiting the Inverse Probability Weighting Method. Communications for Statistical Applications and Methods. 2013;20(6):455–66. (PMID: 10.5351/CSAM.2013.20.6.455285035125426119) Borgan Ø, Keogh R. Nested case–control studies: should one break the matching? Lifetime Data Anal. 2015;21(4):517–41. (PMID: 10.1007/s10985-015-9319-y25608704) Zelic R, Zugna D, Bottai M, Andrén O, Fridfeldt J, Carlsson J, et al. Estimation of Relative and Absolute Risks in a Competing-Risks Setting Using a Nested Case-Control Study Design: Example from the ProMort Study. Am J Epidemiol. 2019;188(6):1165–73. (PMID: 10.1093/aje/kwz026309767898210820) Delcoigne B, Colzani E, Prochazka M, Gagliardi G, Hall P, Abrahamowicz M, et al. Breaking the matching in nested case–control data offered several advantages for risk estimation. J Clin Epidemiol. 2017;1(82):79–86. (PMID: 10.1016/j.jclinepi.2016.11.014) Rentroia-Pacheco B, Tokez S, Bramer EM, Venables ZC, Van De Werken HJG, Bellomo D, et al. Personalised decision making to predict absolute metastatic risk in cutaneous squamous cell carcinoma: development and validation of a clinico-pathological model. eClinicalMedicine. 2023;63:102150. (PMID: 10.1016/j.eclinm.2023.1021503766251910468358) Murphy JD, Olshan AF, Lin FC, Troester MA, Nichols HB, Butt J, et al. A Predictive Model of Noncardia Gastric Adenocarcinoma Risk Using Antibody Response to Helicobacter pylori Proteins and Pepsinogen. Cancer Epidemiol Biomark Prev. 2022;31(4):811–20. (PMID: 10.1158/1055-9965.EPI-21-0869) Hoogeveen RM, Pereira JPB, Nurmohamed NS, Zampoleri V, Bom MJ, Baragetti A, et al. Improved cardiovascular risk prediction using targeted plasma proteomics in primary prevention. Eur Heart J. 2020;41(41):3998–4007. (PMID: 10.1093/eurheartj/ehaa648328080147672529) Reps JM, Ryan PB, Rijnbeek PR, Schuemie MJ. Design matters in patient-level prediction: evaluation of a cohort vs. case-control design when developing predictive models in observational healthcare datasets. Journal of Big Data. 2021;8(1):1–18. (PMID: 10.1186/s40537-021-00501-2) Lee A, Mavaddat N, Wilcox AN, Cunningham AP, Carver T, Hartley S, et al. BOADICEA: a comprehensive breast cancer risk prediction model incorporating genetic and nongenetic risk factors. Genet Med. 2019;21(8):1708–18. (PMID: 10.1038/s41436-018-0406-9306432176687499) Lakeman IMM, Rodríguez-Girondo M, Lee A, Ruiter R, Stricker BH, Wijnant SRA, et al. Validation of the BOADICEA model and a 313-variant polygenic risk score for breast cancer risk prediction in a Dutch prospective cohort. Genet Med. 2020;22(11):1803–11. (PMID: 10.1038/s41436-020-0884-4326245717605432) Zhou QM, Wang X, Zheng Y, Cai T. New weighting methods when cases are only a subset of events in a nested case-control study. Biom J. 2022;64(7):1240–59. (PMID: 10.1002/bimj.2021001943575430910249867) Støer NC, Samuelsen SO. MultipleNCC: Inverse probability weighting of nested case-control data. R Journal. 2016;8(2):5–18. (PMID: 10.32614/RJ-2016-030) Langholz B, Richardson D. Are Nested Case-Control Studies Biased? Epidemiology. 2009;20(3):321–9. (PMID: 10.1097/EDE.0b013e31819e370b192899634194079) Støer NC, Samuelsen SO. Comparison of estimators in nested case-control studies with multiple outcomes. Lifetime Data Anal. 2012;18(3):261–83. (PMID: 10.1007/s10985-012-9214-822382602) Moons KGM, Altman DG, Reitsma JB, Ioannidis JPA, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): Explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73. (PMID: 10.7326/M14-069825560730) Steyerberg EW. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating Second Edition. New York: NY: Springer Nature Switzerland; 2019. (Statistics for Biology and Health). (PMID: 10.1007/978-3-030-16399-0) Vickers AJ, Elkin EB. Decision curve analysis: A novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74. (PMID: 10.1177/0272989X06295361170991942577036) Parikh R, Mathai A, Parikh S, Sekhar GC, Thomas R. Understanding and using sensitivity, specificity and predictive values. Indian J Ophthalmol. 2008;56(1):45–50. (PMID: 10.4103/0301-4738.37595181584032636062) Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: From utopia to empirical data. J Clin Epidemiol. 2016;1(74):167–76. (PMID: 10.1016/j.jclinepi.2015.12.005) McLernon DJ, Giardiello D, Van Calster B, Wynants L, van Geloven N, van Smeden M, et al. Assessing Performance and Clinical Usefulness in Prediction Models With Survival Outcomes: Practical Guidance for Cox Proportional Hazards Models. Ann Intern Med. 2023;176(1):105–14. (PMID: 10.7326/M22-084436571841) Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Medical Inform Decis Mak. 2008;8:53. (PMID: 10.1186/1472-6947-8-53) Pfeiffer RM, Gail MH. Estimating the decision curve and its precision from three study designs. Biom J. 2020;62(3):764–76. (PMID: 10.1002/bimj.20180024031394013) Wieczorek J, Guerin C, McMahon T. K-fold cross-validation for complex sample surveys. In: In: Stat. John Wiley & Sons, Ltd; 2022. p. e454. Arnold M, Morgan E, Rumgay H, Mafra A, Singh D, Laversanne M, et al. Current and future burden of breast cancer: Global statistics for 2020 and 2040. Breast. 2022;1(66):15–23. (PMID: 10.1016/j.breast.2022.08.010) Marmot M, Altman DG, Cameron DA, Dewar JA, Thompson SG, Wilcox M. The benefits and harms of breast cancer screening: an independent review. The Lancet. 2012;380(9855):1778–86. (PMID: 10.1016/S0140-6736(12)61611-0) Ikram MA, Brusselle G, Ghanbari M, Goedegebure A, Ikram MK, Kavousi M, et al. Objectives, design and main findings until 2020 from the Rotterdam Study. European J Epidemiol. 2020;35(5):483–517. (PMID: 10.1007/s10654-020-00640-5) Lee M, Zeleniuch-Jacquotte A, Liu M. Empirical evaluation of sub-cohort sampling designs for risk prediction modeling. J Appl Stat. 2021;48(8):1374–401. (PMID: 10.1080/02664763.2020.186122535706464) Pepe MS, Fan J, Feng Z, Gerds T, Hilden J. The Net Reclassification Index (NRI): A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets. Stat Biosci. 2015;7(2):282–95. (PMID: 10.1007/s12561-014-9118-026504496) Ramspek CL, Teece L, Snell KIE, Evans M, Riley RD, Van Smeden M, et al. Lessons learnt when accounting for competing events in the external validation of time-To-event prognostic models. Int J Epidemiol. 2022;51(2):615–25. (PMID: 10.1093/ije/dyab25634919691) Wolkewitz M, Cooper BS, Palomar-Martinez M, Olaechea-Astigarraga P, Alvarez-Lerma F, Schumacher M. Nested case-control studies in cohorts with competing events. Epidemiology. 2014;25(1):122–5. (PMID: 10.1097/EDE.000000000000002924240653) Sanderson J, Thompson SG, White IR, Aspelund T, Pennells L. Derivation and assessment of risk prediction models using case-cohort data. BMC Med Res Methodol. 2013;13:113. (PMID: 10.1186/1471-2288-13-113240341463848813) Sugiyama M, Krauledat M, Müller KR. Covariate shift adaptation by importance weighted cross validation. J Mach Learn Res. 2007;8:985–1005. |
| معلومات مُعتمدة: | EMCLSH21019 Health~Holland |
| فهرسة مساهمة: | Keywords: Nested case–control study; Prediction model validation; Rare outcomes; Weighted metrics |
| تواريخ الأحداث: | Date Created: 20240517 Date Completed: 20240518 Latest Revision: 20240517 |
| رمز التحديث: | 20240519 |
| DOI: | 10.1186/s12874-024-02213-6 |
| PMID: | 38760688 |
| قاعدة البيانات: | MEDLINE |
Find this article in full text from Springer Verlag. 
Full Text Finder | تدمد: | 1471-2288 |
|---|---|
| DOI: | 10.1186/s12874-024-02213-6 |