تقرير
Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
العنوان: | Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance |
---|---|
المؤلفون: | Lu, Haiquan, Liu, Xiaotian, Zhou, Yefan, Li, Qunli, Keutzer, Kurt, Mahoney, Michael W., Yan, Yujun, Yang, Huanrui, Yang, Yaoqing |
سنة النشر: | 2024 |
المجموعة: | Computer Science Statistics |
مصطلحات موضوعية: | Statistics - Machine Learning, Computer Science - Machine Learning |
الوصف: | Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. |
نوع الوثيقة: | Working Paper |
URL الوصول: | http://arxiv.org/abs/2407.12996 |
رقم الأكسشن: | edsarx.2407.12996 |
قاعدة البيانات: | arXiv |
الوصف غير متاح. |