تقرير
Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
| العنوان: | Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance |
|---|---|
| المؤلفون: | Lu, Haiquan, Liu, Xiaotian, Zhou, Yefan, Li, Qunli, Keutzer, Kurt, Mahoney, Michael W., Yan, Yujun, Yang, Huanrui, Yang, Yaoqing |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science Statistics |
| مصطلحات موضوعية: | Statistics - Machine Learning, Computer Science - Machine Learning |
| الوصف: | Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2407.12996 |
| رقم الأكسشن: | edsarx.2407.12996 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |