تقرير
K-SAM: Sharpness-Aware Minimization at the Speed of SGD
| العنوان: | K-SAM: Sharpness-Aware Minimization at the Speed of SGD |
|---|---|
| المؤلفون: | Ni, Renkun, Chiang, Ping-yeh, Geiping, Jonas, Goldblum, Micah, Wilson, Andrew Gordon, Goldstein, Tom |
| سنة النشر: | 2022 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Computer Science - Computer Vision and Pattern Recognition |
| الوصف: | Sharpness-Aware Minimization (SAM) has recently emerged as a robust technique for improving the accuracy of deep neural networks. However, SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. The computational challenge posed by SAM arises because each iteration requires both ascent and descent steps and thus double the gradient computations. To address this challenge, we propose to compute gradients in both stages of SAM on only the top-k samples with highest loss. K-SAM is simple and extremely easy-to-implement while providing significant generalization boosts over vanilla SGD at little to no additional cost. Comment: 13 pages, 2 figures |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2210.12864 |
| رقم الأكسشن: | edsarx.2210.12864 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |