تقرير
Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
| العنوان: | Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape |
|---|---|
| المؤلفون: | Karhadkar, Kedar, Murray, Michael, Tseran, Hanna, Montúfar, Guido |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science Mathematics Statistics |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Mathematics - Combinatorics, Statistics - Machine Learning |
| الوصف: | We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization. Comment: 40 pages |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2305.19510 |
| رقم الأكسشن: | edsarx.2305.19510 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |