تقرير
Noisy Interpolation Learning with Shallow Univariate ReLU Networks
| العنوان: | Noisy Interpolation Learning with Shallow Univariate ReLU Networks |
|---|---|
| المؤلفون: | Joshi, Nirmit, Vardi, Gal, Srebro, Nathan |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Machine Learning |
| الوصف: | Understanding how overparameterized neural networks generalize despite perfect interpolation of noisy training data is a fundamental question. Mallinar et. al. 2022 noted that neural networks seem to often exhibit ``tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error, but neither does it approach infinity, yielding non-trivial generalization. However, this has not been studied rigorously. We provide the first rigorous analysis of the overfitting behavior of regression with minimum norm ($\ell_2$ of weights), focusing on univariate two-layer ReLU networks. We show overfitting is tempered (with high probability) when measured with respect to the $L_1$ loss, but also show that the situation is more complex than suggested by Mallinar et. al., and overfitting is catastrophic with respect to the $L_2$ loss, or when taking an expectation over the training set. Comment: To appear at ICLR 2024. Updated version with minor changes in the presentation |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2307.15396 |
| رقم الأكسشن: | edsarx.2307.15396 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |