تقرير
Width and Depth Limits Commute in Residual Networks
| العنوان: | Width and Depth Limits Commute in Residual Networks |
|---|---|
| المؤلفون: | Hayou, Soufiane, Yang, Greg |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science Statistics |
| مصطلحات موضوعية: | Statistics - Machine Learning, Computer Science - Machine Learning |
| الوصف: | We show that taking the width and depth to infinity in a deep neural network with skip connections, when branches are scaled by $1/\sqrt{depth}$ (the only nontrivial scaling), result in the same covariance structure no matter how that limit is taken. This explains why the standard infinite-width-then-depth approach provides practical insights even for networks with depth of the same order as width. We also demonstrate that the pre-activations, in this case, have Gaussian distributions which has direct applications in Bayesian deep learning. We conduct extensive simulations that show an excellent match with our theoretical findings. Comment: 24 pages, 8 figures. arXiv admin note: text overlap with arXiv:2210.00688 |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2302.00453 |
| رقم الأكسشن: | edsarx.2302.00453 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |