تقرير
UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization
| العنوان: | UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization |
|---|---|
| المؤلفون: | Jiang, Yiming, Liu, Jinlan, Xu, Dongpo, Mandic, Danilo P. |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science Mathematics |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Mathematics - Numerical Analysis, Mathematics - Optimization and Control |
| الوصف: | Adam-type algorithms have become a preferred choice for optimisation in the deep learning setting, however, despite success, their convergence is still not well understood. To this end, we introduce a unified framework for Adam-type algorithms (called UAdam). This is equipped with a general form of the second-order moment, which makes it possible to include Adam and its variants as special cases, such as NAdam, AMSGrad, AdaBound, AdaFom, and Adan. This is supported by a rigorous convergence analysis of UAdam in the non-convex stochastic setting, showing that UAdam converges to the neighborhood of stationary points with the rate of $\mathcal{O}(1/T)$. Furthermore, the size of neighborhood decreases as $\beta$ increases. Importantly, our analysis only requires the first-order momentum factor to be close enough to 1, without any restrictions on the second-order momentum factor. Theoretical results also show that vanilla Adam can converge by selecting appropriate hyperparameters, which provides a theoretical guarantee for the analysis, applications, and further developments of the whole class of Adam-type algorithms. |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2305.05675 |
| رقم الأكسشن: | edsarx.2305.05675 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |