تقرير
Efficient Training of Deep Equilibrium Models
| العنوان: | Efficient Training of Deep Equilibrium Models |
|---|---|
| المؤلفون: | Nguyen, Bac, Mauch, Lukas |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Computer Science - Artificial Intelligence |
| الوصف: | Deep equilibrium models (DEQs) have proven to be very powerful for learning data representations. The idea is to replace traditional (explicit) feedforward neural networks with an implicit fixed-point equation, which allows to decouple the forward and backward passes. In particular, training DEQ layers becomes very memory-efficient via the implicit function theorem. However, backpropagation through DEQ layers still requires solving an expensive Jacobian-based equation. In this paper, we introduce a simple but effective strategy to avoid this computational burden. Our method relies on the Jacobian approximation of Broyden's method after the forward pass to compute the gradients during the backward pass. Experiments show that simply re-using this approximation can significantly speed up the training while not causing any performance degradation. Comment: ICML 2022 Workshop |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2304.11663 |
| رقم الأكسشن: | edsarx.2304.11663 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |