تقرير
Position Interpolation Improves ALiBi Extrapolation
| العنوان: | Position Interpolation Improves ALiBi Extrapolation |
|---|---|
| المؤلفون: | Al-Khateeb, Faisal, Dey, Nolan, Soboleva, Daria, Hestness, Joel |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Machine Learning |
| الوصف: | Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks. Comment: 4 pages content, 1 page references, 4 figures |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2310.13017 |
| رقم الأكسشن: | edsarx.2310.13017 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |