Neural Network Language Model Compression With Product Quantization and Soft Binarization
| العنوان: | Neural Network Language Model Compression With Product Quantization and Soft Binarization |
|---|---|
| المؤلفون: | Qi Liu, Kai Yu, Kaiyu Shi, Rao Ma |
| المصدر: | IEEE/ACM Transactions on Audio, Speech, and Language Processing. 28:2438-2449 |
| بيانات النشر: | Institute of Electrical and Electronics Engineers (IEEE), 2020. |
| سنة النشر: | 2020 |
| مصطلحات موضوعية: | Word embedding, Acoustics and Ultrasonics, Artificial neural network, Computer science, business.industry, Quantization (signal processing), Word error rate, Pattern recognition, Cartesian product, Matrix decomposition, 030507 speech-language pathology & audiology, 03 medical and health sciences, Computational Mathematics, symbols.namesake, Compression ratio, Computer Science (miscellaneous), symbols, Artificial intelligence, Electrical and Electronic Engineering, 0305 other medical science, business, Subspace topology |
| الوصف: | Large memory consumption of the neural network language models (NN LMs) prohibits their use in many resource-constrained scenarios. Hence, effective NN LM compression approaches that are independent of NN structures are of great interest. However, previous approaches usually achieve a high compression ratio at the cost of obvious performance loss. In this paper, two recently proposed quantization approaches, product quantization (PQ) and soft binarization are effectively combined to address the issue. PQ decomposes word embedding matrices into a Cartesian product of low dimensional subspaces and quantizes each subspace separately. Soft binarization uses a small number of float scalars and the knowledge distillation technique to recover the performance loss during the binarization. Experiments show that the proposed approaches can achieve a high compression ratio, from 70 to over 100, while still maintaining comparable performance to the uncompressed NN LM on both PPL and word error rate criteria. |
| تدمد: | 2329-9304 2329-9290 |
| URL الوصول: | https://explore.openaire.eu/search/publication?articleId=doi_________::4fec87e64e72c1fb47ca8b9b659ebdca https://doi.org/10.1109/taslp.2020.3015659 |
| حقوق: | CLOSED |
| رقم الأكسشن: | edsair.doi...........4fec87e64e72c1fb47ca8b9b659ebdca |
| قاعدة البيانات: | OpenAIRE |
| تدمد: | 23299304 23299290 |
|---|