تقرير
DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models
| العنوان: | DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models |
|---|---|
| المؤلفون: | Liu, Yanming, Peng, Xinyue, Zhang, Yuwei, Ke, Xiaolan, Deng, Songhang, Cao, Jiannan, Ma, Chen, Fu, Mengchen, Zhang, Xuhong, Cheng, Sheng, Wang, Xun, Yin, Jianwei, Du, Tianyu |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Cryptography and Security, Computer Science - Artificial Intelligence, Computer Science - Computation and Language, Computer Science - Machine Learning |
| الوصف: | Large language models have repeatedly shown outstanding performance across diverse applications. However, deploying these models can inadvertently risk user privacy. The significant memory demands during training pose a major challenge in terms of resource consumption. This substantial size places a heavy load on memory resources, raising considerable practical concerns. In this paper, we introduce DP-MemArc, a novel training framework aimed at reducing the memory costs of large language models while emphasizing the protection of user data privacy. DP-MemArc incorporates side network or reversible network designs to support a variety of differential privacy memory-efficient fine-tuning schemes. Our approach not only achieves in memory optimization but also ensures robust privacy protection, keeping user data secure and confidential. Extensive experiments have demonstrated that DP-MemArc effectively provides differential privacy-efficient fine-tuning across different task scenarios. Comment: 9 pages second version |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2406.11087 |
| رقم الأكسشن: | edsarx.2406.11087 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |