تقرير
FedPFT: Federated Proxy Fine-Tuning of Foundation Models
| العنوان: | FedPFT: Federated Proxy Fine-Tuning of Foundation Models |
|---|---|
| المؤلفون: | Peng, Zhaopeng, Fan, Xiaoliang, Chen, Yufan, Wang, Zheng, Pan, Shirui, Wen, Chenglu, Zhang, Ruisheng, Wang, Cheng |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Computer Science - Artificial Intelligence |
| الوصف: | Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, however, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing those crucial neurons. Second, the sub-FM alignment module conducts a two-step distillations-layer-level and neuron-level-before and during FL fine-tuning respectively, to reduce error of gradient by accurately aligning sub-FM with FM under theoretical guarantees. Experimental results on seven commonly used datasets (i.e., four text and three vision) demonstrate the superiority of FedPFT. Comment: Accepted by IJCAI'24 |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2404.11536 |
| رقم الأكسشن: | edsarx.2404.11536 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |