دورية أكاديمية
Clustering swap prediction for image-text pre-training
| العنوان: | Clustering swap prediction for image-text pre-training |
|---|---|
| المؤلفون: | Sun Fayou, Hea Choon Ngo, Yong Wee Sek, Zuqiang Meng |
| المصدر: | Scientific Reports, Vol 14, Iss 1, Pp 1-16 (2024) |
| بيانات النشر: | Nature Portfolio, 2024. |
| سنة النشر: | 2024 |
| المجموعة: | LCC:Medicine LCC:Science |
| مصطلحات موضوعية: | Model pre-training, Clustering learning, Swap prediction, Cluster number, Medicine, Science |
| الوصف: | Abstract It is essential to delve into the strategy of multimodal model pre-training, which is an obvious impact on downstream tasks. Currently, clustering learning has achieved noteworthy benefits in multiple methods. However, due to the availability of open image-text pairs, it is challenging for multimodal with clustering learning. In this paper, we propose an approach that utilizes clustering swap prediction strategy to learn image-text clustering embedding space by interaction prediction between image and text features. Unlike existing models with clustering learning, our method (Clus) allows for an open number of clusters for web-scale alt-text data. Furthermore, in order to train the image and text encoders efficiently, we introduce distillation learning approach and evaluate the performance of the image-encoder in downstream visual tasks. In addition, Clus is pre-trained end-to-end by using large-scale image-text pairs. Specifically, both text and image serve as ground truth for swap prediction, enabling effective representation learning. Concurrently, extensive experiments demonstrate that Clus achieves state-of-the-art performance on multiple downstream fine-tuning and zero-shot tasks (i.e., Image-Text Retrieval, VQA, NLVR2, Image Captioning, Object Detection, and Semantic Segmentation). |
| نوع الوثيقة: | article |
| وصف الملف: | electronic resource |
| اللغة: | English |
| تدمد: | 2045-2322 |
| Relation: | https://doaj.org/toc/2045-2322 |
| DOI: | 10.1038/s41598-024-60832-x |
| URL الوصول: | https://doaj.org/article/a8ac5e46876c44c587def0f7e7f467fa |
| رقم الأكسشن: | edsdoj.8ac5e46876c44c587def0f7e7f467fa |
| قاعدة البيانات: | Directory of Open Access Journals |
Find this article in full text from Springer Verlag. 
Full Text Finder | تدمد: | 20452322 |
|---|---|
| DOI: | 10.1038/s41598-024-60832-x |