When Pairs Meet Triplets: Improving Low-Resource Captioning via Multi-Objective Optimization
| العنوان: | When Pairs Meet Triplets: Improving Low-Resource Captioning via Multi-Objective Optimization |
|---|---|
| المؤلفون: | Yike Wu, Shiwan Zhao, Ying Zhang, Xiaojie Yuan, Zhong Su |
| المصدر: | ACM Transactions on Multimedia Computing, Communications, and Applications. 18:1-20 |
| بيانات النشر: | Association for Computing Machinery (ACM), 2022. |
| سنة النشر: | 2022 |
| مصطلحات موضوعية: | Computer Networks and Communications, Hardware and Architecture, Computer Science::Computation and Language (Computational Linguistics and Natural Language and Speech Processing) |
| الوصف: | Image captioning for low-resource languages has attracted much attention recently. Researchers propose to augment the low-resource caption dataset into (image, rich-resource language, and low-resource language) triplets and develop the dual attention mechanism to exploit the existence of triplets in training to improve the performance. However, datasets in triplet form are usually small due to their high collecting cost. On the other hand, there are already many large-scale datasets, which contain one pair from the triplet, such as caption datasets in the rich-resource language and translation datasets from the rich-resource language to the low-resource language. In this article, we revisit the caption-translation pipeline of the translation-based approach to utilize not only the triplet dataset but also large-scale paired datasets in training. The caption-translation pipeline is composed of two models, one caption model of the rich-resource language and one translation model from the rich-resource language to the low-resource language. Unfortunately, it is not trivial to fully benefit from incorporating both the triplet dataset and paired datasets into the pipeline, due to the gap between the training and testing phases and the instability in the training process. We propose to jointly optimize the two models of the pipeline in an end-to-end manner to bridge the training and testing gap, and introduce two auxiliary training objectives to stabilize the training process. Experimental results show that the proposed method improves significantly over the state-of-the-art methods. |
| تدمد: | 1551-6865 1551-6857 |
| URL الوصول: | https://explore.openaire.eu/search/publication?articleId=doi_________::d1ebe45cfa7e649fcd66282da7c29546 https://doi.org/10.1145/3492325 |
| رقم الأكسشن: | edsair.doi...........d1ebe45cfa7e649fcd66282da7c29546 |
| قاعدة البيانات: | OpenAIRE |
| تدمد: | 15516865 15516857 |
|---|