تقرير
Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training
| العنوان: | Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training |
|---|---|
| المؤلفون: | Zhang, Lisai, Chen, Qingcai, Chen, Zhijian, Han, Yunpeng, Li, Zhonghua, Cao, Zhao |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computer Vision and Pattern Recognition, Computer Science - Computation and Language |
| الوصف: | Fine-grained supervision based on object annotations has been widely used for vision and language pre-training (VLP). However, in real-world application scenarios, aligned multi-modal data is usually in the image-caption format, which only provides coarse-grained supervision. It is not only cost-expensive but also compute-expensive to collect object annotations and build object annotation pre-extractor for different scenarios. In this paper, we propose a fine-grained VLP scheme without object annotations from the linguistic perspective. First, we propose a homonym sentence rewriting (HSR) algorithm to provide token-level supervision. The algorithm replaces a verb/noun/adjective/quantifier word of the caption with its homonyms from WordNet. Correspondingly, we propose refined vision-language modeling (RVLM) framework to exploit the token-level supervision. Three refined tasks, i.e., refined image-text contrastive (RITC), refined image-text matching (RITM), and replace language modeling (RLM) are proposed to learn the fine-grained alignment. Extensive experiments on several downstream tasks demonstrate the superior performance of the proposed method. Comment: Work in progress, v0.2 |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2303.05313 |
| رقم الأكسشن: | edsarx.2303.05313 |
| قاعدة البيانات: | arXiv |
كن أول من يترك تعليقا!