تقرير
NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos
العنوان: | NPF-200: A Multi-Modal Eye Fixation Dataset and Method for Non-Photorealistic Videos |
---|---|
المؤلفون: | Yang, Ziyu, Ren, Sucheng, Wu, Zongwei, Zhao, Nanxuan, Wang, Junle, Qin, Jing, He, Shengfeng |
سنة النشر: | 2023 |
المجموعة: | Computer Science |
مصطلحات موضوعية: | Computer Science - Computer Vision and Pattern Recognition |
الوصف: | Non-photorealistic videos are in demand with the wave of the metaverse, but lack of sufficient research studies. This work aims to take a step forward to understand how humans perceive non-photorealistic videos with eye fixation (\ie, saliency detection), which is critical for enhancing media production, artistic design, and game user experience. To fill in the gap of missing a suitable dataset for this research line, we present NPF-200, the first large-scale multi-modal dataset of purely non-photorealistic videos with eye fixations. Our dataset has three characteristics: 1) it contains soundtracks that are essential according to vision and psychological studies; 2) it includes diverse semantic content and videos are of high-quality; 3) it has rich motions across and within videos. We conduct a series of analyses to gain deeper insights into this task and compare several state-of-the-art methods to explore the gap between natural images and non-photorealistic data. Additionally, as the human attention system tends to extract visual and audio features with different frequencies, we propose a universal frequency-aware multi-modal non-photorealistic saliency detection model called NPSNet, demonstrating the state-of-the-art performance of our task. The results uncover strengths and weaknesses of multi-modal network design and multi-domain training, opening up promising directions for future works. {Our dataset and code can be found at \url{https://github.com/Yangziyu/NPF200}}. Comment: Accepted by ACM MM 2023 |
نوع الوثيقة: | Working Paper |
URL الوصول: | http://arxiv.org/abs/2308.12163 |
رقم الأكسشن: | edsarx.2308.12163 |
قاعدة البيانات: | arXiv |
الوصف غير متاح. |