تقرير
Shuffle & Divide: Contrastive Learning for Long Text
| العنوان: | Shuffle & Divide: Contrastive Learning for Long Text |
|---|---|
| المؤلفون: | Lee, Joonseok, Joe, Seongho, Park, Kyoungwon, Kim, Bogun, Kang, Hoyoung, Park, Jaeseon, Gwon, Youngjune |
| المصدر: | 2022 26th International Conference on Pattern Recognition (ICPR), Montreal, QC, Canada, 2022, pp. 2935-2941 |
| سنة النشر: | 2023 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language, Computer Science - Machine Learning |
| الوصف: | We propose a self-supervised learning method for long text documents based on contrastive learning. A key to our method is Shuffle and Divide (SaD), a simple text augmentation algorithm that sets up a pretext task required for contrastive updates to BERT-based document embedding. SaD splits a document into two sub-documents containing randomly shuffled words in the entire documents. The sub-documents are considered positive examples, leaving all other documents in the corpus as negatives. After SaD, we repeat the contrastive update and clustering phases until convergence. It is naturally a time-consuming, cumbersome task to label text documents, and our method can help alleviate human efforts, which are most expensive resources in AI. We have empirically evaluated our method by performing unsupervised text classification on the 20 Newsgroups, Reuters-21578, BBC, and BBCSport datasets. In particular, our method pushes the current state-of-the-art, SS-SB-MT, on 20 Newsgroups by 20.94% in accuracy. We also achieve the state-of-the-art performance on Reuters-21578 and exceptionally-high accuracy performances (over 95%) for unsupervised classification on the BBC and BBCSport datasets. Comment: Accepted at ICPR 2022 |
| نوع الوثيقة: | Working Paper |
| DOI: | 10.1109/ICPR56361.2022.9956208 |
| URL الوصول: | http://arxiv.org/abs/2304.09374 |
| رقم الأكسشن: | edsarx.2304.09374 |
| قاعدة البيانات: | arXiv |
| DOI: | 10.1109/ICPR56361.2022.9956208 |
|---|