تقرير
Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization
| العنوان: | Multi-View Knowledge Distillation from Crowd Annotations for Out-of-Domain Generalization |
|---|---|
| المؤلفون: | Wright, Dustin, Augenstein, Isabelle |
| سنة النشر: | 2022 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language |
| الوصف: | Selecting an effective training signal for tasks in natural language processing is difficult: expert annotations are expensive, and crowd-sourced annotations may not be reliable. At the same time, recent work in NLP has demonstrated that learning from a distribution over labels acquired from crowd annotations can be effective. However, there are many ways to acquire such a distribution, and the performance allotted by any one method can fluctuate based on the task and the amount of available crowd annotations, making it difficult to know a priori which distribution is best. This paper systematically analyzes this in the out-of-domain setting, adding to the NLP literature which has focused on in-domain evaluation, and proposes new methods for acquiring soft-labels from crowd-annotations by aggregating the distributions produced by existing methods. In particular, we propose to aggregate multiple-views of crowd annotations via temperature scaling and finding their Jensen-Shannon centroid. We demonstrate that these aggregation methods lead to the most consistent performance across four NLP tasks on out-of-domain test sets, mitigating fluctuations in performance from the individual distributions. Additionally, aggregation results in the most consistently well-calibrated uncertainty estimation. We argue that aggregating different views of crowd-annotations is an effective and minimal intervention to acquire soft-labels which induce robust classifiers despite the inconsistency of the individual soft-labeling methods. Comment: 14 pages, 4 figures, 1 table |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2212.09409 |
| رقم الأكسشن: | edsarx.2212.09409 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |