تقرير
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models
| العنوان: | Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models |
|---|---|
| المؤلفون: | Sicilia, Anthony, Kim, Hyunwoo, Chandu, Khyathi Raghavi, Alikhani, Malihe, Hessel, Jack |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Computation and Language, Computer Science - Artificial Intelligence, Computer Science - Machine Learning |
| الوصف: | Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. But even the best human conversationalist cannot perfectly anticipate the trajectory of a dialogue. How well can language models represent inherent uncertainty in conversations? We propose FortUne Dial, an expansion of the long-standing "conversation forecasting" task: instead of just accuracy, evaluation is conducted with uncertainty-aware metrics, effectively enabling abstention on individual instances. We study two ways in which language models potentially represent outcome uncertainty (internally, using scores and directly, using tokens) and propose fine-tuning strategies to improve calibration of both representations. Experiments on eight difficult negotiation corpora demonstrate that our proposed fine-tuning strategies (a traditional supervision strategy and an off-policy reinforcement learning strategy) can calibrate smaller open-source models to compete with pre-trained models 10x their size. Comment: 2 Figures; 7 Tables; 27 pages |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2402.03284 |
| رقم الأكسشن: | edsarx.2402.03284 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |