تقرير
Listwise Reward Estimation for Offline Preference-based Reinforcement Learning
| العنوان: | Listwise Reward Estimation for Offline Preference-based Reinforcement Learning |
|---|---|
| المؤلفون: | Choi, Heewoong, Jung, Sangwon, Ahn, Hongjoon, Moon, Taesup |
| سنة النشر: | 2024 |
| المجموعة: | Computer Science |
| مصطلحات موضوعية: | Computer Science - Machine Learning, Computer Science - Artificial Intelligence |
| الوصف: | In Reinforcement Learning (RL), designing precise reward functions remains to be a challenge, particularly when aligning with human intent. Preference-based RL (PbRL) was introduced to address this problem by learning reward models from human feedback. However, existing PbRL methods have limitations as they often overlook the second-order preference that indicates the relative strength of preference. In this paper, we propose Listwise Reward Estimation (LiRE), a novel approach for offline PbRL that leverages second-order preference information by constructing a Ranked List of Trajectories (RLT), which can be efficiently built by using the same ternary feedback type as traditional methods. To validate the effectiveness of LiRE, we propose a new offline PbRL dataset that objectively reflects the effect of the estimated rewards. Our extensive experiments on the dataset demonstrate the superiority of LiRE, i.e., outperforming state-of-the-art baselines even with modest feedback budgets and enjoying robustness with respect to the number of feedbacks and feedback noise. Our code is available at https://github.com/chwoong/LiRE Comment: 21 pages, ICML 2024 |
| نوع الوثيقة: | Working Paper |
| URL الوصول: | http://arxiv.org/abs/2408.04190 |
| رقم الأكسشن: | edsarx.2408.04190 |
| قاعدة البيانات: | arXiv |
| الوصف غير متاح. |