تقرير
Prior-dependent analysis of posterior sampling reinforcement learning with function approximation
العنوان: | Prior-dependent analysis of posterior sampling reinforcement learning with function approximation |
---|---|
المؤلفون: | Li, Yingru, Luo, Zhi-Quan |
سنة النشر: | 2024 |
المجموعة: | Computer Science Mathematics Statistics |
مصطلحات موضوعية: | Statistics - Machine Learning, Computer Science - Artificial Intelligence, Computer Science - Information Theory, Computer Science - Machine Learning, Mathematics - Statistics Theory |
الوصف: | This work advances randomized exploration in reinforcement learning (RL) with function approximation modeled by linear mixture MDPs. We establish the first prior-dependent Bayesian regret bound for RL with function approximation; and refine the Bayesian regret analysis for posterior sampling reinforcement learning (PSRL), presenting an upper bound of ${\mathcal{O}}(d\sqrt{H^3 T \log T})$, where $d$ represents the dimensionality of the transition kernel, $H$ the planning horizon, and $T$ the total number of interactions. This signifies a methodological enhancement by optimizing the $\mathcal{O}(\sqrt{\log T})$ factor over the previous benchmark (Osband and Van Roy, 2014) specified to linear mixture MDPs. Our approach, leveraging a value-targeted model learning perspective, introduces a decoupling argument and a variance reduction technique, moving beyond traditional analyses reliant on confidence sets and concentration inequalities to formalize Bayesian regret bounds more effectively. Comment: Published in the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) |
نوع الوثيقة: | Working Paper |
URL الوصول: | http://arxiv.org/abs/2403.11175 |
رقم الأكسشن: | edsarx.2403.11175 |
قاعدة البيانات: | arXiv |
الوصف غير متاح. |