Proceedings

Increasingly Cautious Optimism for Practical PAC-MDP Exploration

Liangpeng Zhang, Ke Tang, Xin Yao

Download BibTeX Cite

Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence. Special Track on Machine Learning. Pages 4041-4047.

Abstract

Exploration strategy is an essential part of learning agents in model-based Reinforcement Learning. R-MAX and V-MAX are PAC-MDP strategies proved to have polynomial sample complexity; yet, their exploration behavior tend to be overly cautious in practice. We propose the principle of Increasingly Cautious Optimism (ICO) to automatically cut off unnecessarily cautious exploration, and apply ICO to R-MAX and V-MAX, yielding two new strategies, namely Increasingly Cautious R-MAX (ICR) and Increasingly Cautious V-MAX (ICV). We prove that both ICR and ICV are PACMDP, and show that their improvement is guaranteed by a tighter sample complexity upper bound. Then, we demonstrate their significantly improved performance through empirical results.

Liangpeng Zhang, Ke Tang, and Xin Yao. 2015. Increasingly Cautious Optimism for Practical PAC-MDP Exploration. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI-15. IJCAI, 4041–4047. https://ijcai.org/proceedings/2015/566
Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards