{"type":"ScholarlyArticle","title":"Increasingly Cautious Optimism for Practical PAC-MDP Exploration","authors":[{"name":"Liangpeng Zhang","slug":"liangpeng-zhang","given":"Liangpeng","family":"Zhang","affiliation":""},{"name":"Ke Tang","slug":"ke-tang","given":"Ke","family":"Tang","affiliation":""},{"name":"Xin Yao","slug":"xin-yao","given":"Xin","family":"Yao","affiliation":""}],"abstract":"Exploration strategy is an essential part of learning agents in model-based Reinforcement Learning. R-MAX and V-MAX are PAC-MDP strategies proved to have polynomial sample complexity; yet, their exploration behavior tend to be overly cautious in practice. We propose the principle of Increasingly Cautious Optimism (ICO) to automatically cut off unnecessarily cautious exploration, and apply ICO to R-MAX and V-MAX, yielding two new strategies, namely Increasingly Cautious R-MAX (ICR) and Increasingly Cautious V-MAX (ICV). We prove that both ICR and ICV are PACMDP, and show that their improvement is guaranteed by a tighter sample complexity upper bound. Then, we demonstrate their significantly improved performance through empirical results.","doi":null,"url":"https://ijcai.org/proceedings/2015/566","pdf":"https://ijcai.org/proceedings/2015/0566.pdf","markdown":"https://ijcai.org/proceedings/2015/566.md","bibtex":"https://ijcai.org/proceedings/2015/bibtex/566","volume":{"slug":"2015","title":"Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2015","url":"https://ijcai.org/proceedings/2015/","year":2015,"location":"Buenos Aires, Argentina","dates":"25–31 July 2015","editors":"Qiang Yang and Michael Wooldridge","isbn":"978-1-57735-738-4","publisher":"International Joint Conferences on Artificial Intelligence","doi_prefix":""},"section":"Special Track on Machine Learning","track":"Special Track on Machine Learning","number":566,"pages":{"first":4041,"last":4047},"keywords":[],"videos":[],"references":[],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}