{"type":"ScholarlyArticle","title":"Measuring Structural Similarities in Finite MDPs","authors":[{"name":"Hao Wang","slug":"hao-wang","given":"Hao","family":"Wang","affiliation":"Inception Institute of Artificial Intelligence , UAE; State Key Laboratory for Novel Software Technology , Nanjing University."},{"name":"Shaokang Dong","slug":"shaokang-dong","given":"Shaokang","family":"Dong","affiliation":"State Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Ling Shao","slug":"ling-shao","given":"Ling","family":"Shao","affiliation":"Inception Institute of Artificial Intelligence , UAE; State Key Laboratory for Novel Software Technology , Nanjing University."}],"abstract":"In this paper, we investigate the structural similarities within a finite Markov decision process (MDP). We view a finite MDP as a heterogeneous directed bipartite graph and propose novel measures for state similarity and action similarity in a mutual reinforcement manner. We prove that the state similarity is a metric and the action similarity is a pseudometric. We also establish the connection between the proposed similarity measures and the optimal values of the MDP. Extensive experiments show that the proposed measures are effective.","doi":"10.24963/ijcai.2019/511","url":"https://ijcai.org/proceedings/2019/511","pdf":"https://ijcai.org/proceedings/2019/0511.pdf","markdown":"https://ijcai.org/proceedings/2019/511.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/511","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":511,"pages":{"first":3684,"last":3690},"keywords":["Planning and Scheduling: Markov Decisions Processes","Planning and Scheduling: Planning under Uncertainty","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Reinforcement Learning"],"videos":[],"references":[{"position":1,"raw":"[Abel et al., 2019] David Abel, Dilip Arumugam, Kavosh Asadi, Yuu Jinnai, Michael L. Littman, and Lawson L.S. Wong. State abstraction as compression in apprenticeship learning. In AAAI, 2019.","title":"State abstraction as compression in apprenticeship learning","authors":"David Abel, Dilip Arumugam, Kavosh Asadi, Yuu Jinnai, Michael L Littman, Lawson L S Wong","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Abel, 2019] David Abel. A theory of state abstraction for reinforcement learning. In AAAI, Doctoral Consortium, 2019.","title":"A theory of state abstraction for reinforcement learning","authors":"David Abel","year":"2019","venue":"AAAI, Doctoral Consortium","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ammar et al., 2014] Haitham Bou Ammar, Eric Eaton, Matthew E. Taylor, Decebal Constantin Mocanu, Kurt Driessens, Gerhard Weiss, and Karl Tuyls. An auto- mated measure of MDP similarity for transfer reinforce- ment learning. In AAAI, 2014.","title":"An automated measure of MDP similarity for transfer reinforcement learning","authors":"Haitham Bou Ammar, Eric Eaton, Matthew E Taylor, Constantin Mocanu, Kurt Driessens, Gerhard Weiss, Karl Tuyls","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Antonellis et al., 2008] Ioannis Antonellis, Hector Garcia Molina, and Chi Chao Chang. Simrank++: query rewriting through link analysis of the click graph. In VLDB, 2008.","title":"Simrank++: query rewriting through link analysis of the click graph","authors":"Ioannis Antonellis, Hector Garcia Molina, Chi Chao, Chang","year":"2008","venue":"VLDB","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Brunsch et al., 2013] Tobias Brunsch, Kamiel Cornelissen, Bodo Manthey, and Heiko Röglin. Smoothed analysis of the successive shortest path algorithm. In SODA, 2013.","title":"Smoothed analysis of the successive shortest path algorithm","authors":"Brunsch","year":"2013","venue":"SODA","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Cai et al., 2017] Han Cai, Kan Ren, Weinan Zhang, Klean- this Malialis, Jun Wang, Yong Yu, and Defeng Guo. Real- time bidding by reinforcement learning in display adver- tising. In WSDM, 2017.","title":"Realtime bidding by reinforcement learning in display advertising","authors":"Han Cai, Kan Ren, Weinan Zhang, Kleanthis Malialis, Jun Wang, Yong Yu, Defeng Guo","year":"2017","venue":"WSDM","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Delfour and Zolésio, 2011] Michel C. Delfour and Jean- Paul Zolésio. Shapes and Geometrics: Metrics, Analysis, Differential Calculus, and Optimization. Advances in De- sign and Control. SIAM, 2nd edition, 2011.","title":"Shapes and Geometrics: Metrics, Analysis, Differential Calculus, and Optimization","authors":"C Michel, Jean-Paul Delfour, Zolésio","year":"2011","venue":"Advances in Design and Control","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ferns et al., 2004] Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite Markov decision pro- cesses. In UAI, 2004.","title":"Metrics for finite Markov decision processes","authors":"Ferns","year":"2004","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Givan et al., 2003] Robert Givan, Thomas Dean, and Matthew Greig. Equivalence notions and model minimiza- tion in Markov decision processes. Artificial Intelligence, 147:163-223, 2003.","title":"Equivalence notions and model minimization in Markov decision processes","authors":"Robert Givan, Thomas Dean, Matthew Greig","year":"2003","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gopalan et al., 2017] Nakul Gopalan, Marie desJardins, Michael L. Littman, James MacGlashan, Shawn Squire, Stefanie Tellex, John Winder, and Lawson L.S. Wong. Planning with abstract Markov decision processes. In ICAPS, 2017.","title":"Planning with abstract Markov decision processes","authors":"Nakul Gopalan, Marie Desjardins, Michael L Littman, James Macglashan, Shawn Squire, Stefanie Tellex, John Winder, Lawson L S Wong","year":"2017","venue":"ICAPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Han et al., 2016] Zhenhua Han, Haisheng Tan, Guihai Chen, Rui Wang, Yifan Chen, and Francis C.M. Lau. Dynamic virtual machine management via approximate Markov decision process. In INFOCOM, 2016.","title":"Dynamic virtual machine management via approximate Markov decision process","authors":"Zhenhua Han, Haisheng Tan, Guihai Chen, Rui Wang, Yifan Chen, Francis C M Lau","year":"2016","venue":"INFOCOM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jeh and Widom, 2002] Glen Jeh and Jennifer Widom. Sim- Rank: a measure of structural-context similarity. In KDD, 2002.","title":"Sim-Rank: a measure of structural-context similarity","authors":"Widom Jeh, Glen Jeh, Jennifer Widom","year":"2002","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jeh and Widom, 2003] Glen Jeh and Jennifer Widom. Scal- ing personalized web search. In WWW, 2003.","title":"Scaling personalized web search","authors":"Widom Jeh, Glen Jeh, Jennifer Widom","year":"2003","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Jewell, 1962] William S. Jewell. Optimal flow through net- works. Operations Research, 10(4):476-499, 1962.","title":"Optimal flow through networks","authors":"William S Jewell","year":"1962","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Jin et al., 2014] Ruoming Jin, Victor E. Lee, and Longjie Li. Scalable and axiomatic ranking of network role similarity. TKDD, 8(1):Article No. 3, 2014.","title":"Scalable and axiomatic ranking of network role similarity","authors":"Jin","year":"2014","venue":"TKDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jong and Stone, 2005] Nicholas K. Jong and Peter Stone. State abstraction discovery from irrelevant state variables. In IJCAI, 2005.","title":"State abstraction discovery from irrelevant state variables","authors":"Nicholas K Jong, Peter Stone","year":"2005","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2005/228"},{"position":17,"raw":"[Lazaric et al., 2008] Alessandro Lazaric, Marcello Restelli, and Andrea Bonarini. Transfer of samples in batch rein- forcement learning. In ICML, 2008.","title":"Transfer of samples in batch reinforcement learning","authors":"Alessandro Lazaric, Marcello Restelli, Andrea Bonarini","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Li et al., 2006] Lihong Li, Thomas J. Walsh, and Michael L. Littman. Towards a unified theory of state abstraction for MDPs. In ISAIM, 2006.","title":"Towards a unified theory of state abstraction for MDPs","authors":"Lihong Li, Thomas J Walsh, Michael L Littman","year":"2006","venue":"ISAIM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Lin et al., 2012] Zhenjiang Lin, Michael R. Lyu, and Irwin King. MatchSim: a novel similarity measure based on maximum neighborhood matching. KAIS, 32(1):141-166, 2012.","title":"MatchSim: a novel similarity measure based on maximum neighborhood matching","authors":"Zhenjiang Lin, Michael R Lyu, Irwin King","year":"2012","venue":"KAIS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ramamoorthy et al., 2013] Subramanian Ramamoorthy, M. M. Hassan Mahmud, Majd Hawasly, and Benjamin Rosman. Clustering Markov decision processes for con- tinual transfer. Technical report, University of Edinburgh, 2013.","title":"Clustering Markov decision processes for continual transfer","authors":"M M Subramanian Ramamoorthy, Majd Hassan Mahmud, Benjamin Hawasly, Rosman","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ravindran and Barto, 2003] Balaraman Ravindran and An- drew G. Barto. SDMP homomorphisms: an algebraic ap- proach to abstraction in semi-Markov decision processes. In IJCAI, 2003.","title":"SDMP homomorphisms: an algebraic approach to abstraction in semi-Markov decision processes","authors":"Barto Ravindran, Balaraman Ravindran, Andrew G Barto","year":"2003","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rubner et al., 1998] Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. A metric for distributions with ap- plications to image databases. In ICCV, 1998.","title":"A metric for distributions with applications to image databases","authors":"Yossi Rubner, Carlo Tomasi, Leonidas J Guibas","year":"1998","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sinapov et al., 2015] Jivko Sinapov, Sanmit Narvekar, Mat- teo Leonetti, and Peter Stone. Learning inter-task transfer- ability in the absence of target task samples. In AAMAS, 2015.","title":"Learning inter-task transferability in the absence of target task samples","authors":"Jivko Sinapov, Sanmit Narvekar, Matteo Leonetti, Peter Stone","year":"2015","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Song et al., 2016] Jinhua Song, Yang Gao, Hao Wang, and Bo An. Measuring the distance between finite Markov de- cision processes. In AAMAS, 2016.","title":"Measuring the distance between finite Markov decision processes","authors":"Jinhua Song, Yang Gao, Hao Wang, Bo An","year":"2016","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sorg and Singh, 2009] Jonathan Sorg and Satinder Singh. Transfer via soft homomorphisms. In AAMAS, 2009.","title":"Transfer via soft homomorphisms","authors":"Singh Sorg, Jonathan Sorg, Satinder Singh","year":"2009","venue":"AAMAS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2nd edition, 2018.","title":"Reinforcement Learning: An Introduction","authors":"Richard S Sutton, Andrew G Barto","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Taylor and Stone, 2009] Matthew E. Taylor and Peter Stone. Transfer learning for reinforcement learning do- mains: a survey. JMLR, 10:1633-1685, 2009.","title":"Transfer learning for reinforcement learning domains: a survey","authors":"Matthew E Taylor, Peter Stone","year":"2009","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Taylor and Stone, 2011] Matthew E. Taylor and Peter Stone. An introduction to intertask transfer for reinforce- ment learning. AI Magzine, 32(1):15-34, 2011.","title":"An introduction to intertask transfer for reinforcement learning","authors":"Matthew E Taylor, Peter Stone","year":"2011","venue":"AI Magzine","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Venturini, 2015] Gabriel Martos Venturini. Statistical dis- tances and probability metrics for multivariate data, en- sembles and probability distributions. PhD thesis, Univer- sidad Carlos III de Madrid, 2015.","title":"Statistical distances and probability metrics for multivariate data, ensembles and probability distributions","authors":"Martos Gabriel, Venturini","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xi et al., 2005] Wensi Xi, Edward A. Fox, Weiguo Fan, Benyu Zhang, Zheng Chen, Jun Yan, and Dong Zhuang. SimFusion: measuring similarity using unified relation- ship matrix. In SIGIR, 2005.","title":"SimFusion: measuring similarity using unified relationship matrix","authors":"Wensi Xi, Edward A Fox, Weiguo Fan, Benyu Zhang, Zheng Chen, Jun Yan, Dong Zhuang","year":"2005","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhao et al., 2009] Peixiang Zhao, Jiawei Han, and Yizhou Sun. P-Rank: a comprehensive structrual similarity mea- sure over information networks. In CIKM, 2009.","title":"P-Rank: a comprehensive structrual similarity measure over information networks","authors":"Zhao","year":"2009","venue":"CIKM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}