{"type":"ScholarlyArticle","title":"Information-Theoretic Opacity-Enforcement in Markov Decision Processes","authors":[{"name":"Chongyang Shi","slug":"chongyang-shi","given":"Chongyang","family":"Shi","affiliation":"University of Florida"},{"name":"Yuheng Bu","slug":"yuheng-bu","given":"Yuheng","family":"Bu","affiliation":"University of Florida"},{"name":"Jie Fu","slug":"jie-fu","given":"Jie","family":"Fu","affiliation":"University of Florida"}],"abstract":"The paper studies information-theoretic opacity, an information-flow privacy property, in a setting involving two agents: A planning agent who controls a stochastic system and an observer who partially observes the system states. The goal of the observer is to infer some secret, represented by a random variable, from its partial observations, while the goal of the planning agent is to make the secret maximally opaque to the observer while achieving a satisfactory total return. Modeling the stochastic system using a Markov decision process, two classes of opacity properties are considered---Last-state opacity is to ensure that the observer is uncertain if the last state is in a specific set and initial-state opacity is to ensure that the observer is unsure of the realization of the initial state. As the measure of opacity, we employ the Shannon conditional entropy capturing the information about the secret revealed by the observable. Then, we develop primal-dual policy gradient methods for opacity-enforcement planning subject to constraints on total returns. We propose novel algorithms to compute the policy gradient of entropy for each observation, leveraging message passing within the hidden Markov models. This gradient computation enables us to have stable and fast convergence. We demonstrate our solution of opacity-enforcement control through a grid world example.","doi":"10.24963/ijcai.2024/749","url":"https://ijcai.org/proceedings/2024/749","pdf":"https://ijcai.org/proceedings/2024/0749.pdf","markdown":"https://ijcai.org/proceedings/2024/749.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/749","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Planning and Scheduling","track":"Main Track","number":749,"pages":{"first":6779,"last":6787},"keywords":["Planning and Scheduling: PS: Planning under uncertainty","Planning and Scheduling: PS: Markov decisions processes","Planning and Scheduling: PS: Planning with Incomplete Information","Planning and Scheduling: PS: Planning algorithms"],"videos":[],"references":[{"position":1,"raw":"[Baum et al., 1970] Leonard E Baum, Ted Petrie, George Soules, and Norman Weiss. A maximization technique oc- curring in the statistical analysis of probabilistic functions of markov chains. The annals of mathematical statistics, 41(1):164-171, 1970.","title":"A maximization technique occurring in the statistical analysis of probabilistic functions of markov chains","authors":"Leonard E Baum, Ted Petrie, George Soules, Norman Weiss","year":"1970","venue":"The annals of mathematical statistics","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bérard et al., 2015a] Béatrice Bérard, Krishnendu Chatter- jee, and Nathalie Sznajder. Probabilistic opacity for Markov decision processes. Information Processing Let- ters, 115(1):52-59, January 2015.","title":"Probabilistic opacity for Markov decision processes","authors":"Bérard","year":"2015","venue":"Information Processing Letters","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bérard et al., 2015b] Béatrice Bérard, John Mullins, and Mathieu Sassolas. Quantifying opacity †. Mathematical Structures in Computer Science, 25(2):361-403, February 2015. Publisher: Cambridge University Press.","title":"Quantifying opacity †","authors":"Bérard","year":"2015","venue":"Mathematical Structures in Computer Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cen et al., 2022] Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regulariza- tion. Operations Research, 70(4):2563-2578, 2022.","title":"Fast global convergence of natural policy gradient methods with entropy regularization","authors":"Cen","year":"2022","venue":"Operations Research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dubreil et al., 2008] Jeremy Dubreil, Philippe Darondeau, and Herve Marchand. Opacity enforcing control synthe- sis. In 2008 9th International Workshop on Discrete Event Systems, pages 28-35, 2008.","title":"Opacity enforcing control synthesis","authors":"Jeremy Dubreil, Philippe Darondeau, Herve Marchand","year":"2008","venue":"2008 9th International Workshop on Discrete Event Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Guo et al., 2020] Ye Guo, Xiaoning Jiang, Chen Guo, Shouguang Wang, and Oussama Karoui. Overview of Opacity in Discrete Event Systems. IEEE Access, 8:48731-48741, 2020. Conference Name: IEEE Access.","title":"Overview of Opacity in Discrete Event Systems","authors":"Ye Guo, Xiaoning Jiang, Chen Guo, Shouguang Wang, Oussama Karoui","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Han et al., 2023] X Han, Kuize Zhang, Jiahui Zhang, Zhiwu Li, and Zengqiang Chen. Strong current-state and initial- state opacity of discrete-event systems. Automatica (Ox- ford), 148:110756, 2023.","title":"Strong current-state and initialstate opacity of discrete-event systems","authors":"Han","year":"2023","venue":"Automatica (Oxford)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Keroglou and Hadjicostis, 2016] Christoforos Keroglou and Christoforos N. Hadjicostis. Probabilistic system opacity in discrete event systems. In 2016 13th Interna- tional Workshop on Discrete Event Systems (WODES), pages 379-384, May 2016.","title":"Probabilistic system opacity in discrete event systems","authors":"Christoforos Keroglou, Christoforos N Hadjicostis","year":"2016","venue":"13th International Workshop on Discrete Event Systems (WODES)","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khouzani and Malacaria, 2017] MHR. Khouzani and Pasquale Malacaria. Leakage-minimal design: Univer- sality, limitations, and applications. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 305-317, 2017.","title":"Leakage-minimal design: Universality, limitations, and applications","authors":"Malacaria Khouzani, Mhr Khouzani, Pasquale Malacaria","year":"2017","venue":"IEEE 30th Computer Security Foundations Symposium (CSF)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin, 2011] Feng Lin. Opacity of discrete event systems and its applications. Automatica, 47(3):496-503, 2011.","title":"Opacity of discrete event systems and its applications","authors":"Lin ;, Feng Lin","year":"2011","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ma et al., 2023] Haoxiang Ma, Chongyang Shi, Shuo Han, Michael R. Dorothy, and Jie Fu. Covert planning against imperfect observers, 2023.","title":"Covert planning against imperfect observers","authors":"Haoxiang Ma, Chongyang Shi, Shuo Han, Michael R Dorothy, Jie Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maliah et al., 2017] Shlomi Maliah, Guy Shani, and Roni Stern. Collaborative privacy preserving multi-agent plan- ning. Autonomous Agents and Multi-Agent Systems, 31(3):493-530, may 2017.","title":"Collaborative privacy preserving multi-agent planning","authors":"Shlomi Maliah, Guy Shani, Roni Stern","year":"2017","venue":"Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Marzouqi and Jarvis, 2004] Mohamed Marzouqi and Ray Jarvis. Covert path planning for autonomous robot nav- igation. 01 2004.","title":"Covert path planning for autonomous robot navigation","authors":"Jarvis ; Marzouqi, Mohamed Marzouqi, Ray Jarvis","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Marzouqi and Jarvis, 2005] M. Marzouqi and R.A. Jarvis. Covert path planning in unknown environments with known or suspected sentry location. In 2005 IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems, pages 1772-1778, 2005.","title":"Covert path planning in unknown environments with known or suspected sentry location","authors":"M Marzouqi, R A Jarvis","year":"2005","venue":"IEEE/RSJ International Conference on Intelligent Robots and Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mazaré, 2004] Laurent Mazaré. Using unification for opac- ity properties. 2004.","title":"Using unification for opacity properties","authors":"Laurent Mazaré","year":"2004","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Miura and Zilberstein, 2021] Shuwa Miura and Shlomo Zil- berstein. A unifying framework for observer-aware plan- ning and its complexity. In Cassio de Campos and Mar- loes H. Maathuis, editors, Proceedings of the Thirty- Seventh Conference on Uncertainty in Artificial Intelli- gence, volume 161 of Proceedings of Machine Learning Research, pages 610-620. PMLR, 27-30 Jul 2021.","title":"Shuwa Miura and Shlomo Zilberstein. A unifying framework for observer-aware planning and its complexity","authors":"Zilberstein Miura","year":"2021","venue":"Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Molloy and Nair, 2023] Timothy Molloy and Girish Nair. Smoother entropy for active state trajectory estimation and obfuscation in pomdps. IEEE Transactions on Automatic Control, PP:1-16, 06 2023.","title":"Smoother entropy for active state trajectory estimation and obfuscation in pomdps","authors":"Nair ; ] Timothy Molloy, Girish Molloy, Nair","year":"2023","venue":"IEEE Transactions on Automatic Control","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nachum et al., 2017] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. 2017.","title":"Bridging the gap between value and policy based reinforcement learning","authors":"Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Saboori and Hadjicostis, 2007] Anooshiravan Saboori and Christoforos N Hadjicostis. Notions of security and opac- ity in discrete event systems. In 2007 46th IEEE Confer- ence on Decision and Control, pages 5056-5061. IEEE, 2007.","title":"Anooshiravan Saboori and Christoforos N Hadjicostis. Notions of security and opacity in discrete event systems","authors":"Hadjicostis Saboori","year":"2007","venue":"46th IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saboori and Hadjicostis, 2009] Anooshiravan Saboori and Christoforos N. Hadjicostis. Verification of k-step opacity and analysis of its complexity. In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, pages 205-210, 2009.","title":"Anooshiravan Saboori and Christoforos N. Hadjicostis. Verification of k-step opacity and analysis of its complexity","authors":"Hadjicostis Saboori","year":"2009","venue":"Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Saboori and Hadjicostis, 2013a] Anooshiravan Saboori and Christoforos N Hadjicostis. Current-state opacity formu- lations in probabilistic finite automata. IEEE Transactions on automatic control, 59(1):120-133, 2013.","title":"Anooshiravan Saboori and Christoforos N Hadjicostis. Current-state opacity formulations in probabilistic finite automata","authors":"Hadjicostis Saboori","year":"2013","venue":"IEEE Transactions on automatic control","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Saboori and Hadjicostis, 2013b] Anooshiravan Saboori and Christoforos N. Hadjicostis. Verification of initial-state opacity in security applications of discrete event systems. Information Sciences, 246:115-132, 2013.","title":"Anooshiravan Saboori and Christoforos N. Hadjicostis. Verification of initial-state opacity in security applications of discrete event systems","authors":"Hadjicostis Saboori","year":"2013","venue":"Information Sciences","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shani, 2018] Guy Shani. Advances and challenges in pri- vacy preserving planning. In IJCAI, pages 5719-5723, 2018.","title":"Guy Shani. Advances and challenges in privacy preserving planning","authors":"Shani","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shannon, 1949] C. E. Shannon. Communication theory of secrecy systems. The Bell System Technical Journal, 28(4):656-715, 1949.","title":"Communication theory of secrecy systems","authors":"C E Shannon","year":"1949","venue":"The Bell System Technical Journal","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Shi et al., 2023] Chongyang Shi, Abhishek N. Kulkarni, Hazhar Rahmani, and Jie Fu. Synthesis of opacity- enforcing winning strategies against colluded opponent, 2023.","title":"Synthesis of opacityenforcing winning strategies against colluded opponent","authors":"Chongyang Shi, Abhishek N Kulkarni, Hazhar Rahmani, Jie Fu","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sutton et al., 1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approx- imation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, vol- ume 12. MIT Press, 1999.","title":"Policy gradient methods for reinforcement learning with function approximation","authors":"David Richard S Sutton, Satinder Mcallester, Yishay Singh, Mansour","year":"1999","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Watson, 2011] Paul Watson. A Multi-Level Security Model for PartitioningWorkflows over Federated Clouds. In 2011 IEEE Third International Conference on Cloud Comput- ing Technology and Science, pages 180-188, November 2011.","title":"A Multi-Level Security Model for PartitioningWorkflows over Federated Clouds","authors":"Paul Watson","year":"2011","venue":"IEEE Third International Conference on Cloud Computing Technology and Science","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yasuoka and Terauchi, 2010] H. Yasuoka and T. Terauchi. Quantitative information flow -verification hardness and possibilities. In 2010 IEEE 23rd Computer Security Foun- dations Symposium (CSF 2010), pages 15-27, Los Alami- tos, CA, USA, jul 2010. IEEE Computer Society.","title":"Quantitative information flow -verification hardness and possibilities","authors":"H Yasuoka, T Terauchi","year":"2010","venue":"IEEE 23rd Computer Security Foundations Symposium (CSF 2010)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yin et al., 2019] Xiang Yin, Zhaojian Li, Weilin Wang, and Shaoyuan Li. Infinite-step opacity and K-step opacity of stochastic discrete-event systems. Automatica, 99:266- 274, January 2019.","title":"Infinite-step opacity and K-step opacity of stochastic discrete-event systems","authors":"Xiang Yin, Zhaojian Li, Weilin Wang, Shaoyuan Li","year":"2019","venue":"Automatica","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zeng and Koutny, 2021] Wen Zeng and Maciej Koutny. Quantitative Analysis of Opacity in Cloud Computing Systems. IEEE Transactions on Cloud Computing, 9(3):1210-1219, July 2021.","title":"Quantitative Analysis of Opacity in Cloud Computing Systems","authors":"Koutny ; Zeng, Wen Zeng, Maciej Koutny","year":"2021","venue":"IEEE Transactions on Cloud Computing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}