{"type":"ScholarlyArticle","title":"Formal Verification of Parameterised Neural-symbolic Multi-agent Systems","authors":[{"name":"Panagiotis Kouvaros","slug":"panagiotis-kouvaros","given":"Panagiotis","family":"Kouvaros","affiliation":"Technology and Innovation School , University of Limassol"},{"name":"Elena Botoeva","slug":"elena-botoeva","given":"Elena","family":"Botoeva","affiliation":"School of Computing , University of Kent"},{"name":"Cosmo De Bonis-Campbell","slug":"cosmo-de-bonis-campbell","given":"Cosmo","family":"De Bonis-Campbell","affiliation":"School of Computing , University of Kent"}],"abstract":"We study the problem of verifying multi-agent systems composed of arbitrarily many neural-symbolic agents. We introduce a novel parameterised model, where the parameter denotes the number of agents in the system, each homogeneously constructed from an agent template equipped with a neural network-based perception unit and a traditionally programmed action selection mechanism. We define the verification and emergence identification problems for these models against a bounded fragment of CTL. We put forward an abstraction methodology that enables us to recast both problems to the problem of checking Neural Interpreted Systems with a bounded number of agents. We present an implementation and discuss experimental results obtained on a social dilemma game based on guarding.","doi":"10.24963/ijcai.2024/12","url":"https://ijcai.org/proceedings/2024/12","pdf":"https://ijcai.org/proceedings/2024/0012.pdf","markdown":"https://ijcai.org/proceedings/2024/12.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/12","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Agent-based and Multi-agent Systems","track":"Main Track","number":12,"pages":{"first":103,"last":110},"keywords":["Agent-based and Multi-agent Systems: MAS: Formal verification, validation and synthesis"],"videos":[],"references":[{"position":1,"raw":"[Akintunde et al., 2020a] M. Akintunde, E. Botoeva, P. Kou- varos, and A. Lomuscio. Formal verification of neural agents in non-deterministic environments. In Proceed- ings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS20), page To Ap- pear. ACM, 2020.","title":"Formal verification of neural agents in non-deterministic environments","authors":"Akintunde","year":"2020","venue":"Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS20), page To Appear","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Akintunde et al., 2020b] M. Akintunde, E. Botoeva, P. Kou- varos, and A. Lomuscio. Verifying strategic abilities of neural-symbolic multi-agent systems. In Proceedings of the International Conference on Principles of Knowl- edge Representation and Reasoning (KR20), pages 22-32. AAAI Press, 2020.","title":"Verifying strategic abilities of neural-symbolic multi-agent systems","authors":"Akintunde","year":"2020","venue":"Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning (KR20)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Akintunde et al., 2022] M. Akintunde, E. Botoeva, P. Kou- varos, and A. Lomuscio. Formal verification of neural agents in non-deterministic environments. Journal of Au- tonomous Agents and Multi-Agent Systems, 36(1), 2022.","title":"Formal verification of neural agents in non-deterministic environments","authors":"M Akintunde, E Botoeva, P Kouvaros, A Lomuscio","year":"2022","venue":"Journal of Autonomous Agents and Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Aminof et al., 2016] B. Aminof, A. Murano, S. Rubin, and F. Zuleger. Automatic verification of multi-agent systems in parameterised grid-environments. In Proceedings of the 2016 international conference on autonomous agents & multiagent systems, pages 1190-1199, 2016.","title":"Automatic verification of multi-agent systems in parameterised grid-environments","authors":"B Aminof, A Murano, S Rubin, F Zuleger","year":"2016","venue":"Proceedings of the 2016 international conference on autonomous agents & multiagent systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ball and Kupferman, 2006] Thomas Ball and Orna Kupfer- man. An abstraction-refinement framework for multi- agent systems. In 21st Annual IEEE Symposium on Logic in Computer Science (LICS'06), pages 379-388. IEEE, 2006.","title":"Thomas Ball and Orna Kupferman. An abstraction-refinement framework for multiagent systems","authors":"Kupferman Ball","year":"2006","venue":"21st Annual IEEE Symposium on Logic in Computer Science (LICS'06)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Belardinelli et al., 2017] F. Belardinelli, P. Kouvaros, and A. Lomuscio. Parameterised verification of data-aware multi-agent systems. In Proceedings of the 24th Inter- national Joint Conference on Artificial Intelligence (IJ- CAI17), pages 98-104. AAAI Press, 2017.","title":"Parameterised verification of data-aware multi-agent systems","authors":"F Belardinelli, P Kouvaros, A Lomuscio","year":"2017","venue":"Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJ-CAI17)","doi":null,"url":null,"internal_url":"/proceedings/2017/15"},{"position":7,"raw":"[Bonabeau et al., 1999] E. Bonabeau, M. Dorigo, and G. Theraulaz. Swarm intelligence. Oxford University Press, 1999.","title":"Swarm intelligence","authors":"E Bonabeau, M Dorigo, G Theraulaz","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Felli et al., 2020] P. Felli, A. Gianola, and M. Montali. Smt- based safety verification of parameterised multi-agent sys- tems. arXiv preprint arXiv:2008.04774, 2020.","title":"Smtbased safety verification of parameterised multi-agent systems","authors":"P Felli, A Gianola, M Montali","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gammie and van der Meyden, 2004] P. Gammie and R. van der Meyden. MCK: Model checking the logic of knowledge. In Proceedings of 16th International Conference on Computer Aided Verification (CAV04), volume 3114 of Lecture Notes in Computer Science, pages 479-483. Springer, 2004.","title":"MCK: Model checking the logic of knowledge","authors":"P Gammie, R Van Der Meyden","year":"2004","venue":"Proceedings of 16th International Conference on Computer Aided Verification (CAV04)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gu et al., 2016] Z. Gu, E. Rothberg, and R. Bixby. Gurobi optimizer reference manual. http://www.gurobi.com, 2016.","title":"Gurobi optimizer reference manual","authors":"Z Gu, E Rothberg, R Bixby","year":null,"venue":null,"doi":null,"url":"http://www.gurobi.com,2016","internal_url":null},{"position":11,"raw":"[Hardin, 1968] Garrett Hardin. The tragedy of the commons. Science, 162(3859):1243-1248, 1968.","title":"The tragedy of the commons","authors":"Garrett Hardin","year":"1968","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jamroga et al., 2020] Wojciech Jamroga, Wojciech Penczek, Teofil Sidoruk, Piotr Dembiński, and An- toni Mazurkiewicz. Towards partial order reductions for strategic ability. Journal of Artificial Intelligence Research, 68:817-850, 2020.","title":"Towards partial order reductions for strategic ability","authors":"Wojciech Jamroga, Wojciech Penczek, Teofil Sidoruk, Piotr Dembiński, Antoni Mazurkiewicz","year":"2020","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kacprzak et al., 2004] M. Kacprzak, A. Lomuscio, and W. Penczek. Verification of multiagent systems via un- bounded model checking. In Proceedings of the 3rd In- ternational Conference on Autonomous Agents and Multi- agent Systems (AAMAS04), pages 638-645. ACM, 2004.","title":"Verification of multiagent systems via unbounded model checking","authors":"M Kacprzak, A Lomuscio, W Penczek","year":"2004","venue":"Proceedings of the 3rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS04)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kacprzak et al., 2008] M. Kacprzak, W. Nabialek, A. Niewiadomski, W. Penczek, A. Pólrola, M. Szreter, B. Woźna, and A. Zbrzezny. VerICS 2007 -a model checker for knowledge and real-time. Fundamenta Informaticae, 85(1):313-328, 2008.","title":"VerICS 2007 -a model checker for knowledge and real-time","authors":"M Kacprzak, W Nabialek, A Niewiadomski, W Penczek, A Pólrola, M Szreter, B Woźna, A Zbrzezny","year":"2008","venue":"Fundamenta Informaticae","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kouvaros and Lomuscio, 2015] P. Kouvaros and A. Lomus- cio. Verifying emergent properties of swarms. In Pro- ceedings of the 24th International Joint Conference on Ar- tificial Intelligence (IJCAI15), pages 1083-1089. AAAI Press, 2015.","title":"Verifying emergent properties of swarms","authors":"P Kouvaros, A Lomuscio","year":"2015","venue":"Proceedings of the 24th International Joint Conference on Ar-tificial Intelligence (IJCAI15)","doi":null,"url":null,"internal_url":"/proceedings/2015/157"},{"position":16,"raw":"[Kouvaros and Lomuscio, 2016a] P. Kouvaros and A. Lo- muscio. Parameterised model checking for alternating- time temporal logic. In Proceedings of the 22nd Eu- ropean Conference on Artificial Intelligence (ECAI16), pages 1230-1238. IOS Press, 2016.","title":"Parameterised model checking for alternatingtime temporal logic","authors":"Lomuscio Kouvaros, P Kouvaros, A Lomuscio","year":"2016","venue":"Proceedings of the 22nd European Conference on Artificial Intelligence (ECAI16)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kouvaros and Lomuscio, 2016b] P. Kouvaros and A. Lo- muscio. Parameterised verification for multi-agent sys- tems. Artificial Intelligence, 234:152-189, 2016.","title":"Parameterised verification for multi-agent systems","authors":"Lomuscio Kouvaros, P Kouvaros, A Lomuscio","year":"2016","venue":"Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kouvaros et al., 2018] P. Kouvaros, A. Lomuscio, and E. Pirovano. Symbolic synthesis of fault-tolerance ratios in parameterised multi-agent systems. In Proceedings of the 27th International Joint Conference on Artificial Intel- ligence (IJCAI18), pages 324-330. AAAI Press, 2018.","title":"Symbolic synthesis of fault-tolerance ratios in parameterised multi-agent systems","authors":"P Kouvaros, A Lomuscio, E Pirovano","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI18)","doi":null,"url":null,"internal_url":"/proceedings/2018/45"},{"position":19,"raw":"[Lomuscio and Michaliszyn, 2015] Alessio Lomuscio and Jakub Michaliszyn. Verifying multi-agent systems by model checking three-valued abstractions. In Proceed- ings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 189-198, 2015.","title":"Verifying multi-agent systems by model checking three-valued abstractions","authors":"Alessio Lomuscio, Jakub Michaliszyn","year":"2015","venue":"Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lomuscio et al., 2017] A. Lomuscio, H. Qu, and F. Rai- mondi. MCMAS: A model checker for the verification of multi-agent systems. Software Tools for Technology Trans- fer, 19(1):9-30, 2017.","title":"MCMAS: A model checker for the verification of multi-agent systems","authors":"A Lomuscio, H Qu, F Raimondi","year":"2017","venue":"Software Tools for Technology Transfer","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Milinski et al., 2008] Manfred Milinski, Ralf D. Sommer- feld, Hans-Jürgen Krambeck, Floyd A. Reed, and Jochem Marotzke. The collective-risk social dilemma and the pre- vention of simulated dangerous climate change. Proceed- ings of the National Academy of Sciences, 105(7):2291- 2294, 2008.","title":"The collective-risk social dilemma and the prevention of simulated dangerous climate change","authors":"Manfred Milinski, Ralf D Sommerfeld, Hans-Jürgen Krambeck, Floyd A Reed, Jochem Marotzke","year":"2008","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Pedersen and Dyrkolbotn, 2013] T. Pedersen and SK. Dyrkolbotn. Agents homogeneous: A procedurally anonymous semantics characterizing the homogeneous fragment of atl. In International Conference on Principles and Practice of Multi-Agent Systems, pages 245-259.","title":"Agents homogeneous: A procedurally anonymous semantics characterizing the homogeneous fragment of atl","authors":"T Pedersen, S K Dyrkolbotn","year":null,"venue":"International Conference on Principles and Practice of Multi-Agent Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Penczek and Lomuscio, 2003] Wojciech Penczek and Alessio Lomuscio. Verifying epistemic properties of multi-agent systems via bounded model checking. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pages 209-216, 2003.","title":"Verifying epistemic properties of multi-agent systems via bounded model checking","authors":"Lomuscio Penczek, Wojciech Penczek, Alessio Lomuscio","year":"2003","venue":"Proceedings of the second international joint conference on Autonomous agents and multiagent systems","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Raimondi and Lomuscio, 2005] F. Raimondi and A. Lo- muscio. Automatic verification of multi-agent systems by model checking via OBDDs. Journal of Applied Logic, 5(2):235-251, 2005.","title":"Automatic verification of multi-agent systems by model checking via OBDDs","authors":"F Raimondi, A Lomuscio","year":"2005","venue":"Journal of Applied Logic","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Santos and Pacheco, 2011] Francisco C. Santos and Jorge M. Pacheco. Risk of collective failure provides an escape from the tragedy of the commons. Proceedings of the National Academy of Sciences, 108(26):10421-10425, 2011.","title":"Risk of collective failure provides an escape from the tragedy of the commons","authors":"Santos, Pacheco, Francisco C Santos, Jorge M Pacheco","year":"2011","venue":"Proceedings of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Van Lange et al., 2013] Paul A.M. Van Lange, Jeff Joire- man, Craig D. Parks, and Eric Van Dijk. The psychology of social dilemmas: A review. Organizational Behavior and Human Decision Processes, 120(2):125-141, 2013. Social Dilemmas.","title":"The psychology of social dilemmas: A review","authors":"A M Paul, Jeff Van Lange, Craig D Joireman, Eric Parks, Van Dijk","year":"2013","venue":"Organizational Behavior and Human Decision Processes","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}