{"type":"ScholarlyArticle","title":"Enhancing Network by Reinforcement Learning and Neural Confined Local Search","authors":[{"name":"Qifu Hu","slug":"qifu-hu","given":"Qifu","family":"Hu","affiliation":"Inspur Electronic Information Industry Co. , Ltd"},{"name":"Ruyang Li","slug":"ruyang-li","given":"Ruyang","family":"Li","affiliation":"Inspur Electronic Information Industry Co. , Ltd"},{"name":"Qi Deng","slug":"qi-deng","given":"Qi","family":"Deng","affiliation":"Inspur Electronic Information Industry Co. , Ltd"},{"name":"Yaqian Zhao","slug":"yaqian-zhao","given":"Yaqian","family":"Zhao","affiliation":"Inspur Electronic Information Industry Co. , Ltd"},{"name":"Rengang Li","slug":"rengang-li","given":"Rengang","family":"Li","affiliation":"Inspur Electronic Information Industry Co. , Ltd"}],"abstract":"It has been found that many real networks, such as power grids and the Internet, are non-robust, i.e., attacking a small set of nodes would cause the paralysis of the entire network. Thus, the Network Enhancement Problem~(NEP), i.e., improving the robustness of a given network by modifying its structure, has attracted increasing attention. Heuristics have been proposed to address NEP. However, a hand-engineered heuristic often has significant performance limitations. A recently proposed model solving NEP by reinforcement learning has shown superior performance than heuristics on in-distribution datasets. However, their model shows considerably inferior out-of-distribution generalization ability when enhancing networks against the degree-based targeted attack. In this paper, we propose a more effective model with stronger generalization ability by incorporating domain knowledge including measurements of local network structures and decision criteria of heuristics. We further design a hierarchical attention model to utilize the network structure directly, where the query range changes from local to global. Finally, we propose neural confined local search~(NCLS) to realize the effective search of a large neighborhood, which exploits a learned model to confine the neighborhood to avoid exhaustive enumeration. We conduct extensive experiments on synthetic and real networks to verify the ability of our models.","doi":"10.24963/ijcai.2023/236","url":"https://ijcai.org/proceedings/2023/236","pdf":"https://ijcai.org/proceedings/2023/0236.pdf","markdown":"https://ijcai.org/proceedings/2023/236.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/236","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Data Mining","track":"Main Track","number":236,"pages":{"first":2122,"last":2132},"keywords":["Data Mining: DM: Mining graphs","Machine Learning: ML: Deep reinforcement learning","Search: S: Search and machine learning"],"videos":[],"references":[{"position":1,"raw":"[Aarts et al., 2003] Emile Aarts, Emile HL Aarts, and Jan Karel Lenstra. Local search in combinatorial opti- mization. Princeton University Press, 2003.","title":"Local search in combinatorial optimization","authors":"Aarts","year":"2003","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Albert et al., 2000] Réka Albert, Hawoong Jeong, and Albert-László Barabási. Error and attack tolerance of com- plex networks. nature, 406(6794):378-382, 2000.","title":"Error and attack tolerance of complex networks","authors":"Réka Albert, Hawoong Jeong, Albert-László Barabási","year":"2000","venue":"nature","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barabási and Albert, 1999] Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. science, 286(5439):509-512, 1999.","title":"Emergence of scaling in random networks","authors":"Albert ; Albert-László Barabási, Réka Barabási, Albert","year":"1999","venue":"science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barrat et al., 2004] Alain Barrat, Marc Barthelemy, Ro- mualdo Pastor-Satorras, and Alessandro Vespignani. The architecture of complex weighted networks. Proceedings of the national academy of sciences, 101(11):3747-3752, 2004.","title":"The architecture of complex weighted networks","authors":"Alain Barrat, Marc Barthelemy, Romualdo Pastor-Satorras, Alessandro Vespignani","year":"2004","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bello et al., 2016] Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combi- natorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940, 2016.","title":"Neural combinatorial optimization with reinforcement learning","authors":"Irwan Bello, Hieu Pham, V Quoc, Mohammad Le, Samy Norouzi, Bengio","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Beygelzimer et al., 2005] Alina Beygelzimer, Geoffrey Grinstein, Ralph Linsker, and Irina Rish. Improving network robustness by edge modification. Physica A: Statistical Mechanics and its Applications, 357(3-4):593- 612, 2005.","title":"Improving network robustness by edge modification","authors":"Alina Beygelzimer, Geoffrey Grinstein, Ralph Linsker, Irina Rish","year":"2005","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Burda et al., 2018] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random net- work distillation. arXiv preprint arXiv:1810.12894, 2018.","title":"Exploration by random network distillation","authors":"Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chan et al., 2014] Hau Chan, Leman Akoglu, and Hang- hang Tong. Make it or break it: Manipulating robustness in large networks. In Proceedings of the 2014 SIAM In- ternational Conference on Data Mining, pages 325-333.","title":"Make it or break it: Manipulating robustness in large networks","authors":"Leman Hau Chan, Hanghang Akoglu, Tong","year":null,"venue":"Proceedings of the 2014 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Costa et al., 2007] L da F Costa, Francisco A Rodrigues, Gonzalo Travieso, and Paulino Ribeiro Villas Boas. Char- acterization of complex networks: A survey of measure- ments. Advances in physics, 56(1):167-242, 2007.","title":"Characterization of complex networks: A survey of measurements","authors":"L Da F Costa, Francisco A Rodrigues, Gonzalo Travieso, Paulino Ribeiro, Villas Boas","year":"2007","venue":"Advances in physics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dai et al., 2016] Hanjun Dai, Bo Dai, and Le Song. Dis- criminative embeddings of latent variable models for structured data. In International conference on machine learning, pages 2702-2711. PMLR, 2016.","title":"Discriminative embeddings of latent variable models for structured data","authors":"Dai","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Darvariu et al., 2021] Victor-Alexandru Darvariu, Stephen Hailes, and Mirco Musolesi. Goal-directed graph con- struction using reinforcement learning. Proceedings of the Royal Society A, 477(2254):20210168, 2021.","title":"Goal-directed graph construction using reinforcement learning","authors":"Victor-Alexandru Darvariu, Stephen Hailes, Mirco Musolesi","year":"2021","venue":"Proceedings of the Royal Society A","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Demmel, 1997] James W Demmel. Applied numerical lin- ear algebra. SIAM, 1997.","title":"Applied numerical linear algebra","authors":"James W Demmel","year":"1997","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Deudon et al., 2018] Michel Deudon, Pierre Cournut, Alexandre Lacoste, Yossiri Adulyasak, and Louis-Martin Rousseau. Learning heuristics for the tsp by policy gradient. In International conference on the integration of constraint programming, artificial intelligence, and operations research, pages 170-181. Springer, 2018.","title":"Learning heuristics for the tsp by policy gradient","authors":"Pierre Michel Deudon, Alexandre Cournut, Yossiri Lacoste, Louis-Martin Adulyasak, Rousseau","year":"2018","venue":"International conference on the integration of constraint programming, artificial intelligence, and operations research","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ellens and Kooij, 2013] Wendy Ellens and Robert E Kooij. Graph measures and network robustness. arXiv preprint arXiv:1311.5064, 2013.","title":"Graph measures and network robustness","authors":"Kooij Ellens, Wendy Ellens, Robert E Kooij","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ellens et al., 2011] Wendy Ellens, Floske M Spieksma, Piet Van Mieghem, Almerima Jamakovic, and Robert E Kooij. Effective graph resistance. Linear algebra and its applica- tions, 435(10):2491-2506, 2011.","title":"Effective graph resistance","authors":"Wendy Ellens, M Floske, Piet Spieksma, Almerima Van Mieghem, Robert E Jamakovic, Kooij","year":"2011","venue":"Linear algebra and its applications","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Erdos et al., 1960] Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci, 5(1):17-60, 1960.","title":"On the evolution of random graphs","authors":"Paul Erdos, Alfréd Rényi","year":"1960","venue":"Publ. Math. Inst. Hung. Acad. Sci","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Faloutsos et al., 1999] Michalis Faloutsos, Petros Falout- sos, and Christos Faloutsos. On power-law relationships of the internet topology. ACM SIGCOMM computer com- munication review, 29(4):251-262, 1999.","title":"On power-law relationships of the internet topology","authors":"Michalis Faloutsos, Petros Faloutsos, Christos Faloutsos","year":"1999","venue":"ACM SIGCOMM computer communication review","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Fiedler, 1973] Miroslav Fiedler. Algebraic connectivity of graphs. Czechoslovak mathematical journal, 23(2):298- 305, 1973.","title":"Algebraic connectivity of graphs","authors":"Miroslav Fiedler, Fiedler","year":"1973","venue":"Czechoslovak mathematical journal","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ghosh and Boyd, 2006] Arpita Ghosh and Stephen Boyd. Growing well-connected graphs. In Proceedings of the 45th IEEE Conference on Decision and Control, pages 6605-6611. IEEE, 2006.","title":"Growing well-connected graphs","authors":"Arpita Ghosh, Stephen Boyd","year":"2006","venue":"Proceedings of the 45th IEEE Conference on Decision and Control","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Haarnoja et al., 2018] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on ma- chine learning, pages 1861-1870. PMLR, 2018.","title":"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor","authors":"Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, Sergey Levine","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Holme et al., 2002] Petter Holme, Beom Jun Kim, Chang No Yoon, and Seung Kee Han. Attack vul- nerability of complex networks. Physical review E, 65(5):056109, 2002.","title":"Attack vulnerability of complex networks","authors":"Petter Holme, Jun Beom, Chang Kim, Seung No Yoon, Han Kee","year":"2002","venue":"Physical review E","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Hu et al., 2019] Qifu Hu, Angsheng Li, Jiamou Liu, and Jun Liu. Time-efficient network monitoring through confined search and adaptive evaluation. In Pacific Rim Interna- tional Conference on Artificial Intelligence, pages 633- 646. Springer, 2019.","title":"Time-efficient network monitoring through confined search and adaptive evaluation","authors":"Qifu Hu, Angsheng Li, Jiamou Liu, Jun Liu","year":"2019","venue":"Pacific Rim International Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Khalil et al., 2017] Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial op- timization algorithms over graphs. Advances in neural in- formation processing systems, 30, 2017.","title":"Learning combinatorial optimization algorithms over graphs","authors":"Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, Le Song","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Kool et al., 2018] Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In International Conference on Learning Representations, 2018.","title":"Attention, learn to solve routing problems!","authors":"Kool","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Li et al., 2018] Gang Li, Zhi Feng Hao, Han Huang, and Hang Wei. Maximizing algebraic connectivity via min- imum degree and maximum distance. IEEE Access, 6:41249-41255, 2018.","title":"Maximizing algebraic connectivity via minimum degree and maximum distance","authors":"Gang Li, Feng Zhi, Han Hao, Hang Huang, Wei","year":"2018","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Liu et al., 2016] Jun Liu, Qingyu Xiong, Weiren Shi, Xin Shi, and Kai Wang. Evaluating the importance of nodes in complex networks. Physica A: Statistical Mechanics and its Applications, 452:209-219, 2016.","title":"Evaluating the importance of nodes in complex networks","authors":"Jun Liu, Qingyu Xiong, Weiren Shi, Xin Shi, Kai Wang","year":"2016","venue":"Physica A: Statistical Mechanics and its Applications","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Liu et al., 2017] Jing Liu, Mingxing Zhou, Shuai Wang, and Penghui Liu. A comparative study of network robustness measures. Frontiers of Computer Science, 11(4):568-584, 2017.","title":"A comparative study of network robustness measures","authors":"Jing Liu, Mingxing Zhou, Shuai Wang, Penghui Liu","year":"2017","venue":"Frontiers of Computer Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Lü and Zhou, 2011] Linyuan Lü and Tao Zhou. Link pre- diction in complex networks: A survey. Physica A: statis- tical mechanics and its applications, 390(6):1150-1170, 2011.","title":"Linyuan Lü and Tao Zhou. Link prediction in complex networks: A survey","authors":"Zhou Lü","year":"2011","venue":"Physica A: statistical mechanics and its applications","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Ma et al., 2019] Qiang Ma, Suwen Ge, Danyang He, Dar- shan Thaker, and Iddo Drori. Combinatorial optimization by graph pointer networks and hierarchical reinforcement learning. arXiv preprint arXiv:1911.04936, 2019.","title":"Combinatorial optimization by graph pointer networks and hierarchical reinforcement learning","authors":"Qiang Ma, Suwen Ge, Danyang He, Darshan Thaker, Iddo Drori","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ma et al., 2021] Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Le Zhang, Zhenghua Chen, and Jing Tang. Learning to iteratively solve routing problems with dual- aspect collaborative transformer. Advances in Neural In- formation Processing Systems, 34:11096-11107, 2021.","title":"Learning to iteratively solve routing problems with dualaspect collaborative transformer","authors":"Yining Ma, Jingwen Li, Zhiguang Cao, Wen Song, Le Zhang, Zhenghua Chen, Jing Tang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Mnih et al., 2013] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.","title":"Playing atari with deep reinforcement learning","authors":"Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Nazari et al., 2018] Mohammadreza Nazari, Afshin Oroo- jlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. Advances in neural information processing systems, 31, 2018.","title":"Reinforcement learning for solving the vehicle routing problem","authors":"Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, Martin Takác","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Sánchez Martínez, 2009] Mari Carmen Sánchez Martínez. Robustness optimization via link additions. Master's the- sis, Universitat Politècnica de Catalunya, 2009.","title":"Robustness optimization via link additions","authors":"Mari Carmen, Sánchez Martínez","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Shi et al., 2020] Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509, 2020.","title":"Masked label prediction: Unified message passing model for semi-supervised classification","authors":"Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, Yu Sun","year":"2020","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2021/214"},{"position":36,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in neural in- formation processing systems, 28, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Wang and Rong, 2009] Jian-Wei Wang and Li-Li Rong. Cascade-based attack vulnerability on the us power grid. Safety science, 47(10):1332-1336, 2009.","title":"Cascade-based attack vulnerability on the us power grid","authors":"Jian-Wei Wang, Li-Li Rong","year":"2009","venue":"Safety science","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Wang and Van Mieghem, 2010] Huijuan Wang and Piet Van Mieghem. Algebraic connectivity optimization via link addition. In 3d International ICST Conference on Bio- Inspired Models of Network, Information, and Computing Systems, 2010.","title":"Algebraic connectivity optimization via link addition","authors":"Wang, Van Mieghem, ;, Huijuan Wang, Piet Van Mieghem","year":"2010","venue":"3d International ICST Conference on Bio-Inspired Models of Network, Information, and Computing Systems","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Wang et al., 2014] Xiangrong Wang, Evangelos Pournaras, Robert E Kooij, and Piet Van Mieghem. Improving ro- bustness of complex networks via the effective graph re- sistance. The European Physical Journal B, 87(9):1-12, 2014.","title":"Improving robustness of complex networks via the effective graph resistance","authors":"Xiangrong Wang, Evangelos Pournaras, Robert E Kooij, Piet Van Mieghem","year":"2014","venue":"The European Physical Journal B","doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-256, 1992.","title":"Simple statistical gradient-following algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Wu et al., 2021] Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, and Andrew Lim. Learning improvement heuris- tics for solving routing problems. IEEE transactions on neural networks and learning systems, 33(9):5057-5069, 2021.","title":"Learning improvement heuristics for solving routing problems","authors":"Yaoxin Wu, Wen Song, Zhiguang Cao, Jie Zhang, Andrew Lim","year":"2021","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":43,"raw":"[Zhou et al., 2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wan- cai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106-11115, 2021.","title":"Informer: Beyond efficient transformer for long sequence time-series forecasting","authors":"Zhou","year":"2021","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}