{"type":"ScholarlyArticle","title":"Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation","authors":[{"name":"Shuo Zhang","slug":"shuo-zhang","given":"Shuo","family":"Zhang","affiliation":"Program in Computer Science , The Graduate Center , The City University of New York"},{"name":"Lei Xie","slug":"lei-xie","given":"Lei","family":"Xie","affiliation":"Department of Computer Science , Hunter College , The City University of New York; Helen & Robert Appel Alzheimer's Disease Research Institute , Feil Family Brain & Mind Research Institute , Weill Cornell Medicine , Cornell University"}],"abstract":"Graph Neural Networks (GNNs) are powerful for the representation learning of graph-structured data. Most of the GNNs use a message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information from its neighbors. To achieve a better expressive capability of node influences, attention mechanism has grown to be popular to assign trainable weights to the nodes in aggregation. Though the attention-based GNNs have achieved remarkable results in various tasks, a clear understanding of their discriminative capacities is missing. In this work, we present a theoretical analysis of the representational properties of the GNN that adopts the attention mechanism as an aggregator. Our analysis determines all cases when those attention-based GNNs can always fail to distinguish certain distinct structures. Those cases appear due to the ignorance of cardinality information in attention-based aggregation. To improve the performance of attention-based GNNs, we propose cardinality preserved attention (CPA) models that can be applied to any kind of attention mechanisms. Our experiments on node and graph classification confirm our theoretical analysis and show the competitive performance of our CPA models. The code is available online: https://github.com/zetayue/CPA.","doi":"10.24963/ijcai.2020/194","url":"https://ijcai.org/proceedings/2020/194","pdf":"https://ijcai.org/proceedings/2020/0194.pdf","markdown":"https://ijcai.org/proceedings/2020/194.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/194","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Data Mining","track":"Main track","number":194,"pages":{"first":1395,"last":1402},"keywords":["Data Mining: Mining Graphs, Semi Structured Data, Complex Data","Data Mining: Classification, Semi-Supervised Learning","Machine Learning: Deep Learning","Machine Learning Applications: Networks"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25326/file_25326.m3u8","length_sec":224,"provider":"hls"}],"references":[{"position":1,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pages 1024-1034, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hornik et al., 1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989.","title":"Multilayer feedforward networks are universal approximators","authors":"Kurt Hornik, Maxwell Stinchcombe, Halbert White","year":"1989","venue":"Neural networks","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep net- work training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448-456, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ivanov and Burnaev, 2018] Sergey Ivanov and Evgeny Bur- naev. Anonymous walk embeddings. In International Conference on Machine Learning, pages 2191-2200, 2018.","title":"Anonymous walk embeddings","authors":"Sergey Ivanov, Evgeny Burnaev","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kersting et al., 2020] Kristian Kersting, Nils M. Kriege, Christopher Morris, Petra Mutzel, and Marion Neu- mann. Benchmark data sets for graph kernels, 2020. http://www.graphlearning.io/.","title":"Benchmark data sets for graph kernels","authors":"Kristian Kersting, Nils M Kriege, Christopher Morris, Petra Mutzel, Marion Neumann","year":"2020","venue":null,"doi":null,"url":"http://www.graphlearning.io/","internal_url":null},{"position":7,"raw":"[Kingma and Ba, 2018] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2018.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Rep- resentations, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Knyazev et al., 2019] Boris Knyazev, Graham W Taylor, and Mohamed Amer. Understanding attention and gener- alization in graph neural networks. In Advances in Neural Information Processing Systems, pages 4204-4214, 2019.","title":"Understanding attention and generalization in graph neural networks","authors":"Boris Knyazev, Graham W Taylor, Mohamed Amer","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2016] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph se- quence neural networks. In International Conference on Learning Representations, 2016.","title":"Gated graph sequence neural networks","authors":"Yujia Li, Daniel Tarlow, Marc Brockschmidt, Richard Zemel","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2019] Guohao Li, Matthias Müller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In The IEEE International Conference on Com- puter Vision (ICCV), 2019.","title":"Deepgcns: Can gcns go as deep as cnns?","authors":"Li","year":"2019","venue":"The IEEE International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maron et al., 2019] Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. In Advances in Neural Information Processing Systems, 2019.","title":"Provably powerful graph networks","authors":"Maron","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Morris et al., 2019] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gau- rav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceed- ings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pages 4602-4609, 2019.","title":"Weisfeiler and leman go neural: Higher-order graph neural networks","authors":"Christopher Morris, Martin Ritzert, Matthias Fey, Jan William L Hamilton, Gaurav Eric Lenssen, Martin Rattan, Grohe","year":"2019","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Niepert et al., 2016] Mathias Niepert, Mohamed Ahmed, and Konstantin Kutzkov. Learning convolutional neural networks for graphs. In International conference on ma- chine learning, pages 2014-2023, 2016.","title":"Learning convolutional neural networks for graphs","authors":"Mathias Niepert, Mohamed Ahmed, Konstantin Kutzkov","year":"2016","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Scarselli et al., 2009] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009.","title":"The graph neural network model","authors":"Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, Gabriele Monfardini","year":"2009","venue":"IEEE Transactions on Neural Networks","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Shervashidze et al., 2011] Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(Sep):2539- 2561, 2011.","title":"Weisfeiler-lehman graph kernels","authors":"Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, Karsten M Borgwardt","year":"2011","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Thekumparampil et al., 2018] Kiran K Thekumparampil, Chong Wang, Sewoong Oh, and Li-Jia Li. Attention-based graph neural network for semi-supervised learning. arXiv preprint arXiv:1803.03735, 2018.","title":"Attention-based graph neural network for semi-supervised learning","authors":"Chong Kiran K Thekumparampil, Sewoong Wang, Li-Jia Oh, Li","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In Interna- tional Conference on Learning Representations, 2018.","title":"Graph Attention Networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Weisfeiler and Leman, 1968] B Weisfeiler and A Leman. The reduction of a graph to canonical form and the algebra which appears therein. NTI, Series, 2, 1968.","title":"The reduction of a graph to canonical form and the algebra which appears therein","authors":"B Weisfeiler, Leman","year":"1968","venue":"NTI, Series","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A com- prehensive survey on graph neural networks. IEEE Trans- actions on Neural Networks and Learning Systems, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xinyi and Chen, 2019] Zhang Xinyi and Lihui Chen. Cap- sule graph neural network. In International Conference on Learning Representations, 2019.","title":"Capsule graph neural network","authors":"Zhang Xinyi, Lihui Chen","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2018] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Ma- chine Learning, pages 5449-5458, 2018.","title":"Representation learning on graphs with jumping knowledge networks","authors":"Xu","year":"2018","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In International Conference on Learning Rep- resentations, 2019.","title":"How powerful are graph neural networks","authors":"Xu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ying et al., 2018] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differen- tiable pooling. In Advances in Neural Information Pro- cessing Systems, pages 4805-4815, 2018.","title":"Hierarchical graph representation learning with differentiable pooling","authors":"Ying","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learn- ing architecture for graph classification. In Proceedings of AAAI Conference on Artificial Inteligence, 2018.","title":"An end-to-end deep learning architecture for graph classification","authors":"Zhang","year":"2018","venue":"Proceedings of AAAI Conference on Artificial Inteligence","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2018] Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Graph neu- ral networks: A review of methods and applications. arXiv preprint arXiv:1812.08434, 2018.","title":"Graph neural networks: A review of methods and applications","authors":"Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Maosong Sun","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}