{"type":"ScholarlyArticle","title":"Some Might Say All You Need Is Sum","authors":[{"name":"Eran Rosenbluth","slug":"eran-rosenbluth","given":"Eran","family":"Rosenbluth","affiliation":"RWTH Aachen University"},{"name":"Jan Tönshoff","slug":"jan-tonshoff","given":"Jan","family":"Tönshoff","affiliation":"RWTH Aachen University"},{"name":"Martin Grohe","slug":"martin-grohe","given":"Martin","family":"Grohe","affiliation":"RWTH Aachen University"}],"abstract":"The expressivity of Graph Neural Networks (GNNs) is dependent on the aggregation functions they employ. Theoretical works have pointed towards Sum aggregation GNNs subsuming every other GNNs, while certain practical works have observed a clear advantage to using Mean and Max. An examination of the theoretical guarantee identifies two caveats. First, it is size-restricted, that is, the power of every specific GNN is limited to graphs of a specific size. Successfully processing larger graphs may require an other GNN, and so on. Second, it concerns the power to distinguish non-isomorphic graphs, not the power to approximate general functions on graphs, and the former does not necessarily imply the latter.\r\nIt is desired that a GNN's usability will not be limited to graphs of any specific size. Therefore, we explore the realm of unrestricted-size expressivity. We prove that basic functions, which can be computed exactly by Mean or Max GNNs, are inapproximable by any Sum GNN. We prove that under certain restrictions, every Mean or Max GNN can be approximated by a Sum GNN, but even there, a combination of (Sum, [Mean/Max]) is more expressive than Sum alone. Lastly, we prove further expressivity limitations for GNNs with a broad class of aggregations.","doi":"10.24963/ijcai.2023/464","url":"https://ijcai.org/proceedings/2023/464","pdf":"https://ijcai.org/proceedings/2023/0464.pdf","markdown":"https://ijcai.org/proceedings/2023/464.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/464","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":464,"pages":{"first":4172,"last":4179},"keywords":["Machine Learning: ML: Theory of deep learning","Machine Learning: ML: Sequence and graph learning","Machine Learning: ML: Learning theory"],"videos":[],"references":[{"position":1,"raw":"[Abboud et al., 2021] Ralph Abboud, İsmail İlkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node ini- tialization. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial In- telligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 2112-2118. ijcai.org, 2021.","title":"The surprising power of graph neural networks with random node initialization","authors":"Ralph Abboud, Martin İsmail İlkan Ceylan, Thomas Grohe, Lukasiewicz","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event","doi":null,"url":null,"internal_url":"/proceedings/2021/291"},{"position":2,"raw":"[Barceló et al., 2020a] Pablo Barceló, Egor V Kostylev, Mikael Monet, Jorge Pérez, Juan Reutter, and Juan-Pablo Silva. The logical expressiveness of graph neural net- works. In 8th International Conference on Learning Rep- resentations (ICLR 2020), 2020.","title":"The logical expressiveness of graph neural networks","authors":"Barceló","year":"2020","venue":"8th International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Barceló et al., 2020b] Pablo Barceló, Egor V Kostylev, Mikaël Monet, Jorge Pérez, Juan L Reutter, and Juan- Pablo Silva. The expressive power of graph neural net- works as a query language. ACM SIGMOD Record, 49(2):6-17, 2020.","title":"The expressive power of graph neural networks as a query language","authors":"Barceló","year":"2020","venue":"ACM SIGMOD Record","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Barceló et al., 2021] Pablo Barceló, Floris Geerts, Juan Reutter, and Maksimilian Ryschkov. Graph neural net- works with local graph parameters. Advances in Neural Information Processing Systems, 34:25280-25293, 2021.","title":"Graph neural networks with local graph parameters","authors":"Barceló","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cappart et al., 2021] Quentin Cappart, Didier Chételat, Elias B. Khalil, Andrea Lodi, Christopher Morris, and Petar Velickovic. Combinatorial optimization and reason- ing with graph neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Confer- ence on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 4348-4355.","title":"Combinatorial optimization and reasoning with graph neural networks","authors":"Quentin Cappart, Didier Chételat, Elias B Khalil, Andrea Lodi, Christopher Morris, Petar Velickovic","year":"2021","venue":"Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event","doi":null,"url":null,"internal_url":"/proceedings/2021/595"},{"position":6,"raw":"[Chen et al., 2019] Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. On the equivalence between graph isomorphism testing and function approximation with gnns. Advances in neural information processing sys- tems, 32, 2019.","title":"On the equivalence between graph isomorphism testing and function approximation with gnns","authors":"Chen","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Corso et al., 2020] Gabriele Corso, Luca Cavalleri, Do- minique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances in Neural Information Processing Systems, 33:13260-13271, 2020.","title":"Principal neighbourhood aggregation for graph nets","authors":"Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, Petar Veličković","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geerts and Reutter, 2022] Floris Geerts and Juan L. Reut- ter. Expressiveness and approximation properties of graph neural networks. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.","title":"Expressiveness and approximation properties of graph neural networks","authors":"Reutter ; Geerts, Floris Geerts, Juan L Reutter","year":"2022","venue":"The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gilmer et al., 2017] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In Interna- tional conference on machine learning, pages 1263-1272.","title":"Neural message passing for quantum chemistry","authors":"Justin Gilmer, S Samuel, Patrick F Schoenholz, Oriol Riley, George E Vinyals, Dahl","year":null,"venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"PMLR, 2017.","title":null,"authors":"Pmlr","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Golomb, 1962] Michael Golomb. Lectures on theory of approximation. Argonne National Laboratory, Applied Mathematics Division, 1962.","title":"Lectures on theory of approximation","authors":"Michael Golomb","year":"1962","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Grohe, 2021] Martin Grohe. The logic of graph neural net- works. In 2021 36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS), pages 1-17. IEEE, 2021.","title":"The logic of graph neural networks","authors":"Martin Grohe","year":"2021","venue":"36th Annual ACM/IEEE Symposium on Logic in Computer Science (LICS)","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing sys- tems, 30, 2017.","title":"Inductive representation learning on large graphs","authors":"Hamilton","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kipf and Welling, 2017] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"Thomas N Kipf, Max Welling","year":"2017","venue":"5th International Conference on Learning Representations, ICLR 2017","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Maron et al., 2019] Haggai Maron, Heli Ben-Hamu, Hadar Serviansky, and Yaron Lipman. Provably powerful graph networks. Advances in neural information processing sys- tems, 32, 2019.","title":"Provably powerful graph networks","authors":"Maron","year":"2019","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mayans, 2006] Robert Mayans. The polynomial of best approximation. https://www.maa.org/sites/default/files/ images/upload library/4/vol6/Mayans/Best.html, 2006. Accessed: 2023-06-03.","title":"The polynomial of best approximation","authors":"Robert Mayans","year":"2006","venue":null,"doi":null,"url":"https://www.maa.org/sites/default/files/images/uploadlibrary/4/vol6/Mayans/Best.html","internal_url":null},{"position":17,"raw":"[Morris et al., 2019] Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gau- rav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceed- ings of the AAAI conference on artificial intelligence, vol- ume 33, pages 4602-4609, 2019.","title":"Weisfeiler and leman go neural: Higher-order graph neural networks","authors":"Christopher Morris, Martin Ritzert, Matthias Fey, Jan William L Hamilton, Gaurav Eric Lenssen, Martin Rattan, Grohe","year":"2019","venue":"Proceedings of the AAAI conference on artificial intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Morris et al., 2020] Christopher Morris, Gaurav Rattan, and Petra Mutzel. Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings. Advances in Neural Information Processing Systems, 33:21824-21840, 2020.","title":"Weisfeiler and leman go sparse: Towards scalable higher-order graph embeddings","authors":"Christopher Morris, Gaurav Rattan, Petra Mutzel","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sato et al., 2021] Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neu- ral networks. In Proceedings of the 2021 SIAM Interna- tional Conference on Data Mining (SDM), pages 333-341.","title":"Random features strengthen graph neural networks","authors":"Ryoma Sato, Makoto Yamada, Hisashi Kashima","year":null,"venue":"Proceedings of the 2021 SIAM International Conference on Data Mining (SDM)","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Scarselli et al., 2008] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.","title":"The graph neural network model","authors":"Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, Gabriele Monfardini","year":"2008","venue":"IEEE transactions on neural networks","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tönshoff et al., 2022] Jan Tönshoff, Berke Kisin, Jakob Lindner, and Martin Grohe. One model, any csp: Graph neural networks as fast global search heuristics for con- straint satisfaction. arXiv preprint arXiv:2208.10227, 2022.","title":"One model, any csp: Graph neural networks as fast global search heuristics for constraint satisfaction","authors":"Tönshoff","year":"2022","venue":null,"doi":null,"url":null,"internal_url":"/proceedings/2023/476"},{"position":22,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A com- prehensive survey on graph neural networks. IEEE trans- actions on neural networks and learning systems, 32(1):4- 24, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"IEEE transactions on neural networks and learning systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In 7th International Conference on Learning Rep- resentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.","title":"How powerful are graph neural networks?","authors":"Xu","year":"2019","venue":"7th International Conference on Learning Representations, ICLR 2019","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}