{"type":"ScholarlyArticle","title":"Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction","authors":[{"name":"Mateusz Jurewicz","slug":"mateusz-jurewicz","given":"Mateusz","family":"Jurewicz","affiliation":"IT University of Copenhagen"},{"name":"Leon Derczynski","slug":"leon-derczynski","given":"Leon","family":"Derczynski","affiliation":"IT University of Copenhagen"}],"abstract":"The task of learning to map an input set onto a permuted sequence of its elements is challenging for neural networks. Set-to-sequence problems occur in natural language processing, computer vision and structure prediction, where interactions between elements of large sets define the optimal output. Models must exhibit relational reasoning, handle varying cardinalities and manage combinatorial complexity. Previous attention-based methods require n layers of their set transformations to explicitly represent n-th order relations. Our aim is to enhance their ability to efficiently model higher-order interactions through an additional interdependence component. We propose a novel neural set encoding method called the Set Interdependence Transformer, capable of relating the set's permutation invariant representation to its elements within sets of any cardinality. We combine it with a permutation learning module into a complete, 3-part set-to-sequence model and demonstrate its state-of-the-art performance on a number of tasks. These range from combinatorial optimization problems, through permutation learning challenges on both synthetic and established NLP datasets for sentence ordering, to a novel domain of product catalog structure prediction. Additionally, the network's ability to generalize to unseen sequence lengths is investigated and a comparative empirical analysis of the existing methods' ability to learn higher-order interactions is provided.","doi":"10.24963/ijcai.2022/434","url":"https://ijcai.org/proceedings/2022/434","pdf":"https://ijcai.org/proceedings/2022/0434.pdf","markdown":"https://ijcai.org/proceedings/2022/434.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/434","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":434,"pages":{"first":3128,"last":3134},"keywords":["Machine Learning: Relational Learning","Machine Learning: Structured Prediction","Natural Language Processing: Other","Machine Learning: Attention Models","Machine Learning: Sequence and Graph Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987031","length_sec":75,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985174","length_sec":651,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Ai et al., 2018] Qingyao Ai, Keping Bi, Jiafeng Guo, and W Bruce Croft. Learning a deep listwise context model for ranking refinement. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 135-144, 2018.","title":"Learning a deep listwise context model for ranking refinement","authors":"Ai","year":"2018","venue":"The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cui et al., 2018] Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. Deep attentive sentence ordering net- work. In Proc. EMNLP, pages 4340-4349, 2018.","title":"Deep attentive sentence ordering network","authors":"Baiyun Cui, Yingming Li, Ming Chen, Zhongfei Zhang","year":"2018","venue":"Proc. EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proc. NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proc. NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Diallo et al., 2020] Aïssatou Diallo, Markus Zopf, and Jo- hannes Fürnkranz. Permutation learning via lehmer codes. In ECAI 2020, pages 1095-1102. IOS Press, 2020.","title":"Permutation learning via lehmer codes","authors":"Diallo","year":"2020","venue":"ECAI 2020","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Farag, 2021] Youmna Farag. Neural approaches to dis- course coherence: modeling, evaluation and application. PhD thesis, University of Cambridge, 2021.","title":"Youmna Farag. Neural approaches to discourse coherence: modeling, evaluation and application","authors":"Farag","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hopfield and Tank, 1985] John J Hopfield and David W Tank. \"neural\" computation of decisions in optimization problems. Biological cybernetics, 52(3):141-152, 1985.","title":"neural\" computation of decisions in optimization problems","authors":"J John, David W Hopfield, Tank","year":"1985","venue":"Biological cybernetics","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huang et al., 2020] Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, and Austin Benson. Better set representations for relational reasoning. Proc. NeurIPS, 34, 2020.","title":"Better set representations for relational reasoning","authors":"Qian Huang, Horace He, Abhay Singh, Yan Zhang, Ser-Nam Lim, Austin Benson","year":"2020","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jurewicz and Derczynski, 2021a] Mateusz Jurewicz and Leon Derczynski. Procat: Product catalogue dataset for implicit clustering. Proc. NeurIPS: Track on Datasets and Benchmarks, 35, 2021.","title":"Procat: Product catalogue dataset for implicit clustering","authors":"Derczynski Jurewicz, Mateusz Jurewicz, Leon Derczynski","year":"2021","venue":"Proc. NeurIPS: Track on Datasets and Benchmarks","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jurewicz and Derczynski, 2021b] Mateusz Jurewicz and Leon Derczynski. Set-to-sequence methods in machine learning: a review. Journal of Artificial Intelligence Research, 71:885-924, 2021.","title":"Set-to-sequence methods in machine learning: a review","authors":"Derczynski Jurewicz, Mateusz Jurewicz, Leon Derczynski","year":"2021","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kool et al., 2018] Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In Proc. ICLR, 2018.","title":"Attention, learn to solve routing problems!","authors":"Wouter Kool, Herke Van Hoof, Max Welling","year":"2018","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee and Lee, 2019] Juho Lee and Yoonho Lee. Set trans- former: A framework for attention-based permutation- invariant neural networks. In International Conference on Machine Learning, pages 3744-3753. PMLR, 2019.","title":"Set transformer: A framework for attention-based permutationinvariant neural networks","authors":"Lee, Lee ; ] Juho Lee, Yoonho Lee","year":"2019","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Gao, 2020] Liangcheng Li and Feiyu Gao. An end- to-end ocr text re-organization sequence learning for rich- text detail image comprehension. In ECCV, pages 85-100. Springer, 2020.","title":"An endto-end ocr text re-organization sequence learning for richtext detail image comprehension","authors":"Gao ; Li, Liangcheng Li, Feiyu Gao","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. In CoRR, abs/1711.05101, 2017., 2017.","title":"Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam","authors":"Hutter Loshchilov","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lyu, 2005] Siwei Lyu. A kernel between unordered sets of data: The gaussian mixture approach. In European Con- ference on Machine Learning, pages 255-267. Springer, 2005.","title":"A kernel between unordered sets of data: The gaussian mixture approach","authors":"Siwei Lyu, Lyu","year":"2005","venue":"European Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mostafazadeh and Chambers, 2016] Nasrin Mostafazadeh and Nathan Chambers. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proc. NAACL, pages 839-849, 2016.","title":"Nasrin Mostafazadeh and Nathan Chambers. A corpus and cloze evaluation for deeper understanding of commonsense stories","authors":"Chambers Mostafazadeh","year":"2016","venue":"Proc. NAACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pandey and Chowdary, 2020] Divesh Pandey and C Ravin- dranath Chowdary. Modeling coherence by ordering para- graphs using pointer networks. Neural Networks, 126:36- 41, 2020.","title":"Modeling coherence by ordering paragraphs using pointer networks","authors":"Divesh Pandey, C Ravindranath, Chowdary","year":"2020","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Santa Cruz and Fernando, 2017] Rodrigo Santa Cruz and Basura Fernando. Deeppermnet: Visual permutation learning. In Proc. CVPR, pages 3949-3957, 2017.","title":"Rodrigo Santa Cruz and Basura Fernando. Deeppermnet: Visual permutation learning","authors":"Santa Cruz, Fernando","year":"2017","venue":"Proc. CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Skianis and Konstantinos, 2020] Skianis and Konstantinos. Rep the set: Neural networks for learning set representa- tions. In International conference on artificial intelligence and statistics, pages 1410-1420. PMLR, 2020.","title":"Skianis and Konstantinos. Rep the set: Neural networks for learning set representations","authors":"Konstantinos Skianis","year":"2020","venue":"International conference on artificial intelligence and statistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Proc. NeurIPS, vol- ume 2015-January, pages 2692-2700, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Samy Bengio, and Man- junath Kudlur. Order matters: Sequence to sequence for sets. In Proc. ICLR, pages 1-11, 2016.","title":"Order matters: Sequence to sequence for sets","authors":"Oriol Vinyals, Samy Bengio, Manjunath Kudlur","year":"2016","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vinyals et al., 2019] Oriol Vinyals, Igor Babuschkin, Woj- ciech M. Czarnecki, and Mathieu. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Na- ture, 575(7782):350-354, 2019.","title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","authors":"Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Mathieu","year":"2019","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Wan, 2019] Tianming Wang and Xiaojun Wan. Hierarchical Attention Networks for Sentence Ordering. AAAI, 33:7184-7191, 2019.","title":"Hierarchical Attention Networks for Sentence Ordering","authors":"Wang, Wan ; ] Tianming Wang, Xiaojun Wan","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang and Wang, 2020] Bo Yang and Sen Wang. Robust at- tentional aggregation of deep feature sets for multi-view 3d reconstruction. International Journal of Computer Vi- sion, 128(1):53-73, 2020.","title":"Robust attentional aggregation of deep feature sets for multi-view 3d reconstruction","authors":"Bo Yang, Sen Wang","year":"2020","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yew and Lee, 2020] Zi Jian Yew and Gim Hee Lee. Rpm- net: Robust point matching using learned features. In IEEE/CVF, pages 11824-11833, 2020.","title":"Rpmnet: Robust point matching using learned features","authors":"Jian Zi, Gim Yew, Lee Hee","year":"2020","venue":"IEEE/CVF","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin et al., 2020] Yongjing Yin, Fandong Meng, Jinsong Su, Yubin Ge, Lingeng Song, Jie Zhou, and Jiebo Luo. En- hancing pointer network for sentence ordering with pair- wise ordering predictions. In AAAI, volume 34, pages 9482-9489, 2020.","title":"Enhancing pointer network for sentence ordering with pairwise ordering predictions","authors":"Yongjing Yin, Fandong Meng, Jinsong Su, Yubin Ge, Lingeng Song, Jie Zhou, Jiebo Luo","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2019] Xiang Yu, Ngoc Thang Vu, and Jonas Kuhn. Learning the dyck language with attention-based seq2seq models. In ACL Workshop BlackboxNLP, pages 138-146, 2019.","title":"Learning the dyck language with attention-based seq2seq models","authors":"Xiang Yu, Ngoc Thang Vu, Jonas Kuhn","year":"2019","venue":"ACL Workshop BlackboxNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zaheer et al., 2017] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. In Proc. NeurIPS, vol- ume 30, 2017.","title":"Deep sets","authors":"Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, Alexander Smola","year":"2017","venue":"Proc. NeurIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2019] Yan Zhang, Adam Prügel-Bennett, and Jonathon Hare. Learning representations of sets through optimized permutations. In Proc. ICLR, 2019.","title":"Learning representations of sets through optimized permutations","authors":"Yan Zhang, Adam Prügel-Bennett, Jonathon Hare","year":"2019","venue":"Proc. ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}