{"type":"ScholarlyArticle","title":"Logic Constrained Pointer Networks for Interpretable Textual Similarity","authors":[{"name":"Subhadeep Maji","slug":"subhadeep-maji","given":"Subhadeep","family":"Maji","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Rohan Kumar","slug":"rohan-kumar","given":"Rohan","family":"Kumar","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Manish Bansal","slug":"manish-bansal","given":"Manish","family":"Bansal","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Kalyani Roy","slug":"kalyani-roy","given":"Kalyani","family":"Roy","affiliation":"Indian Institute of Technology , Kharagpur"},{"name":"Pawan Goyal","slug":"pawan-goyal","given":"Pawan","family":"Goyal","affiliation":"Indian Institute of Technology , Kharagpur"}],"abstract":"Systematically discovering semantic relationships in text is an important and extensively studied area in Natural Language Processing, with various tasks such as entailment, semantic similarity, etc. Decomposability of sentence-level scores via subsequence alignments has been proposed as a way to make models more interpretable. We study the problem of aligning components of sentences leading to an interpretable model for semantic textual similarity. In this paper, we introduce a novel pointer network based model with a sentinel gating function to align constituent chunks, which are represented using BERT. We improve this base model with a loss function to equally penalize misalignments in both sentences, ensuring the alignments are bidirectional. Finally, to guide the network with structured external knowledge, we introduce first-order logic constraints based on ConceptNet and syntactic knowledge. The model achieves an F1 score of 97.73 and 96.32 on the benchmark SemEval datasets for the chunk alignment task, showing large improvements over the existing solutions. Source code is available at https://github.com/manishb89/interpretable_sentence_similarity","doi":"10.24963/ijcai.2020/333","url":"https://ijcai.org/proceedings/2020/333","pdf":"https://ijcai.org/proceedings/2020/0333.pdf","markdown":"https://ijcai.org/proceedings/2020/333.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/333","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":333,"pages":{"first":2405,"last":2411},"keywords":["Natural Language Processing: Natural Language Processing","Natural Language Processing: Natural Language Semantics","Machine Learning: Deep Learning: Sequence Modeling"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25421/file_25421.m3u8","length_sec":341,"provider":"hls"}],"references":[{"position":1,"raw":"[Agirre et al., 2012] Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. Semeval-2012 task 6: A pilot on semantic textual similarity. In STARSEM and SemEval, pages 385-393. Association for Computational Linguis- tics, 2012.","title":"Semeval-2012 task 6: A pilot on semantic textual similarity","authors":"Eneko Agirre, Mona Diab, Daniel Cer, Aitor Gonzalez-Agirre","year":"2012","venue":"STARSEM and SemEval","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Agirre et al., 2015] Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. Semeval-2015 task 2: Semantic tex- tual similarity, english, spanish and pilot on interpretabil- ity. In SemEval, pages 252-263, 2015.","title":"Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability","authors":"Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea","year":"2015","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Agirre et al., 2016] Eneko Agirre, Aitor Gonzalez-Agirre, Inigo Lopez-Gazpio, Montse Maritxalar, German Rigau, and Larraitz Uria. Semeval-2016 task 2: Interpretable semantic textual similarity. In SemEval, pages 512-524, 2016.","title":"Semeval-2016 task 2: Interpretable semantic textual similarity","authors":"Eneko Agirre, Aitor Gonzalez-Agirre, Inigo Lopez-Gazpio, Montse Maritxalar, German Rigau, Larraitz Uria","year":"2016","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Banjade et al., 2015] Rajendra Banjade, Nobal Bikram Ni- raula, Nabin Maharjan, Vasile Rus, Dan Stefanescu, Mi- hai Lintean, and Dipesh Gautam. Nerosim: A system for measuring and interpreting semantic textual similarity. In SemEval, pages 164-171, 2015.","title":"Nerosim: A system for measuring and interpreting semantic textual similarity","authors":"Rajendra Banjade, Nobal Bikram Niraula, Nabin Maharjan, Dan Vasile Rus, Mihai Stefanescu, Dipesh Lintean, Gautam","year":"2015","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bär et al., 2012] Daniel Bär, Chris Biemann, Iryna Gurevych, and Torsten Zesch. Ukp: Computing semantic textual similarity by combining multiple content similarity measures. In STARSEM and SemEval, pages 435-440. Association for Computational Linguistics, 2012.","title":"Ukp: Computing semantic textual similarity by combining multiple content similarity measures","authors":"Daniel Bär, Chris Biemann, Iryna Gurevych, Torsten Zesch","year":"2012","venue":"STARSEM and SemEval","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chang et al., 2010] Ming-Wei Chang, Dan Goldwasser, Dan Roth, and Vivek Srikumar. Discriminative learning over constrained latent representations. In NAACL-HLT, pages 429-437. Association for Computational Linguis- tics, 2010.","title":"Discriminative learning over constrained latent representations","authors":"Ming-Wei Chang, Dan Goldwasser, Dan Roth, Vivek Srikumar","year":"2010","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Collobert et al., 2011] Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. Natural language processing (almost) from scratch. JMLR, 12(Aug):2493-2537, 2011.","title":"Natural language processing (almost) from scratch","authors":"Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa","year":"2011","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Conneau et al., 2017] Alexis Conneau, Douwe Kiela, Hol- ger Schwenk, Loïc Barrault, and Antoine Bordes. Super- vised learning of universal sentence representations from natural language inference data. In EMNLP, pages 670- 680, 2017.","title":"Supervised learning of universal sentence representations from natural language inference data","authors":"Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, Antoine Bordes","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. In NeurIPS, pages 2292-2300, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLT, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Franc ¸a et al., 2014] Manoel VM Franc ¸a, Gerson Zaverucha, and Artur S d'Avila Garcez. Fast relational learning us- ing bottom clause propositionalization with artificial neu- ral networks. Machine learning, 94(1):81-104, 2014.","title":"Fast relational learning using bottom clause propositionalization with artificial neural networks","authors":"Franc ¸a","year":"2014","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hänig et al., 2015] Christian Hänig, Robert Remus, and Xose De La Puente. Exb themis: Extensive feature extrac- tion from word alignments for semantic textual similarity. In SemEval, pages 264-268, 2015.","title":"Christian Hänig, Robert Remus, and Xose De La Puente. Exb themis: Extensive feature extraction from word alignments for semantic textual similarity","authors":"Hänig","year":"2015","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Heilman and Smith, 2010] Michael Heilman and Noah A Smith. Tree edit models for recognizing textual entail- ments, paraphrases, and answers to questions. In NAACL- HLT, pages 1011-1019. Association for Computational Linguistics, 2010.","title":"Tree edit models for recognizing textual entailments, paraphrases, and answers to questions","authors":"Smith ; Heilman, Michael Heilman, Noah A Smith","year":"2010","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Honnibal and Montani, 2017] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and in- cremental parsing. To appear, 2017.","title":"Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing","authors":"Montani Honnibal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hu et al., 2016] Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural net- works with logic rules. In ACL, pages 2410-2420, Berlin, Germany, 2016. Association for Computational Linguis- tics.","title":"Harnessing deep neural networks with logic rules","authors":"Hu","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Jawahar et al., 2019] Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does bert learn about the structure of language? In ACL, pages 3651-3657, 2019.","title":"What does bert learn about the structure of language? In ACL","authors":"Ganesh Jawahar, Benoît Sagot, Djamé Seddah","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Jimenez et al., 2012] Sergio Jimenez, Claudia Becerra, and Alexander Gelbukh. Soft cardinality: A parameterized similarity function for text comparison. In STARSEM and SemEval, pages 449-453. Association for Computational Linguistics, 2012.","title":"Soft cardinality: A parameterized similarity function for text comparison","authors":"Sergio Jimenez, Claudia Becerra, Alexander Gelbukh","year":"2012","venue":"STARSEM and SemEval","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kazmi and Schüller, 2016] Mishal Kazmi and Peter Schüller. Inspire at semeval-2016 task 2: Interpretable semantic textual similarity alignment based on answer set programming. In SemEval, pages 1109-1115, 2016.","title":"Inspire at semeval-2016 task 2: Interpretable semantic textual similarity alignment based on answer set programming","authors":"Schüller Kazmi, Mishal Kazmi, Peter Schüller","year":"2016","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Koehn et al., 2003] Philipp Koehn, Franz Josef Och, and Daniel Marcu. Statistical phrase-based translation. In NAACL-HLT, pages 48-54. Association for Computa- tional Linguistics, 2003.","title":"Statistical phrase-based translation","authors":"Philipp Koehn, Franz, Josef Och, Daniel Marcu","year":"2003","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Konopik et al., 2016] Miloslav Konopik, Ondrej Prazak, David Steinberger, and Tomáš Brychcín. Uwb at semeval- 2016 task 2: Interpretable semantic textual similarity with distributional semantics for chunks. In SemEval, pages 803-808, 2016.","title":"Uwb at semeval-2016 task 2: Interpretable semantic textual similarity with distributional semantics for chunks","authors":"Konopik","year":"2016","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Li and Srikumar, 2016] Tao Li and Vivek Srikumar. Ex- ploiting sentence similarities for better alignments. In EMNLP, pages 2193-2203, 2016.","title":"Exploiting sentence similarities for better alignments","authors":"Srikumar ; ] Tao Li, Vivek Li, Srikumar","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Li and Srikumar, 2019] Tao Li and Vivek Srikumar. Aug- menting neural networks with first-order logic. In ACL, pages 292-302, Florence, Italy, July 2019. Association for Computational Linguistics.","title":"Augmenting neural networks with first-order logic","authors":"Srikumar ; Li, Tao Li, Vivek Srikumar","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pagliardini et al., 2018] Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence em- beddings using compositional n-gram features. In NAACL- HLT, pages 528-540, 2018.","title":"Unsupervised learning of sentence embeddings using compositional n-gram features","authors":"Matteo Pagliardini, Prakhar Gupta, Martin Jaggi","year":"2018","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pavlick et al., 2015] Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison- Burch. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classifi- cation. In ACL-IJCNLP, pages 425-430, 2015.","title":"Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification","authors":"Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP-IJCNLP, pages 3973-3983, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Shao, 2017] Yang Shao. Hcti at semeval-2017 task 1: Use convolutional neural network to evaluate semantic textual similarity. In SemEval, pages 130-133, 2017.","title":"Hcti at semeval-2017 task 1: Use convolutional neural network to evaluate semantic textual similarity","authors":"Yang Shao","year":"2017","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Socher et al., 2011] Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. Dy- namic pooling and unfolding recursive autoencoders for paraphrase detection. In NeurIPS, pages 801-809, 2011.","title":"Dynamic pooling and unfolding recursive autoencoders for paraphrase detection","authors":"Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, Andrew Y Ng","year":"2011","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Speer et al., 2017] Robert Speer, Joshua Chin, and Cather- ine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI, 2017.","title":"Conceptnet 5.5: An open multilingual graph of general knowledge","authors":"Robert Speer, Joshua Chin, Catherine Havasi","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sultan et al., 2015] Md Arafat Sultan, Steven Bethard, and Tamara Sumner. Dls@ cu: Sentence similarity from word alignment and semantic vector composition. In SemEval, pages 148-153, 2015.","title":"Dls@ cu: Sentence similarity from word alignment and semantic vector composition","authors":"Md Arafat Sultan, Steven Bethard, Tamara Sumner","year":"2015","venue":"SemEval","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. In ACL-IJCNLP, pages 1556-1566, Beijing, China, 2015. Association for Computational Linguistics.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Towell et al., 1990] Geoffrey G Towell, Jude W Shavlik, and Michiel O Noordewier. Refinement of approximate domain theories by knowledge-based neural networks. In AAAI, volume 861866. Boston, MA, 1990.","title":"Refinement of approximate domain theories by knowledge-based neural networks","authors":"Geoffrey G Towell, Jude W Shavlik, Michiel, Noordewier","year":"1990","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NeurIPS, pages 2692-2700, 2015.","title":"Pointer networks","authors":"Oriol Vinyals, Meire Fortunato, Navdeep Jaitly","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}