{"type":"ScholarlyArticle","title":"Training Naturalized Semantic Parsers with Very Little Data","authors":[{"name":"Subendhu Rongali","slug":"subendhu-rongali","given":"Subendhu","family":"Rongali","affiliation":"University of Massachusetts Amherst; Amazon Alexa AI , New York"},{"name":"Konstantine Arkoudas","slug":"konstantine-arkoudas","given":"Konstantine","family":"Arkoudas","affiliation":"Amazon Alexa AI , New York"},{"name":"Melanie Rubino","slug":"melanie-rubino","given":"Melanie","family":"Rubino","affiliation":"Amazon Alexa AI , New York"},{"name":"Wael Hamza","slug":"wael-hamza","given":"Wael","family":"Hamza","affiliation":"Amazon Alexa AI , New York"}],"abstract":"Semantic parsing is an important NLP problem, particularly for voice assistants such as Alexa and Google Assistant. State-of-the-art (SOTA) semantic parsers are seq2seq architectures based on large language models that have been pretrained on vast amounts of text. To better leverage that pretraining, recent work has explored a reformulation of semantic parsing whereby the output sequences are themselves natural language sentences, but in a controlled fragment of natural language. This approach delivers strong results, particularly for few-shot semantic parsing, which is of key importance in practice and the focus of our paper. We push this line of work forward by introducing an automated methodology that delivers very significant additional improvements by utilizing modest amounts of unannotated data, which is typically easy to obtain. Our method is based on a novel synthesis of four techniques: joint training with auxiliary unsupervised tasks; constrained decoding; self-training; and paraphrasing. We show that this method delivers new SOTA few-shot performance on the Overnight dataset, particularly in very low-resource settings, and very compelling few-shot results on a new semantic parsing dataset.","doi":"10.24963/ijcai.2022/604","url":"https://ijcai.org/proceedings/2022/604","pdf":"https://ijcai.org/proceedings/2022/0604.pdf","markdown":"https://ijcai.org/proceedings/2022/604.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/604","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Natural Language Processing","track":"Main Track","number":604,"pages":{"first":4353,"last":4359},"keywords":["Machine Learning: Few-shot learning","Natural Language Processing: Dialogue and Interactive Systems","Natural Language Processing: Tagging, Chunking, and Parsing","Natural Language Processing: Natural Language Semantics"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986540","length_sec":84,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984683","length_sec":330,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Berant and Liang, 2014] Jonathan Berant and Percy Liang. Semantic parsing via paraphrasing. In ACL (1), pages 1415-1425. ACL (Association for Computer Linguistics), 2014.","title":"Semantic parsing via paraphrasing","authors":"Liang Berant, Jonathan Berant, Percy Liang","year":"2014","venue":"ACL (1)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Berant et al., 2013] J. Berant, A. Chou, R. Frostig, and P. Liang. Semantic parsing on Freebase from question- answer pairs. In Empirical Methods in Natural Language Processing (EMNLP), 2013.","title":"Semantic parsing on Freebase from questionanswer pairs","authors":"J Berant, A Chou, R Frostig, P Liang","year":"2013","venue":"Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brown et al., 2020] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.","title":"Language models are few-shot learners","authors":"Benjamin Tom B Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Arvind Dhariwal, Pranav Neelakantan, Girish Shyam, Amanda Sastry, Askell","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cao et al., 2019] Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. Semantic parsing with dual learning. arXiv preprint arXiv:1907.05343, 2019.","title":"Semantic parsing with dual learning","authors":"Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, Kai Yu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Dong et al., 2018] Li Dong, Chris Quirk, and Mirella Lap- ata. Confidence modeling for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 743-753, Melbourne, Australia, July 2018. Associ- ation for Computational Linguistics.","title":"Confidence modeling for neural semantic parsing","authors":"Li Dong, Chris Quirk, Mirella Lapata","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Einolghozati et al., 2019] Arash Einolghozati, Panupong Pasupat, Sonal Gupta, Rushin Shah, Mrinal Mohit, Mike Lewis, and Luke Zettlemoyer. Improving semantic parsing for task oriented dialog. arXiv preprint arXiv:1902.06000, 2019.","title":"Improving semantic parsing for task oriented dialog","authors":"Arash Einolghozati, Panupong Pasupat, Sonal Gupta, Rushin Shah, Mrinal Mohit, Mike Lewis, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Goldwasser et al., 2011] Dan Goldwasser, Roi Reichart, James Clarke, and Dan Roth. Confidence driven unsuper- vised semantic parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1486-1495, 2011.","title":"Confidence driven unsupervised semantic parsing","authors":"Dan Goldwasser, Roi Reichart, James Clarke, Dan Roth","year":"2011","venue":"Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Krishnamurthy et al., 2017] Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. Neural semantic parsing with type constraints for semi-structured tables. In Proceed- ings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1516-1526, Copen- hagen, Denmark, September 2017. Association for Com- putational Linguistics.","title":"Neural semantic parsing with type constraints for semi-structured tables","authors":"Jayant Krishnamurthy, Pradeep Dasigi, Matt Gardner","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lewis et al., 2019] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: De- noising sequence-to-sequence pre-training for natural lan- guage generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.","title":"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, Luke Zettlemoyer","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McClosky et al., 2006] David McClosky, Eugene Charniak, and Mark Johnson. Effective self-training for parsing. In Proceedings of the Human Language Technology Confer- ence of the NAACL, Main Conference, pages 152-159, 2006.","title":"Effective self-training for parsing","authors":"David Mcclosky, Eugene Charniak, Mark Johnson","year":"2006","venue":"Proceedings of the Human Language Technology Conference of the NAACL, Main Conference","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mihalcea, 2004] Rada Mihalcea. Co-training and self- training for word sense disambiguation. In Proceed- ings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 33-40, 2004. Rongali et al., 2020] Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. Don't parse, generate! a sequence to sequence architecture for task-oriented se- mantic parsing. In Proceedings of The Web Conference 2020, pages 2962-2968, 2020.","title":"Don't parse, generate! a sequence to sequence architecture for task-oriented semantic parsing","authors":"Rada Mihalcea, ; Mihalcea, Rongali","year":"2004","venue":"Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Schick and Schütze, 2021] Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352. Association for Computational Linguistics, June 2021.","title":"It's not just size that matters: Small language models are also few-shot learners","authors":"Schütze Schick, Timo Schick, Hinrich Schütze","year":"2021","venue":"Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Shin et al., 2021] Richard Shin, Christopher H Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil An- tonios Platanios, Adam Pauls, Dan Klein, Jason Eis- ner, and Benjamin Van Durme. Constrained language models yield few-shot semantic parsers. arXiv preprint arXiv:2104.08768, 2021.","title":"Constrained language models yield few-shot semantic parsers","authors":"Richard Shin, Sam Christopher H Lin, Charles Thomson, Subhro Chen, Emmanouil Roy, Adam Antonios Platanios, Dan Pauls, Jason Klein, Benjamin Eisner, Van Durme","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Wang et al., 2015] Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In Proceed- ings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342, 2015.","title":"Building a semantic parser overnight","authors":"Yushi Wang, Jonathan Berant, Percy Liang","year":"2015","venue":"Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Wieting and Gimpel, 2017] John Wieting and Kevin Gim- pel. Paranmt-50m: Pushing the limits of paraphrastic sen- tence embeddings with millions of machine translations. arXiv preprint arXiv:1711.05732, 2017.","title":"Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations","authors":"Gimpel Wieting, John Wieting, Kevin Gimpel","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Wu et al., 2021] Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xunliang Cai. From paraphrasing to semantic parsing: unsupervised semantic parsing via synchronous semantic decoding. arXiv preprint arXiv:2106.06228, 2021.","title":"From paraphrasing to semantic parsing: unsupervised semantic parsing via synchronous semantic decoding","authors":"Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, Xunliang Cai","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Xiao et al., 2019] Chunyang Xiao, Christoph Teichmann, and Konstantine Arkoudas. Grammatical sequence pre- diction for real-time neural semantic parsing. In Proceed- ings of the Workshop on Deep Learning and Formal Lan- guages: Building Bridges, pages 14-23, Florence, August 2019. ACL (Association for Computational Linguistics).","title":"Grammatical sequence prediction for real-time neural semantic parsing","authors":"Chunyang Xiao, Christoph Teichmann, Konstantine Arkoudas","year":"2019","venue":"Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2020] Silei Xu, Sina J Semnani, Giovanni Cam- pagna, and Monica S Lam. Autoqa: From databases to qa semantic parsers with only synthetic training data. arXiv preprint arXiv:2010.04806, 2020.","title":"Autoqa: From databases to qa semantic parsers with only synthetic training data","authors":"Xu","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Yin and Neubig, 2017] Pengcheng Yin and Graham Neu- big. A syntactic neural model for general-purpose code generation. In Regina Barzilay and Min-Yen Kan, edi- tors, Proceedings of the 55th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2017, Vancou- ver, Canada, July 30 -August 4, Volume 1: Long Papers, pages 440-450. Association for Computational Linguis- tics, 2017.","title":"Pengcheng Yin and Graham Neubig. A syntactic neural model for general-purpose code generation","authors":"Neubig Yin","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Yu et al., 2020] Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. Grappa: Grammar- augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845, 2020.","title":"Grappa: Grammaraugmented pre-training for table semantic parsing","authors":"Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, Caiming Xiong","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}