{"type":"ScholarlyArticle","title":"How Far are We from Effective Context Modeling? An Exploratory Study on Semantic Parsing in Context","authors":[{"name":"Qian Liu","slug":"qian-liu","given":"Qian","family":"Liu","affiliation":"School of Computer Science and Engineering , Beihang University , China"},{"name":"Bei Chen","slug":"bei-chen","given":"Bei","family":"Chen","affiliation":"Microsoft Research , Beijing , China"},{"name":"Jiaqi Guo","slug":"jiaqi-guo","given":"Jiaqi","family":"Guo","affiliation":"Xi'an Jiaotong University , Xi'an , China"},{"name":"Jian-Guang Lou","slug":"jian-guang-lou","given":"Jian-Guang","family":"Lou","affiliation":"Microsoft Research , Beijing , China"},{"name":"Bin Zhou","slug":"bin-zhou","given":"Bin","family":"Zhou","affiliation":"School of Computer Science and Engineering , Beihang University , China"},{"name":"Dongmei Zhang","slug":"dongmei-zhang","given":"Dongmei","family":"Zhang","affiliation":"Microsoft Research , Beijing , China"}],"abstract":"Recently semantic parsing in context has received a considerable attention, which is challenging since there are complex contextual phenomena. Previous works verified their proposed methods in limited scenarios, which motivates us to conduct an exploratory study on context modeling methods under real-world semantic parsing in context. We present a grammar-based decoding semantic parser and adapt typical context modeling methods on top of it. We evaluate 13 context modeling methods on two large complex cross-domain datasets, and our best model achieves state-of-the-art performances on both datasets with significant improvements. Furthermore, we summarize the most frequent contextual phenomena, with a fine-grained analysis on representative models, which may shed light on potential research directions. Our code is available at https://github.com/microsoft/ContextualSP.","doi":"10.24963/ijcai.2020/495","url":"https://ijcai.org/proceedings/2020/495","pdf":"https://ijcai.org/proceedings/2020/0495.pdf","markdown":"https://ijcai.org/proceedings/2020/495.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/495","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":495,"pages":{"first":3580,"last":3586},"keywords":["Natural Language Processing: Dialogue","Natural Language Processing: Natural Language Semantics","Natural Language Processing: Natural Language Processing","Natural Language Processing: Coreference Resolution"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24952/file_24952.m3u8","length_sec":280,"provider":"hls"}],"references":[{"position":1,"raw":"[Androutsopoulos et al., 1995] Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch. Natural Lan- guage Interfaces to Databases-An Introduction. Natural language engineering, 1995.","title":"Natural Language Interfaces to Databases-An Introduction","authors":"Ion Androutsopoulos, Graeme D Ritchie, Peter Thanisch","year":"1995","venue":"Natural language engineering","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bertomeu et al., 2006] Núria Bertomeu, Hans Uszkoreit, Anette Frank, Hans-Ulrich Krieger, and Brigitte Jörg. Contextual phenomena and thematic relations in database QA dialogues: results from a Wizard-of-Oz experiment. In NAACL, 2006.","title":"Contextual phenomena and thematic relations in database QA dialogues: results from a Wizard-of-Oz experiment","authors":"Núria Bertomeu, Hans Uszkoreit, Anette Frank, Hans-Ulrich Krieger, Brigitte Jörg","year":"2006","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bogin et al., 2019] Ben Bogin, Jonathan Berant, and Matt Gardner. Representing schema structure with graph neural networks for text-to-SQL parsing. In ACL, 2019.","title":"Representing schema structure with graph neural networks for text-to-SQL parsing","authors":"Ben Bogin, Jonathan Berant, Matt Gardner","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gardner et al., 2018] Matt Gardner, Joel Grus, Mark Neu- mann, and et al. AllenNLP: A deep semantic natural lan- guage processing platform. In ACL, 2018.","title":"AllenNLP: A deep semantic natural language processing platform","authors":"Matt Gardner, Joel Grus, Mark Neumann","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2018] Daya Guo, Duyu Tang, Nan Duan, Ming Zhou, and Jian Yin. Dialog-to-action: conversational ques- tion answering over a large-scale knowledge base. In NIPS, 2018.","title":"Dialog-to-action: conversational question answering over a large-scale knowledge base","authors":"Duyu Daya Guo, Nan Tang, Ming Duan, Jian Zhou, Yin","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Guo et al., 2019] Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. To- wards complex text-to-SQL in cross-domain database with intermediate representation. In ACL, 2019.","title":"Towards complex text-to-SQL in cross-domain database with intermediate representation","authors":"Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, Dongmei Zhang","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hwang et al., 2019] Wonseok Hwang, Jinyeung Yim, Se- unghyun Park, and Minjoon Seo. A comprehensive ex- ploration on WikiSQL with table-aware word contextual- ization. CoRR, 2019.","title":"A comprehensive exploration on WikiSQL with table-aware word contextualization","authors":"Hwang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iyer et al., 2018] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In EMNLP, 2018.","title":"Mapping language to code in programmatic context","authors":"Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Luke Zettlemoyer","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Iyyer et al., 2017] Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. Search-based neural structured learning for se- quential question answering. In ACL, 2017.","title":"Search-based neural structured learning for sequential question answering","authors":"Mohit Iyyer, Wen-Tau Yih, Ming-Wei Chang","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krishnamurthy et al., 2017] Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP, 2017.","title":"Neural semantic parsing with type constraints for semi-structured tables","authors":"Jayant Krishnamurthy, Pradeep Dasigi, Matt Gardner","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lin et al., 2019] Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. Grammar-based neu- ral text-to-SQL generation. CoRR, 2019.","title":"Grammar-based neural text-to-SQL generation","authors":"Kevin Lin, Ben Bogin, Mark Neumann, Jonathan Berant, Matt Gardner","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2019] Qian Liu, Bei Chen, Haoyan Liu, Jian- Guang Lou, Lei Fang, Bin Zhou, and Dongmei Zhang. A split-and-recombine approach for follow-up query analy- sis. In EMNLP-IJCNLP, 2019.","title":"A split-and-recombine approach for follow-up query analysis","authors":"Qian Liu, Bei Chen, Haoyan Liu, Jian-Guang Lou, Lei Fang, Bin Zhou, Dongmei Zhang","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Long et al., 2016] Reginald Long, Panupong Pasupat, and Percy Liang. Simpler context-dependent logical forms via model projections. In ACL, 2016.","title":"Simpler context-dependent logical forms via model projections","authors":"Reginald Long, Panupong Pasupat, Percy Liang","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paszke et al., 2017] Adam Paszke, Sam Gross, Soumith Chintala, and et al. Automatic differentiation in PyTorch. In NIPS, 2017.","title":"Automatic differentiation in PyTorch","authors":"Adam Paszke, Sam Gross, Soumith Chintala","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In EMNLP, 2014.","title":"GloVe: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Sankar et al., 2019] Chinnadhurai Sankar, Sandeep Subra- manian, Chris Pal, Sarath Chandar, and Yoshua Bengio. Do neural dialog systems use the conversation history ef- fectively? An empirical study. 2019.","title":"Do neural dialog systems use the conversation history effectively? An empirical study","authors":"Sankar","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 1997.","title":"Bidirectional recurrent neural networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Trans. Signal Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Suhr et al., 2018] Alane Suhr, Srinivasan Iyer, and Yoav Artzi. Learning to map context-dependent sentences to executable formal queries. In NAACL, 2018.","title":"Learning to map context-dependent sentences to executable formal queries","authors":"Alane Suhr, Srinivasan Iyer, Yoav Artzi","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence to sequence learning with neural networks. In NIPS, 2014.","title":"Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wolf et al., 2019] Thomas Wolf, Lysandre Debut, Victor Sanh, and et al. Huggingface's transformers: State-of-the- art natural language processing. CoRR, 2019.","title":"Huggingface's transformers: State-of-theart natural language processing","authors":"Thomas Wolf, Lysandre Debut, Victor Sanh","year":"2019","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yin and Neubig, 2018] Pengcheng Yin and Graham Neu- big. TRANX: A transition-based neural abstract syn- tax parser for semantic parsing and code generation. In EMNLP, 2018.","title":"TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation","authors":"Neubig Yin, Pengcheng Yin, Graham Neubig","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yu et al., 2018] Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task. In EMNLP, 2018.","title":"SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task","authors":"Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, Dragomir Radev","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2019a] Tao Yu, Rui Zhang, Heyang Er, and et al. CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In EMNLP-IJCNLP, 2019.","title":"CoSQL: A conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases","authors":"Yu","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2019b] Tao Yu, Rui Zhang, Michihiro Yasunaga, and et al. SParC: Cross-domain semantic parsing in con- text. In ACL, 2019.","title":"SParC: Cross-domain semantic parsing in context","authors":"Yu","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zettlemoyer and Collins, 2009] Luke S. Zettlemoyer and Michael Collins. Learning context-dependent mappings from sentences to logical form. In ACL-IJCNLP, 2009.","title":"Learning context-dependent mappings from sentences to logical form","authors":"Collins Zettlemoyer, Luke S Zettlemoyer, Michael Collins","year":"2009","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang et al., 2018] Weinan Zhang, Yiming Cui, Yifa Wang, Qingfu Zhu, Lingzhi Li, Lianqiang Zhou, and Ting Liu. Context-sensitive generation of open-domain conversa- tional responses. In COLING, 2018.","title":"Context-sensitive generation of open-domain conversational responses","authors":"Weinan Zhang, Yiming Cui, Yifa Wang, Qingfu Zhu, Lingzhi Li, Lianqiang Zhou, Ting Liu","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2019] Rui Zhang, Tao Yu, Heyang Er, and et al. Editing-based SQL query generation for cross- domain context-dependent questions. In EMNLP-IJCNLP, 2019.","title":"Editing-based SQL query generation for crossdomain context-dependent questions","authors":"Rui Zhang, Tao Yu, Heyang Er","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}