{"type":"ScholarlyArticle","title":"Exploring Bilingual Parallel Corpora for Syntactically Controllable Paraphrase Generation","authors":[{"name":"Mingtong Liu","slug":"mingtong-liu","given":"Mingtong","family":"Liu","affiliation":"School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , Beijing , China"},{"name":"Erguang Yang","slug":"erguang-yang","given":"Erguang","family":"Yang","affiliation":"School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , Beijing , China"},{"name":"Deyi Xiong","slug":"deyi-xiong","given":"Deyi","family":"Xiong","affiliation":"College of Intelligence and Computing , Tianjin University , Tianjin , China"},{"name":"Yujie Zhang","slug":"yujie-zhang","given":"Yujie","family":"Zhang","affiliation":"School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , Beijing , China"},{"name":"Chen Sheng","slug":"chen-sheng","given":"Chen","family":"Sheng","affiliation":"Lenovo Research AI Lab , Beijing , China"},{"name":"Changjian Hu","slug":"changjian-hu","given":"Changjian","family":"Hu","affiliation":"Lenovo Research AI Lab , Beijing , China"},{"name":"Jinan Xu","slug":"jinan-xu","given":"Jinan","family":"Xu","affiliation":"School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , Beijing , China"},{"name":"Yufeng Chen","slug":"yufeng-chen","given":"Yufeng","family":"Chen","affiliation":"School of Computer Science and Information Technology && Beijing Key Lab of Traffic Data Analysis and Mining , Beijing Jiaotong University , Beijing , China"}],"abstract":"Paraphrase generation is of great importance to many downstream tasks in natural language processing. Recent efforts have focused on generating paraphrases in specific syntactic forms, which, generally, heavily relies on manually annotated paraphrase data that is not easily available for many languages and domains. In this paper, we propose a novel end-to-end framework to leverage existing large-scale bilingual parallel corpora to generate paraphrases under the control of syntactic exemplars. In order to train one model over the two languages of parallel corpora, we embed sentences of them into the same content and style spaces with shared content and style encoders using cross-lingual word embeddings. We propose an adversarial discriminator to disentangle the content and style space, and employ a latent variable to model the syntactic style of a given exemplar in order to guide the two decoders for generation. Additionally, we introduce cycle and masking learning schemes to efficiently train the model. Experiments and analyses demonstrate that the proposed model trained only on bilingual parallel data is capable of generating diverse paraphrases with desirable syntactic styles. Fine-tuning the trained model on a small paraphrase corpus makes it substantially outperform state-of-the-art paraphrase generation models trained on a larger paraphrase dataset.","doi":"10.24963/ijcai.2020/547","url":"https://ijcai.org/proceedings/2020/547","pdf":"https://ijcai.org/proceedings/2020/0547.pdf","markdown":"https://ijcai.org/proceedings/2020/547.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/547","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":547,"pages":{"first":3955,"last":3961},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Processing"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24377/file_24377.m3u8","length_sec":244,"provider":"hls"}],"references":[{"position":1,"raw":"Sem extensive consultation has taken place over the past year. Syn there are our own troops like little ants. Ref there have been extensive consultations over the past year. Gen there are many extensive consultations in the past","title":"troops like little ants","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Sem it is hard for me to imagine where al., 2017] Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th An- nual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 451-462, 2017.","title":"Learning bilingual word embeddings with (almost) no bilingual data","authors":"Mikel Artetxe, Gorka Labaka, Eneko Agirre","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2014] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2015] Samuel R Bowman, Luke Vilnis, O- riol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.","title":"Generating sentences from a continuous space","authors":"Bowman","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, Florence, Italy, July 2019. Association for Computational Linguistics.","title":"Controllable paraphrase generation with a syntactic exemplar","authors":"Mingda Chen, Qingming Tang, Sam Wiseman, Kevin Gimpel","year":"2019","venue":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the NAACL, pages 4171-4186, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"Proceedings of the 2019 Conference of the NAACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dong et al., 2017] Li Dong, Jonathan Mallinson, Siva Red- dy, and Mirella Lapata. Learning to paraphrase for ques- tion answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 875-886, 2017.","title":"Learning to paraphrase for question answering","authors":"Li Dong, Jonathan Mallinson, Siva Reddy, Mirella Lapata","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ficler and Goldberg, 2017] Jessica Ficler and Yoav Gold- berg. Controlling linguistic style aspects in neural lan- guage generation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104, 2017.","title":"Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language generation","authors":"Goldberg Ficler","year":"2017","venue":"Proceedings of the Workshop on Stylistic Variation","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gupta et al., 2018] Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. A deep generative framework for paraphrase generation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.","title":"A deep generative framework for paraphrase generation","authors":"Ankush Gupta, Arvind Agarwal, Prawaan Singh, Piyush Rai","year":"2018","venue":"Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hasan et al., 2016] Sadid A Hasan, Bo Liu, Joey Liu, Ashe- qul Qadir, Kathy Lee, Vivek Datla, Aaditya Prakash, and Oladimeji Farri. Neural clinical paraphrase generation with attention. In Proceedings of the Clinical Natural Lan- guage Processing Workshop, pages 42-53, 2016.","title":"Neural clinical paraphrase generation with attention","authors":"Hasan","year":"2016","venue":"Proceedings of the Clinical Natural Language Processing Workshop","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Iyyer et al., 2018] Mohit Iyyer, John Wieting, Kevin Gim- pel, and Luke Zettlemoyer. Adversarial example genera- tion with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the NAACL, pages 1875-1885, 2018.","title":"Adversarial example generation with syntactically controlled paraphrase networks","authors":"Mohit Iyyer, John Wieting, Kevin Gimpel, Luke Zettlemoyer","year":"2018","venue":"Proceedings of the 2018 Conference of the NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[John et al., 2018] Vineet John, Lili Mou, Hareesh Bahu- leyan, and Olga Vechtomova. Disentangled representa- tion learning for text style transfer. arXiv preprint arX- iv:1808.04339, 2018.","title":"Disentangled representation learning for text style transfer","authors":"Vineet John, Lili Mou, Hareesh Bahuleyan, Olga Vechtomova","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Ba Kingma, Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kingma et al., 2014] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi- supervised learning with deep generative models. In Ad- vances in neural information processing systems, pages 3581-3589, 2014.","title":"Semisupervised learning with deep generative models","authors":"Shakir Durk P Kingma, Danilo Mohamed, Max Jimenez Rezende, Welling","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lample et al., 2018] Guillaume Lample, Sandeep Subrama- nian, Eric Smith, Ludovic Denoyer, Y-Lan Boureau, et al. Multiple-attribute text rewriting. 2018.","title":"Multiple-attribute text rewriting","authors":"Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Y-Lan Boureau","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Be- longie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In European conference on com- puter vision, pages 740-755. Springer, 2014.","title":"Microsoft coco: Common objects in context","authors":"Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, C Lawrence, Zitnick","year":"2014","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Patro et al., 2018] Badri Narayana Patro, Vinod Kumar Kurmi, Sandeep Kumar, and Vinay Namboodiri. Learn- ing semantic sentence embeddings using sequential pair- wise discriminator. In Proceedings of the 27th Internation- al Conference on Computational Linguistics, pages 2715- 2729, 2018.","title":"Learning semantic sentence embeddings using sequential pairwise discriminator","authors":"Badri Narayana Patro, Vinod Kumar Kurmi, Sandeep Kumar, Vinay Namboodiri","year":"2018","venue":"Proceedings of the 27th International Conference on Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Prakash et al., 2016] Aaditya Prakash, Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and O- ladimeji Farri. Neural paraphrase generation with stacked residual lstm networks. arXiv preprint arXiv:1610.03098, 2016.","title":"Neural paraphrase generation with stacked residual lstm networks","authors":"Aaditya Prakash, A Sadid, Kathy Hasan, Vivek Lee, Ashequl Datla, Joey Qadir, Oladimeji Liu, Farri","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2019] Wentao Wang, Zhiting Hu, Zichao Yang, Haoran Shi, Frank Xu, and Eric Xing. Toward unsu- pervised text content manipulation. arXiv preprint arX- iv:1901.09501, 2019.","title":"Toward unsupervised text content manipulation","authors":"Wentao Wang, Zhiting Hu, Zichao Yang, Haoran Shi, Frank Xu, Eric Xing","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wieting and Gimpel, 2018] John Wieting and Kevin Gim- pel. Paranmt-50m: Pushing the limits of paraphrastic sen- tence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Pa- pers), pages 451-462, 2018.","title":"Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations","authors":"Gimpel Wieting, John Wieting, Kevin Gimpel","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wiseman et al., 2018] Sam Wiseman, Stuart Shieber, and Alexander Rush. Learning neural templates for text gen- eration. In Proceedings of the 2018 Conference on Em- pirical Methods in Natural Language Processing, pages 3174-3187, 2018.","title":"Learning neural templates for text generation","authors":"Wiseman","year":"2018","venue":"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhao et al., 2018] Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. Integrating trans- former and paraphrase rules for sentence simplification. In Proceedings of the 2018 Conference on EMNLP, pages 3164-3173, 2018.","title":"Integrating transformer and paraphrase rules for sentence simplification","authors":"Zhao","year":"2018","venue":"Proceedings of the 2018 Conference on EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou et al., 2018] Zhong Zhou, Matthias Sperber, and Alex Waibel. Paraphrases as foreign languages in multilin- gual neural machine translation. arXiv preprint arX- iv:1808.08438, 2018.","title":"Paraphrases as foreign languages in multilingual neural machine translation","authors":"Zhong Zhou, Matthias Sperber, Alex Waibel","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}