{"type":"ScholarlyArticle","title":"Sequence Generation: From Both Sides to the Middle","authors":[{"name":"Long Zhou","slug":"long-zhou","given":"Long","family":"Zhou","affiliation":"University of Chinese Academy of Sciences , Beijing , China; National Laboratory of Pattern Recognition , CASIA , Beijing , China"},{"name":"Jiajun Zhang","slug":"jiajun-zhang","given":"Jiajun","family":"Zhang","affiliation":"University of Chinese Academy of Sciences , Beijing , China; National Laboratory of Pattern Recognition , CASIA , Beijing , China"},{"name":"Chengqing Zong","slug":"chengqing-zong","given":"Chengqing","family":"Zong","affiliation":"University of Chinese Academy of Sciences , Beijing , China; National Laboratory of Pattern Recognition , CASIA , Beijing , China; CAS Center for Excellence in Brain Science and Intelligence Technology , Shanghai , China"},{"name":"Heng Yu","slug":"heng-yu","given":"Heng","family":"Yu","affiliation":"Machine Intelligence Technology Lab , Alibaba Group"}],"abstract":"The encoder-decoder framework has achieved promising process for many sequence generation tasks, such as neural machine translation and text summarization. Such a framework usually generates a sequence token by token from left to right, hence (1) this autoregressive decoding procedure is time-consuming when the output sentence becomes longer, and (2) it lacks the guidance of future context which is crucial to avoid under-translation. To alleviate these issues, we propose a synchronous bidirectional sequence generation (SBSG) model which predicts its outputs from both sides to the middle simultaneously. In the SBSG model, we enable the left-to-right (L2R) and right-to-left (R2L) generation to help and interact with each other by leveraging interactive bidirectional attention network. Experiments on neural machine translation (En-De, Ch-En, and En-Ro) and text summarization tasks show that the proposed model significantly speeds up decoding while improving the generation quality compared to the autoregressive Transformer.","doi":"10.24963/ijcai.2019/760","url":"https://ijcai.org/proceedings/2019/760","pdf":"https://ijcai.org/proceedings/2019/0760.pdf","markdown":"https://ijcai.org/proceedings/2019/760.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/760","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":760,"pages":{"first":5471,"last":5477},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Summarization","Natural Language Processing: Machine Translation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Devlin, 2017] Jacob Devlin. Sharp models on dull hard- ware: Fast and accurate neural machine translation decod- ing on the cpu. In EMNLP, pages 2820-2825, 2017.","title":"Sharp models on dull hardware: Fast and accurate neural machine translation decoding on the cpu","authors":"Jacob Devlin","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Finch and Sumita, 2009] Andrew Finch and Eiichiro Sumi- ta. Bidirectional phrase-based statistical machine transla- tion. In EMNLP, pages 1124-1132, 2009.","title":"Bidirectional phrase-based statistical machine translation","authors":"Sumita Finch","year":"2009","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In ICML, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann Dauphin","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2017] Jiatao Gu, James Bradbury, Caiming X- iong, Victor OK Li, and Richard Socher. Non- autoregressive neural machine translation. arXiv preprint arXiv:1711.02281, 2017.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kaiser et al., 2018] Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Pamar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables. arXiv preprint arXiv:1803.03382, 2018.","title":"Fast decoding in sequence models using discrete latent variables","authors":"Łukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Pamar, Samy Bengio, Jakob Uszkoreit, Noam Shazeer","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Koehn et al., 2007] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Con- stantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In ACL, 2007.","title":"Moses: Open source toolkit for statistical machine translation","authors":"Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, Evan Herbst","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lee et al., 2018] Jason Lee, Elman Mansimov, and Kyunghyun Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In EMNLP, pages 1173-1182, 2018.","title":"Deterministic non-autoregressive neural sequence modeling by iterative refinement","authors":"Jason Lee, Elman Mansimov, Kyunghyun Cho","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Li et al., 2018] Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. Ensure the correctness of the summa- ry: Incorporate entailment knowledge into abstractive sen- tence summarization. In COLING, 2018.","title":"Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization","authors":"Haoran Li, Junnan Zhu, Jiajun Zhang, Chengqing Zong","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2016a] Lemao Liu, Andrew M Finch, Masao Utiyama, and Eiichiro Sumita. Agreement on target- bidirectional lstms for sequence-to-sequence learning. In AAAI, 2016.","title":"Agreement on targetbidirectional lstms for sequence-to-sequence learning","authors":"Liu","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2016b] Lemao Liu, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. Agreement on target- bidirectional neural machine translation. In NAACL, 2016.","title":"Agreement on targetbidirectional neural machine translation","authors":"Liu","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018] Yuchen Liu, Long Zhou, Yining Wang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. A compa- rable study on model averaging, ensembling and reranking in nmt. In NLPCC, pages 299-308, 2018.","title":"A comparable study on model averaging, ensembling and reranking in nmt","authors":"Yuchen Liu, Long Zhou, Yining Wang, Yang Zhao, Jiajun Zhang, Chengqing Zong","year":"2018","venue":"NLPCC","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mi et al., 2016] Haitao Mi, Zhiguo Wang, and Abe Itty- cheriah. Vocabulary manipulation for neural machine translation. In ACL, pages 124-129, 2016.","title":"Vocabulary manipulation for neural machine translation","authors":"Haitao Mi, Zhiguo Wang, Abe Ittycheriah","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Oord et al., 2017] Aaron van den Oord, Yazhe Li, Ig- or Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lock- hart, Luis C Cobo, Florian Stimberg, et al. Paral- lel wavenet: Fast high-fidelity speech synthesis. arXiv preprint arXiv:1711.10433, 2017.","title":"Parallel wavenet: Fast high-fidelity speech synthesis","authors":"Aaron Van Den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Van Den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. Bleu: a methof for auto- matic evaluation of machine translation. In ACL, 2002.","title":"Bleu: a methof for automatic evaluation of machine translation","authors":"Kishore Papineni, Salim Roukos, Todd Ward, Weijing Zhu","year":"2002","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rush et al., 2015] Alexander M. Rush, Sumit Chopra, and Jason Weston. A neural attention model for abstractive sentence summarization. In EMNLP, 2015.","title":"A neural attention model for abstractive sentence summarization","authors":"Alexander M Rush, Sumit Chopra, Jason Weston","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sennrich et al., 2016a] Rico Sennrich, Barry Haddow, and Alexandra Birch. Edinburgh neural machine translation systems for wmt 16. In WMT, 2016.","title":"Edinburgh neural machine translation systems for wmt 16","authors":"Sennrich","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Sennrich et al., 2016b] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, pages 1715-1725, 2016.","title":"Neural machine translation of rare words with subword units","authors":"Sennrich","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Serdyuk et al., 2018] Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. Twin networks: Matching the future for sequence generation. In ICLR, 2018.","title":"Twin networks: Matching the future for sequence generation","authors":"Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. Sequence to sequence learning with neu- ral networks. In NIPS, pages 3104-3112, 2014.","title":"Quoc VV Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neu- ral image caption generator. In CVPR, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Chunqi Wang, Ji Zhang, and Haiqing Chen. Semi-autoregressive neural machine translation. arXiv preprint arXiv:1808.08583, 2018.","title":"Semi-autoregressive neural machine translation","authors":"Wang","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Watanabe and Sumita, 2002] Taro Watanabe and Eiichiro Sumita. Bidirectional decoding for statistical machine translation. In COLING, 2002.","title":"Bidirectional decoding for statistical machine translation","authors":"Sumita Watanabe, Taro Watanabe, Eiichiro Sumita","year":"2002","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. Show, attend and tel- l: Neural image caption generation with visual attention. Computer Science, pages 2048-2057, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio","year":"2015","venue":"Computer Science","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018a] Biao Zhang, Deyi Xiong, and jinsong Su. Accelerating neural transformer via an average atten- tion network. In ACL, pages 1789-1798, 2018.","title":"Accelerating neural transformer via an average attention network","authors":"Zhang","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang et al., 2018b] Xiangwen Zhang, Jinsong Su, Yue Qin, Yang Liu, Rongrong Ji, and Hongji Wang. Asyn- chronous bidirectional decoding for neural machine trans- lation. In AAAI, 2018.","title":"Asynchronous bidirectional decoding for neural machine translation","authors":"Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2017] Qingyu Zhou, Nan Yang, Furu Wei, and Ming Zhou. Selective encoding for abstractive sentence summarization. In ACL, pages 1095-1104, 2017.","title":"Selective encoding for abstractive sentence summarization","authors":"Qingyu Zhou, Nan Yang, Furu Wei, Ming Zhou","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhou et al., 2019] Long Zhou, Jiajun Zhang, and Chengqing Zong. Synchronous bidirectional neural machine translation. In TACL, pages 91-105, 2019.","title":"Synchronous bidirectional neural machine translation","authors":"Long Zhou, Jiajun Zhang, Chengqing Zong","year":"2019","venue":"TACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}