{"type":"ScholarlyArticle","title":"Sharing Attention Weights for Fast Transformer","authors":[{"name":"Tong Xiao","slug":"tong-xiao","given":"Tong","family":"Xiao","affiliation":"Northeastern University , Shenyang , China; NiuTrans Co., Ltd. , Shenyang , China"},{"name":"Yinqiao Li","slug":"yinqiao-li","given":"Yinqiao","family":"Li","affiliation":"Northeastern University , Shenyang , China"},{"name":"Jingbo Zhu","slug":"jingbo-zhu","given":"Jingbo","family":"Zhu","affiliation":"Northeastern University , Shenyang , China; NiuTrans Co., Ltd. , Shenyang , China"},{"name":"Zhengtao Yu","slug":"zhengtao-yu","given":"Zhengtao","family":"Yu","affiliation":"Kunming University of Science and Technology , Kunming , China"},{"name":"Tongran Liu","slug":"tongran-liu","given":"Tongran","family":"Liu","affiliation":"CAS Key Laboratory of Behavioral Science , Institute of Psychology , CAS , Beijing , China"}],"abstract":"Recently, the Transformer machine translation system has shown strong results by stacking attention layers on both the source and target-language sides. But the inference of this model is slow due to the heavy use of dot-product attention in auto-regressive decoding. In this paper we speed up Transformer via a fast and lightweight attention model. More specifically, we share attention weights in adjacent layers and enable the efficient re-use of hidden states in a vertical manner. Moreover, the sharing policy can be jointly learned with the MT model. We test our approach on ten WMT and NIST OpenMT tasks. Experimental results show that it yields an average of 1.3X speed-up (with almost no decrease in BLEU) on top of a state-of-the-art implementation that has already adopted a cache for fast inference. Also, our approach obtains a 1.8X speed-up when it works with the AAN model. This is even 16 times faster than the baseline with no use of the attention cache.","doi":"10.24963/ijcai.2019/735","url":"https://ijcai.org/proceedings/2019/735","pdf":"https://ijcai.org/proceedings/2019/0735.pdf","markdown":"https://ijcai.org/proceedings/2019/735.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/735","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":735,"pages":{"first":5292,"last":5298},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Machine Translation"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In In Proceed- ings of the 3rd International Conference on Learning Representations, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"Proceedings of the 3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Britz et al., 2017] Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc Le. Massive exploration of neural ma- chine translation architectures. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1442-1451, Copenhagen, Den- mark, September 2017.","title":"Massive exploration of neural machine translation architectures","authors":"Denny Britz, Anna Goldie, Minh-Thang Luong, Quoc Le","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dabre and Fujita, 2019] Raj Dabre and Atsushi Fujita. Re- current stacking of layers for compact neural machine translation models. In Proceedings of the 33rd AAAI Con- ference on Artificial Intelligence (AAAI), 2019.","title":"Recurrent stacking of layers for compact neural machine translation models","authors":"Fujita Dabre, Raj Dabre, Atsushi Fujita","year":"2019","venue":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence (AAAI)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolu- tional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 1243-1252, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning, ICML 2017","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gu et al., 2018] Jiatao Gu, James Bradbury, Caiming X- iong, Victor O.K. Li, and Richard Socher. Non- autoregressive neural machine translation. In International Conference on Learning Representations, 2018.","title":"Nonautoregressive neural machine translation","authors":"Gu","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and J- effrey Dean. Distilling the knowledge in a neural net- work. In NIPS Deep Learning and Representation Learn- ing Workshop, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":"NIPS Deep Learning and Representation Learning Workshop","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kingma and Ba, 2015] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.","title":"Adam: A method for stochastic optimization","authors":"P Diederik, Jimmy Kingma, Ba","year":"2015","venue":"3rd International Conference on Learning Representations, ICLR 2015","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[L'Hostis et al., 2016] Gurvan L'Hostis, David Grangier, and Michael Auli. Vocabulary selection strategies for neural machine translation. CoRR, abs/1610.00072, 2016.","title":"Vocabulary selection strategies for neural machine translation","authors":"L' Gurvan, David Hostis, Michael Grangier, Auli","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin, 1991] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Trans. Information Theory, 37(1):145-151, 1991.","title":"Divergence measures based on the shannon entropy","authors":"Jianhua Lin","year":"1991","venue":"IEEE Trans. Information Theory","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation. In Proceed- ings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pages 1412-1421, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":"Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Luong et al., 2016] Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. In 4th International Con- ference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016.","title":"Multi-task sequence to sequence learning","authors":"Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, Lukasz Kaiser","year":"2016","venue":"4th International Conference on Learning Representations, ICLR 2016","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Micikevicius et al., 2018] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -May 3, 2018.","title":"Mixed precision training","authors":"Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, Hao Wu","year":"2018","venue":"6th International Conference on Learning Representations, ICLR 2018","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Quinn and Ballesteros, 2018] Jerry Quinn and Miguel Ballesteros. Pieces of eight: 8-bit neural machine transla- tion. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HTL 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 3 (Industry Papers), pages 114-120, 2018.","title":"Pieces of eight: 8-bit neural machine translation","authors":"Quinn, Ballesteros, Jerry Quinn, Miguel Ballesteros","year":"2018","venue":"Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HTL 2018","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Sankaran et al., 2017] Baskaran Sankaran, Markus Freitag, and Yaser Al-Onaizan. Attention-based vocabulary selec- tion for NMT decoding. CoRR, abs/1706.03824, 2017.","title":"Attention-based vocabulary selection for NMT decoding","authors":"Markus Baskaran Sankaran, Yaser Freitag, Al-Onaizan","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sutskever et al., 2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112, 2014.","title":"Quoc V Le. Sequence to sequence learning with neural networks","authors":"Sutskever","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Szegedy et al., 2016] Christian Szegedy, Vincent Van- houcke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2818-2826, 2016.","title":"Rethinking the inception architecture for computer vision","authors":"Szegedy","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 6000-6010, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Wu et al., 2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine trans- lation. arXiv preprint arXiv:1609.08144, 2016.","title":"Google's neural machine translation system: Bridging the gap between human and machine translation","authors":"Yonghui Wu, Mike Schuster, Zhifeng Chen, V Quoc, Mohammad Le, Wolfgang Norouzi, Maxim Macherey, Yuan Krikun, Qin Cao, Klaus Gao, Macherey","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Xiao et al., 2012] Tong Xiao, Jingbo Zhu, Hao Zhang, and Qiang Li. NiuTrans: An open source toolkit for phrase- based and syntax-based machine translation. In Proceed- ings of the ACL 2012 System Demonstrations, pages 19- 24, Jeju Island, Korea, July 2012.","title":"NiuTrans: An open source toolkit for phrasebased and syntax-based machine translation","authors":"Tong Xiao, Jingbo Zhu, Hao Zhang, Qiang Li","year":"2012","venue":"Proceedings of the ACL 2012 System Demonstrations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Yang et al., 2018] Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. Unsupervised neural machine translation with weight sharing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46-55, 2018.","title":"Unsupervised neural machine translation with weight sharing","authors":"Zhen Yang, Wei Chen, Feng Wang, Bo Xu","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhang et al., 2018] Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average atten- tion network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1789-1798, 2018.","title":"Accelerating neural transformer via an average attention network","authors":"Biao Zhang, Deyi Xiong, Jinsong Su","year":"2018","venue":"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}