{"type":"ScholarlyArticle","title":"Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling","authors":[{"name":"Tao Shen","slug":"tao-shen","given":"Tao","family":"Shen","affiliation":"Centre for Artificial Intelligence , School of Software , University of Technology Sydney"},{"name":"Tianyi Zhou","slug":"tianyi-zhou","given":"Tianyi","family":"Zhou","affiliation":"Paul G. Allen School of Computer Science & Engineering , University of Washington"},{"name":"Guodong Long","slug":"guodong-long","given":"Guodong","family":"Long","affiliation":"Centre for Artificial Intelligence , School of Software , University of Technology Sydney"},{"name":"Jing Jiang","slug":"jing-jiang","given":"Jing","family":"Jiang","affiliation":"Centre for Artificial Intelligence , School of Software , University of Technology Sydney"},{"name":"Sen Wang","slug":"sen-wang","given":"Sen","family":"Wang","affiliation":"School of Information and Communication Technology , Griffith University"},{"name":"Chengqi Zhang","slug":"chengqi-zhang","given":"Chengqi","family":"Zhang","affiliation":"Centre for Artificial Intelligence , School of Software , University of Technology Sydney"}],"abstract":"Many natural language processing tasks solely rely on sparse dependencies between a few tokens in a sentence. Soft attention mechanisms show promising performance in modeling local/global dependencies by soft probabilities between every two tokens, but they are not effective and efficient when applied to long sentences. By contrast, hard attention mechanisms directly select a subset of tokens but are difficult and inefficient to train due to their combinatorial nature. In this paper, we integrate both soft and hard attention into one context fusion model, \"reinforced self-attention (ReSA)\", for the mutual benefit of each other. In ReSA, a hard attention trims a sequence for a soft self-attention to process, while the soft attention feeds reward signals back to facilitate the training of the hard one. For this purpose, we develop a novel hard attention called \"reinforced sequence sampling (RSS)\", selecting tokens in parallel and trained via policy gradient. Using two RSS modules, ReSA efficiently extracts the sparse dependencies between each pair of selected tokens. We finally propose an RNN/CNN-free sentence-encoding model, \"reinforced self-attention network (ReSAN)\", solely based on ReSA.  It achieves state-of-the-art performance on both the Stanford Natural Language Inference (SNLI) and the Sentences Involving Compositional Knowledge (SICK) datasets.","doi":"10.24963/ijcai.2018/604","url":"https://ijcai.org/proceedings/2018/604","pdf":"https://ijcai.org/proceedings/2018/0604.pdf","markdown":"https://ijcai.org/proceedings/2018/604.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/604","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":604,"pages":{"first":4345,"last":4352},"keywords":["Machine Learning: Deep Learning","Natural Language Processing: Natural Language Processing","Machine Learning: Data Mining","Natural Language Processing: Text Classification"],"videos":[],"references":[{"position":1,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bjerva et al., 2014] Johannes Bjerva, Johan Bos, Rob Van der Goot, and Malvina Nissim. The meaning factory: Formal seman- tics for recognizing textual entailment and determining semantic similarity. In SemEval@ COLING, pages 642-646, 2014.","title":"The meaning factory: Formal semantics for recognizing textual entailment and determining semantic similarity","authors":"Bjerva","year":"2014","venue":"SemEval@ COLING","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bowman et al., 2015] Samuel R Bowman, Gabor Angeli, Christo- pher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015.","title":"A large annotated corpus for learning natural language inference","authors":"Gabor Samuel R Bowman, Christopher Angeli, Christopher D Potts, Manning","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bowman et al., 2016] Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. A fast unified model for parsing and sentence understand- ing. In ACL, 2016.","title":"A fast unified model for parsing and sentence understanding","authors":"Jon Samuel R Bowman, Abhinav Gauthier, Raghav Rastogi, Christopher D Gupta, Christopher Manning, Potts","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2017] Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Recurrent neural network- based sentence encoder with gated attention for natural language inference. In RepEval@ EMNLP, 2017.","title":"Recurrent neural networkbased sentence encoder with gated attention for natural language inference","authors":"Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, Diana Inkpen","year":"2017","venue":"RepEval@ EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2017a] Eunsol Choi, Daniel Hewlett, Jakob Uszkor- eit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. Coarse-to-fine question answering for long documents. In ACL, 2017.","title":"Coarse-to-fine question answering for long documents","authors":"Choi","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi et al., 2017b] Jihun Choi, Kang Min Yoo, and Sang-goo Lee. Learning to compose task-specific tree structures. arXiv preprint arXiv:1707.02786, 2017.","title":"Learning to compose task-specific tree structures","authors":"Choi","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Chung et al., 2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evalua- tion of gated recurrent neural networks on sequence modeling. In NIPS, 2014.","title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","authors":"Chung","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dauphin et al., 2016] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolu- tional networks. arXiv preprint arXiv:1612.08083, 2016.","title":"Language modeling with gated convolutional networks","authors":"Angela Yann N Dauphin, Michael Fan, David Auli, Grangier","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Dong et al., 2017] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan. More is less: A more complicated network with less inference complexity. In CVPR, 2017.","title":"More is less: A more complicated network with less inference complexity","authors":"Dong","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gehring et al., 2017] Jonas Gehring, Michael Auli, David Grang- ier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. arXiv preprint arXiv:1705.03122, 2017.","title":"Convolutional sequence to sequence learning","authors":"Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N Dauphin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Glorot and Bengio, 2010] Xavier Glorot and Yoshua Bengio. Un- derstanding the difficulty of training deep feedforward neural net- works. In AISTATS, 2010.","title":"Understanding the difficulty of training deep feedforward neural networks","authors":"Bengio Glorot, Xavier Glorot, Yoshua Bengio","year":"2010","venue":"AISTATS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Graves et al., 2013] Alex Graves, Navdeep Jaitly, and Abdel- rahman Mohamed. Hybrid speech recognition with deep bidi- rectional lstm. In ASRU Workshop, 2013.","title":"Hybrid speech recognition with deep bidirectional lstm","authors":"Alex Graves, Navdeep Jaitly, Abdelrahman Mohamed","year":"2013","venue":"ASRU Workshop","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016a] Di He, Yingce Xia, Tao Qin, Liwei Wang, Neng- hai Yu, Tieyan Liu, and Wei-Ying Ma. Dual learning for machine translation. In NIPS, 2016.","title":"Dual learning for machine translation","authors":"He","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[He et al., 2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Hu et al., 2017] Minghao Hu, Yuxing Peng, and Xipeng Qiu. Re- inforced mnemonic reader for machine comprehension. CoRR, abs/1705.02798, 2017.","title":"Reinforced mnemonic reader for machine comprehension","authors":"Minghao Hu, Yuxing Peng, Xipeng Qiu","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sen- tence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lei et al., 2016] Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In EMNLP, 2016.","title":"Rationalizing neural predictions","authors":"Tao Lei, Regina Barzilay, Tommi Jaakkola","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2017a] Xuijun Li, Yun-Nung Chen, Lihong Li, and Jian- feng Gao. End-to-end task-completion neural dialogue systems. arXiv preprint arXiv:1703.01008, 2017.","title":"End-to-end task-completion neural dialogue systems","authors":"Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Li et al., 2017b] Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. End-to-end adversarial memory network for cross- domain sentiment classification. In IJCAI, 2017.","title":"End-to-end adversarial memory network for crossdomain sentiment classification","authors":"Li","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Liu et al., 2016] Yang Liu, Chengjie Sun, Lei Lin, and Xiao- long Wang. Learning natural language inference using bidi- rectional lstm model and inner-attention. arXiv preprint arXiv:1605.09090, 2016.","title":"Learning natural language inference using bidirectional lstm model and inner-attention","authors":"Yang Liu, Chengjie Sun, Lei Lin, Xiaolong Wang","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Liu et al., 2018] Qian Liu, Heyan Huang, Guangquan Zhang, Yang Gao, Junyu Xuan, and Jie Lu. Semantic structure-based word embedding by incorporating concept convergence and word divergence. In AAAI, 2018.","title":"Semantic structure-based word embedding by incorporating concept convergence and word divergence","authors":"Qian Liu, Heyan Huang, Guangquan Zhang, Yang Gao, Junyu Xuan, Jie Lu","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Luong et al., 2015] Minh-Thang Luong, Hieu Pham, and Christo- pher D Manning. Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025, 2015.","title":"Effective approaches to attention-based neural machine translation","authors":"Minh-Thang Luong, Hieu Pham, Christopher D Manning","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Marelli et al., 2014] Marco Marelli, Stefano Menini, Marco Ba- roni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zampar- elli. A sick cure for the evaluation of compositional distributional semantic models. In LREC, 2014.","title":"A sick cure for the evaluation of compositional distributional semantic models","authors":"Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli","year":"2014","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Munkhdalai and Yu, 2017] Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders. In EACL, 2017.","title":"Tsendsuren Munkhdalai and Hong Yu. Neural semantic encoders","authors":"Yu Munkhdalai","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Nie and Bansal, 2017] Yixin Nie and Mohit Bansal. Shortcut- stacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312, 2017.","title":"Shortcutstacked sentence encoders for multi-domain inference","authors":"Bansal Nie, Yixin Nie, Mohit Bansal","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word repre- sentation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Seo et al., 2017] Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for ma- chine comprehension. In ICLR, 2017.","title":"Bidirectional attention flow for machine comprehension","authors":"Seo","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Shang et al., 2015] Lifeng Shang, Zhengdong Lu, and Hang Li. Neural responding machine for short-text conversation. In ACL, 2015.","title":"Neural responding machine for short-text conversation","authors":"Lifeng Shang, Zhengdong Lu, Hang Li","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Shen et al., 2017] Yelong Shen, Po-Sen Huang, Jianfeng Gao, and Weizhu Chen. Reasonet: Learning to stop reading in machine comprehension. In SIGKDD, 2017.","title":"Reasonet: Learning to stop reading in machine comprehension","authors":"Yelong Shen, Po-Sen Huang, Jianfeng Gao, Weizhu Chen","year":"2017","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Shen et al., 2018] Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. Disan: Directional self- attention network for rnn/cnn-free language understanding. In AAAI, 2018.","title":"Disan: Directional selfattention network for rnn/cnn-free language understanding","authors":"Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, Chengqi Zhang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Socher et al., 2014] Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compo- sitional semantics for finding and describing images with sen- tences. Transactions of the Association for Computational Lin- guistics, 2:207-218, 2014.","title":"Grounded compositional semantics for finding and describing images with sentences","authors":"Richard Socher, Andrej Karpathy, V Quoc, Christopher D Le, Andrew Y Manning, Ng","year":"2014","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Srivastava et al., 2015] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387, 2015.","title":"Highway networks","authors":"Rupesh Kumar Srivastava, Klaus Greff, Jürgen Schmidhuber","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christo- pher D Manning. Improved semantic representations from tree- structured long short-term memory networks. In ACL, 2015.","title":"Improved semantic representations from treestructured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Shazeer, Noam, Niki Par- mar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Williams, 1992] Ronald J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.","title":"Simple statistical gradientfollowing algorithms for connectionist reinforcement learning","authors":"Williams Ronald","year":"1992","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Xu et al., 2015] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.","title":"Show, attend and tell: Neural image caption generation with visual attention","authors":"Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, Yoshua Bengio","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yogatama et al., 2016] Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. Learning to compose words into sentences with reinforcement learning. arXiv preprint arXiv:1611.09100, 2016.","title":"Learning to compose words into sentences with reinforcement learning","authors":"Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, Wang Ling","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yu et al., 2017] Adams Wei Yu, Hongrae Lee, and Quoc V Le. Learning to skim text. arXiv preprint arXiv:1704.06877, 2017.","title":"Learning to skim text","authors":"Adams Wei, Yu, Hongrae Lee, Quoc V Le","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":41,"raw":"[Zhang and Lapata, 2017] Xingxing Zhang and Mirella Lapata. Sentence simplification with deep reinforcement learning. arXiv preprint arXiv:1703.10931, 2017.","title":"Sentence simplification with deep reinforcement learning","authors":"Lapata Zhang, Xingxing Zhang, Mirella Lapata","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":42,"raw":"[Zhao et al., 2014] Jiang Zhao, Tiantian Zhu, and Man Lan. Ecnu: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment. In SemEval@ COL- ING, pages 271-277, 2014.","title":"Ecnu: One stone two birds: Ensemble of heterogenous measures for semantic relatedness and textual entailment","authors":"Jiang Zhao, Tiantian Zhu, Man Lan","year":"2014","venue":"SemEval@ COL-ING","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}