{"type":"ScholarlyArticle","title":"Learning Sentence Representation with Guidance of Human Attention","authors":[{"name":"Shaonan Wang","slug":"shaonan-wang","given":"Shaonan","family":"Wang","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Jiajun Zhang","slug":"jiajun-zhang","given":"Jiajun","family":"Zhang","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China"},{"name":"Chengqing Zong","slug":"chengqing-zong","given":"Chengqing","family":"Zong","affiliation":"National Laboratory of Pattern Recognition , CASIA , Beijing , China; University of Chinese Academy of Sciences , Beijing , China; CAS Center for Excellence in Brain Science and Intelligence Technology , Shanghai , China"}],"abstract":"Recently, much progress has been made in learning general-purpose sentence representations that can be used across domains. However, most of the existing models typically treat each word in a sentence equally. In contrast, extensive studies have proven that human read sentences efficiently by making a sequence of fixation and saccades. This motivates us to improve sentence representations by assigning different weights to the vectors of the component words, which can be treated as an attention mechanism on single sentences. To that end, we propose two novel attention models, in which the attention weights are derived using significant predictors of human reading time, i.e., Surprisal, POS tags and CCG supertags. The extensive experiments demonstrate that the proposed methods significantly improve upon the state-of-the-art sentence representation models.","doi":"10.24963/ijcai.2017/578","url":"https://ijcai.org/proceedings/2017/578","pdf":"https://ijcai.org/proceedings/2017/0578.pdf","markdown":"https://ijcai.org/proceedings/2017/578.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/578","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":578,"pages":{"first":4137,"last":4143},"keywords":["Natural Language Processing: Natural Language Semantics","Natural Language Processing: Natural Language Processing"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2016] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sen- tence embeddings. arXiv, 2016.","title":"A simple but tough-to-beat baseline for sentence embeddings","authors":"Sanjeev Arora, Yingyu Liang, Tengyu Ma","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barrett et al., 2016] Maria Barrett, Joachim Bingel, Frank Keller, and Anders Søgaard. Weakly supervised part-of- speech tagging using eye-tracking data. The 54 th Annual Meeting of the Association for Computational Linguistics, page 579, 2016.","title":"Weakly supervised part-ofspeech tagging using eye-tracking data","authors":"Maria Barrett, Joachim Bingel, Frank Keller, Anders Søgaard","year":"2016","venue":"The 54 th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2015] Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Shiyu Wu, and Xuanjing Huang. Sentence modeling with gated recursive neural network. Proceedings of EMNLP, pages 793-798, 2015.","title":"Sentence modeling with gated recursive neural network","authors":"Chen","year":"2015","venue":"Proceedings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Demberg and Keller, 2008] Vera Demberg and Frank Keller. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193-210, 2008.","title":"Vera Demberg and Frank Keller. Data from eye-tracking corpora as evidence for theories of syntactic processing complexity","authors":"Keller Demberg","year":"2008","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Ganitkevitch et al., 2013] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. PPDB: The paraphrase database. HLT-NAACL, pages 758-764, 2013.","title":"PPDB: The paraphrase database","authors":"Juri Ganitkevitch, Benjamin Van Durme, Chris Callison-Burch","year":"2013","venue":"HLT-NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hale, 2001] John Hale. A probabilistic earley parser as a psycholinguistic model. Proceedings of the NAACL-HLT, pages 1-8, 2001.","title":"A probabilistic earley parser as a psycholinguistic model","authors":"John Hale","year":"2001","venue":"Proceedings of the NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hill et al., 2016a] Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sen- tences from unlabelled data. Proceedings of NAACL-HLT 2016,, pages 1367-1377, 2016.","title":"Learning distributed representations of sentences from unlabelled data","authors":"Hill","year":"2016","venue":"Proceedings of NAACL-HLT 2016","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hill et al., 2016b] Felix Hill, KyungHyun Cho, Anna Ko- rhonen, and Yoshua Bengio. Learning to understand phras- es by embedding the dictionary. Transactions of the Asso- ciation for Computational Linguistics, 4:17-30, 2016.","title":"Learning to understand phrases by embedding the dictionary","authors":"Hill","year":"2016","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jozefowicz et al., 2016] Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. Explor- ing the limits of language modeling. arXiv preprint arX- iv:1602.02410, 2016.","title":"Exploring the limits of language modeling","authors":"Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Keller, 2016] Michael Hahn Frank Keller. Modeling human reading with neural attention. Proceedings of the Confer- ence on Empirical Methods in Natural Language Process- ing, 85(95):95, 2016.","title":"Modeling human reading with neural attention","authors":"Michael Hahn, Frank Keller","year":"2016","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kennedy et al., 2003] Alan Kennedy, Robin Hill, and Joël Pynte. The dundee corpus. Proceedings of the 12 th Euro- pean conference on eye movement, 2003.","title":"The dundee corpus","authors":"Alan Kennedy, Robin Hill, Joël Pynte","year":"2003","venue":"Proceedings of the 12 th European conference on eye movement","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kenter et al., 2016] Tom Kenter, Alexey Borisov, and Maarten de Rijke. Siamese cbow: Optimizing word em- beddings for sentence representations. Proceedings of the 54 th Annual Meeting of the Association for Computational Linguistics, pages 941-951, 2016.","title":"Siamese cbow: Optimizing word embeddings for sentence representations","authors":"Kenter","year":"2016","venue":"Proceedings of the 54 th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. Ad- vances in neural information processing systems, pages 3294-3302, 2015.","title":"Skip-thought vectors","authors":"Ryan Kiros, Yukun Zhu, R Ruslan, Richard Salakhutdinov, Raquel Zemel, Antonio Urtasun, Sanja Torralba, Fidler","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Klerke et al., 2016] Sigrid Klerke, Yoav Goldberg, and An- ders Søgaard. Improving sentence compression by learn- ing to predict gaze. Proceedings of NAACL-HLT, pages 1528-1533, 2016.","title":"Improving sentence compression by learning to predict gaze","authors":"Klerke","year":"2016","venue":"Proceedings of NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Le and Mikolov, 2014] Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. Proceedings of the 31 st International Conference on Ma- chine Learning, pages 1188-1196, 2014.","title":"Distributed representations of sentences and documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"Proceedings of the 31 st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Levy, 2008] Roger Levy. Expectation-based syntactic com- prehension. Cognition, 106(3):1126-1177, 2008.","title":"Expectation-based syntactic comprehension","authors":"Roger Levy","year":"2008","venue":"Cognition","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2016] Zhouhan Lin, Minwei Feng, Ci- cero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A self-attentive sentence embedding. arXiv, 2016.","title":"A self-attentive sentence embedding","authors":"Lin","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ling et al., 2015] Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, and Silvio Amir. Not all contexts are created equal: Better word representations with variable attention. Proceedings of the EMNLP, pages 1367-1372, 2015.","title":"Not all contexts are created equal: Better word representations with variable attention","authors":"Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, Silvio Amir","year":"2015","venue":"Proceedings of the EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arX- iv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mitchell and Lapata, 2010] Jeff Mitchell and Mirella Lap- ata. Composition in distributional models of semantics. Cognitive science, 34(8):1388-1429, 2010.","title":"Composition in distributional models of semantics","authors":"Lapata Mitchell","year":"2010","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rayner, 1998] Keith Rayner. Eye movements in reading and information processing: 20 years of research. Psychologi- cal bulletin, 124(3):372, 1998.","title":"Eye movements in reading and information processing: 20 years of research","authors":"Keith Rayner, Rayner","year":"1998","venue":"Psychological bulletin","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tai et al., 2015] Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representa- tions from tree-structured long short-term memory net- works. Proceedings of the 53 rd Annual Meeting of the Association for Computational Linguistics, pages 1556- 1566, 2015.","title":"Improved semantic representations from tree-structured long short-term memory networks","authors":"Kai Sheng, Tai, Richard Socher, Christopher D Manning","year":"2015","venue":"Proceedings of the 53 rd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang and Zong, 2017] Shaonan Wang and Chengqing Zong. Comparison study on critical components in composition model for phrase representation. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(3):16, 2017.","title":"Comparison study on critical components in composition model for phrase representation","authors":"Wang, Zong ;, Shaonan Wang, Chengqing Zong","year":"2017","venue":"ACM Transactions on Asian and Low-Resource Language Information Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2016] Yashen Wang, Huang Heyan, Feng Chong, Zhou Qiang, Gu Jiahui, and Xiong Gao. Cse: Conceptual sentence embeddings based on attention mod- el. The 54th Annual Meeting of the Association for Com- putational Linguistics, pages 505-515, 2016.","title":"Cse: Conceptual sentence embeddings based on attention model","authors":"Wang","year":"2016","venue":"The 54th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wieting et al., 2016] John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphras- tic sentence embeddings. ICLR, 2016.","title":"Towards universal paraphrastic sentence embeddings","authors":"John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, X- iaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. Proceed- ings of the NAACL, 2016.","title":"Hierarchical attention networks for document classification","authors":"Yang","year":"2016","venue":"Proceedings of the NAACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu and Dredze, 2015] Mo Yu and Mark Dredze. Learning composition models for phrase embeddings. Transactions of the Association for Computational Linguistics, 3:227- 242, 2015.","title":"Learning composition models for phrase embeddings","authors":"Mo Yu, Mark Dredze","year":"2015","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}