{"type":"ScholarlyArticle","title":"Hype-HAN: Hyperbolic Hierarchical Attention Network for Semantic Embedding","authors":[{"name":"Chengkun Zhang","slug":"chengkun-zhang","given":"Chengkun","family":"Zhang","affiliation":"Discipline of Business Analytics , The University of Sydney Business School The University of Sydney , Camperdown , NSW 2006 , Australia"},{"name":"Junbin Gao","slug":"junbin-gao","given":"Junbin","family":"Gao","affiliation":"Discipline of Business Analytics , The University of Sydney Business School The University of Sydney , Camperdown , NSW 2006 , Australia"}],"abstract":"Hyperbolic space is a well-defined space with constant negative curvature. Recent research demonstrates its odds of capturing complex hierarchical structures with its exceptional high capacity and continuous tree-like properties. This paper bridges hyperbolic space's superiority to the power-law structure of documents by introducing a hyperbolic neural network architecture named Hyperbolic Hierarchical Attention Network (Hype-HAN). Hype-HAN defines three levels of embeddings (word/sentence/document) and two layers of hyperbolic attention mechanism (word-to-sentence/sentence-to-document) on Riemannian geometries of the Lorentz model, Klein model and Poincaré model. Situated on the evolving embedding spaces, we utilize both conventional GRUs (Gated Recurrent Units) and hyperbolic GRUs with Möbius operations. Hype-HAN is applied to large scale datasets. The empirical experiments show the effectiveness of our method.","doi":"10.24963/ijcai.2020/552","url":"https://ijcai.org/proceedings/2020/552","pdf":"https://ijcai.org/proceedings/2020/0552.pdf","markdown":"https://ijcai.org/proceedings/2020/552.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/552","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":552,"pages":{"first":3990,"last":3996},"keywords":["Natural Language Processing: Embeddings","Machine Learning: Explainable Machine Learning","Natural Language Processing: Text Classification"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24881/file_24881.m3u8","length_sec":236,"provider":"hls"}],"references":[{"position":1,"raw":"[Bécigneul and Ganea, 2018] G. Bécigneul and O.-E. Ganea. Riemannian adaptive optimization methods. ICLR, 2018.","title":"Riemannian adaptive optimization methods","authors":"G Bécigneul, O.-E Ganea","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cannon et al., 1997] J. W. Cannon, W. J. Floyd, R. Kenyon, and W. R. Parry. Hyperbolic geometry. Flavors of Geom- etry, 1997.","title":"Hyperbolic geometry","authors":"J W Cannon, W J Floyd, R Kenyon, W R Parry","year":"1997","venue":"Flavors of Geometry","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chelba et al., 2017] C. Chelba, M. Norouzi, and S. Bengio. N-gram language modeling using recurrent neural network estimation. arXiv:1703.10724, 2017.","title":"N-gram language modeling using recurrent neural network estimation","authors":"C Chelba, M Norouzi, S Bengio","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2017] J. Cheng, S. Zhao, J. Zhang, I. King, X. Zhang, and H. Wang. Aspect-level sentiment classifica- tion with heat (hierarchical attention) network. In CIKM, pages 97-106, 2017.","title":"Aspect-level sentiment classification with heat (hierarchical attention) network","authors":"J Cheng, S Zhao, J Zhang, I King, X Zhang, H Wang","year":"2017","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Ben- gio. Learning phrase representations using rnn encoder- decoder for statistical machine translation. In EMNLP, pages 1724-1734, 2014.","title":"Learning phrase representations using rnn encoderdecoder for statistical machine translation","authors":"K Cho, B Van Merriënboer, C Gulcehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Conneau et al., 2017] A. Conneau, H. Schwenk, L. Bar- rault, and Y. Lecun. Very deep convolutional networks for text classification. In EACL, pages 1107-1116, 2017.","title":"Very deep convolutional networks for text classification","authors":"A Conneau, H Schwenk, L Barrault, Y Lecun","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Damashek, 1995] M. Damashek. Gauging similarity with n- grams: Language-independent categorization of text. Sci- ence, 267(5199):843-848, 1995.","title":"Gauging similarity with ngrams: Language-independent categorization of text","authors":"M Damashek","year":"1995","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ganea et al., 2018] O. Ganea, G. Bécigneul, and T. Hof- mann. Hyperbolic neural networks. In NIPS, 2018.","title":"Hyperbolic neural networks","authors":"O Ganea, G Bécigneul, T Hofmann","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gao et al., 2017] S. Gao, M. Young, J. Qiu, H. Yoon, J. Christian, P. Fearn, G. Tourassi, and A. Ramanthan. Hierarchical attention networks for information extraction from cancer pathology reports. J Am Med Inform Assoc., 25(3):321-330, 2017.","title":"Hierarchical attention networks for information extraction from cancer pathology reports","authors":"S Gao, M Young, J Qiu, H Yoon, J Christian, P Fearn, G Tourassi, A Ramanthan","year":"2017","venue":"J Am Med Inform Assoc","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Gulcehre et al., 2018] C. Gulcehre, M. Denil, M. Ma- linowski, A. Razavi, R. Pascanu, K. Mo. Hermann, P. Battaglia, V. Bapst, D. Raposo, A. Santoro, and N. de Freitas. Hyperbolic attention networks. ICLR, 2018.","title":"Hyperbolic attention networks","authors":"C Gulcehre, M Denil, M Malinowski, A Razavi, R Pascanu, K Mo, P Hermann, V Battaglia, D Bapst, A Raposo, N Santoro, De Freitas","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Joulin et al., 2017] A. Joulin, É. Grave, P. Bojanowski, and T. Mikolov. Bag of tricks for efficient text classification. In EACL, pages 427-431, 2017.","title":"Bag of tricks for efficient text classification","authors":"A Joulin, É Grave, P Bojanowski, T Mikolov","year":"2017","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kim et al., 2016] Y. Kim, Y. Jernite, D. Sontag, and A. M. Rush. Character-aware neural language models. In AAAI, pages 2741-2749, 2016.","title":"Character-aware neural language models","authors":"Y Kim, Y Jernite, D Sontag, A M Rush","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kim, 2014] Y. Kim. Convolutional neural networks for sen- tence classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Y Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krioukov et al., 2010] D. Krioukov, F. Papadopoulos, M. Kitsak, A. Vahdat, and M. Boguná. Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.","title":"Hyperbolic geometry of complex networks","authors":"D Krioukov, F Papadopoulos, M Kitsak, A Vahdat, M Boguná","year":"2010","venue":"Physical Review E","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Lee and Seung, 1999] D. D. Lee and H. S. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401(6755):788-791, 1999.","title":"Learning the parts of objects by non-negative matrix factorization","authors":"D D Lee, H S Seung","year":"1999","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018] Z. Li, Y. Wei, Y. Zhang, and Q. Yang. Hier- archical attention transfer network for cross-domain senti- ment classification. In AAAI, 2018.","title":"Hierarchical attention transfer network for cross-domain sentiment classification","authors":"Z Li, Y Wei, Y Zhang, Q Yang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lodhi et al., 2002] H. Lodhi, C. Saunders, J. Shawe-Taylor, N. Cristianini, and C. Watkins. Text classification using string kernels. JMLR, 2(Feb):419-444, 2002.","title":"Text classification using string kernels","authors":"H Lodhi, C Saunders, J Shawe-Taylor, N Cristianini, C Watkins","year":"2002","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Nickel and Kiela, 2017] M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. In NIPS, pages 6338-6347, 2017.","title":"Poincaré embeddings for learning hierarchical representations","authors":"M Nickel, D Kiela","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nickel and Kiela, 2018] M. Nickel and D. Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In ICML, pages 3779-3788, 2018.","title":"Learning continuous hierarchies in the lorentz model of hyperbolic geometry","authors":"M Nickel, D Kiela","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Palmer, 1977] S. E. Palmer. Hierarchical structure in per- ceptual representation. Cognitive Psychology, 9(4):441- 474, 1977.","title":"Hierarchical structure in perceptual representation","authors":"S E Palmer","year":"1977","venue":"Cognitive Psychology","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pennington et al., 2014] J. Pennington, R. Socher, and C. Manning. Glove: Global vectors for word represen- tation. In EMNLP, pages 1532-1543, 2014.","title":"Glove: Global vectors for word representation","authors":"J Pennington, R Socher, C Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ramos, 2003] J. Ramos. Using tf-idf to determine word rel- evance in document queries. In ICML, 2003.","title":"Using tf-idf to determine word relevance in document queries","authors":"J Ramos","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Ratcliffe et al., 1994] J. G. Ratcliffe, S. Axler, and K. Ribet. Foundations of Hyperbolic Manifolds. Springer, 1994.","title":"Foundations of Hyperbolic Manifolds","authors":"J G Ratcliffe, S Axler, K Ribet","year":"1994","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ullman, 1996] S. Ullman. High-level vision: Object Recog- nition and Visual Cognition, volume 2. MIT Press Cam- bridge, MA, 1996.","title":"High-level vision: Object Recognition and Visual Cognition","authors":"S Ullman","year":"1996","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wallach, 2006] H. M. Wallach. Topic modeling: beyond bag-of-words. In ICML, pages 977-984, 2006.","title":"Topic modeling: beyond bag-of-words","authors":"H M Wallach","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang and Manning, 2012] S. Wang and C. D. Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In ACL, pages 90-94, 2012.","title":"Baselines and bigrams: Simple, good sentiment and topic classification","authors":"S Wang, C D Manning","year":"2012","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiao and Cho, 2016] Y. Xiao and K. Cho. Efficient character-level document classification by combining con- volution and recurrent layers. arXiv:1602.00367, 2016.","title":"Efficient character-level document classification by combining convolution and recurrent layers","authors":"Y Xiao, K Cho","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xing et al., 2018] C. Xing, Y. Wu, W. Wu, Y. Huang, and M. Zhou. Hierarchical recurrent attention network for re- sponse generation. In AAAI, 2018.","title":"Hierarchical recurrent attention network for response generation","authors":"C Xing, Y Wu, W Wu, Y Huang, M Zhou","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yang et al., 2016] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy. Hierarchical attention networks for document classification. In AACL, 2016.","title":"Hierarchical attention networks for document classification","authors":"Z Yang, D Yang, C Dyer, X He, A Smola, E Hovy","year":"2016","venue":"AACL","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yao et al., 2019] L. Yao, C. Mao, and Y. Luo. Graph con- volutional networks for text classification. In AAAI, vol- ume 33, pages 7370-7377, 2019.","title":"Graph convolutional networks for text classification","authors":"L Yao, C Mao, Y Luo","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Ying et al., 2018] H. Ying, F. Zhuang, F. Zhang, Y. Liu, G. Xu, X. Xie, H. Xiong, and J. Wu. Sequential recom- mender system based on hierarchical attention networks. In IJCAI, pages 3926-3932, 2018.","title":"Sequential recommender system based on hierarchical attention networks","authors":"H Ying, F Zhuang, F Zhang, Y Liu, G Xu, X Xie, H Xiong, J Wu","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/546"},{"position":32,"raw":"[Yogatama et al., 2017] D. Yogatama, C. Dyer, W. Ling, and P. Blunsom. Generative and discriminative text classifica- tion with recurrent neural networks. In ICML, 2017.","title":"Generative and discriminative text classification with recurrent neural networks","authors":"D Yogatama, C Dyer, W Ling, P Blunsom","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang et al., 2015] X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classifica- tion. In NIPS, pages 649-657, 2015.","title":"Character-level convolutional networks for text classification","authors":"Zhang","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhao et al., 2019] Y. Zhao, Y. Shen, and J. Yao. Recur- rent neural network for text classification with hierarchical multiscale dense connections. In IJCAI, pages 5450-5456, 2019.","title":"Recurrent neural network for text classification with hierarchical multiscale dense connections","authors":"Y Zhao, Y Shen, J Yao","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/757"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}