Hype-HAN: Hyperbolic Hierarchical Attention Network for Semantic Embedding

1 Introduction

Semantic embedding is one of the most fundamental tasks of natural language processing (NLP). It aims to represent textual elements with low-dimensional latent variables while preserving their discriminative features. We witness the explosive growth of the internet and the resulting enormous textual content at our disposal. It becomes an important task to develop efficient and effective methods to process such amount of textual information.

Automatic text classification is a ubiquitous application of semantic embedding. With the pre-assigned labels on the training documents, one can complete the task with numerous feature selection techniques and various classifying architecture in favour of different focuses. In terms of traditional approaches, people may encode raw documents as sparse lexical features, such as n-grams [Damashek, 1995], tf-idf [Ramos, 2003] and bag-of-words [Wallach, 2006]. These features can be sent into classical classifiers, such as linear model or kernel methods [Wang and Manning, 2012; Lodhi et al., 2002]. Recently, as computational power develops, neural network based methods have obtained its momentum in the field.

Typical practices include methods founded on convolutional neural network and recurrent neural network [Kim, 2014; Kim et al., 2016; Yao et al., 2019; Zhao et al., 2019]. In particular, a number of researchers realize the natural hierarchical structure of documents and start designing specific neural network architectures acordingly [Yang et al., 2016; Gao et al., 2017; Ying et al., 2018].

However, most of the main-stream methods are defined on Euclidean space, which is not particularly suitable or requiring tremendous embedding size for capturing complex networks. Though large embedding size may not be considered as a major drawback any longer thanks to the light-speed computational devices and the inundant memory, it is still a hard open question to interpret the hierarchical meaning of purely numerical latent variables. This paper tackles this gap by introducing a new network structure to embed the semantic relations on a self-informative manifold.

The hyperbolic manifold is a smooth manifold of constant negative curvature. It was studied in the ancient differential geometry [Cannon et al., 1997] but just started gaining more attention in machine learning field lately [Nickel and Kiela, 2017; Nickel and Kiela, 2018; Ganea et al., 2018; Gulcehre et al., 2018]. With its constant negative curvature, the distance within this type of manifold is self-informative and the resulting space volume grows exponentially even in dimension of two. These characteristics make it particularly advisable for encoding complex networks.

Different from previous hyperbolic NLP practices with manually annotated pair-wise relation, such as [Nickel and Kiela, 2017; Nickel and Kiela, 2018], which consider the supervised word-to-word link hierarchy, or [Ganea et al., 2018], which considers the sentence-to-sentence semantic relation, we regard word-to-sentence and sentence-to-document as two types of natural hierarchical labels. We build our work upon the well-known hierarchical attention architecture (HAN) [Yang et al., 2016; Gao et al., 2017; Cheng et al., 2017; Xing et al., 2018; Li et al., 2018], which considers a knowledge accumulation process according to the structure of the target. We practice this idea and extend it in various ways:

• As complex networks require large embedding size and hierarchical relations are difficult to interpret in Euclidean space, we constrain the computational process in hyperbolic space to incorporate advantages of hyperbolic manifolds. Since hyperbolic coordinate is selfinformative, this enables us to include not only the hierarchical structure of explicit semantic levels (e.g. between words and sentences), but also the implicit parentchild relation among the same level (e.g. among words).
• We encode semantic representations from sequences instead of pre-defined tokens through gated recurrent units (GRUs) [Cho et al., 2014]. As the operational manifolds are evolving along the forward path, to remain the geometric features, we utilize Euclidean GRUs and M¨obius GRUs accordingly. Readers can find a conceptual architecture in Figure 3.
• We believe it is ill-posed to allocate semantic elements in different spaces since sometimes a single word and the whole sentence can be equivalently meaningful. Instead, we design our architecture delicately to ensure representations are aggregated from their sub-trees via hyperbolic attention mechanism.
• We utilize isometric hyperbolic models considering their different geodesic features in favour of computational convenience 1 . Features of word sequences are firstly projected on the Lorentz model. Then, word representations are aggregated on the Klein model via Einstein midpoint. Similar procedure are conducted for sentence accumulation except that sentence encoder is defined on Poincar´e model due to its convenient M¨obius operations.

2 Hyperbolic Geometry Initiation

Hyperbolic space, specifically referred to manifolds with constant negative curvature in this paper, has been studied in differential geometry for long, considered under five isometric models [Cannon et al., 1997]. Among them, Poincar´e ball model , Lorentz model and Klein model have received increasing attention in machine learning community due to their attractive features for modeling complex network.

1 Geometrical properties are fully kept when representations are transformed between isometric models. Readers could refer to [Ratcliffe et al., 1994] for a more detailed explanation.

Poincar´e ball model. The Poincar´e ball model is defined as a Riemannian manifold P n = (Bn , gP ), where Bn = {x ∈ Rn : ‖x‖ < 1}, and themetric tensor gP (x) = ( 2 1-‖x‖2 )2 gE , with the Euclidean metric tensor gE . The distance on this manifold is defined as:

Lorentz model. The Lorentz model is the only unbounded hyperbolic model and is defined as Ln = (Hn , gL) with points constrained by Hn = {x ∈ Rn+1 : 〈x, x〉L = -1, x0 > 0}, where 〈x, y〉L is the Lorentzian scalar product:

and the metric tensor is: gL(x) = diag(-1, 1, ..., 1). The pairwise distance function is given as:

Since all hyperbolic models are isometric, one can project data from Lorentzian coordinates to Poincar´e ball by:

Klein Model. Klein model is also defined on Kn = {x ∈ Rn : ‖x‖ < 1}. The isomorphism between Klein model and Poincar´e ball can be defined through a projection on or from the hemisphere model. People can get Klein points from Poincar´e coordinates by:

or from Lorentzian coordinates by:

3 Hyperbolic Hierarchical Attention Network

In the following, we construct a knowledge accumulation process based on hyperbolic geometry and utilize the attention mechanism to capture the semantic subordination. We treat word-to-sentence and sentence-to-document as two natural hierarchical labels. We enforce latent variables of these semantic elements lie on a hyperbolic space and aggregate them according to a child-parent structure. An abstract architecture of Hype-HAN is demonstrated in Figure 3.

3.1 Hyperbolic Activation

Though conventional RNN-based method has proven its success in modeling sequence data, it suffers from a crucial limitation that the ability to model complicated hierarchies is tightly bounded by the dimensions of latent variables. Meanwhile, the hierarchical relation between components on the same semantic level (word-to-word or sentence-to-sentence) is not delicately taken care of since Euclidean space is not suitable to capture the existence of such power-law distribution. Thus, to utilize the elegant feature of hyperbolic models, we consider constraining latent variables on a Lorentz model.

Figure 2: (a) Points on a unit Poincar´e disk with in-between distances: dP = 0.1 are projected on the Lorentz model. The corresponding Lorentz factor γ(xK) = 1 √1-‖xK‖2 grows when points are closer to the boundary. (b) (i) The geodesics in a Poincar´e disk are circular arcs perpendicular to the boundary (red) while the same line in a Klein model is straight (green) . (ii) Hyperbolic circles have their own hyperbolic radius. A circle in a Poincar´e disk (red) is equivalent to an ellipse (green) in Klein model. But both centers are not at the Euclidean centers of the circle or the ellipse.

Denote the state output of word encoder as hit, we first transform this vector into its polarity form as (d, r) ∈ Rn+1 , where r = ‖hit‖ and d = hit r , i.e. ‖d‖ = 1. Then, the following activation function is conducted to constrain the representation into a valid hyperbolic form:

We can verify the validity of htL = (sinh(r)d, cosh(r)) by checking whether it possesses Lorentzian characteristics from two aspects:

• Each point must have its squared Lorentz norm equal to -1. In our case, it is easy to show 〈htL, htL〉L = -1.
• The volume as embedding space should grow exponentially in terms of a linear increase in radius. In our case,

which also satisfies the condition. This en-powers the network with much larger capacity compared to Euclidean space for modeling sophisticated relations.

By using such activation technique, we have constrained the output of GRUs as hyperbolic points before sending them to the next layer. Since hyperbolic manifold has a continuous tree structure, it enables the components on the same layer of neural network to also share a hierarchical structure (among words or among sentences). Thus, the simple discrete double-level hierarchies (word-to-sentence and sentence-todocument) are en-powered as its continuous analogue.

3.2 Semantic Encoders

Figure 3 shows both word and sentence encoders based on GRUs to capture the semantic representation of sequences.

The intuition of such design is considering that not only different words/sentences have different meaning, same words may also be differentially informative in different context, i.e. the sequence of words plays a significant role in semantic analysis.

Consider the embedding of word-t appeared in sentence-i as xit, which can be initialized from a linear layer or pretrained vectors, the hidden state of xit within the sentence can be constructed with a GRUs layer as:

where ---→ GRU(xit) learns the forward hidden state while ←--- GRU(xit) learns the backward hidden state around the queried word.

Given that latent variables may lie on different manifolds (Euclidean or hyperbolic), GRU architectures used in this work are divided into two streams:

Euclidean GRU Architecture. With slight abuse of notation in this and the following paragraph, we denote input vector at time-t as xt, one can adapt the Euclidean GRU archi-

where σ and ϕ are point-wise non-linearity, ⊙ is point-wise product, W, U and b are parameters, ˜ht is the new state computed from the new sequence, rt decides how much information should be preserved to construct ˜ht, while zt decides the contribution proportion of the past and new information to construct ht.

However, as can be noticed, Euclidean GRU is based on several linear layers with non-linearity manually added. This is not valid on manifold spaces, thus we employ hyperbolic GRUs after latent variables are projected on hyperbolic space.

Hyperbolic GRU Architecture. Ganea [2018] introduced M¨obius operations on Poincar´e model, including M¨obius addition, multiplication and bias, which are the key elements a GRU needs. Thus, M¨obius GRU on the Poincar´e model is defined as:

where log0 is the logarithmic map, ϕ⊗ is the M¨obius nonlinearity and diag(u) is the square diagonal matrix.

The rationale of using different types of GRU architecture is to utmostly remain the geometric features of semantic embeddings. With such design, different semantic elements at different level can all utilize the geodesic features of hyperbolic manifolds.

3.3 Hyperbolic Semantic Aggregation

We now discuss how to conduct the ubiquitous attention mechanism to aggregate hyperbolic individual components.

Word-to-Sentence Aggregation

As discussed in the previous, we activate hidden state of words onto hyperbolic space. It is obvious that words would not contribute equally to the meaning of one sentence. Thus, we need to construct a meaningful aggregation strategy.

Different from Euclidean space with constant zero curvature, hyperbolic attentions are not very obvious to design. Inspired by [Ganea et al., 2018], we exercise the idea of using Einstein midpoint to compute the aggregation weights of attention and demonstrate new interpretation in the context of textual analysis.

Consider the hyperbolic hidden state hitL, in order to utilize the Einstein midpoint defined on Klein model, we first take the transformation (4): hitL → hitK. Then, to construct the attention weights for aggregation, we utilize the hyperbolic power-law characteristics to measure the importance of individual components. We aim to jointly learn a hyperbolic word centroid cwL from all the training documents. cwL can be considered as a baseline for measuring the importance of hyperbolic words based on their mutual distance. The underlying reasoning of such design generally considers two aspects: First, though training documents have been labeled independently, certain hidden states should play a dominant role. Second, traditional documentation methods consider document labels as separate entities, but there may also exist principal and subordinate structure among document labels. For instance, in terms of research paper classification, documents may be labeled as 'Topology', 'Mathematics', 'Social Science' and 'Psychology'. Obviously, 'Mathematics' and 'Topology' may abide by certain hierarchical relation while the same for 'Social Science' and 'Psychology'.

To learn cwL, we consider another layer upon hidden state hit as:

Similar to projecting hit onto hyperbolic space, we use (5) to activate hit ′ as hit ′L. Then the attention weights is computed as αit by:

After capturing the hyperbolic attention weights, semantic meaning of words appearing in the same sentences has been aggregated via Einstein midpoint:

where γ(hitK) is the so-called Lorentz factors, which can be expressed as:

Since Klein and Poincar´e ball model reflect the power-law distribution with its radius, it is not difficult to interpret the above function as a relativistic importance factor.

Sentence-to-Document Aggregation

Similar to the word-to-sentence attention, sentences with more relevant information of the topic can be the clues to classify the documents, thus we use the similar attention mechanism to aggregate sentences as document representation.

However, as we mentioned in the previous that the operational space evolved from Euclidean to hyperbolic manifold, the linear layer (6) will no longer hold on a curved manifold. Thus, we consider a 'linear' layer within Poincar´e space:

No tanh() is applied because this layer has already preserved non-linearity.

We first take: hiP → hiK and h′iP → h′iL via (3) and the reversed (2), then construct the document representation via Einstein midpoint:

where csL is the sentence centroid, which can be randomly initialized and jointly learned with the other network parameters, and vK is the semantic meaning of the whole document summarized from all sentences.

Since vK is a high level representation of the whole document, similar to any other classification task, it can be sent to a hyperbolic 'linear layer' with softmax activation for document classification.

4 Evaluation

With the help of the self-informative and high-capacity hyperbolic manifolds, the proposed method brings forward its main advantages on explainability compared to the other neural network-based practices. We validate Hype-HAN from two aspects: (i) a conventional document classifier, to classify any large-scale datasets with performance on par with the state-of-the-art; and (ii) the core of a hierarchical interpreter or visualization toolbox, to demonstrate the hierarchical structure of the dataset on a tiny dimension through its hyperbolic coordinates.

4.1 Benchmark Comparison

In order to validate our work in the real-world scenario, we test Hype-HAN on some publicly available large-scale benchmark datasets with the same protocols from [Zhang et al., 2015; Yogatama et al., 2017]. The datasets include news classification (AGnews), question/answer categorization (Yahoo Answers), sentiment analysis (Yelp and Amazon) and Wikipedia article classification (DBpedia). The descriptive statistics of the dataset has been shown in table 1.

We consider the best reported benchmarks as baselines, including: N.Bayes and LSTM from [Yogatama et al., 2017], Bow, n-grams, n-TFIDF and ch-CNN from [Zhang et al., 2015], ch-CRNN from [Xiao and Cho, 2016], VDCNN from [Conneau et al., 2017], fastText from [Joulin et al., 2017] and HAN from [Yang et al., 2016].

To assess the generalization ability, we evaluate HypeHAN with the same setting on different datasets. We initialize the word embedding via 'glove-50' [Pennington et al., 2014], and we set the word/sentence hidden state as 50dimension and train the models with manifold-aware Riemannian ADAM [B´ecigneul and Ganea, 2018] with learning rate 0.001. We record the prediction accuracy on the test set around epoch 10-20 on smaller datasets (AGnews, DBpedia, Yelp), and around epoch 5-10 on the large-scale datasets (Yahoo and Amazon). The results are reported in Table 2.

It can be indicated that Hype-HAN outperforms its Euclidean ancestor on all datasets and achieves similar results with the state-of-the-art, especially on datasets related to topic classification, including AGnews, DBpedia, Yahoo answers. However, the performance on sentiment datasets (Yelp and Amazon reviews) is weaker compared to CNN-based methods and fastText(b) with bigram input. We suggest the phenomenon can be explained from several aspects:

• First, sentiment elements may share fewer hierarchical structure compared to ontology and taxonomy, which have straightforward power-law distributions. We conclude this as the major reason for the performance downturn in terms of predicting review ratings.
• Second, if the hierarchical information does not play a dominant role in the dataset, the depth of the network will account for more credits, which explains the advantages of CNN-based methods, such as the Very Deep Convolutional Networks [Conneau et al., 2017].
• Third, as can be noticed, fastText(b) with bigram information is very robust while the version without utilizing bigram information has a large drop on the performance, especially on the sentiment tasks. As discussed by Joulin [2017], adding the bigram information on their architecture has increased the performance by 1-4%. We sug-

gest that using bigram(n-grams) could also capture the sentimental information similar to CNN-based methods. Thus, for future work, one could investigate whether Hype-HAN could benefit from such extra information since it has been reported that RNN-based LSTM cell with dropout could be the best model for encoding the n-gram state [Chelba et al., 2017].

4.2 Hyperbolic Interpretation

More importantly, compared to the other neural networkbased classifier, Hype-HAN brings forward its main advantages on explainability, thus the usage of the proposed method is not limited for traditional classification tasks. Benefit from the self-informative hyperbolic coordinates, the learned semantic representations can be utilized directly for generating insights.

We scrape 1000 scientific abstracts from arXiv.org with keywords: 'manifold' and 'supermassive black hole' respectively. We split 2000 samples into 1600/400 training-test set and use only three-dimensional embedding space for both Euclidean and M¨obius GRU to visualize the knowledge accumulation process. This experiment is conducted without a GPU and the embedding converges in 11 epochs with 99.7% prediction accuracy (default hyperparameter setting with the benchmark experiment).

We record the Poincar´e hidden states of a random minibatch at epoch 10 and visualize the points at the word, sentence and document level. As shown in Figure 5, the embedding of different semantic elements can be well-represented in the same hyperbolic space, which validates our hypothesis. Meanwhile, one could notice that the attention mechanism, based on Einstein midpoint (8) and (12), plays a prominent role in the aggregation step. Since the attention is computed based on Klein coordinates (with straight geodesics), the direction of the vector would dominate the process and motivate embeddings to evolve along separate directions.

Sent. 87: we prove conforml positive mass theorems for asymptotically flat manifolds with charge.

Sent. 65: pbhs much larger than this might have important consequences even if they provide only a small faction of the dark matter.

Sent. 118: We study the interaction between the geodesic orbit (g.o.~ in short) property and certain flag curvature condition. Sent. 95: outflow size-luminosity relationenergetic gas outflows driven by active galactic nuclei (agns) are considered as one of the mechanisms.

Figure 6: Sentence embeddings from one batch during the training process. It could be noticed that sentences with keywords: 'manifolds' and 'dark matter' have a larger Lorentz factor.

To demonstrate the hierarchical structure of the semantic embeddings, we randomly extract the sentence representations from a mini-batch and visualize it in a Poincar´e ball. Different from Euclidean variables, one could investigate the importance of semantic elements based on Lorentz factor (or radius). As shown in Figure 6, sentences with keywords: 'manifolds' and 'dark matter', are closer to the boundary of the defined space. Thus, apart from the traditional classifier usage, we suggest that one could utilize hyperbolic embeddings for more insights generation tasks, such as (i) lowdimensional representation for hierarchical visualization, and (ii) statistical methods to summarize the power-law distributions.

5 RelatedWork

Hierarchical structures are important in various of representation and logical reasoning tasks. Researchers reveal that perceptual representation in human brain may have a hierarchical relation [Palmer, 1977] and there is also physiological evidence supporting the parts-based representation theory in object recognition [Ullman, 1996]. People are inspired from such intuition and build practical algorithms. For instance, Lee and Seung [1999] use non-negative matrix factorization to represent images based on their partial information. Hyperbolic geometry, receives growing attention in hierarchical network embedding and machine learning field recently. For instance, Krioukov et al. [2010] demonstrates that heterogeneous degree distributions and strong clustering of complex network can be modeled with hyperbolic geometry. Nickel and Kiela [2017; 2018] discuss the supervised word-to-word link prediction tasks, and Ganea et al. [2018] considers the sentence-to-sentence textual entailment in hyperbolic space. We are inspired from these works and reformulate the problem as a manifold-based hierarchical knowledge accumulation process.

6 Conclusion

In this work, we practice a hierarchical embedding method: Hype-HAN based on three types of hyperbolic manifolds. The proposed method is evaluated with large-scale datasets for text classification tasks. Hype-HAN can be used as (i) the core of a hierarchical interpreter or visualizer without explicitly storing the discrete tree structure, or (2) a conventional document classifier with its performance on par with the state-of-the-art.