--- title: "F-HMTC: Detecting Financial Events for Investment Decisions Based on Neural Hierarchical Multi-Label Text Classification" authors: - "Xin Liang" - "Dawei Cheng" - "Fangzhou Yang" - "Yifeng Luo" - "Weining Qian" - "Aoying Zhou" doi: "10.24963/ijcai.2020/619" url: "https://ijcai.org/proceedings/2020/619" pdf: "https://ijcai.org/proceedings/2020/0619.pdf" venue: "Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence" year: 2020 pages: "4490-4496" track: "Special Track on AI in FinTech" keywords: - "Foundation for AI in FinTech: Deep learning and representation for FinTech" - "Foundation for AI in FinTech: Data mining and knowledge discovery for FinTech" --- # F-HMTC: Detecting Financial Events for Investment Decisions Based on Neural Hierarchical Multi-Label Text Classification Xin Liang, Dawei Cheng, Fangzhou Yang, Yifeng Luo, Weining Qian, Aoying Zhou *Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence*, pages 4490-4496. DOI: [10.24963/ijcai.2020/619](https://doi.org/10.24963/ijcai.2020/619) ## Abstract The share prices of listed companies in the stock trading market are prone to be influenced by various events. Performing event detection could help people to timely identify investment risks and opportunities accompanying these events. The financial events inherently present hierarchical structures, which could be represented as tree-structured schemes in real-life applications, and detecting events could be modeled as a hierarchical multi-label text classification problem, where an event is designated to a tree node with a sequence of hierarchical event category labels. Conventional hierarchical multi-label text classification methods usually ignore the hierarchical relationships existing in the event classification scheme, and treat the hierarchical labels associated with an event as uniform labels, where correct or wrong label predictions are assigned with equal rewards or penalties. In this paper, we propose a neural hierarchical multi-label text classification method, namely F-HMTC, for a financial application scenario with massive event category labels. F-HMTC learns the latent features based on bidirectional encoder representations from transformers, and directly maps them to hierarchical labels with a delicate hierarchy-based loss layer. We conduct extensive experiments on a private financial dataset with elaborately-annotated labels, and F-HMTC consistently outperforms state-of-art baselines by substantial margins. We will release both the source codes and dataset on the first author's repository. ## Full text ## 1 Introduction Stock trading is an important kind of financial activity concerning investing and financing, and a lot of people and institutions are involved as investors in the stock trading market, buying and selling stock shares to gain margin profits. The share prices of listed companies are generally supported by several key drivers concerned with these companies, such as sales volumes, quarterly/annual revenues, gross profit margins, net incomes and earnings per share, etc. While these key drivers are usually influenced by various factors, such as politics, policies and macroeconomy, and changes of these factors would eventually result in changes of key drivers of supporting share prices. The status changes of a factor that influences the key drivers are usually presented as formal documents, such as news articles and policy statements, and the status changes of factors are deemed as events that would eventually affect the share prices of listed companies. ∗Corresponding Author 1 https://github.com/finint/F-HMTC Since the share prices of a listed company may experience frequent ups and downs, investors in the stock trading markets need to face routine investment risks and opportunities. Since implementing event detection could help them to identify the investment risks and opportunities as early as possible, people are focusing on analyzing the documents related with events, so as to perceive the status changes of factors, and get prepared to avoid deleterious risks and seize beneficial opportunities, aiming to gain as many margin profits as possible. With detected events, people often perform backtesting to verify the connections between the events and the variation trends of stock share prices, and thus derive investment rules or tactics from these established connections. With the derived investment rules or tactics, people could leverage event detection to predict the variation trends of stock share prices, assuming that similar events would exert similar impacts on the variation trends of stock share prices. Based on the detected events, people could make their investment decisions, and some devoted to quantitative investment may even directly buy or sell stock shares, via following the investment rules programmed in their automated trading systems. It is thus crucial to implement accurate and reliable event detection mechanisms for these event-driven investment applications, since they rely on detected events to make investment decisions. In actual application scenarios, the financial event scheme is often presented and organized with hierarchies, where a hierarchy denotes a sequence of event categories, and lower hierarchies denote finer event categories. As events could be classified with various hierarchies of categories, an event could be assigned with multiple category labels, indicating the hierarchical categories it should be classified with. Thus the financial event detection problem could be modeled as a hierarchical multi-label text classification (HMTC) problem[Wu et al., 2016], where the whole event scheme could be denoted as a category tree, and an event instance can be associated with multiple hierarchical categories[Triguero and Vens, 2016] and represented as an unbranched path covering the contiguous hierarchical categories within the event scheme, from a top hierarchy node to a bottom hierarchy node[Sun and Lim, 2001]. Figure 1: An investment analysis and market prediction framework based on event detection. ![Image](https://ijcai.org/proceedings/2020/fulltext/619/image_000000_2dada9c924b4fedbee6e7d4b73de3a4dd7fb089baa7fa125a3c805a0c9268e35.png) Most HMTC methods assume that categories are independent, and dismiss the structure dependencies existing between category hierarchies, while others utilizing ensemble-based methods to simulate the hierarchical structures are computationally inefficient when dealing with large-scale numbers of categories. Besides, equal weights are given to all categories in these HMTC methods for false-positive penalties, while the parent, brother and child category labels associated with a category label should actually be treated discriminatively in real applications, since they denote different hierarchical levels and thus contain various amounts of information for predicting the label. Although [Wu et al., 2019] considered hierarchical dependency information, they did not distinguish the impacts of different dependency relationships. Sometimes it is more acceptable to predict a label as its ground-truth parent hierarchy label than to predict it as some sibling label, since the parent label is a super hierarchical category of the children labels, and sibling labels within the same hierarchy are mutually exclusive. In this paper, we propose an event detection mechanism based on neural hierarchical multi-label text classification. We employ bidirectional encoder representations from Transformers to learn deep representations for documents, and then embed the tree-structured event category scheme into a neural prediction layer. Moreover, we propose an industrial evaluation metric, and employ the metric for loss computation, where losses of wrong predictions for labels across different hierarchies are computed discriminatively. We finally conduct extensive experiments on our industrial dataset to evaluate our model, and we achieve remarkable performance improvements, compared with the state-of-art baseline methods. Our main contributions include: 1. We model financial event detection as a hierarchical multi-label text classification problem, and propose a neural event detection model, namely F-HMTC, for our real-life industrial application scenario. 2. We propose an industrial evaluation metric, and leverage the metric for loss evaluation with discriminative hierarchical prediction losses. 3. We conduct extensive experiments on our private financial dataset with elaborately-annotated labels, and we will release the dataset for public research. ## 2 Application Background In this paper, we leverage event detection to help people make informed investment decisions in the application scenario illustrated in Fig.1. Here financial documents including news reports, industry research reports and announcements of listed companies, are collected for event detection based on a tree-structured event scheme, where a document may be associated with multiple event paths. Then the detected events are used for event inference with the extracted event elements and their embedding representations. We implement event inference via combining two kinds of knowledge graphs, namely the enterprise knowledge graph and the event evolutionary graph. The enterprise knowledge graph describes the relationships of entities related to all listed companies, including their major shareholders, senior managers, suppliers, partners and competitor companies, etc. and is mainly used to infer which companies shall be influenced by a detected event. The event evolutionary graph describes the connections between various events and their consequences, and is mainly used to infer how the detected event would influence the key drivers and ultimately the stock prices of the listed companies. The final investment analysis and market prediction results could be provided to industry researchers and investment advisory service subscribers, and they can make their own investment decisions based on the provided results. In this application scenario, the hierarchical event scheme is constructed as an event category tree [Sun and Lim, 2001], where a tree node denotes an event category, and a descendant of the node denotes a finer-granularity event category. Here a document can be associated with multiple event paths of various depths in the event category tree, where an event path denotes a sequence of contiguous event hierarchies. As we could deduce the complete event path with the bottommost node contained in an event path, given the readily-built event category tree, we thus could just leverage the bottommost node to denote the whole event path, and the event hierarchies could just be represented with the finest-granularity event contained in the event hierarchies. For example, a document is associated with two event paths as Fig.2 shows, and they could just be denoted by node m and node n respectively. Denoting the event hierarchies with the finestgranularity events could reduce the workload of annotating training dataset, where a document instance is annotated with the finest-granularity event categories. Figure 2: Two event paths associated with a document. ![Image](https://ijcai.org/proceedings/2020/fulltext/619/image_000001_c5319c9be4aac4fc9e30a6d5754aa8263a3c25e763da7afaa5ed9c3745290a1e.png) ## 3 Methodology The architecture of our model for hierarchical multi-label text classification is illustrated in Fig.3. The overall neural networks are presented in the right part, which mainly consists of an encoding network and a label prediction network. The encoding network contains 12 encoding layers, and each layer is composed of multiple basic units, each of which is a stack of six identical encoder modules as shown in the left part. The label prediction network is a shallow feedforward neural network.We mainly illustrate the encoding network and model optimization objective in rest of this section. ## 3.1 Encoding Network For each document, the encoding network extracts the highlevel features with bidirectional Transformer[Devlin et al., 2018] encoders, from the document's various input embeddings. For a document x = {c1, c2, · · · , cm}, where ci denotes a character included, we represent the document into an embedding matrix M ∈ R|V|×d , where V denotes the fixed-sized character vocabulary and d denotes the dimension size of the character embeddings. The overall embedding, denoted as Eci , for a character is the concatenation of the character's token and position embeddings. The pre-trained Word2Vec [Mikolov et al., 2013] Chinese character embeddings are leveraged as the token embeddings, and a learned positional embedding matrix Mp ∈ Rs×d is leveraged as position embeddings, where s denotes the length of character sequence supported in the model. The embedding matrix of a document is fed to each of the Transformer encoders of the first encoding layer for feature extraction, and then the output vectors of these Transformer encoders are fed to each of the Transformer encoders of the second encoding layer. Each encoding layer is fullyconnected to the next encoding layer in the encoding network, and the vectors of encoded hidden states output by the last encoding layer are fed to the prediction network for classification label prediction. Figure 3: The architecture of F-HMTC. ![Image](https://ijcai.org/proceedings/2020/fulltext/619/image_000002_81ed7797f2d82f64a63a43109e5a816fb3e9b844a0b1ea24b660bcf318f95f18.png) A basic encoder module contained in Transformer encoders implements the multi-head attention defined as: where headi = Attention(QW Q i ,KW K i , VW V i ) (i ∈ [1, h]), h denotes the number of parallel attention layers, Con(headi) denotes the concatenation of h heads, and parameter matrices W O ∈ Rhd×dmodel , W Q i ,W K i ,W V i ∈ Rdmodel×d . Because all of the queries, keys and values come from the same input, Eci , their dimension sizes are all set to d. Besides the attention sub-layers, our encoder module includes a position-wise feed-forward network, consisting of two linear transformations with ReLU activations. ## 3.2 Optimization Objective We define two vectors for a document instance di, namely a target vector yi and a prediction vector ˆ yi. The target vector is a multi-hot 0-1 vector of N dimensions, with bits corresponding to the document's finest-granularity event categories set to 1, and other bits set to 0, where N is the overall number of hierarchical event categories. The prediction vector ˆ yi is an n-dimensional vector of real values ranging from 0 to 1, and a component of ˆ yi denotes the probability that corresponding event category is assumed to be the document's finest-granularity event category, and the event categories corresponding to components that exceed some predefined threshold are predicted as the document's event categories. For document di, we then define dist(yi,t, ˆ yi) to denote the distance between the t-th component of document's target vector and its prediction vector, just as: where αtj is the penalty coefficient associated with the distance between the target hierarchy and predicted hierarchy. Supposing target event category corresponding to yi,t falls in the l-th hierarchy of the event category tree and the predicted event category corresponding to ˆ yi,j falls in the k-th layer, the penalty coefficient αtj is defined as: where Label(ˆ yi,j) denotes the label corresponding to ˆ yi,j , Anc(c) and Des(c) respectively denote the ancestor labels and descendant labels of label c. We define the hierarchical multi-label distance (HMD) as the prediction loss for di as: As we often face the label imbalance problem in the hierarchical classification task, where finer-granularity event categories are annotated with fewer label instances, we introduce the recursive regularization[Gopal and Yang, 2013; Peng et al., 2018] to improve the model performance with such fewer finer-granularity event labels. As parent and child event categories possess hierarchical relationships, considering hierarchical dependencies between labels could encourage the parameters of child event categories to approach that of the parent event category, which could be learned with more label instances. Formally, we denote total parameter matrix of the final fully-connected prediction network as W= {wli : li ∈ L}, where wli denotes the parameter vector of event category li, lj i refers to a child event category of li and L denotes the whole label set. Then the recursive regularization is defined as: The model's overall optimization objective is defined as: where d1,..., dZ denote all the training documents, and C is the weight decay hyper-parameter. The MSE (i.e. Mean Square Error) optimization objective function is a special case of the objective function we propose, when the penalty coefficient matrix is the identity matrix and C = 0. ## 4 Experimental Evaluations ## 4.1 Experimental Settings We invited senior professionals to sort out a hierarchical event scheme, which contains 98 event categories spreading across the event category nodes of the constructed event category tree of depth 7, based on our application scenario illustrated in Sec.2, and had our personnel with industrial research expertise annotate a Chinese financial dataset (CN-Fin) to evaluate our model's performance. We totally collected more than 500, 000 news documents from some major Chinese financial websites, and have financial labors annotate these documents with the defined event hierarchies, and finally chose 35, 000 news documents with high-quality event labels as the CN-Fin dataset. The dataset statistics are presented in Table.2. In our model implementation, we set the hidden size to 768, self-attention head size to 12 and dropout rate to 0.1, learning rate to 1e-4, β1 = 0.9, β2 = 0.999, L2 weight decay to 0.001. Via data driven analysis and cross validation, we set αa to 0.0002, αd to 0.0003, αo to 0.01 and αp to 1.1, with αo > α6d > α6a . For recursive regularization, we set regularization weight decay C to 0.00005, as suggested in [Peng et al., 2018]. We compare the proposed method with the following stateof-the-art text classification methods: - TextCNN [Kim, 2014]: which leverages a convolutional neural network for sentence-level classification. - FastText[Joulin et al., 2017]: which applies n-gram features to capture some partial information about the local word order for text classification. - HAN[Yang et al., 2016]: which leverages the Hierarchical Attention Network to capture basic insights about document structure to classify documents. - Transformer[Vaswani et al., 2017]: which leverages global dependencies with attention mechanisms for text classification. - Star-Transformer[Guo et al., 2019], which is a lightweight alternative of Transformer with a starshaped topology to reduce classification complexity. - HSVM[Cai and Hofmann, 2004], which is a classic hierarchical classification method. The baseline models, except HSVM, are implemented with the Tencent open-source toolkit 2 for neural hierarchical multi-label text classification, and we implement HSVM according to the method proposed in [Cai and Hofmann, 2004]. For documents contained in the CN-Fin dataset, we perform word segmentation with Jieba 3 , and then train 300dimensional character embeddings on the dataset according to the skip-gram Word2Vec [Mikolov et al., 2013] model, with the window size set to five, and the trained character embeddings are leveraged to train all models as initial input. ## 4.2 EvaluationMetrics We benchmark the model's performance with three kinds of metrics, namely the accuracy metric, the micro-averaged hPRF metrics (hPrecision, hRecall and hF-score) and the HMDScore we define based on our actual application scenario. The PRF (precision, recall and F-score) metrics are commonly used for evaluating classification performance. We do not leverage the PRF metrics, as they are not suitable for hierarchical text classification tasks, where wrong classification predictions could not be clearly discriminated with these metrics. hPRF metrics are proposed by [Kiritchenko et al., 2005] for evaluating hierarchical text classification models, where hierarchical classification predictions are discriminatively considered. Assuming document di with target label set Ci is predicted with the lable set C ′ i , we extend Ci and C ′ i with the corresponding ancestor labels, and get: ˆ Ci = {⋃ck∈Ci Anc(ck)}, ˆ C ′ i = {⋃ck∈C′ i Anc(ck)}, where ck is an event category label. Then the micro-averaged hP and hR are defined as: 2 https://github.com/Tencent/NeuralNLP-NeuralClassifier 3 https://github.com/fxsjy/jieba Table 1: Hierarchical distributions of prediction results and HMDSCores achieved by various models. Models with 'HR' denote that recursive regularization is used for these models. "Exact Match" denotes the ratio of prediction labels exactly matching the target labels, and "Parent", "Grandparent", "Child", and "Grandchild" respectively denote the ratio of prediction labels being parent, grandparent, child and grandchild labels of the target labels, while "Others" denotes the ratio of other prediction labels. | Models | Exact Match | Parent | Grandparent | Child | Grandchild | Others | HMDScore | |---------------------|---------------|----------|---------------|---------|--------------|----------|------------| | TextCNN | 45.97% | 8.17 % | 2.41 % | 7.58 % | 1.86 % | 34.01% | 0.5945 | | FastText | 50.01% | 8.59 % | 1.06 % | 7.00 % | 1.67 % | 31.67% | 0.6113 | | HAN | 51.18% | 10.21% | 1.83 % | 7.99% | 1.84 % | 26.95% | 0.6276 | | Transformer | 50.48% | 10.01% | 2.62 % | 7.26% | 1.83 % | 27.80% | 0.6158 | | Star-Transformer | 44.09% | 8.24 % | 1.90 % | 6.98% | 1.76 % | 37.03% | 0.5902 | | TextCNN-HR | 46.69% | 7.68 % | 1.76 % | 7.05% | 1.65 % | 35.17% | 0.6250 | | FastText-HR | 51.35% | 8.34 % | 0.92 % | 6.40% | 1.48 % | 31.52% | 0.6367 | | HAN-HR | 53.62% | 9.64 % | 1.85 % | 6.97% | 1.56 % | 26.37% | 0.6440 | | Transformer-HR | 51.89% | 9.37 % | 2.31 % | 6.77% | 1.58 % | 28.06% | 0.6395 | | Star-Transformer-HR | 45.28% | 8.38 % | 1.77 % | 5.51% | 1.39 % | 37.67% | 0.6174 | | HSVM | 37.13% | 7.00 % | 0.76 % | 5.06% | 1.43 % | 48.61% | 0.4804 | | F-HMTC | 57.42%∗ | 7.39 % | 1.16 % | 6.25% | 1.39% | 26.38% | 0.7102∗ | Table 2: Dataset statistics. #s denotes the number of sentences, and #w denotes the number of words. | CN-Fin | Train | Test | |-------------|---------|---------| | #Vocabulary | 268,200 | 268,200 | | #Documents | 31,306 | 4,104 | | Average #s | 16.4 | 17.8 | | Max #s | 706 | 593 | | Average #w | 168.9 | 415.6 | | Max #w | 60,614 | 57,534 | hF-score is defined as: In our application scenario, we deem a document's target labels and the corresponding parent labels as acceptable prediction results, and tend to exactly measure the hierarchical differences between the target labels and prediction labels. We define the hierarchical multi-label distance score (HMDScore) for performance evaluation, based on the hierarchical multi-label distance (HMD) aforementioned, just as: Table 3: hPRF performance achieved by various models. | Models | hPresion | hRecall | hF-score | |---------------------|------------|-----------|------------| | TextCNN | 0.5529 | 0.5642 | 0.5585 | | FastText | 0.5278 | 0.6397 | 0.5784 | | HAN | 0.5518 | 0.6593 | 0.6008 | | Transformer | 0.6139 | 0.5780 | 0.5954 | | Star-Transformer | 0.5462 | 0.5568 | 0.5515 | | TextCNN-HR | 0.5630 | 0.5754 | 0.5691 | | FastText-HR | 0.5607 | 0.6048 | 0.5819 | | HAN-HR | 0.5728 | 0.6495 | 0.6087 | | Transformer-HR | 0.5602 | 0.6391 | 0.5971 | | Star-Transformer-HR | 0.5301 | 0.5385 | 0.5343 | | HSVM | 0.3328 | 0.8250 | 0.4743 | | F-HMTC | 0.8009 | 0.6023 | 0.6876∗ | ## 4.3 Experimental Results and Analysis We benchmark the performance of our model, and compare it with the baseline models with and without recursive regularization. The experimental results of evaluating prediction accuracy and HMDScore are presented in Table.1, and experimental results of evaluating hPRF performance are presented in Table.3. From Table.1 and Table.3, we can see that our model outperforms all baseline models with or without recursive regularization. Among all the baseline models, HAN achieves the best performance with all the three kinds of evaluation metrics, while the performance of our model, namely F-HMTC, leads that of HAN by substantial margins, especially on HMDScore and hF-score. As far as prediction accuracy is concerned, F-HMTC yields the most "Exact Match" prediction results than all baseline models, and leads TextCNN by over 10% of "Exact Match" prediction labels, and leads HAN without recursive regularization by over 6.2% of "Exact Match" prediction labels. As target labels' parent labels are deemed as acceptable prediction results, the overall accuracy of acceptable predictions achieved by F-HMTC only leads HAN without recursive regularization by 3.4%, and only leads HAN with recursive regularization by about 1.5%. However, F-HMTC could provide more accurate finer-granularity event detection results for our application scenario with its leading accuracy of "Exact Match" predictions. The reason why F-HMTC's performance leads all baseline models is straight-forward, as leveraging hierarchical dependency information helps to make better label predictions. One thing worth noting is that introducing recursive regularization could improve the performance of baseline models, this also validates our assumption that hierarchical dependency information could improve the performance of hierarchical text classification models. Figure 4: F-HMTC's performance on hF-score and HMDScore affected by the magnitude of hyper-parameters. ![Image](https://ijcai.org/proceedings/2020/fulltext/619/image_000003_7e4d2f10b50b9fec74c9b7111dc85b78eab94c80af2fc9772aea7686be79ca7e.png) We finally explore how F-HMTC's performance is influenced by the α series of hyper-parameters, associated with hierarchical dependencies, via evaluating its hF-scores and HMDScores when αa is set with various orders of magnitude, where the orders of magnitude of other α parameters are set accordingly, and the results are presented in Figure.4. We can see that the α series of parameters play an important role in controlling the intensity of the penalty in the model. As the order of magnitude becomes larger, the model's performance will get better until the order of magnitude exceeds 10-4 , when the model's performance begins to decline. This verifies our assumption that hierarchical dependency information is indeed supplementary to the semantic information hidden within the document texts, while having hierarchical dependency information dominate the overall semantic information will degrade the model's performance, when αa is set above some critical value. ## 5 RelatedWork Text classification is one of the fundamental tasks of natural language processing. Traditional methods are mostly based on feature engineering and feature selection to obtain good features for text classification [Aggarwal and Zhai, 2012]. Support Vector Machines and Naive Bayes have already been used on text classification tasks for a long time [Joachims, 1998; McCallum et al., 1998]. Dimensionality reduction methods like Latent Dirichlet Allocation [Blei et al., 2003] has been used to reduce the feature space which can be better than bag-of-word with few features. As leveraging label dependencies can greatly improve the performance of multi-label hierarchical classification[Sun and Lim, 2001; Cai and Hofmann, 2004], [Wu et al., 2016] proposes MLForest to learn an ensemble of hierarchical multi-label classifiers to reveal the intrinsic label dependencies. Deep learning has been adopted for text classification, and pre-trained language models, such as Word2Vec [Mikolov et al., 2013], FastText [Joulin et al., 2017] and BERT [Devlin et al., 2018], are leveraged in text classification models to improve documents' representations. While [Rousseau et al., 2015; Peng et al., 2018] use co-occurrence matrix to convert texts to graphs and apply graph mining algorithms to get frequent sub-graphs to define new features for text classification. CNN has also been applied for text classification using kernels to capture the documents' latent semantic information [Kim, 2014], and [Conneau et al., 2016] designs VDCNN to operate directly at the character level with smaller convolutional kernels. [Liu et al., 2016] proposes an RNN-based model for long document classification, which jointly learns across multiple related tasks via multi-task learning. In addition, [Wang, 2018] proposes a disconnected recurrent neural network to restrict the hidden states of each step to cover the words near the current position, by limiting the distance of information flow in RNN. Models based on attention mechanisms, such as hierarchical attentions [Yang et al., 2016] and self-attentions [Vaswani et al., 2017], have been introduced for text classification recently. To reduce the dependencies of Transformers on largescale training datasets, due to its heavy structure with fullyconnected attention connections, [Guo et al., 2019] proposes the Star-Transformer by replacing the fully-connected structure with a star-shaped topology. All of these methods could be further improved for HMTC tasks with massive numbers of hierarchical categories, because they often assume that categories are independent and thus dismiss the hierarchical dependencies existing between various categories. ## 6 Conclusions In this paper, we model the financial event detection as a hierarchical multi-label text classification problem, and propose the F-HMTC event detection model for our industrial application scenario. F-HMTC leverages bidirectional encoders from Transformers for document representations, and implements a carefully-designed hierarchy-based loss layer for model optimization. The extensive experiments demonstrate that F-HMTC can achieve better classification performance, compared with baseline text classification models. We can conclude that the distance function we introduce with penalty coefficients can provide more insights for the hierarchical multi-label text classification problem. ## Acknowledgments This work was supported by the National Key R&D Program of China (2018YFC0831904) and the Joint Research Program of SeekData Inc. 4 and ECNU. 4 www.seek-data.com ## References 1. [Aggarwal and Zhai, 2012] Charu C Aggarwal and ChengX- iang Zhai. A survey of text classification algorithms. In Mining text data, pages 163-222. Springer, 2012. 2. [Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003. 3. [Cai and Hofmann, 2004] Lijuan Cai and Thomas Hofmann. Hierarchical document categorization with support vector machines. In Proceedings of the thirteenth ACM interna- tional conference on Information and knowledge manage- ment, pages 78-87. ACM, 2004. 4. [Conneau et al., 2016] Alexis Conneau, Holger Schwenk, Loïc Barrault, and Yann Lecun. Very deep convolu- tional networks for text classification. arXiv preprint arXiv:1606.01781, 2016. 5. [Devlin et al., 2018] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805, 2018. 6. [Gopal and Yang, 2013] Siddharth Gopal and Yiming Yang. Recursive regularization for large-scale classification with hierarchical and graphical dependencies. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 257-265. ACM, 2013. 7. [Guo et al., 2019] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. Star- transformer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pages 1315-1325, 2019. 8. [Joachims, 1998] Thorsten Joachims. Text categorization with support vector machines: Learning with many rele- vant features. In European conference on machine learn- ing, pages 137-142. Springer, 1998. 9. [Joulin et al., 2017] Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. Bag of tricks for effi- cient text classification. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427-431, 2017. 10. [Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882, 2014. 11. [Kiritchenko et al., 2005] Svetlana Kiritchenko, Stan Matwin, and A Fazel Famili. Functional annotation of genes using hierarchical text categorization. In Proceed- ings of the BioLINK SIG: Linking Literature, Information and Knowledge for Biology (held at ISMB-05. 12. [Liu et al., 2016] Pengfei Liu, Xipeng Qiu, and Xuan- jing Huang. Recurrent neural network for text clas- sification with multi-task learning. arXiv preprint arXiv:1605.05101, 2016. 13. [McCallum et al., 1998] Andrew McCallum, Kamal Nigam, et al. A comparison of event models for naive bayes text classification. In AAAI-98 workshop on learning for text categorization, volume 752, pages 41-48. Citeseer, 1998. 14. [Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. 15. [Peng et al., 2018] Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. Large-scale hierarchical text classification with recursively regularized deep graph-cnn. In Proceed- ings of the 2018 World Wide Web Conference, pages 1063- 1072. International World Wide Web Conferences Steer- ing Committee, 2018. 16. [Rousseau et al., 2015] Franc ¸ois Rousseau, Emmanouil Ki- agias, and Michalis Vazirgiannis. Text categorization as a graph classification problem. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1702-1712, 2015. 17. [Sun and Lim, 2001] Aixin Sun and Ee-Peng Lim. Hierar- chical text classification and evaluation. In Proceedings of IEEE International Conference on Data Mining, pages 521-528. IEEE, 2001. 18. [Triguero and Vens, 2016] Isaac Triguero and Celine Vens. Labelling strategies for hierarchical multi-label classifica- tion techniques. Pattern Recognition, 56:170-183, 2016. 19. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 5998-6008, 2017. 20. [Wang, 2018] Baoxin Wang. Disconnected recurrent neural networks for text categorization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2311-2320, 2018. 21. [Wu et al., 2016] Qingyao Wu, Mingkui Tan, Hengjie Song, Jian Chen, and Michael K Ng. Ml-forest: a multi- label tree ensemble method for multi-label classification. IEEE transactions on knowledge and data engineering, 28(10):2665-2680, 2016. 22. [Wu et al., 2019] Cinna Wu, Mark Tygert, and Yann LeCun. A hierarchical loss and its problems when classifying non- hierarchically. PLoS ONE, 14(12):e0226222-e0226222, 2019. 23. [Yang et al., 2016] Zichao Yang, Diyi Yang, Chris Dyer, Xi- aodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Pro- ceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, 2016.