{"type":"ScholarlyArticle","title":"Domain-Hierarchy Adaptation via Chain of Iterative Reasoning for Few-shot Hierarchical Text Classification","authors":[{"name":"Ke Ji","slug":"ke-ji","given":"Ke","family":"Ji","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Peng Wang","slug":"peng-wang","given":"Peng","family":"Wang","affiliation":"School of Computer Science and Engineering , Southeast University; Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) , Ministry of Education"},{"name":"Wenjun Ke","slug":"wenjun-ke","given":"Wenjun","family":"Ke","affiliation":"School of Computer Science and Engineering , Southeast University; Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) , Ministry of Education"},{"name":"Guozheng Li","slug":"guozheng-li","given":"Guozheng","family":"Li","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Jiajun Liu","slug":"jiajun-liu","given":"Jiajun","family":"Liu","affiliation":"School of Computer Science and Engineering , Southeast University"},{"name":"Jingsheng Gao","slug":"jingsheng-gao","given":"Jingsheng","family":"Gao","affiliation":"School of Electronic Information and Electrical Engineering , Shanghai Jiao Tong University"},{"name":"Ziyu Shang","slug":"ziyu-shang","given":"Ziyu","family":"Shang","affiliation":"School of Computer Science and Engineering , Southeast University"}],"abstract":"Recently, various pre-trained language models (PLMs) have been proposed to prove their impressive performances on a wide range of few-shot tasks. However, limited by the unstructured prior knowledge in PLMs, it is difficult to maintain consistent performance on complex hierarchically dependent tasks, especially when the downstream data is extremely scarce. The main challenge is how to transfer the unstructured semantic space in PLMs to the downstream domain hierarchy. Unlike previous work on hierarchical text classification (HTC) which directly performs multi-label classification or uses graph neural network (GNN) to inject label hierarchy, in this work, we study the HTC problem under a few-shot setting to adapt knowledge in PLMs from an unstructured manner to the downstream hierarchy. Technically, we design a simple yet effective method named Hierarchical Iterative Conditional Random Field (HierICRF) to search the most domain-challenging directions and exquisitely crafts domain-hierarchy adaptation as a hierarchical iterative language modeling problem, and then it encourages the model to make hierarchical consistency self-correction during the inference, thereby achieving knowledge transfer with hierarchical consistency preservation. We perform HierICRF on various architectures, and extensive experiments on two popular HTC datasets demonstrate that prompt with HierICRF significantly boosts the few-shot HTC performance with an average Micro-F1 by 28.80% to 1.50% and Macro-F1 by 36.29% to 1.5% over the previous state-of-the-art (SOTA) baselines under few-shot settings (1->16), while remaining SOTA hierarchical consistency performance.","doi":"10.24963/ijcai.2024/698","url":"https://ijcai.org/proceedings/2024/698","pdf":"https://ijcai.org/proceedings/2024/0698.pdf","markdown":"https://ijcai.org/proceedings/2024/698.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/698","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Natural Language Processing","track":"Main Track","number":698,"pages":{"first":6315,"last":6323},"keywords":["Natural Language Processing: NLP: Applications","Natural Language Processing: NLP: Text classification"],"videos":[],"references":[{"position":1,"raw":"[Brown et al., 2020] Tom Brown, Benjamin Mann, and Nick et al. Ryder. Language models are few-shot learners. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"Tom Brown, Benjamin Mann, Nick","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cevahir and Murakami, 2016] Ali Cevahir and Koji Mu- rakami. Large-scale multi-class and hierarchical product categorization for an e-commerce giant. In COLING, 2016.","title":"Ali Cevahir and Koji Murakami. Large-scale multi-class and hierarchical product categorization for an e-commerce giant","authors":"Murakami Cevahir","year":"2016","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Haibin Chen, Qianli Ma, Zhenxi Lin, et al. Hierarchy-aware label semantics matching network for hi- erarchical text classification. In ACL, 2021.","title":"Hierarchy-aware label semantics matching network for hierarchical text classification","authors":"Haibin Chen, Qianli Ma, Zhenxi Lin","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chung et al., 2022] Hyung Won Chung, Le Hou, Shayne Longpre, et al. Scaling instruction-finetuned language mod- els. arXiv preprint arXiv:2210.11416, 2022.","title":"Scaling instruction-finetuned language models","authors":"Chung Hyung Won, Le Hou, Shayne Longpre","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, et al. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Forney, 1973] G David Forney. The viterbi algorithm. Pro- ceedings of the IEEE, 61(3):268-278, 1973.","title":"The viterbi algorithm","authors":"David Forney","year":"1973","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao et al., 2021] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learn- ers. In ACL, 2021.","title":"Making pre-trained language models better few-shot learners","authors":"Tianyu Gao, Adam Fisch, Danqi Chen","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Irsan and Khodra, 2019] Ivana Clairine Irsan and Masayu Leylia Khodra. Hierarchical multi-label news article classification with distributed semantic model based features. International Journal of Advances in Intelligent Informatics, 5(1):40-47, 2019.","title":"Ivana Clairine Irsan and Masayu Leylia Khodra. Hierarchical multi-label news article classification with distributed semantic model based features","authors":"Khodra Irsan","year":"2019","venue":"International Journal of Advances in Intelligent Informatics","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ji et al., 2023] Ke Ji, Yixin Lian, Jingsheng Gao, et al. Hier- archical verbalizer for few-shot hierarchical text classifica- tion. In ACL, 2023.","title":"Hierarchical verbalizer for few-shot hierarchical text classification","authors":"Ke Ji, Yixin Lian, Jingsheng Gao","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kowsari et al., 2017] Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, et al. Hdltex: Hierarchical deep learning for text classification. In ICMLA, 2017.","title":"Hdltex: Hierarchical deep learning for text classification","authors":"Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa","year":"2017","venue":"ICMLA","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kowsari et al., 2019] Kamran Kowsari, Kiana Ja- fari Meimandi, Mojtaba Heidarysafa, et al. Text classification algorithms: A survey. Information, 10(4):150, 2019.","title":"Text classification algorithms: A survey","authors":"Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa","year":"2019","venue":"Information","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2022] Guozheng Li, Xu Chen, Peng Wang, Jiafeng Xie, and Qiqing Luo. Fastre: Towards fast relation ex- traction with convolutional encoder and improved cascade binary tagging framework. In IJCAI, 2022.","title":"Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework","authors":"Guozheng Li, Xu Chen, Peng Wang, Jiafeng Xie, Qiqing Luo","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/583"},{"position":13,"raw":"[Li et al., 2023] Guozheng Li, Peng Wang, and Wenjun Ke. Revisiting large language models as zero-shot relation ex- tractors. In Findings of EMNLP, 2023.","title":"Revisiting large language models as zero-shot relation extractors","authors":"Guozheng Li, Peng Wang, Wenjun Ke","year":"2023","venue":"Findings of EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2024] Guozheng Li, Wenjun Ke, Peng Wang, Zi- jie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, and Qiqing Luo. Unlocking instructive in-context learning with tabular prompting for relational triple extraction. arXiv preprint arXiv:2402.13741, 2024.","title":"Unlocking instructive in-context learning with tabular prompting for relational triple extraction","authors":"Guozheng Li, Wenjun Ke, Peng Wang, Zijie Xu, Ke Ji, Jiajun Liu, Ziyu Shang, Qiqing Luo","year":"2024","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Mao et al., 2019] Yuning Mao, Jingjing Tian, Jiawei Han, et al. Hierarchical text classification with reinforced label assignment. In EMNLP, 2019.","title":"Hierarchical text classification with reinforced label assignment","authors":"Mao","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Nadeau and Sekine, 2007] David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3-26, 2007.","title":"A survey of named entity recognition and classification","authors":"Sekine Nadeau, David Nadeau, Satoshi Sekine","year":"2007","venue":"Lingvisticae Investigationes","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Petroni et al., 2019] Fabio Petroni, Tim Rocktäschel, Sebas- tian Riedel, et al. Language models as knowledge bases? In EMNLP, 2019.","title":"Language models as knowledge bases?","authors":"Fabio Petroni, Tim Rocktäschel, Sebastian Riedel","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Radford et al., 2018] Alec Radford, Karthik Narasimhan, Tim Salimans, et al. Improving language understanding by generative pre-training. 2018.","title":"Improving language understanding by generative pre-training","authors":"Alec Radford, Karthik Narasimhan, Tim Salimans","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Raffel et al., 2020] Colin Raffel, Noam Shazeer, Adam Roberts, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.","title":"Exploring the limits of transfer learning with a unified text-to-text transformer","authors":"Colin Raffel, Noam Shazeer, Adam Roberts","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Scao et al., 2022] Teven Le Scao, Angela Fan, Christopher Akiki, et al. Bloom: A 176b-parameter open-access multi- lingual language model. arXiv preprint arXiv:2211.05100, 2022.","title":"Teven Le Scao","authors":"Scao","year":"2022","venue":"Bloom: A 176b-parameter open-access multilingual language model","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schick and Schütze, 2021] Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL, 2021.","title":"Exploiting cloze-questions for few-shot text classification and natural language inference","authors":"Schütze Schick, Timo Schick, Hinrich Schütze","year":"2021","venue":"EACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sinha et al., 2018] Koustuv Sinha, Yue Dong, Jackie Chi Kit Cheung, et al. A hierarchical neural attention-based text classifier. In EMNLP, 2018.","title":"A hierarchical neural attention-based text classifier","authors":"Sinha","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sutton et al., 2012] Charles Sutton, Andrew McCallum, et al. An introduction to conditional random fields. Foundations and Trends® in Machine Learning, 4(4):267-373, 2012.","title":"An introduction to conditional random fields","authors":"Charles Sutton, Andrew Mccallum","year":"2012","venue":"Foundations and Trends® in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2021] Xuepeng Wang, Li Zhao, Bing Liu, et al. Concept-based label embedding via dynamic routing for hierarchical text classification. In ACL, 2021.","title":"Concept-based label embedding via dynamic routing for hierarchical text classification","authors":"Wang","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2022a] Zihan Wang, Peiyi Wang, Lianzhe Huang, et al. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classi- fication. In ACL, 2022.","title":"Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification","authors":"Wang","year":"2022","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2022b] Zihan Wang, Peiyi Wang, Tianyu Liu, et al. Hpt: Hierarchy-aware prompt tuning for hierarchical text classification. In EMNLP, 2022.","title":"Hpt: Hierarchy-aware prompt tuning for hierarchical text classification","authors":"Wang","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2023] Peng Wang, Tong Shao, Ke Ji, Guozheng Li, and Wenjun Ke. fmlre: A low-resource relation extrac- tion model based on feature mapping similarity calculation. In AAAI, 2023.","title":"fmlre: A low-resource relation extraction model based on feature mapping similarity calculation","authors":"Peng Wang, Tong Shao, Ke Ji, Guozheng Li, Wenjun Ke","year":"2023","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Wei et al., 2022] Jason Wei, Xuezhi Wang, Dale Schuur- mans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reason- ing in large language models. NeurIPS, 35:24824-24837, 2022.","title":"Chain-of-thought prompting elicits reasoning in large language models","authors":"Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, V Quoc, Denny Le, Zhou","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Wu et al., 2019] Jiawei Wu, Wenhan Xiong, and William Yang Wang. Learning to learn and predict: A meta-learning approach for multi-label classification. In EMNLP, 2019.","title":"Learning to learn and predict: A meta-learning approach for multi-label classification","authors":"Jiawei Wu, Wenhan Xiong, William Yang, Wang","year":"2019","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Xu and Barbosa, 2018] Peng Xu and Denilson Barbosa. Neural fine-grained entity type classification with hierarchy- aware loss. In NAACL, 2018.","title":"Neural fine-grained entity type classification with hierarchyaware loss","authors":"Peng Xu, Denilson Barbosa","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yu et al., 2022] Chao Yu, Yi Shen, and Yue Mao. Con- strained sequence-to-tree generation for hierarchical text classification. In SIGIR, 2022.","title":"Constrained sequence-to-tree generation for hierarchical text classification","authors":"Chao Yu, Yi Shen, Yue Mao","year":"2022","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang et al., 2022] Xinyi Zhang, Jiahao Xu, Charlie Soh, and Lihui Chen. La-hcn: label-based attention for hierar- chical multi-label text classification neural network. Expert Systems with Applications, 187:115922, 2022.","title":"La-hcn: label-based attention for hierarchical multi-label text classification neural network","authors":"Xinyi Zhang, Jiahao Xu, Charlie Soh, Lihui Chen","year":"2022","venue":"Expert Systems with Applications","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhou et al., 2020] Jie Zhou, Chunping Ma, Dingkun Long, et al. Hierarchy-aware global model for hierarchical text classification. In ACL, 2020.","title":"Hierarchy-aware global model for hierarchical text classification","authors":"Jie Zhou, Chunping Ma, Dingkun Long","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhu et al., 2023] Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxi- ang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. Solving math word problems via cooperative reasoning induced language models. In ACL, 2023.","title":"Solving math word problems via cooperative reasoning induced language models","authors":"Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, Yujiu Yang","year":"2023","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}