{"type":"ScholarlyArticle","title":"Weakly-supervised Text Classification with Wasserstein Barycenters Regularization","authors":[{"name":"Jihong Ouyang","slug":"jihong-ouyang","given":"Jihong","family":"Ouyang","affiliation":"College of Computer Science and Technology , Jilin University , China; Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education , Jilin University , China"},{"name":"Yiming Wang","slug":"yiming-wang","given":"Yiming","family":"Wang","affiliation":"College of Computer Science and Technology , Jilin University , China; Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education , Jilin University , China"},{"name":"Ximing Li","slug":"ximing-li","given":"Ximing","family":"Li","affiliation":"College of Computer Science and Technology , Jilin University , China; Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education , Jilin University , China"},{"name":"Changchun Li","slug":"changchun-li","given":"Changchun","family":"Li","affiliation":"College of Computer Science and Technology , Jilin University , China; Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education , Jilin University , China"}],"abstract":"Weakly-supervised text classification aims to train predictive models with unlabeled texts and a few representative words of classes, referred to as category words, rather than labeled texts. These weak supervisions are much more cheaper and easy to collect in real-world scenarios. To resolve this task, we propose a novel deep classification model, namely Weakly-supervised Text Classification with Wasserstein Barycenter Regularization (WTC-WBR). Specifically, we initialize the pseudo-labels of texts by using the category word occurrences, and formulate a weakly self-training framework to iteratively update the weakly-supervised targets by combining the pseudo-labels with the sharpened predictions. Most importantly, we suggest a Wasserstein barycenter regularization with the weakly-supervised targets on the deep feature space. The intuition is that the texts tend to be close to the corresponding Wasserstein barycenter indicated by weakly-supervised targets. Another benefit is that the regularization can capture the geometric information of deep feature space to boost the discriminative power of deep features. Experimental results demonstrate that WTC-WBR outperforms the existing weakly-supervised baselines, and achieves comparable performance to semi-supervised and supervised baselines.","doi":"10.24963/ijcai.2022/468","url":"https://ijcai.org/proceedings/2022/468","pdf":"https://ijcai.org/proceedings/2022/0468.pdf","markdown":"https://ijcai.org/proceedings/2022/468.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/468","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":468,"pages":{"first":3373,"last":3379},"keywords":["Natural Language Processing: Text Classification","Machine Learning: Weakly Supervised Learning","Machine Learning: Classification"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986509","length_sec":79,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984652","length_sec":295,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Agueh and Carlier, 2011] Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space. SIAM Jour- nal on Mathematical Analysisb, 43(2):904-924, 2011.","title":"Martial Agueh and Guillaume Carlier. Barycenters in the wasserstein space","authors":"Carlier Agueh","year":"2011","venue":"SIAM Journal on Mathematical Analysisb","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bogachev and Kolesnikov, 2012] Vladimir Igorevich Bo- gachev and Aleksandr Viktorovich Kolesnikov. The monge-kantorovich problem: achievements, connec- tions, and perspectives. Russian Mathematical Surveys, 67(5):785-890, 2012.","title":"The monge-kantorovich problem: achievements, connections, and perspectives","authors":"Vladimir Igorevich, Bogachev, Aleksandr Viktorovich, Kolesnikov","year":"2012","venue":"Russian Mathematical Surveys","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chang et al., 2008] Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. Importance of seman- tic representation: Dataless classification. In AAAI, pages 830-835, 2008.","title":"Importance of semantic representation: Dataless classification","authors":"Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, Vivek Srikumar","year":"2008","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2020] Jiaao Chen, Zichao Yang, and Diyi Yang. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In ACL, pages 2147-2157, 2020.","title":"Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification","authors":"Chen","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chi et al., 2019] Jinjin Chi, Jihong Ouyang, Ximing Li, Yang Wang, and Meng Wang. Approximate optimal trans- port for continuous densities with copulas. In IJCAI, pages 2165-2171, 2019.","title":"Approximate optimal transport for continuous densities with copulas","authors":"Jinjin Chi, Jihong Ouyang, Ximing Li, Yang Wang, Meng Wang","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2019/300"},{"position":6,"raw":"[Chizat et al., 2018] Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and Franc ¸ois-Xavier Vialard. Scaling algo- rithms for unbalanced optimal transport problems. Math- ematics of Computation, 87(314):2563-2609, 2018.","title":"Scaling algorithms for unbalanced optimal transport problems","authors":"Chizat","year":"2018","venue":"Mathematics of Computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. In NeurIPS, pages 2292-2300, 2013.","title":"Sinkhorn distances: Lightspeed computation of optimal transport","authors":"Marco Cuturi","year":"2013","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understand- ing. In NAACL, pages 4171-4186, 2019.","title":"BERT: pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Druck et al., 2008] Gregory Druck, Gideon Mann, and An- drew McCallum. Learning from labeled features using generalized expectation criteria. In SIGIR, pages 595-602, 2008.","title":"Learning from labeled features using generalized expectation criteria","authors":"Gregory Druck, Gideon Mann, Andrew Mccallum","year":"2008","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kusner et al., 2015] Matt Kusner, Yu Sun, Nicholas Kolkin, and Kilian Weinberger. From word embeddings to docu- ment distances. In ICML, pages 957-966, 2015.","title":"From word embeddings to document distances","authors":"Matt Kusner, Yu Sun, Nicholas Kolkin, Kilian Weinberger","year":"2015","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lan et al., 2020] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of lan- guage representations. In ICLR, 2020.","title":"Albert: A lite bert for self-supervised learning of language representations","authors":"Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut","year":"2020","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li and Yang, 2018] Ximing Li and Bo Yang. A pseudo label based dataless naive bayes algorithm for text classification with seed words. In COLING, pages 1908-1917, 2018.","title":"A pseudo label based dataless naive bayes algorithm for text classification with seed words","authors":"Ximing Li, Bo Yang","year":"2018","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Chenliang Li, Jian Xing, Aixin Sun, and Zongyang Ma. Effective document labeling with very few seed words: a topic modeling approach. In CIKM, pages 85-94, 2016.","title":"Effective document labeling with very few seed words: a topic modeling approach","authors":"Li","year":"2016","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2018] Ximing Li, Changchun Li, Jinjin Chi, Ji- hong Ouyang, and Chenliang Li. Dataless text classifica- tion: A topic modeling approach with document manifold. In CIKM, pages 973-982, 2018.","title":"Dataless text classification: A topic modeling approach with document manifold","authors":"Ximing Li, Changchun Li, Jinjin Chi, Jihong Ouyang, Chenliang Li","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2020] Changchun Li, Ximing Li, Jihong Ouyang, and Yiming Wang. Semantics-assisted wasserstein learn- ing for topic and word embeddings. In ICDM, pages 292- 301, 2020.","title":"Semantics-assisted wasserstein learning for topic and word embeddings","authors":"Changchun Li, Ximing Li, Jihong Ouyang, Yiming Wang","year":"2020","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2021] Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S. Yu, and Lifang He. A survey on text classification: from shallow to deep learning. arXiv:2008.00364, 2021.","title":"A survey on text classification: from shallow to deep learning","authors":"Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, Lifang He","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mekala and Shang, 2020] Dheeraj Mekala and Jingbo Shang. Contextualized weak supervision for text classification. In ACL, pages 323-333, 2020.","title":"Contextualized weak supervision for text classification","authors":"Dheeraj Mekala, Jingbo Shang","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Meng et al., 2018] Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. Weakly-supervised neural text classifi- cation. In CIKM, pages 983-992, 2018.","title":"Weakly-supervised neural text classification","authors":"Yu Meng, Jiaming Shen, Chao Zhang, Jiawei Han","year":"2018","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Meng et al., 2020] Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text classification using label names only: A language model self-training approach. In EMNLP, pages 9006- 9017, 2020.","title":"Text classification using label names only: A language model self-training approach","authors":"Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, Jiawei Han","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Schmitz et al., 2018] Morgan A Schmitz, Matthieu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, and Jean-Luc Starck. Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning. SIAM Journal on Imaging Sciences, 11(1):643-678, 2018.","title":"Wasserstein dictionary learning: Optimal transport-based unsupervised nonlinear dictionary learning","authors":"Morgan A Schmitz, Matthieu Heitz, Nicolas Bonneel, Fred Ngole, David Coeurjolly, Marco Cuturi, Gabriel Peyré, Jean-Luc Starck","year":"2018","venue":"SIAM Journal on Imaging Sciences","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2021] Zihan Wang, Dheeraj Mekala, and Jingbo Shang. X-class: Text classification with extremely weak supervision. In NAACL, pages 3043-3053, 2021.","title":"X-class: Text classification with extremely weak supervision","authors":"Zihan Wang, Dheeraj Mekala, Jingbo Shang","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xie et al., 2019] Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848, 2019.","title":"Unsupervised data augmentation for consistency training","authors":"Xie","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Yang et al., 2019] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, pages 5754-5764, 2019.","title":"Xlnet: Generalized autoregressive pretraining for language understanding","authors":"Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G Carbonell, Ruslan Salakhutdinov, V Quoc, Le","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2021] Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. Weakly-supervised text classification based on keyword graph. In EMNLP, page 2803-2813, 2021.","title":"Weakly-supervised text classification based on keyword graph","authors":"Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, Shuigeng Zhou","year":"2021","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhou, 2018] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-53, 2018.","title":"A brief introduction to weakly supervised learning","authors":"Zhi-Hua Zhou","year":"2018","venue":"National Science Review","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}