{"type":"ScholarlyArticle","title":"Unbiased Risk Estimator to Multi-Labeled Complementary Label Learning","authors":[{"name":"Yi Gao","slug":"yi-gao","given":"Yi","family":"Gao","affiliation":"School of Cyber Science and Engineering , Southeast University , China; Key Laboratory of Computer Network and Information Integration (Southeast University) , Ministry of Education , China"},{"name":"Miao Xu","slug":"miao-xu","given":"Miao","family":"Xu","affiliation":"University of Queensland , Australia"},{"name":"Min-Ling Zhang","slug":"min-ling-zhang","given":"Min-Ling","family":"Zhang","affiliation":"School of Computer Science and Engineering , Southeast University , China; Key Laboratory of Computer Network and Information Integration (Southeast University) , Ministry of Education , China"}],"abstract":"Multi-label learning (MLL) usually requires assigning multiple relevant labels to each instance. While a fully supervised MLL dataset needs a large amount of labeling effort, using complementary labels can help alleviate this burden. However, current approaches to learning from complementary labels are mainly designed for multi-class learning and assume that each instance has a single relevant label. This means that these approaches cannot be easily applied to MLL when only complementary labels are provided, where the number of relevant labels is unknown and can vary across instances. In this paper, we first propose the unbiased risk estimator for the multi-labeled complementary label learning (MLCLL) problem. We also provide an estimation error bound to ensure the convergence of the empirical risk estimator. In some cases, the unbiased estimator may give unbounded gradients for certain loss functions and result in overfitting. To mitigate this problem, we improve the risk estimator by minimizing a proper loss function, which has been shown to improve gradient updates. Our experimental results demonstrate the effectiveness of the proposed approach on various datasets.","doi":"10.24963/ijcai.2023/415","url":"https://ijcai.org/proceedings/2023/415","pdf":"https://ijcai.org/proceedings/2023/0415.pdf","markdown":"https://ijcai.org/proceedings/2023/415.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/415","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":415,"pages":{"first":3732,"last":3740},"keywords":["Machine Learning: ML: Weakly supervised learning","Machine Learning: ML: Classification","Machine Learning: ML: Multi-label"],"videos":[],"references":[{"position":1,"raw":"[Bartlett and Mendelson, 2002] Peter L. Bartlett and Sha- har Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. J. Mach. Learn. Res., 2111:224-240, 2002.","title":"Rademacher and gaussian complexities: Risk bounds and structural results","authors":"Peter L Bartlett, Shahar Mendelson","year":"2002","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boutell et al., 2004] Matthew R. Boutell, Jie-Bo Luo, Xi- Peng Shen, and Christopher M. Brown. Learning multi- label scene classification. Pattern Recognit., 37(9):1757- 1771, 2004.","title":"Learning multilabel scene classification","authors":"Matthew R Boutell, Jie-Bo Luo, Xi-Peng Shen, Christopher M Brown","year":"2004","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chou et al., 2020] Yu-Ting Chou, Gang Niu, Hsuan-Tien Lin, and Masashi Sugiyama. Unbiased risk estimators can mislead: A case study of learning with complementary la- bels. In Proceedings of the 37th International Conference on Machine Learning, pages 1929-1938, Virtual Event, 2020.","title":"Unbiased risk estimators can mislead: A case study of learning with complementary labels","authors":"Yu-Ting Chou, Gang Niu, Hsuan-Tien Lin, Masashi Sugiyama","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Elisseeff and Weston, 2001] André Elisseeff and Jason We- ston. A kernel method for multi-labelled classification. In Advances in Neural Information Processing Systems 14, pages 681-687, Vancouver, Canada, 2001.","title":"A kernel method for multi-labelled classification","authors":"Weston ; Elisseeff, André Elisseeff, Jason Weston","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2020] Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, and Masashi Sugiyama. Learning with mul- tiple complementary labels. In Proceedings of the 37th International Conference on Machine Learning, pages 3072-3081, Virtual Event, 2020.","title":"Learning with multiple complementary labels","authors":"Lei Feng, Takuo Kaneko, Bo Han, Gang Niu, Bo An, Masashi Sugiyama","year":"2020","venue":"Proceedings of the 37th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fürnkranz et al., 2008] Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, and Klaus Brinker. Multilabel classification via calibrated label ranking. Machine learning, 73(2):133-153, 2008.","title":"Multilabel classification via calibrated label ranking","authors":"Johannes Fürnkranz, Eyke Hüllermeier, Eneldo Loza Mencía, Klaus Brinker","year":"2008","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gao and Zhang, 2021] Yi Gao and Min-Ling Zhang. Dis- criminative complementary-label learning with weighted loss. In Proceedings of the 38th International Conference on Machine Learning, pages 3587-3597, Virtual Event, 2021.","title":"Discriminative complementary-label learning with weighted loss","authors":"Zhang ; ] Gao, Yi Gao, Min-Ling Zhang","year":"2021","venue":"Proceedings of the 38th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gerych et al., 2021] Walter Gerych, Thomas Hartvigsen, Luke Buquicchio, Emmanuel Agu, and Elke A. Runden- steiner. Recurrent bayesian classifier chains for exact multi-label classification. In Advances in Neural Informa- tion Processing Systems 34, pages 15981-15992, Virtual Event, 2021.","title":"Rundensteiner. Recurrent bayesian classifier chains for exact multi-label classification","authors":"Walter Gerych, Thomas Hartvigsen, Luke Buquicchio, Emmanuel Agu, Elke A","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Ghosh et al., 2017] Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 1919-1925, San Francisco, CA, 2017.","title":"Robust loss functions under label noise for deep neural networks","authors":"Aritra Ghosh, Himanshu Kumar, P S Sastry","year":"2017","venue":"Proceedings of the 31st AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ishida et al., 2017] Takashi Ishida, Gang Niu, Wei-Hua Hu, and Masashi Sugiyama. Learning from complementary la- bels. In Advances in Neural Information Processing Sys- tems 30, pages 5639-5649, Long Beach, CA, 2017.","title":"Learning from complementary labels","authors":"Takashi Ishida, Gang Niu, Wei-Hua Hu, Masashi Sugiyama","year":"2017","venue":"Advances in Neural Information Processing Systems 30","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Ishida et al., 2019] Takashi Ishida, Gang Niu, Aditya Kr- ishna Menon, and Masashi Sugiyama. Complementary- label learning for arbitrary losses and models. In Pro- ceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, pages 2971-2980, Long Beach, CA, 2019.","title":"Complementarylabel learning for arbitrary losses and models","authors":"Ishida","year":"2019","venue":"Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Yun-Cheng Li, Yale Song, and Jie-Bo Luo. Improving pairwise ranking for multi-label image classifi- cation. In Proceedings of 2017 IEEE conference on com- puter vision and pattern recognition, pages 3617-3625, Honolulu, HI, 2017.","title":"Improving pairwise ranking for multi-label image classification","authors":"Yun-Cheng Li, Yale Song, Jie-Bo Luo","year":"2017","venue":"Proceedings of 2017 IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2022] Ze-Yu Ma, Xiao Luo, Ying-Jie Chen, Mi- Xiao Hou, Jin-Xing Li, Ming-Hua Deng, and Guang- Ming Lu. Improved deep unsupervised hashing with fine- grained semantic similarity mining for multi-label image retrieval. In Proceedings of the 31st International Joint Conference on Artificial Intelligence, pages 1254-1260, Vienna, Austria, 2022.","title":"Improved deep unsupervised hashing with finegrained semantic similarity mining for multi-label image retrieval","authors":"Ze-Yu Ma, Xiao Luo, Ying-Jie Chen, Mi-Xiao Hou, Jin-Xing Li, Ming-Hua Deng, Guang-Ming Lu","year":"2022","venue":"Proceedings of the 31st International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maltoudoglou et al., 2022] Lysimachos Maltoudoglou, An- dreas Paisios, Ladislav Lenc, Jirí Martínek, Pavel Král, and Harris Papadopoulos. Well-calibrated confidence measures for multi-label text classification with a large number of labels. Pattern Recognit., 122:108271, 2022.","title":"Well-calibrated confidence measures for multi-label text classification with a large number of labels","authors":"Lysimachos Maltoudoglou, Andreas Paisios, Ladislav Lenc, Jirí Martínek, Pavel Král, Harris Papadopoulos","year":"2022","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Ze-Ming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun-Jie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neu- ral Information Processing Systems 32, pages 8024-8035, Vancouver, Canada, 2019.","title":"Pytorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Ze-Ming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Jun-Jie Bai, Soumith Chintala","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Read et al., 2011] Jesse Read, Bernhard Pfahringer, Geoff Holmes, and Eibe Frank. Classifier chains for multi-label classification. Mach. Learn., 85(3):333-359, 2011.","title":"Classifier chains for multi-label classification","authors":"Read","year":"2011","venue":"Mach. Learn","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sun et al., 2019] Li-Juan Sun, Song-He Feng, Tao Wang, Cong-Yan Lang, and Yi Jin. Partial multi-label learning by low-rank and sparse decomposition. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, pages 5016-5023, Honolulu, HI, 2019.","title":"Partial multi-label learning by low-rank and sparse decomposition","authors":"Li-Juan Sun, Song-He Feng, Tao Wang, Cong-Yan Lang, Yi Jin","year":"2019","venue":"Proceedings of the 33rd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2022] Li-Juan Sun, Song-He Feng, Jun Liu, Geng-Yu Lyu, and Cong-Yan Lang. Global-local label correlation for partial multi-label learning. IEEE Trans. Multim., 24:581-593, 2022.","title":"Global-local label correlation for partial multi-label learning","authors":"Li-Juan Sun, Song-He Feng, Jun Liu, Geng-Yu Lyu, Cong-Yan Lang","year":"2022","venue":"IEEE Trans. Multim","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tsoumakas et al., 2011] Grigorios Tsoumakas, Ioannis Katakis, and Ioannis P. Vlahavas. Random k-labelsets for multilabel classification. IEEE Trans. Knowl. Data Eng., 23(7):1079-1089, 2011.","title":"Random k-labelsets for multilabel classification","authors":"Grigorios Tsoumakas, Ioannis Katakis, Ioannis P Vlahavas","year":"2011","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Vapnik, 1991] Vladimir Vapnik. Principles of risk mini- mization for learning theory. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 831-838, 1991.","title":"Principles of risk minimization for learning theory","authors":"Vapnik","year":"1991","venue":"Proceedings of Advances in Neural Information Processing Systems (NIPS)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2018] Zhi-Rong Wu, Yuan-Jun Xiong, Stella X. Yu, and Da-Hua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018.","title":"Unsupervised feature learning via non-parametric instance discrimination","authors":"Zhi-Rong Wu, Yuan-Jun Xiong, Stella X Yu, Da-Hua Lin","year":"2018","venue":"Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xie and Huang, 2018] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pages 4302-4309, New Orleans, LA, 2018.","title":"Partial multi-label learning","authors":"Huang Xie, Ming-Kun Xie, Sheng-Jun Huang","year":"2018","venue":"Proceedings of the 32nd AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xie and Huang, 2020] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. In Proceedings of 34th AAAI Conference on Artificial Intelligence, pages 6454-6461, New York, NY, 2020.","title":"Partial multi-label learning with noisy label identification","authors":"Huang Xie, Ming-Kun Xie, Sheng-Jun Huang","year":"2020","venue":"Proceedings of 34th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xu et al., 2020] Yan-Wu Xu, Ming-Ming Gong, Jun-Xiang Chen, Tong-Liang Liu, Kun Zhang, and Kayhan Bat- manghelich. Generative-discriminative complementary learning. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 6526-6533, New York, NY, 2020.","title":"Generative-discriminative complementary learning","authors":"Yan-Wu Xu, Ming-Ming Gong, Jun-Xiang Chen, Tong-Liang Liu, Kun Zhang, Kayhan Batmanghelich","year":"2020","venue":"Proceedings of the 34th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2022a] Cheng-Yin Xu, Zeng-Hao Chai, Zheng- Zhuo Xu, Chun Yuan, Yan-Bo Fan, and Jue Wang. Hyp 2 loss: Beyond hypersphere metric space for multi-label im- age retrieval. In Proceedings of the 30th ACM Interna- tional Conference on Multimedia, pages 3173-3184, Lis- boa, Portugal, 2022.","title":"Hyp 2 loss: Beyond hypersphere metric space for multi-label image retrieval","authors":"Xu","year":"2022","venue":"Proceedings of the 30th ACM International Conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2022b] Ning Xu, Cong-Yu Qiao, Jia-Qi Lv, Xin Geng, and Min-Ling Zhang. One positive label is suffi- cient: Single-positive multi-label learning with label en- hancement. In Advances in Neural Information Processing Systems 35, in press, 2022.","title":"One positive label is sufficient: Single-positive multi-label learning with label enhancement","authors":"Xu","year":"2022","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yu et al., 2018a] Guo-Xian Yu, Xia Chen, Carlotta Domeni- coni, Jun Wang, Zhao Li, Zi-Li Zhang, and Xin-Dong Wu. Feature-induced partial multi-label learning. In Proceed- ings of 2018 IEEE International Conference on Data Min- ing, pages 1398-1403, Singapore, 2018.","title":"Feature-induced partial multi-label learning","authors":"Yu","year":"2018","venue":"Proceedings of 2018 IEEE International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yu et al., 2018b] Xi-Yu Yu, Tong-Liang Liu, Ming-Ming Gong, and Da-Cheng Tao. Learning with biased comple- mentary labels. In Proceedings of the 15th European Con- ference on Computer Vision, pages 69-85, Munich, Ger- many, 2018.","title":"Learning with biased complementary labels","authors":"Yu","year":"2018","venue":"Proceedings of the 15th European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhang and Wu, 2015] Min-Ling Zhang and Lei Wu. Lift: Multi-label learning with label-specific features. IEEE Trans. Pattern Anal. Mach. Intell., 37(1):107-120, 2015.","title":"Lift: Multi-label learning with label-specific features","authors":"Wu ; ] Min-Ling Zhang, Lei Zhang, Wu","year":"2015","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zhang and Zhou, 2006] Min-Ling Zhang and Zhi-Hua Zhou. Multilabel neural networks with applications to functional genomics and text categorization. IEEE transactions on Knowledge and Data Engineering, 18(10):1338-1351, 2006.","title":"Multilabel neural networks with applications to functional genomics and text categorization","authors":"Zhou ; ] Min-Ling Zhang, Zhi-Hua Zhang, Zhou","year":"2006","venue":"IEEE transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang and Zhou, 2007] Min-Ling Zhang and Zhi-Hua Zhou. ML-KNN: A lazy learning approach to multi-label learning. Pattern Recognit., 40(7):2038-2048, 2007.","title":"ML-KNN: A lazy learning approach to multi-label learning","authors":"Zhou ; ] Min-Ling Zhang, Zhi-Hua Zhang, Zhou","year":"2007","venue":"Pattern Recognit","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Zhang and Zhou, 2014a] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng., 26(8):1819-1837, 2014.","title":"A review on multi-label learning algorithms","authors":"Zhou Zhang, Min-Ling Zhang, Zhi-Hua Zhou","year":"2014","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Zhang and Zhou, 2014b] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng., 26(8):1819-1837, 2014.","title":"A review on multi-label learning algorithms","authors":"Zhou Zhang, Min-Ling Zhang, Zhi-Hua Zhou","year":"2014","venue":"IEEE Trans. Knowl. Data Eng","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Zhang et al., 2018] Min-Ling Zhang, Yu-Kun Li, Xu-Ying Liu, and Xin Geng. Binary relevance for multi-label learn- ing: an overview. Frontiers Comput. Sci., 12(2):191-202, 2018.","title":"Binary relevance for multi-label learning: an overview","authors":"Zhang","year":"2018","venue":"Frontiers Comput. Sci","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Zhang et al., 2022] Yu Zhang, Zhi-Hong Shen, Chieh-Han Wu, Bo-Ya Xie, Jun-Heng Hao, Ye-Yi Wang, Kuan-San Wang, and Jia-Wei Han. Metadata-induced contrastive learning for zero-shot multi-label text classification. In Proceedings of the ACM Web Conference 2022, pages 3162-3173, Virtual Event, 2022.","title":"Metadata-induced contrastive learning for zero-shot multi-label text classification","authors":"Yu Zhang, Zhi-Hong Shen, Chieh-Han Wu, Bo-Ya Xie, Jun-Heng Hao, Ye-Yi Wang, Kuan-San Wang, Jia-Wei Han","year":"2022","venue":"Proceedings of the ACM Web Conference 2022","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhao et al., 2021] Wen-Ting Zhao, Shu-Feng Kong, Jun- Wen Bai, Daniel Fink, and Carla P. Gomes. HOT-VAE: learning high-order label correlation for multi-label clas- sification via attention-based variational autoencoders. In Proceedings of 35th AAAI Conference on Artificial Intelli- gence, pages 15016-15024, Virtual Event, 2021.","title":"HOT-VAE: learning high-order label correlation for multi-label classification via attention-based variational autoencoders","authors":"Wen-Ting Zhao, Shu-Feng Kong, Jun-Wen Bai, Daniel Fink, Carla P Gomes","year":"2021","venue":"Proceedings of 35th AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhou, 2018] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National Science Review, 5(1):44-53, 2018.","title":"A brief introduction to weakly supervised learning","authors":"Zhi-Hua Zhou","year":"2018","venue":"National Science Review","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Zhou, 2022] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 9(8):nwac123, 2022.","title":"Open-environment machine learning","authors":"Zhi-Hua Zhou","year":"2022","venue":"National Science Review","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}