--- title: "WPML3CP: Wasserstein Partial Multi-Label Learning with Dual Label Correlation Perspectives" authors: - "Ximing Li" - "Yuanchao Dai" - "Bing Wang" - "Changchun Li" - "Renchu Guan" - "Fangming Gu" - "Jihong Ouyang" doi: "10.24963/ijcai.2024/495" url: "https://ijcai.org/proceedings/2024/495" pdf: "https://ijcai.org/proceedings/2024/0495.pdf" venue: "Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence" year: 2024 pages: "4479-4487" track: "Main Track" keywords: - "Machine Learning: ML: Weakly supervised learning" - "Machine Learning: ML: Classification" - "Machine Learning: ML: Self-supervised Learning" --- # WPML3CP: Wasserstein Partial Multi-Label Learning with Dual Label Correlation Perspectives Ximing Li, Yuanchao Dai, Bing Wang, Changchun Li, Renchu Guan, Fangming Gu, Jihong Ouyang *Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence*, pages 4479-4487. DOI: [10.24963/ijcai.2024/495](https://doi.org/10.24963/ijcai.2024/495) ## Abstract Partial multi-label learning (PMLL) refers to a weakly-supervised classification problem, where each instance is associated with a set of candidate labels, covering its ground-truth labels but also with irrelevant ones. The current methodology of PMLL is to estimate the ground-truth confidences of candidate labels, i.e., the likelihood of a candidate label being a ground-truth one, and induce the multi-label predictor with them, rather than the candidate labels. In this paper, we aim to estimate precise ground-truth confidences by leveraging precise label correlations, which are also required to estimate. To this end, we propose to capture label correlations from both measuring and modeling perspectives. Specifically, we measure the loss between ground-truth confidences and predictions by employing the Wasserstein distance involving label correlations; and form a label correlation-aware regularization to constraint predictive parameters. The two techniques are coupled to promote precise estimations of label correlations. Upon these ideas, we propose a novel PMLL method, namely Wasserstein Partial Multi-Label Learning with dual Label Correlation Perspectives (WPML3CP). We conduct extensive experiments on several benchmark datasets. Empirical results demonstrate that WPML3CP can outperform the existing PMLL baselines. ## Full text ## 1 Introduction Multi-label learning (MLL) [Zhang and Zhou, 2014a] refers to inducing multi-label predictors from the precisely labeled training dataset, where each instance is annotated with its ground-truth labels. However, in many real-world scenarios, e.g., annotations from crowdsourcing platforms, only partially valid training dataset is available due to various reasons [Li et al., 2021; Li et al., 2022; Feng et al., 2022; Li et al., 2023; Xie et al., 2023], such as the difficulty of accurate supervision collection and human effort cost reduction, etc. A common situation is that each instance is associated with a set of candidate labels, which cover the groundtruth labels but also with some irrelevant ones, e.g., the image instance illustrated in Fig.1. Learning with such a partially valid training dataset, formally referred to as partial multilabel learning (PMLL), is naturally much more challenging than the traditional MLL since even the number of groundtruth labels is unknown. ∗Corresponding author. Figure 1: An example of partial multi-label learning. The example image instance is annotated with 8 candidate labels, but only 5 of them are ground-truth ones. ![Image](https://ijcai.org/proceedings/2024/fulltext/495/image_000000_765aa37697495e3832c87cf51a0fe6318554671b62e3209f77872d348f6c5c4d.png) To handle the emerging task of PMLL, many methods have been recently developed, where the basic idea is to estimate the ground-truth confidences of candidate labels, i.e., the likelihood of a candidate label being a ground-truth one, and induce the multi-label predictor with them rather than candidate labels [Xie and Huang, 2018a; Sun et al., 2019; Li et al., 2020; Li and Wang, 2020; Xie et al., 2021; Sun et al., 2022]. Naturally, the primary philosophy of PMLL is how to estimate precise ground-truth confidences, and to our knowledge, the prevalent spirit is to promote it by exploiting the correlations among labels. For example, the GLC method formulates the ground-truth confidences by integrating a label correlation matrix [Sun et al., 2022]; the MUSER method decomposes ground-truth confidences into two parts, including a low-dimensional label matrix and a label correlation matrix [Li et al., 2020]. Although this spirit is empirically effective in prior literature, it raises a new problem that label correlations are also required to estimate, and imprecise estimations may result in imprecise ground-truth confidence. Accordingly, in this paper, we concentrate on estimating precise label correlations to promote the precision of groundtruth confidence. To this end, we propose to capture label correlations from both measuring and modeling perspectives. Specifically, we measure the loss between groundtruth confidences and predictions by employing the Wasserstein distance involving label correlations; and form a label correlation-aware regularization to constrain predictive parameters. The two techniques are coupled to promote precise estimations of label correlations. Upon these ideas, we propose a novel PMLL method, namely Wasserstein Partial Multi-Label Learning with dual Label Correlation Perspectives (WPML3 CP), which is optimized within the framework of the augmented Lagrange multiplier method. To evaluate the performance of WPML3 CP, we conduct extensive experiments on both real-world and synthetic datasets. Experimental results demonstrate that WPML3 CP can outperform the existing PMLL baselines in most cases. To sum up, the main contributions of this paper are presented as follows: - We propose a novel PMLL method named WPML3 CP, which promotes precise ground-truth confidences by capturing label correlations from both measuring and modeling perspectives. - We efficiently optimize the proposed WPML3 CP method with the framework of augmented Lagrange multiplier. - We empirically validate the effectiveness of WPML3 CP by comparing with the existing PMLL baselines on both real-world and synthetic datasets. ## 2 RelatedWorks ## 2.1 Partial Multi-Label Learning Recently, several PMLL methods have been proposed. Generally, they mainly focus on estimating the ground-truth confidences, so as to capture more accurate supervised signals. At present, the methods to solve PMLL are mainly divided into unified framework method and two-stage method. Specifically, the unified framework method aims to integrate label confidence learning and multi-classification model into a joint framework, then the unified framework is optimized by iterative strategy. The PMLL framework proposed by [Xie and Huang, 2018b] is built on a ranking loss objective weighted by the ground-truth confidence, which is regularized through either label correlation or feature prototype, and gives two versions, named PML-lc and PML-fp, respectively. Then both fPML [Yu et al., 2018] and PML-LRS [Sun et al., 2019] utilize the low-rank decomposition to capture the ground-truth label matrix from the observed candidate labels. MUSER [Li et al., 2020] combines label decomposition and feature mapping to enhance the robustness of the classifier against redundant labels and noisy features. Another recent PML-NI [Xie and Huang, 2021] method jointly learns the multi-label classifier and noise label identifier which combines the label correlation exploitation and feature-induced noise model. In order to make full use of label correlation information to deal with noise labels, GLC [Sun et al., 2022] uses low-rank representation and label manifold regularizer to capture the global and local label correlation, respectively. Recently, graph-based methods have been widely used to deal with PMLL problems. PARD [Hang and Zhang, 2023] first applies the probabilistic graphical model to disambiguate the candidate labels. PLAIN [Wang et al., 2023] constructs the instance graph and the label graph in a data-driven way, and calculates similarity to generate pseudo-labels through label propagation. Different from the unified framework method, the twostage method restores the ground-truth label distribution or learns a label confidence matrix in the first stage, and then builds a multi-classification prediction model based on the recovered label distribution or label confidence matrix in the second stage. PARTICLE [Zhang and Fang, 2021] first estimates the ground-truth confidences by iterating label propagation to select credible labels. Then, it induces the prediction model with these credible labels using either VLS or MAP, respectively. In addition to PARTICLE's work, PMLLD [Xu et al., 2020] uses the topological information of feature space and the correlation between labels to recover the label distribution. PML-MD [Xie et al., 2021] uses confidenceweighted ranking loss to process the PMLL data in the first stage, where confidences are estimated adaptively using a meta-learning-based approach. Considering that estimating label confidences may introduce a lot of uncertainty, PAMB [Liu et al., 2023] uses error-correcting output code (ECOC) technique to transform PMLL problems into multiple binary learning problems. ## 2.2 Learning withWasserstein Distance The Wasserstein distance [Villani, 2008] has drawn great attention in the machine learning community, and been widely applied to various learning tasks [Cuturi and Doucet, 2014; Rubner et al., 2000; Arjovsky et al., 2017; Frogner et al., 2015; Kusner et al., 2015; Huang et al., 2016; Zhao and Zhou, 2018]. Recently, the authors of [Frogner et al., 2015] propose a novel learning framework, which refers to the Wasserstein distance as the loss function. Orthogonal to WPML3 CP, this framework is also solved using entropic regularization approximation [Cuturi, 2013], however, it is only applicable to precisely annotated datasets. Another representative work proposed in [Zhao and Zhou, 2018] employs Wasserstein loss for the task of label distribution learning. It introduces a kernel biased regularization to simultaneously leverage the label distribution and induce label correlations. However, it is also assumed that the ground-truth supervision is known. ## 3 Preliminaries In this section, we first briefly review the Wasserstein distance and then introduce its entropic regularization. The Wasserstein distance [Bogachev and Kolesnikov, 2012] is a special case of optimal transport distance [Villani, 2008], also referred as earth mover's distance [Rubner et al., 2000]. Formally, given any two probability measures µ(u), ν(v) on a space K (i.e., u, v ∈ K) and a cost function c : K×K → R, the optimal transport distance measures the cheapest way to transport the mass in µ to match that in ν: where Π(µ, ν) denotes the set of all joint probability measures on K × K with marginals µ(u) and ν(v). The Wasserstein distance is the case of Eq.(1), in which the cost function c is specified by drK, the r-th power of a metric dK on K with r ∈ [1,∞). Discrete Wasserstein distance. When K is a finite set of size |K| = K, both µ(u) and ν(v) are discrete distributions (i.e., histograms) in the simplex ∆K , i.e., {µ, ν ∈ RK+ | µ⊤1 = 1, ν ⊤1 = 1}, where 1 denotes the all-one vector. Given a cost matrix MK computed by the r-th power of pairwise distances over K, i.e., MK = [drK(ui, vj)]ij ∈ RK×K , the discrete Wasserstein distance between µ(u) and ν(v) can be written as: where ⟨·, ·⟩ denotes the Frobenius dot-product, and U(µ, ν) = {T ∈ RK×K + |T1 = µ,T⊤1 = ν} is the transportation polytope. Regularized Wasserstein distance. The discrete Wasserstein distance of Eq.(2) is not smooth with respect to its arguments µ, ν [Cuturi and Doucet, 2014], and its computation requires costly O(K3 logK) time [Pele and Werman, 2009]. To handle those problems, the authors of [Cuturi, 2013] incorporate an entropic regularizer into Eq.(2), leading to the regularized Wasserstein distance: where H(T) = -⟨T, logT⟩ is the (strictly concave) entropy function and λ > 0 is the regularization parameter. Accordingly, given models with parameters of interest, i.e., denoted by {µ, ν,MK}, we can optimize them by the Sinkhorn's algorithm with O(K2 ) complexity [Cuturi, 2013; Cuturi and Doucet, 2014]. The optimization details are introduced in the Appendix 1 . ## 4 ProposedWPML3 CP Method In this section, we introduce the proposed PMLL method named WPML3 CP. Problem formulation of PMLL. Formally, given a partially valid training dataset D = {(xi, yi)}i=n i=1 with n instances and l category labels, let xi ∈ Rd and yi ∈ {0, 1}l denote the d-dimensional feature vector and the candidate label set associated with the i-th instance, respectively. For yi, the candidate label is signed by 1, otherwise 0. For each instance xi, its ground-truth label set y∗ i ∈ {0, 1}l is unknown, but it is covered by yi, i.e., y∗ i ⪯ yi. The objective of PMLL is to induce a classifier over D, which can predict the labels for unseen instances. [1 https://github.com/daidai1118/WPML3CP/blob/main/ WPML3CP Appendix.pdf](https://github.com/daidai1118/WPML3CP/blob/main/WPML3CP_Appendix.pdf) ## 4.1 Method Description For each instance xi, its (normalized) ground-truth confidence is defined by pi ∈ Rl such that 0 ⪯ pi ⪯ yi, p⊤i 1 = 1, where 0 denotes the all-zero vector. And the label correlation matrix is denoted by C ∈ Rl×l + with each c:,j being the coefficient vector for label j with respect to labels l. The basic idea of WPML3 CP is to estimate precise p by capturing precise C from both measuring and modeling perspectives. Specifically, from the measuring perspective, we specify each pi as a discrete distribution, and measure the loss between it and the prediction model by exploiting the regularized Wasserstein distance Wλ(·). Toward this goal, we employ a normalized linear model f(xi|W) = Wxi, W = [w1, · · · ,wl]⊤ ∈ Rl×d such that (Wxi)⊤1 = 1, ensuring the prediction also as a discrete distribution. From the modeling perspective, we form a label correlation-aware manifold regularization term to constrain the parameter W of the model. Upon the above ideas, we formulate the objective of WPML3 CP with a squared Frobenius norm of W as follows: where m(·) = 1 - sigmoid(·) is utilized to transform the label correlation matrix C to the cost matrix of Wasserstein distances; and {β1, β2} are the regularization parameters. Since the constraints of Eq.(4) tend to increase the optimization complexity, we leverage two surrogate heuristics to eliminate them. First, for the two equality constraints of pi and Wxi, we exploit the softmax function s(·) to directly satisfy them. Specifically, for each pi we introduce an auxiliary softmax parameter qi, i.e., pi = s(qi), and therefore learn Q = [q1, · · · , qn]⊤ ∈ Rn×l , instead of P. Second, for the inequality constraint of pi, we consider the irrelevant labels existed in the observed candidate label matrix Y = [y1, · · · , yn]⊤, and decompose it into the matrix Q and a noise matrix E. Considering that the ground-truth labels are correlated in the multi-label learning case and the noise is always sparse in many real-world scenarios, we introduce a nuclear norm regularization for Q and ℓ1-norm regularization for E. Accordingly, we achieve the final objective of WPML3 CP with respect to {W, Q, C, E} below: Here, the equality constraint of Eq.(5) can be eliminated easily by employing the augmented Lagrange multiplier technique [Zhang et al., 2017]. ## 4.2 Optimization The objective of WPML3 CP in Eq.(5) refers to four parameters of interest, i.e., {W,Q,C,E}. We optimize them with the gradient decent approach under the augmented Lagrange multiplier framework. Specifically, the augmented Lagrangian of Eq.(5) is given with an auxiliary variable H: where L = diag(C1) - C is the Laplacian matrix of C. According to the LADMAP method [Lin et al., 2011], it can be rewritten as: Accordingly, we employ the gradient decent approach to optimize {W, C, H}, whose gradients can be easily calculated with some simple derivations and the Sinkhorn algorithm, and update {Q, E} as well as {Y1, Y2, µ1, µ2} with the linear ADM method following [Liu et al., 2010]. Due to the space limitation, the optimization details are presented in the Appendix 2 . Initialization of label correlation matrix C: We initialize the label correlation matrix C with the cosine similarities among label prototypes, which are calculated by averaging the training features X = [x1, · · · , xn]⊤ weighted with Y. Specifically, given the label prototypes O = Y⊤X = [o1, · · · , ol]⊤, the initialization of the label correlation matrix C is given by: ## 5 Experiment In this section, we empirically evaluate the proposed WPML3 CP method. Datasets. To thoroughly evaluate WPML3 CP, we employ 5 real-world PMLL datasets, 3 including Music emotion, Mirflickr, YeastBP, YeastCC, and YeastMF, and generate synthetic PMLL datasets by using 4 MLL datasets, 4 including Birds, Medical, Genbase, and Entertainment. For clarity, we present the detailed characteristics of all datasets in Table 1. [2 https://github.com/daidai1118/WPML3CP/blob/main/](https://github.com/daidai1118/WPML3CP/blob/main/WPML3CP_Appendix.pdf) [WPML3CP Appendix.pdf](https://github.com/daidai1118/WPML3CP/blob/main/WPML3CP_Appendix.pdf) [3 http://palm.seu.edu.cn/zhangml/](http://palm.seu.edu.cn/zhangml/) 4 http://mulan.sourceforge.net/datasets-mlc.html, https://github. com/fcharte/mldr.datasets Table 1: Detailed characteristics of datasets. |S|: the number of instances of dataset S. dim(S): the number of features. L(S): the number of category labels. LCard(S): the average number of labels per instance. | Dataset | |S| | dim(S) | L(S) | LCard(S) | Domain | |---------------|-----------------|----------|--------|------------|----------| | Music emotion | 6,833 | 98 | 11 | 2.42 | Music | | Mirflickr | 10,433 | 100 | 7 | 1.77 | Images | | YeastBP | 6,139 | 6,139 | 217 | 5.537 | Biology | | YeastCC | 6,139 | 6,139 | 50 | 1.348 | Biology | | YeastMF | 6,139 | 6,139 | 39 | 1.005 | Biology | | Birds | 645 | 260 | 19 | 1.014 | Audio | | Medical | 978 | 1,449 | 45 | 1.140 | Text | | Genbase | 662 | 1,186 | 27 | 1.252 | Biology | | Entertainment | 12,730 | 32,001 | 21 | 1.414 | Text | Besides, the synthetic versions are generated by the following policies. For each instance, we randomly choose some irrelevant labels, and then mix them with the ground-truth ones, leading to the set of candidate labels. The number of irrelevant noisy labels is denoted by m%∥y∗ i ∥0, where the value of m is varied over [50, 100, 150]. Accordingly, in our experiments, we generate 12 synthetic PMLL datasets in total. Baseline methods. We employ 7 the state-of-the-art PMLL methods for comparison. They include PAMB [Liu et al., 2023], GLC [Sun et al., 2022], PMLNI [Xie and Huang, 2021], PML-MD [Xie et al., 2021], PML-LRS [Sun et al., 2019], PAR-MAP and PAR-VLS [Zhang and Fang, 2021]. We set the parameters as suggested in their original papers. For our WPML3 CP, across Mirflickr and Music emotion the parameters are set as: β1 = 1, β2 = 10-3 , β3 = 10-1 , and β4 = 1, and across YeastBP, YeastCC and YeastMF the parameters are set as: β1 = 10-3 , β2 = 10-1 , β3 = 102 , and β4 = 102 . Furthermore, about the parameter λ of the regularized Wasserstein distance, we set λ = 1 for Mirflickr and λ = 50 for others. Evaluation metrics. We employ 5 widely-used MLL metrics to measure the performance, including Average Precision (AP), Coverage Error (CError), Ranking Loss (RLoss), Macro Averaging F1 (Macro F1) and Micro Averaging F1 (Micro F1). Those evaluation metrics cover both rankingand binary-based ones and their detailed definitions can refer to the descriptions in [Zhang and Zhou, 2014b]. For CError and RLoss, the smaller value implies better performance (denoted by ↓); and for AP, Macro F1, and Micro F1, the larger value is better (denoted by ↑). ## 5.1 Results on Real-world Datasets For each dataset, we conduct five-fold cross-validation and report the average results in Tables 2 and 3. We can clearly observe that WPML3 CP can outperform the baseline methods in most cases. Across all datasets and evaluation criteria on real-world datasets, WPML3 CP ranks 1st in 80% and 2nd in 12% cases. More detailed observations are made below. Comparisons via the perspective of evaluation criteria: We can observe that, on the ranking based metrics (i.e., AP, CError and RLoss), WPML3 CP ranks first in 73.3% cases. Specifically, WPML3 CP dominates all seven PML baselines on CError and performs similarly on AP and Rloss: It outperforms most baselines on these criteria but slightly worse than PAMB in music emotion and PML-MD in mirflickr. On binary prediction-based metrics (i.e., Macro F1 and Micro F1), WPML3 CP ranks first in 90% cases. Specifically, WPML3 CP dominates all seven PML baselines under these two criteria, except in music emotion where PML-MD outperforms WPML3 CP over Micro F1. This probably because that our method primarily guiding prediction model generation based on label correlation, resulting in many irrelevant labels sharing similar features being predicted as correlated. As a result, the binary prediction is perceived as more accurate but ranks slightly lower. In particular, PAMB performs very well in music emotion on three ranking based metrics, but poorly on two binary prediction based metrics, which indicates that the binary decomposition technique still has some limitations for PMLL problems. Table 2: Experimental results of each comparing method (mean ± std) in terms of AP, CError, RLoss, Macro F1 and Micro F1, where the best performance is shown in boldface. | Dataset | WPML3 CP | PAMB | GLC | PML-NI | PML-MD | PML-LRS | PAR-MAP | PAR-VLS | |-------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------|---------------------------------------------------| | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | | YeastBP YeastCC YeastMF Mirflickr Music emotion | .552±.009 .836±.007 .779±.009 .858±.006 .623±.004 | .353±.206 .653±.133 .638±.111 .753±.038 .642±.008 | .466±.016 .790±.018 .703±.008 .765±.008 .515±.005 | .402±.007 .833±.008 .770±.001 .789±.008 .599±.007 | .218±.013 .785±.009 .736±.016 .859±.007 .636±.012 | .432±.002 .722±.007 .685±.009 .793±.001 .617±.008 | .386±.203 .684±.121 .688±.108 .795±.181 .620±.010 | .332±.012 .630±.014 .603±.016 .673±.024 .585±.008 | | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | | YeastBP YeastCC YeastMF Mirflickr Music | .281±.005 .090±.007 .104±.009 .209±.004 .405±.001 | .560±.189 .271±.119 .249±.096 .276±.050 .407±.007 | .325±.012 .124±.012 .149±.008 .289±.006 .494±.004 | .416±.011 .101±.003 .119±.010 .229±.005 .411±.007 | .572±.010 .140±.011 .149±.016 .220±.005 .409±.007 | .410±.008 .171±.004 .190±.005 .231±.001 .408±.006 | .452±.193 .219±.104 .184±.085 .242±.055 .423±.006 | .689±.012 .376±.015 .397±.017 .315±.004 .431±.007 | | emotion | | | | | | | | | | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | | YeastBP YeastCC YeastMF Mirflickr Music emotion | .146±.002 .063±.004 .088±.008 .094±.004 .242±.003 | .389±.119 .231±.103 .235±.083 .147±.039 .234±.006 | .197±.008 .089±.009 .132±.005 .185±.007 .345±.006 | .218±.005 .072±.004 .105±.007 .123±.006 | .300±.013 .101±.012 .121±.012 .089±.005 .239±.011 | .258±.005 .142±.002 .180±.006 .122±.001 .243±.005 | .254±.100 .158±.068 .153±.062 .134±.124 .241±.005 | .693±.012 .396±.015 .423±.017 .230±.017 .268±.007 | | YeastBP YeastCC YeastMF Mirflickr | .339±.012 .536±.008 .404±.018 .607±.016 | .010±.006 .010±.004 .014±.007 .567±.079 | .016±.001 .069±.004 | .250±.008 Macro F1 ↑ .297±.020 .281±.017 | .149±.009 .187±.025 .144±.036 .533±.019 | .181±.008 .252±.021 .197±.013 .571±.005 | .015±.007 .017±.006 .019±.007 .346±.079 | .004±.002 .017±.002 .019±.009 | | | | .046±.026 .033±.007 .037±.014 | .035±.006 .157±.006 | Micro F1 ↑ .311±.018 .297±.018 .204±.014 | .211±.006 .271±.022 | .388±.010 | .078±.019 .079±.007 | .008±.004 | | YeastCC YeastMF Mirflickr | .429±.012 .384±.008 | .256±.008 | .053±.001 | .197±.013 .564±.014 .286±.013 | .206±.044 | .184±.007 | | .480±.012 | | | .748±.008 .529±.007 | .743±.042 .386±.005 | .079±.009 .553±.006 .238±.003 | | .748±.011 .538±.008 | | .531±.189 | | | Music emotion | | | .044±.006 | | | .263±.021 .223±.014 | | | | | | | | | .388±.023 | | .253±.009 | .205±.006 | | Music emotion | | | | | | | | | | YeastBP | .569±.013 .418±.022 | | | .685±.008 .458±.012 | | | .080±.028 | | | | | | .152±.013 | | | | | .045±.006 | | Average Rank | | | | | | .659±.002 | | .056±.008 .667±.007 | | | 1.32 | 5.68 | 5.32 | 3.28 | 3.36 | 3.96 | 5.36 | 7.52 | | | | | | | | .524±.009 | .446±.009 | .425±.008 | Comparisons via the perspective of datasets: We can observe that, on the datasets with a large number of features (i.e. YeastBP, YeastCC and YeastMF), WPML3 CP shows the best under all evaluation metrics. On datasets with a small number of features (i.e. Mirflickr and Music emotion), the average rank of WPML3 CP ranks first, slightly better than that of PML-MD. This might be caused by the too large average number of labels per instance of Yeast compared with the number of category labels, leading to the very sparse label vector of each instance. Fortunately, our method fully considers the label correlation from both measuring and modeling. ## 5.2 Results on Synthetic Datasets Across all datasets and evaluation criteria on synthetic datasets, WPML3 CP ranks 1st in 76.7% and 2nd in 20% cases. We can observe that WPML3 CP consistently outperforms all baselines for both medical and entertainment datasets in terms of all five evaluation metrics. For genbase, WPML3 CP only has a small performance gap compared with PAR-MAP on CError and RLoss metrics but still competitive. In addition, PAMB outperforms WPML3 CP over Macro F1 metric. One exception is in birds, where PML-NI outperforms our method over CError and RLoss evaluation metrics. This may be caused by the noise label identifier proposed in PML-NI, which leads to the improvement of performance. Besides, the performance gain of WPML3 CP tends to be more significant with the proportions of irrelevant labels increasing. Additionally, we examine whether the performance gain of WPML3 CP is statistically significant. To answer this question, for each PMLL dataset and evaluation metric, we conduct a pairwise t-test over the results of the corresponding five train/test splits. The resulting win/tie/loss counts over 425 statistical tests (17 PMLL datasets × 5 evaluation metrics × 5-fold cross-validation) are reported in Table 4. In summary, out of all the tests, WPML3 CP significantly outperforms recent PMLL algorithms in 88.9% (PAMB), 96.5% (GLC), 77.4% (PML-NI), 88% (PML-MD), 91.8% (PMLLRS), 80.2% (PAR-MAP) and 96% (PAR-VLS) cases, respectively. This further indicates the effectiveness of WPML3 CP. Table 3: Results of each comparing method (mean ± std) in terms of AP, CError, RLoss, Macro F1 and Micro F1, where the best performance is shown in boldface. The notation "AN.#Per" denotes the percentage of ground-truth labels added to each instance as irrelevant noisy labels. | Dataset | AN.#Per | WPML3 CP | PAMB | GLC | PML-NI | PML-MD | PML-LRS | PAR-MAP | PAR-VLS | |----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------|----------------------------------------------------------| | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | AP ↑ | | | | .777±.030 | .760±.030 | .684±.035 | .786±.020 | .728±.035 | .639±.040 | .752±.024 | .734±.027 | | birds | 50 100 | .787±.024 | .743±.032 | .682±.023 | .781±.031 | .745±.042 | .671±.014 | .737±.010 | .745±.020 | | | 150 | .775±.035 | .683±.024 | .678±.023 | .757±.036 | .733±.025 | .696±.021 | .754±.018 | .734±.029 | | | | .477±.017 | .288±.044 | .421±.044 | .455±.020 | .376±.059 | .415±.034 | .453±.099 | .418±.107 | | genbase | 50 | | | | | | | | | | | 100 150 | .481±.020 .466±.017 | .211±.037 .345±.055 | .420±.027 .356±.050 | .451±.035 .444±.037 | .381±.066 .396±.058 | .422±.045 .396±.053 | .458±.097 .453±.097 | .426±.107 .429±.113 | | | 50 | .900±.028 | .486±.038 | .657±.020 | .888±.016 | .820±.028 | .494±.025 | .810±.014 | .746±.030 | | medical | 100 | .894±.032 | .482±.039 | .653±.028 | .883±.013 | .788±.042 | .497±.020 | .801±.025 | .747±.028 | | | 150 | .890±.031 | .482±.033 | .658±.010 | .863±.015 | .779±.025 | .492±.074 | .813±.014 | .711±.024 | | | 50 | .697±.028 | - | .611±.006 | .662±.009 | .405±.058 | .358±.003 | .583±.011 | .533±.006 | | entertainment | 100 | .694±.011 | - | .610±.004 | .650±.009 | .445±.047 | .345±.003 | .570±.009 | .522±.006 | | | 150 | .691±.034 | - | .607±.005 | .620±.005 | .386±.053 | .306±.005 | .539±.004 | .518±.005 | | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | CError ↓ | | | 50 | .135±.021 | .147±.020 | .187±.038 | .136±.018 | .166±.029 | .289±.032 | .113±.021 | .171±.027 | | birds | 100 | .138±.019 | .170±.027 | .187±.008 | .141±.020 | .161±.025 | .148±.021 | .108±.008 | .172±.019 | | | 150 | .145±.025 | .193±.022 | .194±.014 | .151±.030 | .163±.014 | .197±.017 | .103±.014 | .179±.023 | | | | | | .176±.008 | .192±.026 | | .247±.038 | .156±.035 | .222±.065 | | genbase | 50 | .153±.020 | .322±.052 | .196±.017 .192±.017 | | .210±.020 | .216±.053 | .150±.035 | .209±.054 | | | 100 150 | .147±.019 .163±.020 | .362±.064 .319±.058 | .215±.012 | .177±.024 .196±.010 | .217±.024 | .273±.021 | .155±.034 | .197±.059 | | | | | .188±.015 | .071±.008 | .038±.002 | .062±.014 | .322±.028 | .060±.008 | .099±.011 | | medical | 50 100 | .030±.015 | .198±.026 | .077±.013 | .036±.009 | .067±.011 | .307±.036 | .074±.022 | .113±.023 | | | 150 | .030±.013 .034±.016 | | | .045±.006 | .079±.010 | .313±.058 | .070±.013 | .134±.017 | | | 50 | .129±.012 | - | .149±.003 | .172±.004 | .284±.068 | .400±.007 | .171±.005 | .206±.003 | | | | | .168±.019 | .076±.018 | | | | | | | entertainment | 100 | .128±.016 | | .151±.003 | .183±.001 | .246±.050 | .403±.004 | .179±.002 | .222±.002 | | | 150 | .138±.014 | - - | .153±.005 | .199±.006 | .261±.014 | .425±.005 | .199±.002 | .229±.003 | | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | RLoss ↓ | | | 50 | .105±.019 | .117±.009 | .154±.022 | .102±.015 | .131±.018 | .252±.028 | .090±.014 | .133±.017 | | birds | 100 | .105±.013 | .131±.016 | .156±.012 | .108±.019 | .127±.023 | .205±.011 | .090±.004 | .132±.010 | | | 150 | .110±.016 | .160±.012 | .157±.012 | | .130±.010 | .161±.011 | .083±.009 | .140±.013 | | | | | | | .116±.020 .157±.009 | | | | | | genbase | 50 | .136±.018 | .301±.047 | .175±.019 | | .170±.019 | .233±.040 | .138±.029 | .194±.053 | | | 100 | .132±.013 | .354±.060 | .171±.015 | .156±.016 .167±.010 | .184±.015 | .200±.054 | .133±.027 | .192±.046 | | | 150 | .142±.016 | .300±.052 | .191±.013 | | .195±.026 | .253±.016 | .135±.026 | .177±.051 | | | 50 | .020±.011 | .172±.015 | .058±.012 | .026±.003 | .048±.012 | .293±.022 | .045±.005 .059±.010 | .080±.007 .090±.017 | | medical | 100 | .020±.011 | .177±.023 | .063±.010 | .026±.007 | .052±.011 | .283±.035 | | | | | 150 | .024±.012 | | .061±.003 | .033±.004 | .061±.010 | .360±.007 | .135±.005 | .105±.013 | | | 50 | .092±.013 | .151±.018 | | .127±.003 | .240±.070 | .033±.004 | .053±.009 | .165±.002 | | entertainment | 100 | .093±.018 | - - | .106±.002 .106±.003 | .138±.001 | .204±.049 | .365±.005 | .141±.002 | .179±.002 | | | 150 | .100±.013 | - | .109±.002 | .153±.004 | .224±.014 | .385±.004 | .162±.002 | .185±.003 | | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | Macro F1 ↑ | | | | | .173±.038 | .277±.041 | .211±.047 | .097±.012 | | .189±.029 | | | birds | 50 100 | .267±.016 .305±.019 | .151±.019 | .031±.005 .030±.005 | .297±.043 .206±.034 | | .102±.019 | .187±.028 | .099±.022 .072±.016 | | | | .291±.046 | .105±.021 | .025±.011 | .213±.033 | .205±.036 | .179±.036 | .205±.012 | .104±.011 | | | 150 | | | | | | | | | | genbase | 50 100 | .272±.034 .272±.033 | .243±.060 .204±.050 | .209±.073 .231±.059 | .219±.068 .210±.029 | .253±.092 .236±.045 | .256±.043 .260±.079 | .310±.067 .309±.069 | .287±.072 .287±.072 | | | 150 | .273±.034 | .226±.054 | .233±.088 | .258±.085 | .257±.043 | .244±.062 | .312±.071 | .287±.072 | | | 50 | .368±.028 | .029±.007 | .058±.004 | .270±.020 | .260±.018 | .040±.003 | .223±.032 | .140±.016 | | | 100 | .346±.016 | .029±.008 | .053±.009 | .252±.017 | .215±.037 | .040±.002 | .207±.020 | .141±.014 | | medical | 150 | .341±.028 | .029±.008 | .060±.007 | .263±.011 | .194±.017 | .047±.008 | .224±.017 | .154±.019 | | | 50 | .295±.021 | - | .101±.003 | .236±.007 | .109±.031 | .160±.003 | .179±.007 | .152±.002 | | entertainment | 100 | .290±.022 | - | .102±.002 | .217±.004 | .115±.031 | .155±.000 | .175±.006 | .148±.006 | | | | | | | | | .127±.010 | .184±.003 | | | | | .272±.034 | | | | | | | | | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | 150 - .101±.003 .206±.006 .105±.018 .150±.003 Micro F1 ↑ | | | 50 | .483±.044 | .278±.028 | .177±.029 | | .400±.053 | .256±.035 | .242±.017 | .260±.030 | | birds | 100 | .498±.045 | .256±.029 | .174±.023 | .418±.034 | .428±.054 | .297±.019 | .249±.030 | .193±.063 | | | 150 | .480±.057 | .171±.032 | .163±.063 | .399±.053 .331±.052 | .396±.063 | .328±.037 | .251±.025 | .210±.018 | | | 50 | .264±.017 | .129±.031 | .236±.042 | .169±.110 | | .241±.026 | .251±.116 | .012±.001 | | | 100 | .245±.017 | .130±.045 | | .191±.063 | .142±.086 | | | .012±.001 | | genbase | | .277±.021 | .127±.021 | .236±.022 | .176±.065 | | .243±.037 | .259±.111 | | | | 150 | | | .147±.072 | .242±.039 | .220±.067 | .217±.062 | .247±.108 | .012±.001 | | medical | 50 | .835±.036 | .291±.045 | .442±.020 | .758±.025 .734±.030 | .390±.025 | | .583±.029 | .433±.086 | | | 100 150 | .830±.036 .820±.037 | .289±.042 .287±.044 | .438±.031 | .751±.025 .706±.025 | .701±.049 .682±.031 | .385±.016 .377±.060 | .585±.030 .571±.016 | .460±.051 .463±.085 | | | 50 | .563±.042 | - | .361±.009 | .392±.011 .256±.042 .361±.008 | | .242±.003 | .376±.008 | .347±.009 | | | | | | .444±.022 | | | | | .332±.008 | | | 150 | .556±.015 .543±.040 | - - | .360±.004 .361±.007 | | .312±.063 .243±.090 | .233±.002 .188±.009 | .366±.008 .338±.005 | .324±.005 | | entertainment | 100 | | | .334±.003 | 4.24 | 6.36 | | 2.82 | 5.44 | | Average | | 1.40 | | 6.11 | | | | | | | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Rank 6.87 2.71 | Figure 2: Sensitivity analysis of regularization parameters {β1, β2, β3, β4} on two real PMLL datasets Mirflickr (top) and YeastCC (bottom). ![Image](https://ijcai.org/proceedings/2024/fulltext/495/image_000001_ed5fc553b31d6f621ae0ae87b7ffa8d7b42260bddeba6f4eaff6fa185594d991.png) Table 4: Win/tie/loss counts of pairwise t-test between WPML3 CP and each comparing method. | Baseline Method | AP | CError | RLoss | Macro F1 | Micro F1 | Total | |-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------|----------------------------------------------------|----------------------------------------------------|-----------------------------------------------------|--------------------------------------------------|--------------------------------------------------------------------| | PAMB [Liu et al., 2023] GLC [Sun et al., 2022] PML-NI [Xie and Huang, 2021] PML-MD [Xie et al., 2021] PML-LRS [Sun et al., 2019] PAR-MAP[Fang and Zhang, 2019] PAR-VLS [Fang and Zhang, 2019] | 74/3/8 85/0/0 54/7/24 72/6/7 82/3/0 63/4/18 79/0/6 | 76/0/9 85/0/0 55/5/25 82/3/0 83/1/1 62/1/22 79/0/6 | 76/0/9 85/0/0 60/3/22 77/1/7 82/0/3 58/3/24 82/0/3 | 68/3/14 76/9/0 78/6/1 70/15/0 70/3/12 76/6/3 84/0/1 | 84/0/1 79/3/3 82/0/3 73/8/4 73/5/7 82/0/3 84/0/1 | 378/6/41 410/12/3 329/21/75 374/33/18 390/12/23 341/14/70 408/0/17 | ## 5.3 Parameter Evaluation In this subsection, we evaluate the sensitivity of the regularization parameter {β1, β2, β3, β4} of WPML3 CP. To this end, we examine the impact of varying {β1, β2, β3} values across the range {10i |i = -3, · · · , 3} and varying β4 values across the range {10i |i = -1, · · · , 5} on two real PMLL datasets Mirflickr and YeastCC, measured by the ranking based metric (i.e., AP) and the binary prediction based metric (i.e., Macro F1). The results are presented in Figure 2. Roughly, we can observe that WPML3 CP tends to achieve higher scores with relatively smaller values of {β1, β2} and higher value of β4, and excepts a smaller (higher) value of β3 on the dataset with a large (small) number of features. We empirically recommend selecting the values of {β1, β2} from 10-3 to 101 , β3 inversely proportional to dim(S) and β4 from 10-1 to 103 . ## 5.4 Ablation Study In this subsection, we conduct ablative studies of different strategies (terms in the loss) by setting the corresponding regularization parameters to zero (see Figure 2). Specifically, {β1 = 0, β3 = 0, β4 = 0} represents the versions without the manifold regularization, ∥Q∥∗, and ∥E∥1 terms, respectively. Furthermore, to validate the effectiveness of Wasserstein distance, we compare substituting it with Euclidean distance in Table 5, serving as an ablation experiment for the label correlation-aware Wasserstein distance. The results on all real datasets demonstrate the superiority of the Wasserstein distance over the Euclidean distance. Table 5: Ablation study results about Wasserstein distance. | Dataset | Ver. | AP | CError | RLoss | MacroF1 | MicroF1 | |---------------|------------|---------------------|---------------------|---------------------|---------------------|---------------------| | YeastBP | Eucli Wass | .370±.009 .552±.009 | .504±.008 .281±.005 | .286±.004 .146±.002 | .031±.003 .339±.012 | .114±.007 .384±.008 | | YeastMF | Eucli Wass | .650±.003 .779±.009 | .215±.005 .104±.009 | .199±.006 .088±.008 | .024±.005 .404±.018 | .065±.010 .418±.022 | | YeastCC | Eucli Wass | .661±.008 .836±.007 | .251±.014 .090±.007 | .195±.005 .063±.004 | .035±.011 .536±.008 | .091±.016 .569±.013 | | Mirflickr | Eucli Wass | .790±.009 .858±.006 | .228±.005 .209±.004 | .122±.006 .094±.004 | .577±.010 .607±.016 | .659±.008 .748±.008 | | Music emotion | Eucli Wass | .617±.004 .623±.004 | .408±.003 .405±.001 | .243±.003 .242±.003 | .389±.008 .429±.012 | .526±.004 .529±.007 | ## 6 Conclusion In this paper, we investigate the paradigm of PMLL with noise supervised signals. A novel PMLL method, namely WPML3 CP, is proposed. In WPML3 CP, we specify the ground-truth confidence as the normalized discrete distribution, i.e., latent label distribution, to describe the probability of a candidate label being a ground-truth one. Considering it as a latent distribution variable, we jointly learn it and a normalized linear prediction model by minimizing the robust regularized Wasserstein distance. To reduce the optimization complexity, we eliminate the normalized constraints by exploiting the softmax function, leading to a surrogate objective without any constraint. The gradient descent with Adam is used to solve it. Extensive experimental results demonstrate that WPML3 CP outperforms the state-of-the-art baseline algorithms, and specifically it works well with high proportions of irrelevant labels. ## Acknowledgements We would like to acknowledge support for this project from the National Science and Technology Major Project (No.2021ZD0112501), the National Natural Science Foundation of China (No.62276113), and China Postdoctoral Science Foundation (No.2022M721321). ## References 1. [Arjovsky et al., 2017] Martín Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In ICML, pages 214-223, 2017. 2. [Bogachev and Kolesnikov, 2012] Vladimir Igorevich Bo- gachev and Aleksandr Viktorovich Kolesnikov. The monge-kantorovich problem: Achievements, connec- tions, and perspectives. Russian Mathematical Surveys, 67(5):785-890, 2012. 3. [Cuturi and Doucet, 2014] Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. In ICML, pages 685-693, 2014. 4. [Cuturi, 2013] Marco Cuturi. Sinkhorn distances: Light- speed computation of optimal transport. In NeurIPS, pages 2292-2300, 2013. 5. [Fang and Zhang, 2019] Jun-Peng Fang and Min-Ling Zhang. Partial multi-label learning via credible label elicitation. In AAAI, pages 3518-3525, 2019. 6. [Feng et al., 2022] Lei Feng, Jun Huang, Senlin Shu, and Bo An. Regularized matrix factorization for multilabel learning with missing labels. IEEE TCYB, 52(5):3710- 3721, 2022. 7. [Frogner et al., 2015] Charlie Frogner, Chiyuan Zhang, Hos- sein Mobahi an Mauricio Araya-Polo, and Tomaso Pogglo. Learning with a wasserstein loss. In NeurIPS, pages 2053- 2061, 2015. 8. [Hang and Zhang, 2023] Jun-Yi Hang and Min-Ling Zhang. Partial multi-label learning with probabilistic graphical disambiguation. In NeurIPS, 2023. 9. [Huang et al., 2016] Gao Huang, Chuan Guo, Matt J. Kus- ner, Yu Sun, Kilian Q. Weinberger, and Fei Sha. Super- vised word mover's distance. In NeurIPS, pages 4862- 4870, 2016. 10. [Kusner et al., 2015] Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. From word embeddings to document distances. In ICML, pages 957-966, 2015. 11. [Li and Wang, 2020] Ximing Li and Yang Wang. Recovering accurate labeling information from partially valid data for effective multi-label learning. In IJCAI, pages 1373-1380, 2020. 12. [Li et al., 2020] Ziwei Li, Gengyu Lyu, and Songhe Feng. Partial multi-label learning via multi-subspace representa- tion. In IJCAI, pages 2612-2618, 2020. 13. [Li et al., 2021] Changchun Li, Ximing Li, and Jihong Ouyang. Semi-supervised text classification with balanced deep representation distributions. In ACL-IJCNLP, pages 5044-5053, 2021. 14. [Li et al., 2022] Changchun Li, Ximing Li, Lei Feng, and Ji- hong Ouyang. Who is your right mixup partner in positive and unlabeled learning. In ICLR, 2022. 15. [Li et al., 2023] Ximing Li, Yuanzhi Jiang, Changchun Li, Yiyuan Wang, and Jihong Ouyang. Learning with partial labels from semi-supervised perspective. In AAAI, pages 8666-8674, 2023. 16. [Lin et al., 2011] Zhouchen Lin, Risheng Liu, and Zhixun Su. Linearized alternating direction method with adaptive penalty for low-rank representation. In NeurIPS, pages 612-620, 2011. 17. [Liu et al., 2010] Guangcan Liu, Zhouchen Lin, and Yong Yu. Robust subspace segmentation by low-rank represen- tation. In ICML, pages 663-670, 2010. 18. [Liu et al., 2023] Bing-Qing Liu, Bin-Bin Jia, and Min-Ling Zhang. Towards enabling binary decomposition for partial multi-label learning. IEEE TPAMI, 2023. 19. [Pele and Werman, 2009] Ofir Pele and Michael Werman. Fast and robust earth mover's distances. In ICCV, pages 460-467, 2009. 20. [Rubner et al., 2000] Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. The earth mover's distance as a metric for image retrieval. IJCV, 40(2):99-121, 2000. 21. [Sun et al., 2019] Lijuan Sun, Songhe Feng, Tao Wang, Con- gyan Lang, and Yi Jin. Partial multi-label learning with low-rank and sparse decomposition. In AAAI, pages 5016- 5023, 2019. 22. [Sun et al., 2022] Lijuan Sun, Songhe Feng, Jun Liu, Gengyu Lyu, and Congyan Lang. Global-local label corre- lation for partial multi-label learning. IEEE Transactions on Multimedia, 24:581-593, 2022. 23. [Villani, 2008] Cédric Villani. Optimal Transport: Old and New. Springer Berlin Heidelberg, 2008. 24. [Wang et al., 2023] Haobo Wang, Shisong Yang, Gengyu Lyu, Weiwei Liu, Tianlei Hu, Ke Chen, Songhe Feng, and Gang Chen. Deep partial multi-label learning with graph disambiguation. arXiv preprint arXiv:2305.05882, 2023. 25. [Xie and Huang, 2018a] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In AAAI, pages 4302-4309, 2018. 26. [Xie and Huang, 2018b] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In AAAI, pages 4302-4309, 2018. 27. [Xie and Huang, 2021] Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning with noisy label identification. IEEE TPAMI, 44(7):3676-3687, 2021. 28. [Xie et al., 2021] Ming-Kun Xie, Feng Sun, and Sheng-Jun Huang. Partial multi-label learning with meta disambigua- tion. In ACM SIGKDD, pages 1904-1912, 2021. 29. [Xie et al., 2023] Ming-Kun Xie, Jia-Hao Xiao, Hao-Zhe Liu, Gang Niu, Masashi Sugiyama, and Sheng-Jun Huang. Class-distribution-aware pseudo-labeling for semi-supervised multi-label learning. In NeurIPS, 2023. 30. [Xu et al., 2020] Ning Xu, Yun-Peng Liu, and Xin Geng. Partial multi-label learning with label distribution. In AAAI, pages 6510-6517, 2020. 31. [Yu et al., 2018] Guoxian Yu, Xia Chen, Carlotta Domeni- coni, Jun Wang, Zhao Li, Zili Zhang, and Xindong Wu. Feature-induced partial multi-label learning. In IEEE ICDM, pages 1398-1403, 2018. 32. [Zhang and Fang, 2021] Min-Ling Zhang and Jun-Peng Fang. Partial multi-label learning via credible label elic- itation. IEEE TPAMI, 43(10):3587-3599, 2021. 33. [Zhang and Zhou, 2014a] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE TKDE, 26(8):1819-1837, 2014. 34. [Zhang and Zhou, 2014b] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE TKDE, 26(8):1819-1837, 2014. 35. [Zhang et al., 2017] Yongqiang Zhang, Daming Shi, Junbin Gao, and Dansong Cheng. Low-rank-sparse subspace rep- resentation for robust regression. In IEEE CVPR, pages 2972-2981, 2017. 36. [Zhao and Zhou, 2018] Peng Zhao and Zhi-Hua Zhou. Label distribution learning by optimal transport. In AAAI, pages 4506-4513, 2018.