Common-Individual Semantic Fusion for Multi-View Multi-Label Learning

1 Introduction

Multi-View Multi-Label Learning (MVML) learns from the training data, where each object is represented by several heterogeneous feature representations and associated with multiple class labels simultaneously [Bickel and Scheffer, 2004; Wu et al., 2019; Wu et al., 2020; Lyu et al., 2022a]. Recently, such learning paradigm has been widely used in many realworld applications. For example, in the task of news webpage classification, one news webpage can be represented by diverse channel information including video, image and text, while at the same time it is annotated with multiple class labels such as FIFA World Cup, Sports and Lionel Messi. Multiview multi-label learning provides an effective framework to learn a desired multi-label classifier from such heterogeneous data and predict proper labels for unseen examples.

∗Zhen Yang and Songhe Feng are the corresponding authors.

The key to deal with multi-view multi-label data lies in how to effectively integrate these heterogenous features while ensuring all relevant labels can be characterized comprehensively. A general strategy is to learn a latent multi-view subspace representation, which can characterize multiple semantic labels as much intact as possible. [Zhang et al., 2018a] proposes a matrix factorization based shared subspace representation method, which employs Hilbert-Schmidt independence criterion to strengthen its ability of consensus semantic characterization. [Zhao et al., 2023] also seeks for a latent low-dimensional representation, while it focuses on each specific view and employs the structural view-label consistency information to enhance the expressions of view-specific semantics. [Lu et al., 2023] proposes a bipartite graph based multi-view embedding representation method, and it imposes a joint low-rank constraint on both the embedding representation matrix and multi-label classifier matrix to enhance its robustness toward the labels with dependencies. Obviously, the above MVML methods just leverage the multi-view consensus and complementary relationship in the feature space and they formulate multi-view subspace as implicit semanticaware representation.

In this paper, different from traditional feature-level multiview fusion MVML methods, we take the first attempt to conduct multi-view fusion under the guidance of semantic fusion, and propose a Common-Individual Semantic Fusion MultiView Multi-Label Learning Method (CISF). Specifically, we first assume each view corresponds to one view-specific label set and each of them contains two different kinds of semantics - Common Semantics & Individual Semantics. The common semantics refers to the core semantic labels shared by multiple views, which reflects the consensus information across different views. The individual semantics refers to the exclusive semantic labels owned by each specific view, which characterizes the complementary information of diverse views. Afterwards, we separately introduce the low-rank and sparse constraint to highlight the label co-occurrence relationship of the common semantics and the exclusive semantic representation of the individual semantics. Finally, we embed an adaptive global label correlations to enhance the semantic integrity for improving the performance of the final multi-label model.

In summary, the main contribution of our paper lie in the following aspects:

• We propose a new Common-Individual Semantic Fusion Multi-View Multi-Label Learning Method (CISF). To the best of our knowledge, it is the first attempt to directly leverage multi-view fusion on the semantic space.
• Considering that single common semantics cannot characterize all relevant labels, we simultaneously consider the commonality and individuality of multi-view data, and introduce an adaptive global label correlations to enhance the semantic integrity of the final model.
• We provide theoretical guarantee for the strict convexity of CISF by properly setting parameters and develop an alternative optimization algorithm to solve it. Extensive results have verified the superiority of our method.

2 RelatedWork

2.1 Multi-View Learning (MVL)

Multi-View Learning aims to learn a desired multi-view representation from different views by leveraging the consensus and complementary information across heterogeneous features [Bickel and Scheffer, 2004; Wang et al., 2016; Zhang et al., 2017; Gu et al., 2023]. A core purpose of multi-view learning is to encapsulate multi-view information of different views to learn a share or common representation for clustering. Based on the way to exploit multiview information, existing multi-view learning methods can be roughly divided into the following categories: canonical correlation analysis [Andrew et al., 2013; Wang et al., 2015], multi-view subspace clustering [Gao et al., 2015; Cao et al., 2015; Kang et al., 2020; Wang et al., 2021], multi-view matrix factorization [Liu et al., 2013; Zhao et al., 2017], and deep multi-view clustering [Li et al., 2019; Xu et al., 2023]. Besides, there are also many other multiview learning methods for different tasks, such as retrieval [Yan et al., 2020], recommendation [Flanagan et al., 2021] and classification [Han et al., 2022; Lyu et al., 2022a], etc.

2.2 Multi-Label Learning (MLL)

In Multi-Label Learning, each instance is represented by a single feature vector and annotated with multiple valid labels [Wen et al., 2022]. Label correlation is a fundamental challenge to be utilized for improving the performance of multilabel learning. Based on the order of label correlations being exploited for model training, existing MLL methods can be roughly grouped into three categories: first-order correlations [Zhang et al., 2018b], second-order correlations [Madjarov et al., 2010; Li et al., 2017] and high-order correlations [Burkhardt and Kramer, 2018]. The above methods mainly are formulated under full supervised settings while such phenomenon may not hold in real-world scenarios due to expensive annotation efforts. Recently, some weakly supervised MLL frameworks are proposed and have been widely used in many applications, such as semi-supervised MLL [Wang et al., 2020], MLL with missing labels [Zhu et al., 2018], partial multi-label learning [Lyu et al., 2020; Li et al., 2021; Lyu et al., 2022b; Wang et al., 2023], etc.

2.3 Multi-ViewMulti-Label Learning (MVML)

In Multi-View Multi-Label Learning, each instance is represented by several heterogeneous features and associated with multiple valid labels [Lyu et al., 2024]. Obviously, such paradigm can be regard as an integration of multiview learning and multi-label learning, and the key to deal with MVML data lies in how to integrate heterogeneous features effectively while realize multi-label classification accurately. [Zhang et al., 2020] propose a sparse feature selection MVML method, which exploits both view relations and label correlations to select discriminative features for further multi-label model training. [Wu et al., 2019] propose a viewspecific MVML method named SIMM, which simultaneously leverages shared subspace exploitation and view-specific information extraction to enhance the performance of multilabel classifier. Except for the above MVML methods, there are also some weakly supervised MVML methods, including MVML with missing labels [Huang et al., 2019], MVML with missing views [Tan et al., 2018], non-aligned MVML [Zhao et al., 2023; Zhong et al., 2024], etc.

3 The ProposedMethod

3.1 Notations

Formally speaking, we denote X = Rd1 ×Rd2 . . . ×RdV as the feature space with V views and Y = {c1, c2, . . . , cq} as the label space with q class labels, where dt (1 ≤ t ≤ T) is the feature dimension of t-th view. Given the multi-view multi-label training data D = {(Xi, yi)|1 ≤ i ≤ n} with n instances, where each Xi ∈ X is represented by V feature vectors [x(1) i ; x(2) i ; . . . ; x(V ) i ] and yi ∈ {0, 1}q×1 is the label vector associated with Xi, our proposed CISF aims to integrate these heterogeneous representations from different views to construct a robust multi-label classifier f : X ↦→ 2Y and further predicts some proper labels for unseen instances.

3.2 Formulation

Consistences and complementarities are two key ingredients for boosting multi-view multi-label learning. Existing MVML methods generally advocates different view to predict the same label results to use consistent information across different views and simultaneously considers the different contribution weights of each specific view to learn the complementary information among different views. Eq. (1) illustrates a general MVML framework:

where Y ∈ Rq×n is the label matrix, X(i) ∈ Rdi×n is the feature matrix of i-th view, W (i) ∈ Rq×di is the mapping matrix from features to semantics, Γ(·) is the regularization term and µ(i) is the weight of i-th view.

According to Eq.(1), we can easily see that existing MVML methods mainly formulate multi-view consistent and complementary information in the feature space. Basically, they target to seek one or several good view that can characterize all relevant labels as completely as possible. Obviously, such multi-view fusion strategy cannot respect every view sufficiently and even lead the semantic representation of some rare labels to be overwhelmed by the core labels.

Hence, in this paper, we directly take the multi-view fusion in the semantic space and explicitly measure multi-view consistences and complementarities in objective fusion model. Specifically, we assume that each view corresponds to a viewspecific label set Y (i) ∈ Rq×n and each label set contains two different kinds of semantics - Common Semantics C(i) ∈ Rq×n and Individual Semantics D(i) ∈ Rq×n , i.e.,

The common semantics C(i) ∈ Rq×n×V refers to the shared semantic labels represented by all V views and it reflects the consensus semantic information across different views. In our model, we introduce Hilbert-Schmidt Independence Criterion [Gretton et al., 2005] to constrain the semantic consistence of the V different views, i.e.,

The individual semanticsD(i) ∈ Rq×n×V refers to the exclusive semantic labels owned by each specific view, which characterizes the diversities and complementarities among different views. Based on the assumption that the individual semantics is exclusive for each specific view and the diversity is also sparse across different views, we measure such semantic diversities and complementarities by minimizing the sum of the product of each pair of individual semantics, i.e.,

Except for the above semantics consistences and complementarities, another inherent property of learning frommultiview multi-label data is how to utilize label correlations. Different from previous fixed label co-occurrence relationships, we try to leverage a dynamic label correlations S ∈ Rq×q and recover all relevant labels by minimizing

In addition, in order to construct the direct correspondences from features to semantics and obtain a desired multi-label classifier for unseen examples prediction simultaneously, we introduce two feature mapping matrices W (i) C ,W (i) D ∈ Rq×di that correspond to the common semantics C(i) and the individual semantics D(i) respectively, i.e., C(i) =

W (i) C X(i) and D(i) = W (i) D X(i) . By integrating the above functions (2)-(5), we can obtain the final framework of our proposed CISF method as follows:

Here, the common feature mapping matrix W (i) C is constrained with nuclear norm to preserve its low-rank property, since its represented shared semantic labels tend to have statistical co-occurrence. The exclusive feature mapping matrix is constrained with F-norm, since the diversities of exclusive semantics are always expressed as sparse. The weights µ(i) are defined by inverse distance weighting strategy to avoid undesired hyperparameters [Nie et al., 2016].

3.3 Optimization

To optimize (6) conveniently, we introduce an additional variable constraint A(i) = SW (i) C X(i) and convert (6) to its Augmented Lagrange Multiplier (ALM) form as follows:

Obviously, the above function involves four variables W (i) C , W (i) D , A(i) and S, which cannot be optimized simultaneously. Therefore, we adopt the alternating minimization strategy and update these variables iteratively.

Update W (i) C with other variables fixed. We can calculate W (i) C by minimizing the following objective function:

where Θ = 2µ(i) (Y - SW (i) D X(i) ) + (λ(i)C(i) +M(i) ) and ζ (i) = (2µ(i) + λ(i) )/2. According to [Zhu et al., 2010], (8) has the closed form solution and the variable W (i) C can

Algorithm 1 The Training Process of CISF

Inputs:

D : MVML training data {(x(v) i , yi)|i ∈ [n], v ∈ [V ]};

α, β, γ and η: the trade-off parameters;

Imax : the number of maximum iterations;

x(i) ∗ : the unseen example.

Process:

1. while t < Imax do
2. for i = 1, 2, . . . , V do
3. UpdateW (i) C by solving (8);
4. end for
5. for i = 1, 2, . . . , V do
6. UpdateW (i) D by solving (9);
7. end for
8. for i = 1, 2, . . . , V do
9. Update A(i) by solving (10);
10. end for
11. Update S by solving (11);
12. for i = 1, 2, . . . , V do
13. UpdateM (i) and λ(i) by Eq. (12);
14. end for
15. if converge then
16. break;
17. end if;

19. end while;

Output:

be optimized following W (i) C = S γ 2ζ(i) ( Θ 2ζ (i) ), where S is the singular value thresholding.

Update W (i) D with other variables fixed. The variable W (i) D can be updated following:

We take the derivative of (9) with respect toW (i) D to 0. Afterwards, based on KKT conditions, we can easily update W (i) D in an iterative manner [Tan et al., 2021].

Update A(i) with other variables fixed. The optimization subproblem with regard to A(i) can be reformulated as:

Here, HSIC(A(i) , A(j) ) = (n - 1)-2T r (HK(i)HK(j) ), K(i) = A(i)⊤A(i) is the Gram matrix and H centers it to have zero mean. Theorem 1 (in section 5.3) guarantees the subproblem (10) to be convex and the optimal solution could be obtained by setting its derivative with respect to A(i) to 0. Update S with other variables fixed. The variable S can be updated by solving the following sub-problem:

Similar to (9), we also take the derivative of (11) with respect to S to 0, and then we can obtain its closed-form solution.

UpdateM (i) and λ(i) with other variables fixed. Finally, we update the Lagrange multiplier matrices M(i) and penalty scalars λ(i) following:

During the process of model training, we first initialize the required variables, and then repeat the above steps until the algorithm converges or reaches the maximum iterations. Finally, we make prediction for unseen instance following y∗ = ∑Vi=1 µ(i)S(W (i) C +W (i) D )x(i) ∗ . Algorithm 1 summarizes the whole procedure of our proposed CISF method.

4 Experiments

4.1 Experimental Setting

To evaluate the performance of our proposed CISF method, we implement experiments on seven widely-used MVML data sets, including Emotions, Scene, Corel5k, Espgame, Pascal, Iaprtc12 and Mirflickr data sets. Table 1 summarizes the detailed characteristics of the above data sets.

Meanwhile, we compare our proposed CISF with the following five state-of-the-art MVML methods, including LSPC [Szymanski et al., 2016], FIMAN [Wu et al., 2020], ICM2L [Tan et al., 2021], BEMVL [Lu et al., 2023] and NAIM3L [Li and Chen, 2022]. The configured parameters of the above methods are set according to the suggestions in their corresponding literature.

In addition, five popular multi-label evaluation metrics are employed to measure the performance of each comparing method, including Hamming Loss (H-L), Ranking Loss (R-L), One Error (O-E), Coverage (COV) and Average Precision (AP) [Zhang and Zhou, 2013]. For each dataset, we randomly select 70% examples for training, 10% examples for parameter tuning and 20% examples for evaluation, where each algorithm is run 5 times independently. The codes and data sets are provided in https://gengyulyu.github.io/homepage/.

4.2 Experimental Results

Table 2 illustrates the experimental comparisons between our proposed CISF and other five comparing methods on all evaluation metrics, where the average metrics results and standard deviations are recorded respectively. According to Table 2, out of 210 (7 data sets × 6 methods × 5 metrics) statistical comparisons can make the following observations:

• Among all five comparing methods, our proposed CISF method is superior to LSPC, FIMAN and ICM2L in almost all cases, and it also outperforms BEMVL and NAIM3L in 91.42% and 88.57% cases, respectively.
• Among all employed evaluation metrics, our proposed CISF achieves the best performance in 97.14% cases on Hamming Loss, Ranking Loss and Average Precision metrics. And on One Error and Coverage metrics, it is also superior to other methods over 94% cases.
• Among all employed datasets, CISF outperforms almost all comparing methods on Emotions, Scene, Pascal, Iaprtc12 and Mirflickr datasets. And it also achieve superior performance against other comparing methods over 82% cases on Corel5k and Espgame data sets.
• Overall, our proposed CISF method can achieve competitive performance against other feature-fusion based MVML methods, which demonstrates the effective of our proposed multi-view semantic-fusion strategy.

In order to comprehensively evaluate the superiority of CISF, Friedman test [Demˇsar, 2006] is utilized as the statistical test to analyze the relative performance among the comparing algorithms. According to Table 3, the null hypothesis of distinguishable performance among the comparing algorithms is rejected at 0.05 significance level. Thus, we further employ the post-hoc Bonferroni-Dunn test [Demˇsar, 2006] to show the relative performance among the comparing algorithms. Figure 3 illustrates the CD diagrams on each evaluation metric, where the average rank of each algorithm is marked along the axis. According to Figure 3, it is observed that CISF always ranks 1st on all evaluation metrics.

5 Further Analysis

5.1 Ablation Study

In order to evaluate the effect of the each components of our proposed CISF, we conduct the Ablation Study between CISF and its three degenerated algorithms CISFnC, CISFnI and CISFnL, where each degenerated algorithm ignores the common semantics, individual semantics and label correlations, respectively. Table 4 reports the experimental comparison between these methods on Emotions data set. According to Table 4, we can find that CISFnI achieves better performance than CISFnC method, which shows that common semantics has greater contribution than individual semantics to the effectiveness of learning model. Besides, our proposed CISF is superior to both CISFnC and CISFnI methods, which demonstrates that the common semantics and individual semantics can jointly improve the performance of MVML model.

5.2 Parameter Sensitivity

We study the sensitivity analysis of our proposed CISF with respect to its four employed parameters α, β, γ and η. Figure 4 shows the performance of CISF under different parameter configurations on Emotions data set. According to Figure 4, we can find that α and γ usually have great influence on the performance of the proposed model, and we select the optimal values of them from {10-3 , 10-2 , . . . , 102 } and {0.01, 0.05, . . . , 10}, respectively. Meanwhile, other parameters often follow the optimal configurations β = 0.1 and η = 100 but vary with minor adjustments on different data sets. In addition, in our experiments, the value of λmax is set to 1e6 and the maximum iterations Imax is set to 50.

5.3 Complexity Analysis

At each iteration, the computational cost mainly comes from the derivative calculation and singular value decomposition

(SVD) operations, and the complexity of our optimization procedure mainly comes from the optimization of three subproblems with regard to W (i) C , W (i) D and A(i) . For simplicity, we suppose the dimensionality of each view is d. And the complexity of these sub-problems are O(dnr), O(Tnqd) and O(q2d+qdn+q3 ) respectively, where r is the rank of W (i) C and T is the iteration number for updatingW (i) D .

5.4 Convergence Analysis

The convergence of the whole optimization problem (6) depends on how to guarantee the subproblem (10) is convex, especially its HSIC term is negative. Theorem 1 provides the theoretical guarantee for the convexity of (10) under proper parameter setting. Besides, Figure 5 illustrates the convergence curves on Emotions and Scene data sets, which also empirically demonstrates the convergence of our model.

Theorem 1: The problem (10) is convex given the parameter λ(i) ≥ 8q(V -1)αµiµj , where V is the number of views. Proof: Due to the page limitation, we provided the proof of Theorem 1 in https://gengyulyu.github.io/homepage/.

6 Conclusion

In this paper, we proposed a Common-Individual Semantic Fusion Multi-View Multi-Label Learning Method. Different from previous feature-fusion basedMVMLmethods, it is the first attempt to conduct multi-view fusion under the guidance of the semantic fusion, where both common semantics and individual semantics are simultaneously incorporated into the multi-view fusion process to learn a desired multi-label classification model. Extensive experimental results on various MVML datasets has demonstrated the effectiveness of our proposed multi-view semantic-fusion strategy.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No. 2023YFB3107100), the National Natural Science Foundation of China (No. 62306020), the China Postdoctoral Science Foundation (No. 2022M720320), the Beijing Postdoctoral Science Foundation (No. 2023-zz-78), the Chaoyang Postdoctoral Science Foundation, the Fundamental Research Funds for the Central universities (No. 2022JBZY019), the Beijing Natural Science Foundation (No. 4242046), the Major Research Plan of National Natural Science Foundation of China (No. 92167102).