Weakly-Supervised Multi-view Multi-instance Multi-label Learning

1 Introduction

Multi-viewMulti-instance Multi-label (M3) objects (or bags) are characterized by heterogeneous feature views, including diverse instances, and are simultaneously annotated with nonexclusive labels. For example, in Figure 1, a video is represented by text and image views, where each text (image) bag includes diverse instances (paragraphs or animals) and is annotated with several semantic labels (e.g., seagull, water, and sky). Multi-view Multi-instance Multi-label Learning (M3L) [Nguyen et al., 2013] can simultaneously model bags, instances of bags, and their non-exclusive labels to learn a predictive model to project multiple views of bags (and instances) into the label space, which reflects the semantic meaning of the bags (instances). Due to its capability of modeling complex objects in the real-world, M3L has attracted increasing research interest [Yang et al., 2018; Xing et al., 2019].

∗Corresponding author, guoxian85@gmail.com. This work is supported by NSFC (61872300 and 61873214).

Traditional M3L approaches typically assume that the entire data is mapped across views, and the label annotation of objects is complete. Both assumptions are often violated in practical M3L tasks. As an example, in Figure 1, the mapping of a given bag across different views is only partially given. Moreover, the bags have missing annotations, and the number of bags in two views is different. In fact, such weakly-supervised multi-view data are universal in many domains. For example, for medicine development, the relation of a pill and its compounds with the therapy (adverse) effects is typically partially known. However, to the best of our knowledge, none of the existing M3L methods has studied the partial correspondence of M3 data. The incomplete annotation problem [Xu and Zhou, 2017; Tan et al., 2018; Xing et al., 2018] has also not been investigated. We term these two types of information as weakly-supervised information, which restricts the effectiveness and application, or even the adaption, of existing M3L approaches.

To address the weakly-supervised M3 problem, we introduce a weakly-supervised M3L approach (WSM3L) based on multimodal dictionary learning [Mandal and Biswas, 2016; Liu et al., 2018a]. WSM3L introduces a unified objective function to seek the matches between bags across multiple views and to predict labels of bags. It uses the heterogeneous features of bags to learn a multi-view coordinated dictionary (representation space) and individual encoding vectors of bags for each view. Then the feature similarity derived from the encoding vectors and label similarity of bags are leveraged to seek matches between bags across views. Besides, it jointly replenishes the labels of a bag using the labels of its neighborhood bags, distributes the labels of bags to instances, and reversely aggregates the labels of instances to their originating bags. In this way, WSM3L can predict the labels of bags, and also the labels of instances in a coherent fashion. The main contributions of this work are as follows:

(i) WSM3L can handle not only weakly-paired (or even completely-unpaired) bags across views, but also partially annotated training bags. To the best of our knowledge, none of the existing M3L approaches can simultaneously make well usage of these two types of weakly-supervising information. (ii) A matching solution based on labels and features of bags is introduced to discover their correspondence across views. We also introduce a unified objective function to seek the match between bags, to replenish missing labels of bags, to push the bag-level labels to instances, and reversely aggregate the labels of instances to their affiliated bags in a coordinated fashion. (iii) WSM3L significantly outperforms state-of-the-art M3L approaches [Nguyen et al., 2014; Li et al., 2017; Xing et al., 2019], multi-instance multi-label weak-label learning [Yang et al., 2013], and weakly-paired multi-modal learning [Lampert and Kr¨omer, 2010; Liu et al., 2018a] in different practical settings. In addition, WSM3L can work in open settings (i.e., with different numbers of bags across views and with completely unpaired multi-view bags), in which the competitive methods cannot be applied.

2 RelatedWork

Multi-instance multi-label learning (M2L) [Zhou et al., 2012; Huang et al., 2019] deals with complex interrelations between bags, instances, and labels. M3L is more difficult, and less well-studied, than M2L, due to the additional heterogeneous feature views and complicated correlations across views. [Nguyen et al., 2013] introduced a Latent Dirichlet Allocation [Blei et al., 2003] based M3L approach, which separately explores the visual-label topics from the visual view and the text-label topics from the text view, and then performs prediction by forcing the label consistency between the two views. [Nguyen et al., 2014] proposed an M3L approach (MIMLmix) that uses a hierarchical Bayesian network and variational inference to leverage multiple feature views. [Li et al., 2017] developed a multi-view multiinstance learning (M2IL) algorithm, which considers different intrinsic structures between instances of a bag across views, and exploits sparse representation [Rubinstein et al., 2010] and multi-view dictionary learning [Wu et al., 2016; Gao et al., 2015] for bag-level label prediction. [Yang et al., 2018] introduced a deep neural network based approach, which separately applies a deep network for each view, and keeps the bag-level predictions across views consistent. Furthermore, a semi-supervised deep M3L approach [Yang et al., 2019] is introduced to leverage label correlation and unlabeled instances for bag-level prediction. The aforementioned M3L approaches only consider limited types of inter-relations and intra-relations between bags, and between instances and labels, which in fact carry important contextual information for M3L to explore. [Xing et al., 2019] recently introduced a collaborative matrix factorization based solution (M3Lcmf), which first constructs multiple inter(intra)-relational data matrices of bags, of instances, and of labels, to capture diverse intrinsic relations among them, and then collaboratively factorizes the matrices into low-rank ones to merge them and to coherently predict the bag(instance)-label associations.

The above M3L solutions optimistically assume that bags are completely paired across heterogeneous views, and are also comprehensively annotated. However, these two assumptions are often violated in practical M3L scenarios. Our study expands the flexibility and capability of M3L by designing a weakly-supervised M3L approach (WSM3L).

3 ProposedMethod

Without loss of generality, we assume bags (or instances) have V feature views, and each view has nv bag sets X v = {Xv1 ,Xv2 , · · · ,Xvnv }. Xvi = [xvi,j ]mvi j=1 , a matrix, denotes the i-th bag in the v-th view includes mvi ≥ 1 instances, where xv ∈ Rdv (v = 1, 2, · · · , V ) is the feature space of instances in the v-th view. Yv i ∈ Rq encodes the currently known labels of Xvi . Yv iq′ = 1 if Xvi is annotated with the q′ -th label, Yv iq′ = 0 otherwise. All bags belong to the same label space and paired bags share a same subset of labels. For an M3 dataset with completely paired bags, {Yv }Vv=1 is identical across all the views, but not so for an M3 dataset with weaklypaired (completely-unpaired) bags. The task of M3L is to learn a predictive function f({Xv }Vv=1 , {Yv }Vv=1 ) → Rq .

The correspondence between bags in M3L is the basis for multi-view data fusion. For weakly-paired M3 data, a simple bypass solution is to exclude unpaired bags and only use the known paired bags across views to train the predictive model. However, these excluded bags (and their member instances) also convey important context information for the task, and disregarding them may distort the underlying data distribution. To make use of as many bags as possible, we first seek matches between bags across views. Different techniques [Zhang et al., 2015; Mandal and Biswas, 2016] can be used to this end, and here we adopt multi-modal dictionary learning [Monaci et al., 2007], which has been successfully adopted to capture and correlate heterogeneous features across modalities [Mandal and Biswas, 2016; Liu et al., 2018b]. The multi-modal dictionary learning technique provides an effective strategy to unify multi-modal data, since each view can be generated from the shared dictionary with individual encoding vectors. As such, the heterogeneous feature vectors are reformulated as comparable encoding vectors. Multi-modal dictionary learning on two feature views [Monaci et al., 2007; Liu et al., 2018a] is formulated as follows:

where D1 ∈ Rd1×d and D2 ∈ Rd2×d are the dictionaries, and E1 ∈ Rd×n1 and E2 ∈ Rd×n2 are the coding matrices of the two views, respectively. d is the dictionary size, which can be specified by the designer. The constraint term C(E1E2 ) has different forms [Mandal and Biswas, 2016], and it can be used to incorporate inter(intra)-modal relations.

3.1 Matching Bags Across Views

To explore the complementary information across views and the matches between bags, we first learn a shared dictionary for bags across views, which also gives a unified representational space for bags. In addition, we seek an encoding matrix of bags per view. Since the same bag may have different number of instances in different views, we first project the instances' features of a bag onto a bag feature vector like [Zhou and Zhang, 2007] for dictionary learning. Thus, Xv used in the following equations is a matrix storing the projected features of bags in the v-th view. To learn a shared dictionary, we project the feature views onto the same dimensional space: (Pv )T Xv , where Pv ∈ Rdv×s is the projection matrix of the v-th view with Pv (Pv )T = I ∈ Rs×s . We then use (Pv )T Xv to seek the shared dictionary and the encoding matrix of bags for each view as follows:

where D ∈ Rs×d is the shared dictionary of bags across views, and ds′ ∈ Rd is the dictionary vector of D. Ev ∈ Rd×nv is the coding matrix of bags (and instances therein) of the v-th view. In this way, bags across different views are comparable in the representational space, which is configured by the shared dictionary. Mvw ∈ Rnv×nw records the mapping information between bags of the v-th view and w-th view. The term ∑Vv=1,w̸=v ‖ (Ev )T -Mvw (Ew )T ‖2F is introduced to force matched bags having similar encoding vectors.

Existing multi-modal learning methods match objects across views solely using the features [Lampert and Kr¨omer, 2010; Mandal and Biswas, 2016], or labels of objects [Liu et al., 2018b]. In contrast, we leverage both label and feature information to improve the matching process. To match bags across views, we leverage the label and feature information of pairwise bags (Xvi and Xwj ) as follows:

where fea(Evi ,Ewj ) and lab( ˜Yv i , ˜Yw j ) are the feature-based and label-based similarity between Xvi and Xwj , respectively. Two bags may be annotated with the same set of labels, which give a lab( ˜Yv i , ˜Yw j ) = 1 and result in a large match score m(Xvi ,Xwj ). However, these two bags may not be the best match, since they may have a moderate feature similarity. Given that, we add a small constant ϵ = 0.01. The larger the feature-based and label-based similarities, and the more consistent between these two similarities, the more likely these two bags will be matched. To quantify the label and feature similarities between bags, we use the cosine similarity for its simplicity and effectiveness, other similarity metrics can also be used here. Since the feature and label vectors of our used datasets are all nonnegative, thus our cosine similarity actually locates in [0,1].

Based on m(Xvi , Xwj ), we can specify the matching matrix Mvw between bags of the v-th and w-th views as follows:

where p encodes the previously known matched information of bags across views, p(Xvi ,Xwj ) = 1 if Xvi and Xwj are known paired; p(Xvi ,Xwj ) = 0, otherwise. The first condition shows two matched cases of Xvi and Xwj : (i) Xvi and Xwj are known matched in advance (i.e., p(Xvi ,Xwj ) = 1); (ii) Xvi and Xwj are calculated to have the maximum value of m(Xvi ,Xwj ). If Xvi and Xwj meet the first condition, we set Mvw ij = 1; Mvw ij = 0, otherwise. As such, WSM3L can not only incorporate the known paired bags to deal with weaklypaired bags, but also deal with completely-unpaired bags, by leveraging feature and label similarities of bags.

3.2 Replenishing Labels of Bags

Most existing M3L approaches typically assume complete label annotations of bags, i.e., no missing labels. However, in practice, the annotation is indeed incomplete. Since each feature view has its distinctiveness and bags across views are only partially paired, we first replenish the missing labels of bags per view. We assume that missing labels of a bag can be replenished based on the labels of its neighborhood bags as follows:

where ˜Yv ∈ Rnv×q represents the replenished label sets of bags in the v-th view. Av ∈ Rnv×nv is the adjacency matrix of the k nearest neighborhood (kNN) graph of bags in the v-th view, and it's specified as follows:

where Xvi ∈ Nk(Xvj ) is one of the k nearest neighbors of Xvj , and the neighborhood relationship between bags is determined by the cosine similarity.

3.3 Distribution and Aggregation of Labels

In multi-instance learning, a bag includes one or more instances, and its label set depends on the labels of its instances

[Zhou et al., 2012]. Multi-instance learning typically uses the bag-instance relations to predict the labels of bags; some approaches can also identify the labels of instances [Carbonneau et al., 2018]. To perform label prediction for both bags and instances, we introduce a term to distribute the labels of bags to instances, and reversely aggregate instance-level labels for bags, as shown in the following:

where ˜Yv are the replenished label sets of bags in the v-th view. Unlike Eq. (6), the computation of ˜Yv is coordinated with the matched bags (viaMvw ) from other views. Λv ∈ Rnv×nv is a diagonal matrix with Λv (i, i) = 1/mb(v, i), where mb(v, i) counts the number of matched bags of Xvi , including itself. Rw ∈ Rnw×mw stores the inter-associations between nw bags and mw instances in the w-th view. Rw (i, j) = 1 if the i-th bag includes the j-th instance; Rw (i, j) = 0, otherwise. Fv ∈ Rmv×dv stores the feature vectors of instances of the vth view, andWv ∈ Rdv×q is the projection matrix for the v-th view, Zv ∈ Rmv×q is the predicted label matrix of instances in the v-th view, which can be obtained by Fv and the optimized Wv . As a result, our proposed WSM3L makes predictions for instances, and also aggregates instance labels at the bag-label. Meanwhile, it combines the replenished labels of bags across views.

3.4 The Unified Objective Function

To coordinate the match between bags across views and label replenishment, and to coherently dispatch the bag-level labels to instances and aggregate the instance-level labels onto bags, let Ω = {D, Ev , Pv , ˜Yv ,Wv }, we formulate a unified objective function as follows:

where L1 aims to control the data fidelity and to explore match across bags. L2 and L3 target to replenish and predict the labels of bags at bag-level and instance-level, respectively. Notice that L1 and L3 share the same match information. The parameter α balances the importance of L1 and the latter two terms. Eq. (8) makes the potential match between bags across views, label replenishment, bag-level and instance-level label prediction in a coordinated fashion. Thus, both the weakly-paired bags and incomplete labels of weakly-supervised learning on M3 data are jointly accounted for.

To compute D, Ev , Pv , ˜Yv and Wv , we adopt an alternative optimization technique following the idea of the alternating direction method of multipliers (ADMM) [Boyd and Vandenberghe, 2004]. Since directly optimizing the discrete indicator match matrix Mvw is NP-hard, we update it based on the updated Ev , ˜Yv in each iteration. Suppose t is the maximum number of iterations, the time complexity of our model is O(tV [sdnv +V d(nv)2 +d2nv +mvq +V (nv)2mv]). Our preliminary study shows that WSM3L generally converges within 50 iterations on the used datasets. We give the optimization procedure as a supplementary file.

To this end, WSM3L predicts the labels of a new bag Xh by integrating the aggregated labels from its instances and the known labels of it neighborhood training bags across views (if any) as follows:

where V(Xh) collects the observed views of Xh, xvh,j represents the j-th instance feature vector of Xvh , andWv is the optimized coefficient matrix for instance-label prediction in the v-th view. The first term targets to aggregate the prediction from instance-level, and the second term aims to integrate the prediction from neighborhood training bags across views.

4 Experiments

4.1 Experimental Setup

We design experiments to study the performance of WSM3L on completely-paired bags, weakly-paired bags and completely-unpaired bags across views, respectively. We collect eight publicly available multi-instance multi-label datasets and one real M3 dataset from different domains for the experiments. The details of these datasets are listed in Table 1. The first four datasets 1 and Isoform dataset [Yu et al., 2020] are used to evaluate the predicted labels of bags, since baglevel labels are available one. The last four datasets have instance-level labels for evaluation [Briggs et al., 2012].

To evaluate the effectiveness of the proposed WSM3L, four widely-used multi-label evaluation metrics are adopted to evaluate the performance from different perspectives, including Hamming Loss (HL), Ranking Loss (RL), Average Precision (AP), and macro AUC (Area Under receiver operating Curve) (mAUC). Due to the page limit, the formal definition of these metrics is omitted here and can be found in [Zhang and Zhou, 2014]. The smaller the values of HL and RL, the better the performance is. As such, to be consistent with the other evaluation metrics, we report 1-RL and 1-HL instead. For the latter metrics, larger values indicate better performance.

4.2 Results on Completely PairedMulti-viewData

We randomly select 70% of the bags of a dataset to train the model, and use the remaining 30% for testing. For the eight multi-instance multi-label datasets, we randomly divide the original features of each bag into two sets of equal size, each providing one view. We then randomly mask 30% of the label information of each bag in the training set, to study the performance of WSM3L on bags annotated with incomplete labels. For multi-view methods (i.e., MIMLmix [Nguyen et al., 2014], M2IL[Li et al., 2017] and M3Lcmf[Xing et al., 2019]), we use the same datasets as our method, and for MIMLwel[Yang et al., 2013], we directly use the collected datsets. Besides, the input parameters of all comparing methods used in this paper are specified (or optimized) as suggested by the authors in their papers or shared codes. For reference, we also report the results of WSM3L(cL), which does not mask any label but uses complete labels. The input parameters of WSM3L are set as follows: d = 160, s = 150, k = 30 and α = 1. Tables 2 and 3 report the results of the comparing methods on baglevel and instance-level label prediction, respectively. Only MIMLmix and M3Lcmf can make instance-level prediction, so Table 3 does not report results of other compared methods.

1 http://lamda.nju.edu.cn/CH.Data.ashx

Our proposed WSM3L generally outperforms the comparing methods across different datasets and evaluation metrics, on both the bag-level and instance-level label prediction tasks. We used the signed-rank test [Demˇsar, 2006] to check the significance of the results between WSM3L and the other methods, and all the p-values are smaller than 0.037. WSM3L frequently outperforms other M3L methods, which shows the effectiveness of WSM3L on completely paired multi-view datasets. Both M2 IL and WSM3L learn a shared dictionary across views, and WSM3L performs much better than M2 IL. This observation shows that WSM3L can learn a more adaptive dictionary. WSM3L outperforms MIMLwel which demonstrates the effectiveness of WSM3L on replenishing the labels of bags. WSM3L obtains a slightly lower performance than WSM3L(cL), which indicates the effectiveness of WSM3L on replenishing labels and also suggests WSM3L is not so sensitive to missing labels of training bags.

The results on instance-level prediction again expresses the effectiveness of the proposed WSM3L in distributing the bag-level labels to instances, which in turn boosts the accuracy of bag-level label prediction. WSM3L sometimes loses to M3Lcmf and MIMLmix on the Birds dataset. The possible reason is that each bag in Birds has a large number of instances, WSM3L does not concretely use the relations between instances or labels as these compared methods, which boost the performance but result in a more complicated model.

4.3 Results onWeakly-pairedMulti-viewData

Based on the previous 70-30% split, we simulate three settings for weakly-supervised M3 data. In the first setting, we randomly mask the correspondence between 30% of the training bags, and then randomly remove 30% of the labels of the training bags. The second setting is the same as the previous one, with the additional removal of 30% of the bags in one view, to investigate the flexibility of WSM3L when different numbers of bags are present across views. In the third setting, we completely mask all the mappings between bags across views. For the last two settings, none of the comparing methods in Table 2 can be applied. We report the results of WSM3L(dB) and WSM3L(uB) in the last two columns of Table 4. WSM3L(dB) and WSM3L(uB) correspond to the case with different numbers of bags across views and to the case with completely unpaired bags across views, respectively. For a comprehensive comparison, two multi-view dictionary learning methods for weakly-paired data, WMCA (weakly-paired maximum covariance analysis) [Lampert and Kr¨omer, 2010] and MFCDL (Multimodal Fusion via Common Dictionary Learning) [Liu et al., 2018a] are also included for comparison in the first setting. Since WMCA does not provide the label likelihoods as other comparing methods, which are required by 1-RL and mAUC, only the results of 1-HL and AP are reported in Table 4.

We have the following observations:

(i) In the first setting, all comparing methods use all the training bags, and WSM3L achieves the best performance and holds comparable results with itself on completely paired bags in Table 2. This observation shows the effectiveness of WSM3L on learning from weakly-paired M3 data. Both WSM3L, WMCA and MFCDL can work on weakly-paired multi-view data. WMCA adopts the maximum covariance analysis to match bags. WSM3L achieves a better performance than WMCA. This observation shows the effectiveness of WSM3L on handling multi-view weakly-paired data. Both WSM3L and MFCDL learn a shared dictionary of multiple views, WSM3L outperforms MFCDL, which facts the effectiveness of WSM3L on matching bags across views.

(ii) In the second setting, WSM3L(dB) operates with training data where some bags are missing in one view. WSM3L(dB) obtains a slightly lower performance compared to WSM3L in the first setting. This result shows that WSM3L can also work well in the case of bags with a different number of feature views.

(iii) In the third setting, WSM3L(uB), which operates on completely unpaired bags, achieves a performance comparable to the first setting. This shows that our strategy is reliable in finding the matching between bags. Overall these results prove the effectiveness of WSM3L on M3 data in different open settings.

4.4 Ablation Study

Four variants of WSM3L are designed to further explore the different contribution components of WSM3L with the setting of 70/30% split of training/testing set, and 30% correspondence between training bags randomly masked. The description of these variants is as follows:

• (i) WSM3L(Bag): only uses neighbourhood information of bags to replenish missing labels of bags.
• (ii) WSM3L(Ins): only considers the aggregated instance predictions to predict the labels of bags.
• (iii) WSM3L(nFea)): only uses the label similarity for bag matching.
• (iv) WSM3L(nMat): does not match bags across view.

From Figure 2, we observe that WSM3L outperforms its variants, which separately disregard different components of WSM3L. WSM3L(Bag) and WSMEL(Ins) are two components for label prediction of bags, they ignore the aggregated label predictions from instances and the predictions from neighbourhood bags, respectively. WSM3L outperforms them both. This observation suggests the significance of integrating these two types of label predictions in WSM3L. WSM3L(nMat) does not seek the matches between bags across views, and WSM3L(nFea) ignores the feature similarity of bags during the bag matching process. Both of them are outperformed by WSM3L, and WSM3L(nMat) achieves the lowest performance. These observations manifest the necessity of matching bags in M3L and the contribution of leveraging feature similarity and label similarity for matching bags. There is a small margin between WSM3L(nFea) and WSM3L in 1-RL for Drosophila melanogaster and Saccharomyces cerevisiae datasets. The reason of such phenomenon is that these two datasets have a relatively large label space, which causes a low distinction of 1-RL.

From these results, we can safely say that these components of WSM3L indeed deal with the multiplicity of learning on weakly-supervised M3 data.

We further investigated the sensitivity of four input parameters (i.e., α, k, s and d). We run WSM3L with different input values of α, k, combinations of s and d in the range of [10-2 , 103 ], [0, 100] and [50, 300], respectively. We summarize the observations here: (i) α maintains a relatively stable and good performance when α > 0.1, which suggests the importance of label replenishment; (ii) WSM3L achieves the lowest performance when k = 0, it then rises as k increases and has a good performance when k is close to 30; (iii) WSM3L achieves a stable performance under a wide range of combinations of d and s, and it achieves a good performance with d and s in [150, 250]. From these results, we can conclude that WSM3L is relatively robust to α, k, s and d. These results are given in the supplementary file(mlda.swu.edu.cn/WSM3L).

5 Conclusions

In this paper, we proposed a weakly-supervised multi-view multi-instance multi-label learning approach (WSM3L), which extends the flexibility of M3L on practical M3 data, whose matches between bags across views are partially (or completely) unknown, and the labels of bags are incomplete. WSM3L outperforms existing M3L algorithms under different practical settings, some of which existing M3L methods cannot handle.