Comprehensive Semi-Supervised Multi-Modal Learning

1 Introduction

In most real-world applications, data are collected from diverse sources and exhibit heterogeneous properties, e.g., during driverless driving, cars collect information from different sensors; in medical testing, doctors collect information from different inspections, etc. These variable group information is defined as Multiple Modality in literature. In contrast to single modal learning, multi-modal learning mainly exploits the consistent and complementary properties among different modalities and improves the learning performance, which attracts increasing attentions and is widely studied in terms of algorithms and theories [Xu et al., 2013; Sridharan and Kakade, 2008; Wang and Zhou, 2013]. Formally, existing algorithms can be categorized into two classes: 1) co-training style; 2) co-regularize style.

∗Contact Author

Co-training [Blum and Mitchell, 1998] style methods always use complementary principle to label unlabeled data for each other. Relatively, co-regularize [Brefeld et al., 2006] style methods exploit unlabeled data with consistency principle. In the literature on multi-modal learning, these methods always assume that each modality is sufficient for classification independently. With this assumption, theoretically, Balcan et al. [2004] proved that if each modal classifier is never "confident but wrong", the ε-expansion can guarantee the success of co-training; Sridharan and Kakade [2008] presented an information theoretic framework for co-regularization which showed excess error between the output hypothesis of co-regularization and the optimal classifier. Nevertheless, in many practical applications, it is very difficult to meet the sufficiency assumption. Consequently, using the previous multimodal methods directly will degenerate the performance on the contrary, even without the effect of single modality [Tao et al., 2018]. Considering this realistic problem, Wang and Zhou [2013] proved that if different modalities have large diversities, co-training algorithm may be able to improve the learning performance, while co-regularize based methods will lose efficacy. Yet co-training has a lot of hyperparameters for adjusting, and it is difficult to satisfy the label noise assumption in reality. Thus, the applicabilities of these style methods are very limited in practice.

In this paper, we are committed to reformulate coregularize based methods. The modal insufficiency mainly leads to the divergences among different modalities. These are partial instances with consistent modalities, while other instances are with diverse modalities, i.e., the modalities are inconsistent, and exist strong/weak modalities [Yang et al., 2015]. Therefore, forcing the predictions of different modalities to be consistent as previous methods will lead to simply minimize the disagreements rather than the optimal classifiers, result in increasing learning confusion and reducing diversity. And in turn, it will degrade single modal performance. Ensemble results are also affected. To solve this much more challenging but practical problem, we present the analysis on multi-modal learning with insufficient modalities, i.e., the consistent and divergent modalities should be treated differently, and propose a novel Comprehensive Multi-Modal Learning (CMML) framework. Specifically, CMML processes the insufficiency with the instance level attention mechanism. On this basis, a diversity regularization is applied to discover complementarities among modalities, and a novel robust consistency metric is designed considering the divergent multi-modal data. In consequence, different modalities can predict accurately for the same task while preserving the diversity.

2 RelatedWork

Multi-modal approaches aim to improve the single modal and overall performance by using the heterogeneities of different modalities. Crucially, this can be done by using unlabeled data, which is out of reach with single modality. Thus, multimodal learning always denotes to semi-supervised learning. The majority multi-modal methods can be divided into two categories: co-training style and co-regularize style.

Co-training [Blum and Mitchell, 1998] is one of the earliest multi-modal methods, which uses initial labeled data to learn two weak hypotheses and allows them to label confident instances for each other, thus improving each modal performance. Blum and Mitchell [1998] proved that co-training can boost the performance of weak classifiers to arbitrarily high level with sufficient modalities. This assumption is too strong to meet in real applications, i.e., the classifier may not correctly make prediction. Thus, Wang and Zhou [2013] proved that co-training may also be effective under the insufficient setting with large diversity. Yet the co-training is still difficult for tuning the hyper-parameters and it ignores the consensus principle, which limits the practical applicability of such methods.

Another style methods is co-regularize [Brefeld et al., 2006], which directly minimizes the disagreements over different modal predictions. The intuition of these methods is that the optimal classifiers of various modalities are compatible with each other. Co-regularize style algorithms are widely researched such as co-regularize [Brefeld et al., 2006], ARM [Yang et al., 2015] and SLIM [Yang et al., 2018b]. It is worthy noting that the basic assumption is that different modalities can provide almost the same predictions. Unfortunately, in practical applications, it is unreasonable that each modality can provide sufficient information, in other words, there exist divergences among different modalities. Consequently, the single modal and overall performance may even degrade.

As a matter of fact, learning comprehensive multi-modal methods considering the modal insufficiency is a relatively new topic. Intensively, we propose a novel Comprehensive Multi-Modal Learning (CMML) framework considering both the consistency and diversity properties in an unified framework. CMML can well consider the sufficiency of each instance on different modalities, while novel diversity regularization and robust consistency metric are designed cogently for promoting the performance.

3 ProposedMethod

Insufficient multi-modal data leads to the divergences among modalities, thus we need to reflect the diversity and robust consistency among different modalities on the basis of measuring the sufficiency, rather than simply use consistency regularization as before.

3.1 Notations

Without any loss of generality, there are N instances with multi-modal information, including Nl labeled examples, i.e., Xl = {(x1, y1), (x2, y2), · · · , (xNl , yNl )}, and Nu unlabeled instances, i.e., Xu = {xNl+1, · · · , xN}. xi = {x1i , x2i , · · · , xMi } with M modalities, m-th modality is with dm-dimensional representation, y ∈ {0, 1}C , C is the class number. The goal is to learn M discriminative models for every modalities: fm : Rdm → RC . Basically, the learning targets are: 1) better single modal result, the result of each modality will not degrade; 2) better ensemble result, the overall result is better than the best single modal result.

3.2 SufficiencyMeasure

The supervise loss term of previous method is always constituted by mean/max voting with different modal classifiers, while is leaves the importance of each instance on different modalities without consideration. Therefore, we utilize the instance level attention mechanism to automatically learn the attention weights. Thus, the loss can be reformulated as:

here considering the distinguished performance of deep models, f utilizes the deep model for each modality in this paper, e.g., convolutional neural network for image modality, long short term memory for text modality. αi,j is the attention weight for i-th instance on j-th modality, which can be learned by an extra attention network: αi,j =

h(fj (xji )) ∑Mm=1 h(fm(xmi )) , h(·) is the extra neural network, i.e., we utilize two layer shallow fully connected here. The α can represent the instance level sufficiency if the instance volume is large as [Wang and Zhou, 2013]. At the end of each round, the weights α are normalized as ∑j αi,j = 1. Moreover, it is notable that the attention can also be applied on the feature embedding layer, i.e., ˆ f = αi,jxlp ijW, where xlp ij is the embedding for xij of feature representation layer, W denotes the fully connected matrix to the prediction layer.

3.3 DiversityMeasure

The previous co-regularize based methods usually require the prediction probabilities of different modalities to be consistent. While in the insufficient setting, the multi-modal ensemble result using consistency regularization is even worse than single modal result. This is ultimately because minimizing the disagreement among modalities will result in the loss of the divergence. In other words, for the insufficient multi-modal data, partial data of each modality may contain the knowledge that other modalities lack, while the consistency regularization will lose the complementary information instead, i.e., weak modality will affect the strong modality to some extent in the learning procedure. Therefore, similar to ensemble learning, the divergences among different modalities can be regarded as the diversities among different modal classifiers.

Without any loss of generality, given M trained classifiers F = {fm(xi)}Mm=1 for all modalities. In this work, we measure diversities of different modalities based on pairwise difference as the diversity [Li et al., 2012], and the definition is given as follows:

Definition 1 Given a set of N examples D = {{(xmi , ym i )}Nl i=1 , {(xmi )}Ni=Nl+1 }Mm=1 , the diversities of different modalities on D is defined as:

where sim(·, ·) is the pairwise difference between two modalities as:

where the sim can be any convex function here, we utilize the cosine here, thus the difference sim(fi, fj) falls into the interval [-1, 1], and equals to 1/-1 only if two modal classifiers always make the same/opposite predictions on the same instances, and the larger sim, the larger Com. In consequence, since the diversity is based on the pairwise differences, it reveals that the smaller Com(F), the larger diversity of the modal classifier set F. Moreover, different from defining the diversity in the parameter space [Yu et al., 2011], here the diversity measure defines in the output space, thus can cover various kinds of individual classifier.

On the other hand, it is easy to find that this diversity measure is closely related to complementary measure, which aims to utilize the preponderant information of each modality to improver the performance. Thereby, intergrading multiple modalities with diversity regularization can describe the data more comprehensively and accurately in the insufficient setting. Moreover, the supervised loss and diversity regularization are also adversarial here. In detail, the supervised loss means that the classifiers of different modalities must predict similarly on the same label, while the diversity regularization increases the diversities over various modalities. Thus, considering the tradeoff between empirical error and diversity, better generalization performance can be expected.

3.4 Robust ConsistencyMeasure

The consistency principle maximizes agreement on multiple modalities, and it has been demonstrated the connection between the consensus of two modal hypotheses respectively and their error rates as [Dasgupta et al., 2001]:

which reflects that the probability of the disagreement of two independent hypotheses upper bounds the error rate of either hypothesis. Thus by minimizing the disagreement of the two hypotheses, the error rate of each hypothesis will be minimized. However, the basic assumption behind the theory is that each modality is sufficient for prediction, while different modal feature representations are always insufficient in practice. And it has been proved that co-regularization based methods, which utilize the consistency principle, prefers to output false hypotheses rather than output optimal classifiers [Wang and Zhou, 2013].

To solve this problem, we turn to utilize the robust regression loss, i.e., Huber Loss [Huber and others, 1964], instead of the square loss in previous methods. For consistency calculation, we also adopt cosine for convenient as the diversity measure mentioned in last section. The modified huber loss can be defined as:

Differing from the squared loss which has a disadvantage of the tendency being dominated by outliers (disagree on different modalities), huber loss is quadratic for small values of |2-cos(fi, fj)|, linear for large |2-cos(fi, fj)|, with equal values and slopes of the different sections at the two points where |2 - cos(fi, fj)| = δ, and it has the differentiable extension, the δ is set as 1 in the experiments. In result, the robust consistency metric will more incline to constrain the instances with consistent modalities, neglecting the inconsistent instances. Thus, with the Eq. 3, the consistency regularization can be rewritten as:

3.5 Comprehensive Multi-Modal Learning

In this section, we will reformulate the co-regularize based methods in insufficient setting with the three paradigms mentioned above. The basic intuition of previous co-regularize style methods is that the complexity of learning problem can be reduced by eliminating hypotheses from each modality that do not agree [Sridharan and Kakade, 2008]. Representative multi-modal semi-supervised learning method as coregularize is:

As mentioned above, it reveals that co-regularize will lose efficacy under the insufficient scenario. While in CMML, the attention mechanism can calculate the instance level insufficiency, the robust consistency metric and diversity measure can process consistent or divergent modalities separately, thus it can make full use of multi-source data. Therefore, in the more realistic setting, combing the Eq. 1, Eq. 2 and Eq. 4, previous approaches can be rewritten dramatically:

it is notable that we utilize the deep network for different modalities in this paper.

4 Experiment

In this section, we validate the effectiveness of our proposed CMML approach. Specifically, most of current large-scale multi-modal datasets are image-text multi-label classification for deep networks, thus we conduct the experiments on realworld multi-label task.

4.1 Datasets and Configurations

Without any loss of generality, we first experiment on 4 public real-world datasets, i.e., FLICKR25K [Huiskes and Lew, 2008], IAPR TC-12 [Escalante et al., 2010], MS-COCO [Lin et al., 2014] and NUS-WIDE [Chua et al., 2009]. Besides, we also experiment on 1 real-world complex article dataset, i.e., WKG Game-Hub [Yang et al., 2018a]:

• FLICKR25K: consists of 25,000 images collected from Flickr website, each image is associated with several textual tags. The text for each instance is represented by a 1386-dimensional bag-of-words vector. Each point is manually annotated with 24 labels. We select 23,600 pairs that belong to the 10 most frequent concepts;
• IAPRTC-12: consists of 20,000 image-text pairs which are annotated 255 labels. The text for each point is represented by a 2912-dimensional bag-of-words vector;
• NUS-WIDE: contains 260,648 web images, and images are associated with textual tags where each point is annotated with 81 concept labels. We select 195,834 image-text pairs that belong to the 21 most frequent concepts. The text for each point is represented by a 1000dimensional bag-of-words vector;
• MS-COCO: contains 82,783 training, 40,504 validation image-text pairs which belong to 91 categories. We select 38,000 image-text pairs that belong to the 20 most frequent concepts. Text is represented by 2912dimensional bag-of-words vector;
• WKG Game-Hub: consists of 13,750 articles collected from the Game-Hub of " Strike of Kings" with 1744 concept labels. We select 11,000 image-text pairs that belong to the 54 most frequent concepts. Each article contains several images and content paragraphs. The text for each point is represented by a 300-dimensional word2vector vector.

For each dataset, we randomly select 33% of the data for test set and the remaining instances are used for training. And for training data, we randomly choose 30% as the labeled data, and the left 70% as unlabeled ones. Image encoder is implemented with Resnet18 [He et al., 2015], the text utilizes fully connected network. The parameter λ in the training phase is tuned in {0.1, 0.2, · · · , 0.9}. When the variation between the objective values of Eq. 6 is less than 10-4 in iterations, we consider CMML converges. For all compared methods, the parameters are tuned as original papers. 6 common multi-label measurement criteria are recorded, i.e., Coverage, Ranking Loss, Average Precision, Macro AUC, example AUC and Micro AUC. We run the following experiments with the implementation of an environment on NVIDIA K80 GPUs server, and our model can be trained about 290 images per second with a single K80 GPGPU.

4.2 ComparedMethods

To evaluate the performance of our proposed approach, firstly, we adopt ablation study to verify the effectiveness of the proposed CMML. Specifically, we define 5 different varieties of CMML (Attf ), i.e., Atto, Maxf , Maxo, Meanf , Meano, in which Att/Max/Mean mean attention mechanism/max pooling/mean pooling, f represents the feature embedding layer and o represents label output layer. Thus, e.g., Attf denotes that we utilize the attention mechanism on the feature embedding layer for different modalities, then with a fully connected matrix for final prediction and so on. Besides, for validate the effectiveness of the diversity measure and consistency measure, we construct another 2 comparison methods using only one of the measurements alone, i.e., Com, Rδ.

Moreover, we compare CMML with the state-of-theart multi-modal/multi-label/multi-modal multi-label methods. For multi-modal comparing methods, we treat each label independently, i.e., for each label, a method trains classifiers using different modalities; for multi-label comparison methods, we concatenate multi-modal data as the unified input. In detail, multi-modal methods include: Cotrade [Zhang and Zhou, 2011], Co-regularize [Brefeld et al., 2006], WNH [Wang et al., 2013], ARM [Yang et al., 2015], SLIM [Yang et al., 2018b]; multi-label method includes: DeepMIML [Feng and Zhou, 2017]; multi-modal multi-label methods include: CS3G [Ye et al., 2016], M3DN [Yang et al., 2018a]:

• Co-trade: is a novel co-training algorithm by reliably communicating labeling information between different modalities;

• Co-regularize: minimizes the disagreement over different modal data with the unlabeled data;
• WNH: combines all modal values from different modalities together and then uses l2,1-norm to regularize the modality selection process and finally gives the results;
• ARM: extracts the discriminative feature subspace of weak modality while regularizing the strong predictor;
• SLIM: exploits more extrinsic information from unlabeled data for classification and clustering;
• DeepMIML: exploits deep neural network to generate instance representation for MIML;
• CS3G: handles types of interactions between multiple labels and utilizes the data from different modalities;
• M3DN: imposes the modal consistency on bag-level prediction by requiring that bag-based prediction of different modalities generate similar label correlation.

4.3 Ablation Study

We first explore which variant method will yield better results, and the results are listed in Fig. 1. Due to page limitation, we only list 4 public datasets here. The results reveal that the CMML, i.e., Attf , achieves the best on most datasets for different performance measures, while it reveals that it achieves better performance to use both the diversity regularization and robust consistency metric than using them independently. This verifies the effectiveness of attention mechanism, diversity regularization and robust consistency metric for solving the insufficient problem.

4.4 Multi-label Classification

CMML is firstly compared with the state-of-the-art methods on 4 benchmark datasets, results are listed in Tab. 1. Results of deep learning methods only give the best results as [LeCun et al., 2015], other comparison methods give the results with mean and std. The notation "N/A" means a method cannot Heroes need to develop slowly in the early stage Rδ = 0.87

The skin of Valkyrie is still very beautiful, there is a feeling of angels, especially when walking.

Rδ = 0.83

Rδ = 0.80

Figure 2: Examples of inconsistent instances using the robust consistency metric fromWKG Game-Hub dataset.

give a result in 60 hours. From the results, it is obviously that our CMML can achieve the best performance in most datasets with different performance measures, except the Coverage on FLICKR25K, which reveals that the CMML approach is a high-competitive multi-modal learning method considering the modal insufficiency. Besides, for the real-world complex article classification dataset, comparison results against compared methods are listed in Tab. 2. Similarly, it can be found that CMML approach also gets the best results on overall criteria, which validates the effectiveness of our method in solving complex article classification problem. Considering limitation of the paper, we only give the comparison results of single modality on WKG-Hub dataset, which is with modal insufficiency in practice. In Tab. 2, the first partition is the results using only image or text information, and ensemble results. The last partition is the results of CCML method for image (CMMLI), text (CMMLT ) predictions and ensemble (CMML) results. We can find that not only the results of single modalities are not degraded on various criteria, but also the overall results are also improved, thus achieved our goal, i.e., better single modal and overall performance. While other multi-modal methods (e.g., CoReg, CoTra, WNH, SLIM) are lower than single modal results on some criteria without considering the modal insufficiency.

4.5 Overcome Insufficiency

Insufficiency leads to divergences of different modalities, and there will exist inconsistent instances. However, the robust consistency measure can overcome the insufficiency. Thus, we conduct more experiments, and locate the inconsistent instances with the δ in maximum order. Fig. 2 exhibits several illustrative examples of the inconsistent instances. Qualitatively, we find that using robust consistency measure will achieve better performance from the classification results, in addition, it also demonstrates the phenomenon from the sorted examples.

5 Conclusion

In real-world applications, multi-modal data are often insufficient, which leads to the failure of previous multi-modal learning methods based on sufficiency assumption. In this paper, under the insufficient scenario, we developed a novel Comprehensive Multi-Modal Learning (CMML) framework. Specifically, we proposed the attention mechanism to learn the instance level sufficiency for each instance on different modalities. Also, novel diversity regularization and robust consistency metrics are designed for discovering insufficient modalities. Finally, experiments on real-world data obtain remarkable results for our method in terms of various criteria.

Acknowledgments

This work is partially supported by National Key R&D Program of China (2018YFB1004300), NSFC(61773198), NSF IIS-1814510, and Collaborative Innovation Center of Novel Software Technology and Industrialization of NJU, Jiangsu.