Auxiliary Information Regularized Machine for Multiple Modality Feature Learning

1 Introduction

With the development of data collection techniques, multiple modal data can be acquired from many applications, e.g., modern mobile phones with types of sensors can collect sensor signals from multiple channels. In order to utilize the information from different modalities, more and more attentions have been paid to multi-modal learning. Ngiam et al.[2011] applied deep networks to learn features over multiple modal data; Zhai et al.[2013] used multi-modal method for efficient large-scale similarity search; Nguyen et al.[2013] proposed the M3LDA to annotate image regions and provide a promising way to understand the relation between input patterns and output semantics. The main assumption behind these methods is that each modality contains sufficient information for target tasks and is with equal importance.

Nevertheless, different modalities are of various importance under specific circumstance. In this paper, we denote the important modalities with particular tasks by strong modality. On the contrary, the modalities with less importance are weak modality. E.g., in identification systems, recognizer can more easily tell the ID with the fingerprints provided, so we denote the fingerprints as strong modality, while features like face descriptors and gaits are usually less accurate, they are weak modalities in this task. It is notable that strong modal features can lead to a better performance, nevertheless, are more expensive, therefore a group of serialized feature extraction methods were proposed. These methods extract weak modal features firstly, and then extract more strong modal features gradually to improve the performance and reduce the overall cost as well. Marcialis et al.[2010] proposed a serial fusion technique for multiple biometric modal features through extracting gaits information and face information step by step; Zhang et al.[2014] addressed the serialized multi-modal learning techniques in a semi-supervised learning scenario. These methods handle strong and weak modalities independently while leaving the fact of unsatisfied performance on weak modality unexplained. In this paper, we consider the issue of unsatisfied classification performance on weak modality, and attribute the phenomena into two different reasons:

∗This research was supported by NSFC(61273301, 61305067).

• Weak modality can be with insufficient information, e.g., face with occluded illumination conditions;
• Weak modality contains helpful information, but these information are 'hidden' behind other irrelevant factors, e.g., noise, irrelevant features.

In the second situation, how to extract the discriminative features of weak modality by feature learning and improve the multi-modal learning classification performance are the urgent problem. In this paper, we will mainly focus on this situation for discussion.

We therefore propose the ARM (Auxiliary information Regularized Machine) approach to extract the most discriminative features of weak modality with the auxiliary information of strong modality in multi-modal learning. In supervised learning, lots of feature extraction methods have been proposed, we can easily use sufficient supervised information in these methods, e.g., face recognition with supervised feature extraction methods. However, here is usually in semisupervised scenarios when performing multi-modal learning, and it is difficult to improve the performance with a limited amount of labeled data, while it is easy to gather the strong modal information corresponding to weak modality. In this paper, we consider the auxiliary information provided by strong modality, and utilize them to help weak modality extract more discriminative features. Meanwhile, through the consistency of classifiers on strong and weak modality, we want to get better multi-modal learning performance. The effectiveness of the proposed ARM approach is validated in our empirical investigations.

Section 2 is related work. Section 3 presents our approach, and the algorithm is presented in Section 4. Section 5 reports our experiments. Finally, Section 6 concludes.

2 RelatedWork

Our ARM approach can handle multiple modal information in semi-supervised scenarios as well as extracting informative features of weak modality by feature learning. Therefore our work is closely related to: multi-modal learning, semisupervised learning and feature learning.

The exploitation of multiple modalities has attracted many attentions recently. The mainstream of the multi-modal learning models handle multi-modal information in parallel or serialized style, especially for biometric learning tasks, e.g., [Hong and Jain, 1998], [Zhou et al., 2005], [Zhang and Li, 2014]. However, these methods handle each modality independently first and then fuse them with late fusion strategies, without considering the correlation between different modalities, while in recent, Kiros et al.[2014] jointly learn word representations and image features by convolutional network.

Many studies have made efforts on the exploitation of multi-modal learning with unlabeled data. Modern multimodal semi-supervised learning methods mainly derive from co-training methods. Co-training [Blum and Mitchell, 1998] is the most famous multi-modal semi-supervised learning method, which trains two classifiers separately on two modalities and then uses them to label unlabeled instances for each other. Co-training style semi-supervised learning approaches have been well developed in decades, e.g., [Kiritchenko and Matwin, 2011], [Zhang and Zhou, 2011]. However, previous methods mainly focus on how to label the unlabeled data between different modalities while leaving the importance of different modalities unconsidered.

In feature learning research, conventional feature learning methods liking PCA [Jolliffe, 2005], LPP [He and Partha, 2003], LDA [Fisher, 1936] etc. aim to extract features in a low dimension space. Feature learning methods are also used in many applications, e.g., [Xu et al., 2010], [Guo and Xue, 2013], [Xie et al., 2014]. However, most existing feature learning methods require supervised information. Yet it is hard to get enough labeled examples in practice.

To the best of our knowledge, previous multi-modal methods assume that each modality would be sufficient for classification without differences of 'strong' or 'weak'. But in practice, it is difficult to build a classifier just using weak modality with limited labeled data, so we turn to utilize auxiliary information provided by the strong modality. In this paper, we proposed an Auxiliary information Regularized Machine (ARM) which utilizes the multi-modal information in semisupervised scenario. Moreover, ARM can exploit the weak modal features more extensively, extract the useful feature subspaces from weak modalities. Therefore, with ARM we can only extract/utilize the relatively inexpensive weak modal features, and obtain an improved weak modal prediction performance during the test phase.

3 ProposedMethod

Suppose we have N examples, where labeled examples are denoted by {(xi, yi)}N1 i=1 and unlabeled instances are {xj}Nj=N1+1 , where xi ∈ Rd , and yi ∈ {0, 1} which denotes the class label of the i-th sample. d is the feature dimension. Meanwhile, in multi-modal learning, instance space can be denoted as two parts without overlap, V = {V d1 1 , V d2 2 }, where V d1 1 is the feature space derived from weak modality and V d2 2 is the features of strongmodality, d = d1+d2. Consequently, each instance xi can be denoted as (xi,v1 , xi,v2 ).

Auxiliary Information Regularized Machine(ARM)

Auxiliary information Regularized Machine (ARM) aims to improve the multi-modal learning performance while extracting the most discriminative weak modal feature subspace at the same time. As a consequence, we can get more accurate predictions with either only the weak or the strong modal features provided in the test phase.

In detail, ARM can be decomposed into two targets: first, with the feature subspace learned by ARM on weak modality, we can predict a instance only with the raw features from weak modality, and this requires extracting the most discriminative feature subspace in weak modality. Conventional feature learning methods often require lots of labeled data to supervise the learning process, while in multi-modal learning scenario, the volume of data is usually larger than that of single modal learning, nevertheless, the labeled examples are very limited. Therefore, in order to extract more discriminative feature from the weak modality, ARM needs to make full use of information from strong modality. Besides, it is also expected ARM can achieve even better performance when the weak as well as the strong modal data are provided in the test phase, and this requires the consistency of predictions built on the strong and weak modality. Thus, we can formulate the ARMmodel as:

where Fv2 = {fv2 (x1,v2 ), fv2 (x2,v2 ), . . . , fv2 (xN,v2 )} ∈ RN , fv2 (xi,v2 ) is the strong modal prediction value for xi,v2 . Lv1 is the Laplacian matrix on weak modality. ˜Xv1 = {xi,v1 , i = 1, · · · ,N1} is the matrix of weak modal labeled examples. The label information is encoded in the vector Y = [y1, y2, . . . , yN1 ] ∈ RN1 , where yi = 1 if xi belongs to positive class, and yi = 0 is negative. ω is the feature extraction matrix for weak modality. λ1 and λ2 are the balance factors, and constraints yifv2 ≥ 1 is a hard margin for strong modal labeled examples.

The ‖Fv2 ‖22 is the structure risk of strong modal predictor fv2 in the function space, and ‖ ˜X⊤ v1 ω - Y ‖2F acts as linear dimensionality reduction on weak modality [Ye, 2007].

The main targets of ARM are closely related to the 2nd term of Eq. 1. In this term, Lv1 is defined as following:

where without any loss of generality, we represent the weak modal instance matrix Xv1 ∈ Rd1×N as X⊤ v1 ω in the discriminant feature space, which is expected to extract the most informative features for classification, and Wv1 is similarity matrix of weak modal instances, denoted by the inner product of instances in the discriminant feature space: Wv1 = 〈X⊤ v1 ω, X⊤ v1 ω〉. Dv1 is the diagonal matrix induced from the Wv1 , Dv1,i is the diagonal entry. Dv1,i = ∑j Wv1,i,j, where Wv1,i,j is the (i, j) element of matrix Wv1 . When ω is provided, the 2nd term is equivalent to:

which reveals that similar weak modal instances in the projected feature space should have similar predictions with strong modal predictor.

By reciting the 2nd term of Eq. 1, it can be found that when fv2 is provided or obtained with high accuracy, we can get a better projection matrix ω by minimizing the 2nd term. In this way, the strong modal information acts like a supervisor, to guide the weak modal feature extraction. The labels of weak modal instances should also be considered, and thus we introduce the 3rd term which is a variant of linear discriminant analysis [Ye, 2007]. On the other hand, when a better feature space for weak modality is provided, we can trust the similarities calculated on the weak modal feature space, and then the 2nd term of Eq. 1 actually becomes a manifold regularizer for the strong modal predictor according to [Belkin et al., 2006].

As a matter of fact, we not only encounter with binary problems in many practical applications, and the ARM formulation can be easily extended into multi-class case where we have c classes:

where F c v2 = {f c v2 (x1,v2 ) . . . f c v2 (xN,v2 )} ∈ RN×c , yc i ∈ Rc is a multi-class label vector, yc i (j) = 1 indicates the i-th instance belongs to the j-th class, ◦ is element wise product operator, 1 is the all one vector, ω ∈ Rd1×c , tr(·) is the trace operator. The constraints yc i ◦F c v2 ≥ 1 are still hard margins, since strong modal examples should be classified correctly. While herein Eq. 4, the normalized label matrix can be represented by (Y Y ⊤)- 12 Y [Ye, 2007]. Inspired by [Melacci and Belkin, 2011], the strong modal predictor can be denoted as:

where α = {αc j , j = 1, · · · ,N} ∈ RN×c , K(·, xi,v2 ) is the i-th column of strong modal kernel matrix for both labeled and unlabeled data, αc j ∈ Rc . The Eq. 4 is reformed as: arg min F c v2 ,ω tr (α⊤Kα) + λ1 tr (α⊤KLv1 Kα) (5) + λ2‖ ˜X⊤ v1 ω - (Y Y ⊤)- 1 2 Y ‖2F s.t. yc i ◦ (α⊤K(·, xi,v2 ) +b) ≥ 1, ∀i ∈ {1, . . . ,N1}.

4 Algorithm

The 2nd term in Eq. 5 involves the product of the weak modal feature extraction matrix ω and the strong modal predictor α, which makes the formulation not joint convex. Consequently, the formulation cannot be optimized easily. The alternative descent algorithm is utilized for solving this problem, and we will reveal the physical meanings of each step in corresponding subsection. Specifically, we first optimize the objective function respect to ω when α is fixed, then optimize α while making ω fixed. We provide the optimization process below:

Fix α, Update ω

We define ̂ Fi,j = ‖(f c v2 (xi,v2 ) - f c v2 (xj,v2 ))‖22 , and M = Xv1 ̂ F X⊤ v1 . According to [Ye, 2007], the 3rd term of Eq. 5 can be written as LDA. When α is fixed, the term tr(α⊤Kα) is not related to ω, thus Eq. 5 can be equivalently written as:

where SW is the within-class variance, and SB is the between-class variance.

However, ̂ F may not always be positive definite and ̂ F is not guaranteed to be reversible. To overcome this, we define ˜ F to be the inner product of strong modal instances, which can be represented by ˜ Fi,j = f c v2 (xi,v2 )⊤f c v2 (xj,v2 ) and define ˜ M = Xv1 ˜ F X⊤ v1 , and yield the main target function, which aims to get a better feature projection with the help of strong modality:

Eq. 7 has closed-form solution for ω, or ω can be obtained by the generalized eigenvalue of (λ1 ˜ M + λ2SB)ω = λSW ω.

From the aspect of dimensionality reduction or feature extraction, we treat the extra strong modal information as supervision to help learning the discriminative projection matrix ω for weak modality, i.e., we consider the predictor fv2 as pseudo labels and make kernel alignments between the strong modal predictions and the weak modality in the projected feature space in this step.

Fix ω, Update α

When ω is fixed, note that the 3rd term in Eq. 5 is not related to the predictor fv2 . So Eq. 5 can be addressed as the following sub-problems:

where we can solve it via efficient quadratic programming(QP) method.

When ω is fixed, in the 2nd term of Eq. 8, the strong modal predictor α treats the KLv1 K, which is a hybrid of K and Laplacian matrix Lv1 as a regularizer, and we expect the Laplacian regularizer defined on weak modality can boost the performance of strong modality.

Above procedures are repeated iteratively until convergence, the ARM algorithm is shown in Algorithm 1. The algorithm updates the parameters, and experiments show the objective value can be decreased gradually.

Prediction

In the prediction procedure, from Algorithm 1, it can be found that we can predict either with the strong modal predictor α or by kNN with the weak modal linear projection matrix ω. In detail, for strong modality, we can use

for predicting xi,v2 . For weak modality, we first use the ω to extract the latent feature representation of both training data and testing data on weak modality, then use kNN for classification on weak modality, since most feature extraction methods employ kNN as the classifier.

Algorithm 1 The ARM method

Require: Xlv1 ,Xu v1 ,Xlv2 ,Xu v2 ,λ1, λ2, Y ;

• 1: Initialize ω0 ← I;
• 2: Initialize α0 ←Eq. 8 with fixed ω = I;
• 3: while true do
• 4: Functobj ←calculate obj. value in Eq. 5 with αt , ωt ;
• 5: Fix αt ,update ωt+1 ←Eq. 7;
• 6: Fix ωt+1 ,update αt+1 ←Eq. 8;
• 7: Funct+1 obj ← calculate obj. in Eq. 5 with αt+1 , ωt+1 ;
• 8: if |Funct+1 obj -Functobj | ≤ threshold then
• 9: Break;
• 10: end if
• 11: end while
• 12: return α, ω;

5 Experiment

In this section, we first introduce the datasets in brief and then give the empirical results of ARM and compared methods.

Datasets and Configurations

ARM can be adopted for many applications with multi-modal features. In this paper, we use the datasets from image categorization , webpage classification and the biometric tasks in our empirical investigations.

For image categorization, two real-world image datasets are used, i.e., Nus [Chua et al., 2009] and Msra [Wang et al., 2009]. Nus subset contains 9,109 images of 10 categories, and has 6 groups of features extracted. Msra subset contains 10,680 images of 9 categories, and has 7 groups of features are extracted. We partition all the features into strong modal features and weak modal features. More specifically, in Nus, color histogram features are weak modality while the rest are strong modal features. Examples from Professional and Movie are selected for balanced binary classification; In Msra, HSV color histogram are weak modal features and the rest are strong modal features, and 2 balanced categories are selected for classification as well.

The WebKB dataset contains webpages collected from 4 universities: Wisconsin, Washington, Cornell and Texas. These webpages are about five categories, i.e., student, project, course, stuff and faculty, and described with two modalities: the content and the citation. We consider the content as weak modality and the citation as strong modality. Five subsets are constructed fromWebKB, i.e., Winsconsin; Washington; Cornell and Texas (denoted as Wins., Wash., Corn. and Texas in tables), which use the instances from each single university, and WebKB that combines all universities data is also tested in our experiments. The first 4 datasets are for binary classification, which aim to tell the differences between student and stuff vs. the rest categories. While WebKB is a multi-class classification task, which aims to identify the differences between each university.

In Biometric task, we construct the virtual faces and gaits dataset, in which faces are strong modality and gaits are weak modality. The CMU PIE is used for face modality construction and the gaits dataset is collected by SDUML [Zhang et al., 2014]. PIE contains more than 750,000 images of 337 people. In the constructed faces and gaits dataset, 25 virtual users are picked out by assigning the same identity to the users from PIE and SDUML. Gait sequences contain 25 users with 40 gait sequences per user. For each gait sequences, we choose the first 50 frames in each video to represent the gait sequence. For each of the 25 virtual users, 150 frames are randomly chosen and the contour features of gait sequences are extracted; for face modality, 150 images are randomly selected per person. This dataset is denoted as Biom. in tables.

For all datasets in our experiments, we randomly select 66% instances for training, and the remains are used for testing. The labeled ratio is set to 30% according to [Zhang et al., 2014]. We repeat this for 30 times, the average accuracy and std. of predictions are recorded as classification performance. The parameter λ1 and λ2 in the training phase is tuned in {10-2 , 10-1 , 1, 10, 102}. Empirically, when the variations between the objective value of Eq. 5 is less than 10-5 in iteration, we treat ARM converges. HIK kernel is used in ARM.

Comparing with Multi-Modal Learning Methods

ARM is essentially a multi-modal learning algorithm. It should be compared with the state of the art multi-modal learning methods. In our empirical investigations, 4 multimodal learning methods are compared:

KCCA (kernel CCA) [Hardoon et al., 2004] is first used to extract the latent feature representation of both the strong modality and weak modality, and then kNN is performed for the final classification;

Co-Training is a famous multi-modal semi-supervised learning method, and co-training usually employs Naive Bayes as the base learner;

CoTRADE As a Co-Training style algorithms, CoTRADE

chooses the predictions with authentic high confidence for labeling information communication based on a particular designed data editing techniques;

Co-Regularization In Co-Regularization [Sindhwani and Niyogi, 2005], it aims to reduce the divergence of two predictors on different modalities. We follow [Sindhwani and Niyogi, 2005] and use the Laplacian SVM as the classifier.

Table 1 records the prediction accuracies (avg.± std.) of weak/strong modalities of the ARM and compared methods. ARM is tested with a kernel classifier fv2 on strong modality as well as kNN on weak modality. As a consequence, we can either directly use the kernel classifier or employ kNN on the reduced dimensions(feature spaces) as the classification method during the test phase. The former result is denoted by 'strong modality' in Table 1, while the latter is denoted by 'weak modality' in the same table.

Table 1 clearly reveals that on all 6 binary classification datasets, the ARM average accuracies of 'strong modality' are the best. While comparing to KCCA (weak modality), Co-Training, CoTRADE, Co-Regularization, on Msra, Winsconsin, Cornell and Texas, i.e., 4 of the 6 binary classification datasets, the 'weak modality' performance of ARM are also the best. On multi-class classification, the average accuracies of ARM (on both the weak modality and strong modality) are the best among all of the compared methods except for the weak modality performance on WebKB.

Comparing with Dimensionality Reduction Methods on Weak Modality

ARM employs the information from strong modality to help extract a better feature space of weak modality, thus, it is closely related to dimensionality reduction from the aspect of weak modality. To validate the effectiveness of utilizing strong modality for weak modal feature extraction. We compare ARM with dimensionality reduction methods, i.e.,

Linear dimensionality reduction methods such as LPP, LDA, PCA, CSFS [Chang et al., 2014] are used to extract the reduced subspace, and then kNN is performed for obtaining the final classification result. Table 2 records the weak modal performance of ARM and compared methods. It clearly reveals that on 6 binary classification datasets and 2 multi-class datasets, the performance of ARM are the best.

Comparing with Kernel Methods on Strong Modality

ARM trains a kernel classifier on strong modality. As a matter of fact, if the test examples are only with strong modality, we can employ the kernel classifier trained by ARM to predict the instances. As a consequence, ARM should be compared with the kernel methods. In our empirical study, 3 kernel methods are compared, i.e.,

Kernel methods, such as LibSVM [Chang and Lin, 2011], LapSVM [Belkin et al., 2006], are compared with ARM in additional experiments. It is notable that here LibSVM1 represents LibSVM with HIK kernel and LibSVM2 represents LibSVM with polynomial kernel. Table 3 records the strong modal performance of the ARM and compared methods. It clearly reveals that on Msra, Washington, Cornell and Texas, i.e., 4 binary datasets and all multi-class datasets, ARM has achieved the best performance.

Investigation on Stability of Parameter

In order to explore the influence of parameters λ1 and λ2, more experiments are conducted. We first fix the λ1 while tuning λ2 in {10-2 , 10-1 , 1, 10, 102} and record the average accuracy in the first row of Fig. 1, then we fix the λ2 while tuning λ1 in {10-2 , 10-1 , 1, 10, 102} and get the average accuracy in the second row of Fig. 1. Due to the page limits, we only list 4 datasets for verification, i.e., Msra, Winsconsin, Cornell and WebKB. From these figures, we can find that ARM achieves a stable performance on each dataset, which indicates the insensitivity of ARM to parameters.

Empirical Investigation on Convergence

To investigate the convergence of ARM iterations empirically. The objective function value, i.e., the value of Eq. 5 and the classification performance of ARM in each iteration are recorded. Due to the page limits, results on only 4 dataset are plotted in Fig. 2. It clearly reveals that the objective function value decreases as the iterations increase, and the classification performance is stable after several iterations on different datasets in Fig. 2. Moreover, these additional experiments result indicates that our ARM can converge very fast, i.e., on most datasets, ARM converges after 3 rounds.

6 Conclusion

In this paper, we focus on multi-modal classification and present a novel method ARM. We first analyze the phenomenon of the unsatisfied classification performance on weak modality and attribute the reasons into lack of information or disturbance of informative weak modal features. We claim that different modalities should be treated with different strategies, and consequently proposed the ARM approach. ARM can perform feature extraction on weak modal features by treating the auxiliary information from strong modality as supervision, meanwhile it can also improve the classification performance on strong modality by organizing the weak modal features as a regularizer. Empirical results on real world datasets clearly validate the effectiveness of ARM, and show that ARM can extract the most discriminative feature subspace on weak modality while successfully regularize the strong modal predictor at the same time. In the current setting of ARM, only two modalities are considered. How to distinguish more than two modalities by classification capacities and integrate multiple weak or strong modalities in the ARM framework can be an interesting future work.