Modal Consistency based Pre-Trained Multi-Model Reuse

1 Introduction

Machine learning techniques have achieved great success in many applications. Nowadays, many machine learning models are integrated as a part of functional software. Similar to the developments of software engineering research, machine learning researchers pay more attentions on the reuse of pre-trained learners in real applications [Zhou, 2016; Pan and Yang, 2010].

∗This work was supported by NSFC (61632004, 61673201), Huawei Fund (YBN2017030027) and the Collaborative Innovation Center of Novel Software Technology and Industrialization.

Model reuse is with different levels: transfer learning techniques reuse models by obtaining middle level representations [Wei et al., 2016; Long and Wang, 2015; Isele et al., 2016] for cross domain flexibility of models. However, it is notable that in transfer learning both source domain and target domain data are required during the training phase, i.e., transfer learning is a technical engineer oriented reuse.

Nevertheless, learnware describes a substantially different paradigm of model reuse: pre-trained models are ready with specifications which are sufficient for users. In general, users can fetch a group of similar models for their task. Note that there are no source domain training examples in this kind of model reuse, and pre-trained learners cannot be modified in their model essentially. This type of reuse paradigm is more practical in real cases since most users can only understand the specifications instead of managing the skills of training/refining the model parameters. In addition, although Zhou [2016] reserves the interface of retaining model in learnware, users cannot afford re-training on the target domain in practice for lacking of their own techniques and experiences, computational consumptions, and more important, the lack of labeled data in target domain. Thus, the learnware paradigm tends to propose a re-training free and general user oriented model reuse.

How to reuse multi-model predictions becomes the most important problem in this case. Since ideally, one can directly find a pre-trained model with the exact same specification as the target domain requirement. However, due to the differences between the environments of pre-training and the user's target domain, it is more common that one cannot find a "perfect" pre-trained model but a set of analogous models. We denote this problem as Multi-Model Reuse (MMR) problem in the context. MMR problem is similar to the scene when a customer chooses the same type of goods in the supermarket. Ideally, if the customer is well experienced and with demands precisely described, he can directly buy the appointed goods. However, a more real case is that he may take more time on hesitating for equal confidence assignments to the set of items. In MMR problem, users can also use some ensemble techniques, e.g., the majority voting, for the confidence assigned equally when there is no prior knowledge provided. Moreover, in ensemble learning, we can use weighted voting for better performance since model-wise confidence can be obtained with evaluation on the ground truth. Nevertheless, in MMR problem, labeled data are absent, and the confidence re-weighting cannot be obtained directly. Similar phenomenon occurs in the market example, it is also ridiculous to ask the feelings before customers buy items.

Nevertheless, one in a supermarket can eventually make the decision by reading the specifications, comparing the "detail" and "consistency" of those descriptions. The "consistency" in production specification can be described from multiple aspects (modalities). In multi-modal learning, researchers have explored the consistency of predictions from different modalities for semi-supervised learning [Blum and Mitchell, 1998; Wang et al., 2016], clustering [Li et al., 2014; Xu et al., 2015], theoretical analysis [Wang and Zhou, 2013] etc., and these show that the consistency among multiple modalities can be very informative in the situation where precision evaluation cannot be applied.

In this paper, we propose the Pre-trained Multi-Model Reuse method, PM2 R, to tackle the MMR problem under the multi-modal learning scenario. Different from ensemble methods, no training data and validation instances are provided in the MMR problem, thus PM2 R is a re-training free strategy. Comparing to the multi-modal late fusion methods, the MMR emphasizes user oriented reuse and requires the pre-trained models fixed, therefore, PM2 R is built with no modification on pre-trained models. PM2 R can be implemented from various ways, yet in this paper, a belief propagation style updating is demonstrated and the effectiveness of this PM2 R implementation is shown in the experiment.

The rest of this paper starts from introduction of related work. Then we propose our approach, followed by experiments and conclusion.

2 RelatedWork

This work focuses on reuse of multiple models and is related to multi-classifier systems, transfer learning and multi-modal learning. In this section, we first review these topics and figure out the main differences between these existing work and our new settings eventually.

Multi-classifier systems or know as ensemble learning are proposed to increase generalization abilities of single type of base classifiers. Breiman [1996] proposed Bagging, which trains a component model from several training sets generated from the original training set. Schapire [1990] proposed Boosting, which generates a series of component neural networks whose training sets are determined by the performance of former ones. Zhou et al. [2002] analyzed the ensemble learning, pointed that with a portion of selected base learners, the ensemble learner can be with even higher generalization ability than using all base learners and consequently proposed the GASEN approach which selects a portion of neural networks based on the evolved weights to make up the ensemble; [Ueda, 2000] optimized linear weights to combine component predictions based on statistical pattern recognition theory. Stacking [Wolpert, 1992] is a special kind of ensemble which can be regarded as a complicated vote scheme generated by a second level classifier. The majority voting can be used for our MMR problem for lacking of prior knowledge on pre-trained models, however, an obvious gap between MMR

problem and ensemble techniques lie in the availability of labeled training data and the existence of modifiable models. In the MMR setting, no labeled data can be revisited since there are rarely labeled training examples stored in learnware repositories, and no apparent adjustments can be made by inexperienced users.

Transfer learning concentrates on solving the prediction tasks on target domains. In recent, many transfer learning approaches are proposed for expanding the scope of learning applications, e.g., Tan et al. [2017] proposed a selective learning algorithm to solve the distant domain transfer learning problem, Kandemir [2015] performed knowledge transfer by projecting the target data onto the source domain and linearly combining its representations on the source and target domain manifolds. However, similar issues in ensemble learning are also appeared on solving MMR with transfer learning: it requires both source and target labeled data for transfer learning, while in MMR setting, only unlabeled data can be gathered by model reuse users (denoted as user domain data without any confusion).

Without labeled examples in user domain, model evaluation cannot be precisely established in MMR setting, and consequently, it may be hard to re-weight the confidences of multiple models with classification precision. Applications with complicated object descriptions are usually brought forward the problem of multi-modal learning. In multi-modal learning, different modalities can provide uncorrelated predictions and consistency becomes an important evaluation than precision. Blum and Mitchell [1998] trained two classifiers separately on two modalities and then uses them to label unlabeled instances for each other, Wang and Zhou [2013] presented the theoretical analysis on co-training when neither modality is sufficient. Multi-modal learning approaches can be categorized into 3 types, i.e., pre-fusion, subspace, late-fusion, and late fusion is more closed to MMR problem, e.g., Wang et al. [2013] proposed the WNH to integrate all modal features and learn the corresponding weights for every modality, Ye et al. [2015] proposed RANC in a hybrid fusion manner. However, both in WNH and RANC, the classifiers of different modalities are adjusted in different iterations during the training phase, which is not agreed with the MMR setting.

There have been some model reuse approaches, e.g., FMR [Yang et al., 2017] which tries to integrate the discriminative ability of fixed models into one deep network, and eventually can be applied in various applications. However, in our MMR settings, a different barrier of model reuse is tackled, i.e., weighting or selection of proper multiple models for reuse is mainly focused.

We propose the PM2 R approach for classifying new instances directly from a collection of responses from different pre-trained models with various modalities. Note that PM2 R can be implemented from various ways, yet a belief propagation style updating is demonstrated in this paper.

3 ProposedMethod

3.1 Notations

In this paper, without any loss of generality, suppose there are N instances on the user's side, which are denoted by D = {x1, x2, · · · , xN}. Each instance has d-dimensional inputs as raw features, i.e., xi = [xi1 , xi2 , · · · , xid ] ∈ Rd . Meanwhile, in multi-modal learning, instance space can be denoted as, at least M parts without any overlap, v = {v1, v2, · · · , vM}, where vm ∈ Rdm is raw features from j-th modality, d = d1 + d2 + · · · + dM. Therefore, in multi-modal settings, instance xi can be further denoted as (xi,v1 , xi,v2 , · · · , xi,vM ), where xi,vm denotes the raw feature representation of the m-th modality of i-th instance. In this paper, those multiple modalities are homogeneous ones, e.g., face photographs of the same person with different expressions or from different poses. Suppose there are C pretrained models in "learnware market" chosen and denoted as {f1, f2, · · · , fC}. The corresponding prediction of user's side instances on a concerned modality is Ami,j = fj(xi,vm ), and Ami,j ∈ {-1, +1}.

3.2 The Pre-trainedMulti-Model Reuse Method

PM2 R focuses on the MMR problem under the homogeneous multi-modal learning scenario. Different from ensemble methods, there are no training data or validation set provided in the MMR settings. Meanwhile, the MMR emphasizes user oriented reuse and requires the pre-trained models fixed, therefore, PM2 R is built with no modification on pre-trained models. In practice, as a re-training free strategy, PM2 R only has the corresponding predictions of user's side as inputs. More specially, for the m-th modality, we can denote the prediction values of all the instances as Am ∈ RN×C , where

That is, for each modality, C classifiers are invoked for predicting and returns N predictions for all instances. Assignment bipartite graph Gm(Xm, V, Em) is further used for representing the responses between instances and models in detail, where the superscript m represents the m-th modality, Xm is the set of instances on the m-th modality, V is the set of pre-trained models, and (im, j) ∈ Em means the j-th classifier makes prediction on instance i on modality m. Eventually, there are M assignment bi-graphs {G1 ,G2 , · · · ,GM } due to the multi-modal setting. The overall flowchart is shown in Fig. 1.

To solve the MMR problem under this setting, we claim that the modal consistency should be considered and PM2 R can be implemented in various ways. Yet in this paper, a selfconsistent weight propagation strategy is used for passing the weights between the two sides of bi-graph, and a message passing style algorithm operating two types of messages are proposed for seeking the equilibrium state of the weight propagation system. In detail, two types of messages are used, i.e., the instance messages {zim→j}, (im, j) ∈ Em capturing how likely instance i being predicted as positive on the m-th modality, and the model messages {yj→im} capturing how reliable the j-th model is.

Implementation by Considering Multi-Modal as One

First, we can concatenate the prediction values of each modality, denoted as A = [A1 ,A2 , · · · ,AM ] ∈ RN×(CM) . Besides, the set of edge assignment between the pre-trained model j and predict instance im of different modalities can be concatenated as E = E1 ∪E2 · · ·∪EM as well. Specially, the instance messages can be denoted as {zi→j}, (i, j) ∈ E, and the model messages can be represented as {yj→i}. In each round, all the messages are updated as follows:

where the first is generally the weighted majority voting according to how reliable every model is, and the second is updating the reliability according to how many times the model agreed. It is notable that the inherent updating of model messages utilizes all the instance with multi-modal information. This approach is denoted as the PM2 Rone in the following text. The pseudo code of PM2 Rone is given in Algorithm 1.

Implementation by Considering Consistency Explicitly

Besides considering multi-modal as one, a more reasonable approach lies considering the consistency between different modalities. The PM2 Rone method implicitly uses multi-modal information, while we also implement PM2 R by utilizing the consistency explicitly between different modalities in iterations. In PM2 R all messages are updated separately according to the consistency index function I(zim→j ∑m zim→j ≥ 0). I(stat.) = 1 iff. the statement stat. is true, which figures out whether modal consistency meets. In this version, Am and Em are independent for different modalities. In detail, after passing the message from model to instance, the instance messages of each modality, i.e., zi1→j, · · · , ziM →j, are re-consisted and updated according to following equations:

The pseudo code is described in Algorithm 2.

Algorithm 1 The pseudo code of PM2 Rone

Input:

A ∈ RN×(CM) is outputs of C classifiers on M modalities E = E1 ∪E2 · · · ∪EM , is the union of all linkages between multi-modal features and pre-trained models

K is the maxIter

Output:

{p1, p2, · · · , pN}: the final predictions for all N instances

• 1: for all (i, j) ∈ E do
• 2: Initialize model messages y0 j→i with random values under Normal distribution
• 3: end for
• 4: for k = 1, 2, · · · ,K do
• 5: for all (i, j) ∈ E do
• 6: zk i→j = ∑j ′ ̸=j Ai,j ′ y(k-1) j ′ →i
• 7: end for
• 8: for all (i, j) ∈ E do
• 9: yk j→i = ∑i′ ̸=i Ai′ ,j zk i′ →j
• 10: end for
• 11: end for
• 12: for all i do
• 13: pi = sign (∑j Ai,jyK j→i),
• 14: end for

4 Experiment

In the MMR problem, PM2 Rone/PM2 R can classify new instances directly from a collection of predictions provided by a bundle of different pre-trained models with various modalities. Besides, PM2 Rone/PM2 R are built without modification on pre-trained models.

In this section, we will provide the empirical investigations and performances of PM2 Rone/PM2 R from the users' side of model reuse. In particular, investigations on both synthetic dataset and real-world tasks, i.e., gender classification, are studied. The real-world task is to predict the gender of people with different modalities, i.e., in different expressions, poses, etc. There are 3 different subsets in gender prediction task.

Some of the ensemble methods without re-training can be used in MMR problem, thus, PM2 Rone/PM2 R are compared to the widely used ensemble method, i.e., Majority Voting (denoted as MV). For PM2 Rone/PM2 R are related to late fusion style multi-modal learning approaches, WNH, RLF, RANC are also compared in our experiments. The test sets are drawn from the users' side data with bootstrap, and repeated for 30 times for each task. For both synthetic and real world task, the average accuracies and standard deviations are recorded. In detail, the compared methods are listed as:

• MV: classifies instances from different modalities with the collected predictions provided by different pretrained models via majority voting, i.e., suppose there are M modalities and C classifiers, then the majority voting is performed on M × C prediction values;
• WNH: combines all prediction values from different modalities together and then uses l2,1-norm to regular-

Algorithm 2 The pseudo code of PM2 R

Input:

• {A1 ,A2 , · · · ,AM } ∈ RN×C are outputs of C classifiers on M modalities seperately
• {E1 , E2 , · · · , EM } are the linkages between different modal instances and pre-trained models

K is the maxIter

Output:

{p1, p2, · · · , pN}: the final predictions for all N instances

• 1: Initialize y0 j→im, m ∈ {1, 2, · · · ,M} as steps 1-3 in Algorithm 1
• 2: for k = 1, 2, · · · ,K do
• 3: Update zk im→j , m ∈ {1, 2, · · · ,M} as steps 5-7 in Algorithm 1
• 4: for all (im, j) ∈ Em do
• 5: Update zk im→j as in Eq. 2
• 6: end for
• 7: Update yk j→im, m ∈ {1, 2, · · · ,M} as steps 8-10 in Algorithm 1
• 8: end for

ize the modality selection process and finally gives the prediction [Wang et al., 2013];

• RLF: minimizes the rank of indicator matrix to maximize the modal consistency and fuse the predicted confidence scores of multiple models [Ye et al., 2012]. Note that RLF can only provide the final prediction rather than predictions on single modality;
• RANC: truncated low-rank regularized approach for fusing the prediction values, modal consistency is considered through truncated nuclear norm regularization. Final fusing steps in [Ye et al., 2012] are used;
• BPE: belief propagation style method. In implementation, belief propagation is invoked on every modality, and then majority vote across different modalities for final predictions [Yedidia et al., 2003].

4.1 Synthetic Data

Synthetic data are generated according to [Khetan and Oh, 2016]. For every modality, the prediction confidence of instances i is parameterized by qi ∈ [0, 1] which represents how likely the instance is positive. More specifically, when j-th pre-trained model predicts the i-th instance on m-th modality as positive, we assign the instance as positive with probability qi. Hence, when qi is close to 0.5, the instances on the concerned modality can be regarded as confusing and is difficult to correctly classify. Meanwhile, the j-th pre-trained model is parameterized by pj ∈ [0, 1], which represents how trustable the model is. The closer pj approaches one, the more trustable this model is. pj closed to 0.5 indicates the model providing a random guess. With the qi and pj defined above, we can get the expected prediction values of Ami,j as:

In synthetic experiments, we can directly generate the prediction values Ami,j . In practice, we generated the values for 1000 instances with 5 different modalities. The synthetic label for each instance is sampled i.i.d. from a Bernoulli distribution with parameter 0.5. Meanwhile, the difficulty qi is sampled from a Beta distribution with mean 0.75 (for positive instances) or 0.25 (for negative instances), and the variance is configured as 0.04. We re-sample the task difficulty for each modality to make the qi varies between different modalities. Then we construct 70 pre-trained models whose reliability pj are sampled from a Beta distribution with mean 0.6 and variance 0.01. Pre-trained models are sampled only once for all modalities. Finally, the prediction values of different modalities are calculated based on Eq. 3, and denoted as {A1 ,A2 , · · · ,AM }. PM2 Rone/PM2 R are invoked for the prediction of model reuse, and prediction accuracies are recorded in Table 1. From the table, it reveals that for both single modality and final prediction, PM2 Rone/PM2 R almost always achieve the best classification performance on mean accuracy comparing to all other methods.

4.2 Gender Classification

In the real-world gender classification task, one of the most widely used multi-modal datasets is used, i.e., CASPEAL [Gao et al., 2008] is constructed by Chinese Academy of Sciences (CAS). The CAS-PEAL is with large-scale face images from different sources of variations, specifically, the variations include poses, expressions, accessories, etc. The whole dataset contains 99,594 images from 1040 individuals (595 males and 445 females). The CAS-PEAL is naturally divided into 4 subsets with different variations, i.e., personal poses (denoted as Pos.) set containing 1038 instances with different poses/modalities, expressions (denoted as Exp.) set containing 376 instances with different expressions/modalities, accessories (denoted as Acc.) set including 434 instances with different accessories/modalities. For each subset in Pos., there are 9 different shooting slopes, i.e., there are 9 cameras spaced equally in a horizontal semicircular shelf to simultaneously capture images across different angles in one shot. Besides, each subset also contains two group of visual directions, i.e., all people are asked to look up and down to capture 18 images in another two shots, therefore, we can divide the Pos. set into 21 modalities. We also divide Exp. set into 5 modalities, and similarly, 6 modalities for Acc. set (photoed with 3 different glasses or with 3 different caps). More detailed descriptions on subsets categorization can be found in [Gao et al., 2008].

40 pre-trained models are generated on Pos. set, i.e., we extract traditional BOW features, Fisher vectors features, LBP features, HOG features from Pos., then train 12 random forest models with different number of trees, 24 support vector models with different kernel methods or costs using these features respectively, besides, 4 deep models are also included, which are trained with the raw features. Note that we can treat the pre-trained models as a bundle of black-boxes in whole from the "learnware market" side, while from the aspect of users' side, all 40 models are pre-trained and unmodifiable. Users of learnware models can only input raw pictures from source datasets, e.g., Exp. set, consequently the black-boxes of pre-trained models will output the prediction values, and eventually the final predictions are obtained with PM2 Rone/PM2 R together with compared methods. For facilitating the notations, this setting is denoted as Pos.-Exp, we also have investigations of Pos.-Acc. and Acc.-Exp..

To demonstrate the generalization ability of PM2 Rone /PM2 R. We conduct more experiments with external data sources. We utilize the WIKI [Rothe et al., 2015] dataset, which is also a face dataset with the same input size, for models pre-training, and predict with Pos., Exp., Acc. sets separately. WIKI is one of the largest datasets of face images publicly available with gender and age labels. It contains 62,328 profiled images from pages of people from Wikipedia with their profiles date of birth, name, gender and all images associated with that person. Therefore, we also have following settings as WIKI-Pos., WIKI-Exp. and WIKI-Acc..

From Table 2 to Table 4, it can be observed that although the models on single modality don't provide satisfactory performance in partial subsets due to the dis-matching of users side tasks, PM2 R can almost always achieve the best performance on the final precision, except for Acc.-Exp. set.

This phenomenon clearly reveals the effectiveness of considering modal consistency in model reuse. Without considering the modal consistency explicitly, PM2 Rone becomes inferior. The performance of RANC is superior on single modality in most case, however, the final prediction performance of RANC is not as well as PM2 R. This may be simply because RANC is a multi-modal learning method and do not solve the problem of striding over the gap between the source (original models) and target (users side) domains.

4.3 Influences of Number of Models for Reuse

In order to explore the influence on the number of pre-trained models, more experiments are conducted. In this section, the number of modality in each investigation is fixed, while the number of pre-trained models varies in {5, 10, · · · , 40}. The average errors and standard deviations are recorded in Fig. 2. Due to the page limits, we only list 4 datasets for verification, i.e., Pos.-Acc., Pos.-Exp., WIKI-Acc. and WIKI-Exp.. From these figures, it clearly shows that PM2 R achieves the best performance when the number of models is larger than 15. However, without explicitly considering in the modal consistency, the performance of PM2 Rone is inferior to PM2 R. Besides, we can also find that PM2 R achieves a stable performance fast, and the errors of our model reuse strategies decrease faster than compared methods, as the number of pretrained models increases. It is notable that with the number of models increasing, the error would not decrease without limits, especially for PM2 Rone, the error may increase after the number of models is over a threshold. This may indicate it is wise to select a proper number of models rather than all feasible models in learnware repositories.

5 Conclusion

Model Reuse has attracted many attentions in machine learning communities recently. Multi-Model Reuse (MMR) problem is one of the challenge issues in Learnware. Aiming at the problem of the final prediction acquisition from multiple prediction values in MMR, we follow the learnware principle and propose a multi-model reuse method under multi-modal scenario: PM2 R. Different from multi-classifiers ensemble, there are only pre-trained models rather than the whole training sets provided in this setting. The difficulty of MMR lies lacking the evaluation on reliability for pre-trained models, PM2 R solves this by utilizing the potential consistency on different modalities. Experiments on synthetic and real-world datasets validate the effectiveness of PM2 R under this realistic setting. It is interesting to expand PM2 R style approaches to multi-class scenario and provide theoretical analysis on both generalization abilities and convergence.