Multi-Task Clustering with Model Relation Learning

1 Introduction

Multi-task clustering, which improves the clustering performance of each task by transferring knowledge among the related tasks, receives increasing attention recently. Most existing multi-task clustering methods make an ideal assumption that the tasks are completely related, i.e., the tasks share the same latent categories. However, in many real applications, the tasks are usually partially related, i.e., only parts of the latent categories are shared among the tasks. Transferring knowledge among the partially related tasks without considering the task relatedness may cause negative transfer [Pan and Yang, 2010], which degrades the clustering performance. For example, we have two tasks to cluster the news from two news agencies by topics, one agency reports the news of Education, Sport and Science, the other agency reports the news of Education, Sport and Politics. Only the news on the topics of Education and Sport are useful for multi-task knowledge transfer, and transferring the knowledge of the news on the topics of Science and Politics may cause negative effect.

In the multi-task clustering literatures, only two methods can automatically assess the task relatedness. (1) DMTRC [Zhang, 2015] formulates the task relatedness as the covariance matrix by the Gaussian prior. But it assumes that the cluster numbers in all the tasks are the same and the label marginal distribution in each task is evenly distributed, which is too restrictive. (2) SAMTC [Zhang et al., 2016] first finds possibly related clusters between each pair of tasks and obtains a pair of subtasks, then it further learns the relatedness between each pair of subtasks. But it only transfers the instance knowledge between each pair of subtasks, which may lose the useful instance knowledge not in the subtasks.

In light of the limitations of the existing multi-task clustering methods, in this paper, we propose a multi-task clustering with model relation learning (MTCMRL) method, which can automatically learn the model parameter relatedness between each pair of tasks. The overall objective function of MTCMRL consists of two parts. (1) Within-task clustering: this part is to find the clusters within each task and update the clusters by knowledge transfer from the other tasks. We introduce linear regression model into symmetric nonnegative matrix factorization [Kuang et al., 2012] to accomplish this goal. The symmetric nonnegative matrix factorization captures the cluster structure embedded into the similarity matrix. The linear regression model provides a condition for transferring the knowledge of model parameters across the tasks. (2) Cross-task relatedness learning: this part is to learn the model parameter relatedness between the clusters in each pair of tasks, then the model parameter of each task can be updated by those of the other tasks. The basic intuition is that if the tasks are related, there are some clusters related to each other among these tasks. Thus the model parameters used for predicting the cluster indicator among these tasks should have some relatedness. We further present an effective alternating algorithm to solve the non-convex optimization problem. Experimental results show the superiority of the proposed method over traditional single-task clustering methods and existing multi-task clustering methods.

2 RelatedWork

Multi-Task Clustering: Most multi-task clustering methods make an ideal assumption that the tasks are completely related, i.e., they transfer knowledge among the tasks without considering the task relatedness. LSSMTC [Gu and Zhou, 2009] learns a subspace where the related tasks have the same centroids. LNKMTC and LSKMTC [Gu et al., 2011] learn a kernel space where the task distributions are close to each other. MCDA [Zhang and Zhou, 2012] learns a subspace where the task distributions are close to each other. ITCC [Xie et al., 2012] learns the feature relatedness among the related tasks by information theoretic co-clustering. MBC [Zhang and Zhang, 2010] and its improved methods (S-MBC and SMKC) [Zhang et al., 2015] alternatively update the clusters and learn the relationship between clusters of different tasks. SMT-NMF [Al-Stouhi and Reddy, 2014] introduces an intertask bias to reweight the distance between any two samples in different tasks. DMTFC [Zhang, 2015] learns the feature relatedness through Gaussian prior. MTCTKI [Zhang et al., 2017] transfers instance knowledge among the tasks under a shared subspace where the task distributions are close to each other. MTSC [Yang et al., 2015] learns a lower dimensional feature space by incorporating an l2,p-norm regularization term over the model parameters. However, in the real world, the tasks are usually partially related. Transferring knowledge among partially related tasks without considering the task relatedness may cause negative transfer [Pan and Yang, 2010], which degrades the clustering performance.

To our knowledge, in the multi-task clustering literatures, only DMTRC [Zhang, 2015] and SAMTC [Zhang et al., 2016] can automatically assess the task relatedness, but they have some limitations. DMTRC formulates the task relatedness as the covariance matrix by the Gaussian prior. But it assumes that the tasks have the same cluster number and the label marginal distribution in each task is evenly distributed, which limits its applicability. SAMTC first finds the related clusters between each pair of tasks and obtains a pair of subtasks, then it further learns the relatedness between each pair of subtasks, finally it constructs the similarity matrix for each task by exploiting the instances from the other subtasks. But it only transfers the instance knowledge between each pair of subtasks, which may lose the useful instance knowledge not in the subtasks.

Besides the multi-task clustering (unsupervised multi-task learning) methods which can automatically assess the task relatedness, many supervised multi-task learning methods are proposed for learning the task relatedness [Zhang and Yang, 2017]. A recent method in [Murugesan et al., 2017] learns both the task relationship matrix and feature relationship matrix by co-clustering the tasks and features. But it requires true labels to learn the task cluster matrix. In this paper, we concentrate on learning the task relatedness without labels.

Multi-View Learning: Different from multi-task learning which deals with the data from multiple tasks, multiview learning [Zhao et al., 2017] deals with the data which have the features from different views. Multi-view learning improves the learning performance by maximizing the agreement on multiple distinct views. There are mainly three representative ways for multi-view learning. Cotraining [Blum and Mitchell, 1998; Appice and Malerba, 2016] uses the most confident predictions in each view to iteratively train the learning model for all the views. Coregularization [Kumar et al., 2011; Minh et al., 2016] takes the disagreement between the classification or clustering results of any two views as a regularization term in the objective function. Canonical correlation analysis [Hotelling, 1936; Podosinnikova et al., 2016] aims to find the linear projections which have maximum correlation among different views. Recently, multi-view learning has been successfully applied to the gene network reconstruction [Ceci et al., 2015].

3 The ProposedMethod

3.1 ProblemFormulation

We are given T clustering tasks, each with a set of data points, i.e., Xt = {xt1 , xt2 , . . . , xtnt} ∈ Rd×nt (t = 1, . . . , T ), where nt is the number of data points in the t-th task, d is the dimensionality of the feature vectors. The similarity matrix of the t-th task is M t ∈ Rnt×nt . Each data set Xt is to be partitioned into ht clusters, i.e., C t = {C t1 ,C t2 , . . . ,C tht}. The cluster indicator of the t-th task is Y t ∈ Rnt×ht .

3.2 Within-Task Clustering

This component is to find the clusters within each task and update the clusters by knowledge transfer from the other tasks. We introduce linear regression model (LRM) into symmetric nonnegative matrix factorization (SNMF) to accomplish this goal. SNMF partitions the data points of each task into clusters, LRM provides a condition for transferring the knowledge of model parameters across the tasks.

Denote the parameter of the linear regression model in the t-th task as W t ∈ Rd×ht , the objective function of withintask clustering for the t-th task is

where M t is the similarity matrix of the t-th task, which can be computed by the similarity metric that accommodates to the data set. For example, we would use cosine similarity for document data sets, and adopt the Itakura-Saito divergence for music data sets. λ is a trade-off parameter to control the importance of LRM. µ is a regularization parameter for LRM.

In Eq.(1), the first term clusters the data points in the t-th task by SNMF. SNMF is a clustering technique based on the similarity matrix, which has the following advantages. (1) The cluster indicator Y t computed by SNMF is nonnegative, thus we can directly get the clustering results without postclustering on Y t . (2) SNMF retains the near-orthogonality of columns of the cluster indicator Y t , i.e., (Y t )T Y t = I, which is important for data clustering. The second term uses LRM to refine the cluster indicator Y t by learning a model parameter W t on the data Xt . LRM is a predictive approach which fits the output variable by using the linear combination of the features of the input sample. By updating the model parameter W t through cross-task relatedness learning, the cluster indicator Y t will be improved. The third term is a regularization term for the model parameter W t of LRM.

3.3 Cross-Task Relatedness Learning

This component is to learn the cross-task relatedness. Specifically, we learn the task relatedness by computing the model parameter relatedness between any two clusters in each pair of tasks. The basic intuition is that if the tasks are related, there are some clusters related to each other among these tasks. Therefore, we explore the relation of the clusters in different tasks by learning the relatedness of the model parameters corresponding to these clusters, then the model parameter of each task can be updated by those of the other tasks.

Denote the cluster model parameter relatedness between the t-th task and the s-th task as G(t,s) ∈ Rht×hs , the objective of learning G(t,s) is

where W t i ∈ Rd×1 is the i-th column of W t , which is a model parameter corresponding to the i-th cluster of the t-th task. G(t,s) ij is the similarity between the i-th cluster of the t-th task and the j-th cluster of the s-th task. The first termmeans that the smaller the squared Euclidean distance between W t i and W s j is, the higher the similarity between them is. The second term is to avoid the trivial solution that only the parameters W t i and W s j with the minimum squared Euclidean distance have the similarity 1, otherwise they have the similarity 0. β is a regularization parameter.

In Eq.(2), a larger G(t,s) ij means that the model parameters between the i-th cluster in the t-th task and the j-th cluster in the s-th task have a higher relatedness. If G(t,s) ij equals to 0, the model parameters between the i-th cluster in the t-th task and the j-th cluster in the s-th task have no relatedness, thus no model parameter knowledge will be transferred between these two clusters.

3.4 The Overall Objective Function

We integrate within-task clustering and cross-task relatedness learning into the overall objective function as follows.

where α is a trade-off parameter to control the importance of cross-task relatedness learning. According to Theorem

2 in [Ding and He, 2005], the orthogonality constraint (Y t )T Y t = I is retained in SNMF, thus we do not need to incorporate this constraint into optimization.

3.5 Optimization

Optimizing Eq.(3) is with respect to variables Y t , W t and G(t,s) . We alternatively optimize each variable by fixing the other variables.

Optimizing G(t,s) : Given Y t and W t , optimizing Eq.(3) with respect to G(t,s) is equivalent to optimize

Eq.(4) can be rewritten as

where g = (G(t,s) 1,: ,G(t,s) 2,: , . . . ,G(t,s) ht ,: )T , where G(t,s) i,: is the i-th row of G(t,s) . f = (A(t,s) 1,: ,A(t,s) 2,: , . . . ,A(t,s) ht ,: )T , where A(t,s) ij = ||W t i -W s j ||22 , A(t,s) i,: is the i-th row of A(t,s) . g and f are both ht × hs dimensional vectors. P = 2βI ∈ R(ht×hs )×(ht×hs ) . Eq.(5) is a convex quadratic programming problem, which can be solved by the quadratic programming optimizer such as the QUAD function in MATLAB.

Optimizing W t : Given Y t and G(t,s) , optimizing Eq.(3) with respect toW t is equivalent to optimize

By omitting the constant, Eq.(6) can be rewritten as

where H(t,s) is a diagonal matrix with H(t,s) ii = ∑hs j=1 G(t,s) ij .

Eq.(7) can be transformed to a convex formula with constraint.

The Lagrangian function of Eq.(8) is

where Γt ∈ Rnt×ht is a Lagrangian multiplier. Eq.(9) is convex with respect to W t and ξ t . According to Karush-KuhnTucker (KKT) condition ∂L(W t ,ξt ) ∂W t = 0 and ∂L(W t ,ξt ) ∂ξt = 0 [Boyd and Vandenberghe, 2004], we have

By introducing Eq.(10) and Eq.(11) into Eq.(9), we get the dual problem [Boyd and Vandenberghe, 2004] of Eq.(8).

Setting ∂JΓt ∂Γt = 0, we have

Eq.(13) is the Sylvester equation, which can be solved by the off-the-shelf LYAP function in MATLAB.

Optimizing Y t : Given W t and G(t,s) , optimizing Eq.(3) with respect to Y t is equivalent to optimize

The Lagrangian function of Eq.(14) is

Algorithm 1 MTCMRL

Input: T tasks {Xt }Tt=1 , cluster number of each task {ht }Tt=1 . The parameters λ, µ, α and β. Initializing Mt (t = 1, . . . , T ) by the similarity metric which accommodates to the data set. Initializing Y t (t = 1, . . . , T ) by the k-means method, then setting Y t = Y t + 0.2. Initializing W t (t = 1, . . . , T ) as an nt × ht matrix of ones.

Output: Partitions {Ct }Tt=1 . repeat for t = 1 to T do for s = 1 to T do if s ̸= t then Compute the parameter relatedness G(t,s) by Eq.(5). end if end for Compute the model parameter W t by Eq.(10). Compute the cluster indicator Y t by Eq.(19). end for until Eq.(3) is convergent.

where △ ∈ Rnt×ht is a Lagrangian multiplier. Setting ∂L(Y t ) ∂Y t = 0, we have

According to KKT condition △ijY t ij = 0, we get

By introducing Q = (Xt )T W t and Q = Q+ - Q- [Ding et al., 2010], where Q+ = (|Q|+Q)/2 and Q- = (|Q|-Q)/2. Eq.(17) can be rewritten as

Eq.(18) leads to the following multiplicative update formula

When initializing the cluster indicator Y t , we follow the traditional nonnegative matrix factorization methods, i.e., initializing Y t by the k-means method, then setting Y t = Y t + 0.2 [Ding et al., 2006].

The overall optimizing process of MTCMRL is listed in Algorithm 1. The clustering performance will be affected by different initializations of W t . Generally we can initialize it as a fixed matrix which is an nt × ht matrix of ones.

3.6 Time Complexity Analysis

Denote n as the sample number in each task, d as the feature number, ht and hs as the cluster numbers of the t-th task and the s-th task, T as the task number, ˜ I as the iterations of k-means, ˆ I as the iterations of MTCMRL. The time complexity of the initialization process for running k-means and computing the similarity matrix M t is O(T ˜ Ihtdn + T n2d). The time complexity of computing G(t,s) is O(ˆ IT 2 (hths )3 ) through quadratic programming, but since the cluster numbers of the tasks are usually much smaller compared with the sample number n and feature number d, optimizing G(t,s) usually does not cost too much time. If the cluster number becomes very larger, we can use the Frank-Wolfe method [Wen et al., 2015] instead of using the quadratic programming. Then the time complexity of computing G(t,s) can be reduced to O(ˆ IT 2hths ). The time complexity of computing W t is O(ˆ IT (dhths + dnht )). The time complexity of computing Y t is O(ˆ IT (n2ht + ndht )). Since ht , hs , T , ˜ I, ˆ I are usually much smaller than n and d, the overall time complexity of MTCMRL by omitting them is O(n2d). This time complexity is the same as that of the traditional clustering method kernel k-means [Dhillon et al., 2004].

4 Experiments

4.1 Methods and EvaluationMetrics

We compare MTCMRL with the single-task clustering methods k-means, symmetric nonnegative matrix factorization (SNMF) [Kuang et al., 2012], constrained Laplacian rank graph-based clustering (CLR) [Nie et al., 2016] and symmetric nonnegative matrix factorization with linear regression model (SNMF+LR), which is a single-task version of MTCMRL with α = 0, β = 0. We also compare MTCMRL with the multi-task clustering methods: the shared subspace learning multi-task clustering (LSSMTC) method [Gu and Zhou, 2009], the smart multi-task Bregman and Kernel clustering (S-MBC and S-MKC) methods [Zhang et al., 2015], the convex discriminative multi-task feature clustering (DMTFC) method [Zhang, 2015], the convex discriminative multi-task relationship clustering (DMTRC) method [Zhang, 2015], the multi-task spectral clustering (MTSC) method [Yang et al., 2015], and the self-adapted multi-task clustering (SAMTC) method [Zhang et al., 2016].

We use two performance measures in [Xu et al., 2003]: clustering accuracy (Acc) and normalized mutual information Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18)

(NMI) to evaluate the clustering performance.

4.2 Data Sets

WebKB4 1 : This data set contains web pages collected from computer science department websites at 4 universities: Cornell, Texas, Washington and Wisconsin. They are divided into 7 categories, we choose 4 most populous categories such as course, faculty, project and student for clustering.

20NewsGroups 2 : This data set consists of the news documents under 20 categories, we choose 4 most populous root categories such as comp, rec, sci and talk for clustering.

Reuters 3 : This data set is composed of the news documents under 135 categories from the Reuters newswire, we choose 4 most populous root categories such as economic index, energy, food and metal for clustering.

There are three typical cases for multi-task data setting. The first case is that the tasks are completely related (i.e., the tasks share the same categories), we use the WebKB4 data set to represent this case. The second case is that the tasks are partially related (i.e., the tasks share parts of categories) and the cluster numbers in all the tasks are not the same, we use the 20NewsGroups data set to represent this case. The third case is that the tasks are partially related and the cluster numbers in all the tasks are the same, we use the Reuters data set to represent this case. The details of constitutions of these data sets are shown in Table 1.

1 http://www.cs.cmu.edu/afs/cs.cmu.edu/project/theo-20/www/data/

2 http://qwone.com/∼jason/20Newsgroups/

3 http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html

4.3 Parameter Investigation

We investigate the impact of the parameters λ, µ, α and β on the clustering performance for MTCMRL. For each data set, we repeat the MTCMRL method 10 times by setting one parameter to search the grid and fixing the other parameters, then compute the average Acc and NMI of all the tasks under this parameter. More specifically, we successively set one parameter to search the grid {2-2 , 2-1 , 20 , 21 , 22 } by fixing λ, α = 22 and µ, β = 2-1 . From the parameter investigation in Figure 1, it can be seen that MTCMRL is more sensitive to the linear regression model trade-off parameter λ and the cross-task relatedness learning trade-off parameter α than their corresponding regularization parameters µ and β. The span of clustering performance with respect to λ and α is 2% ∼ 5%, whereas the span of clustering performance with respect to µ and β is less than 2%. λ and α control the main body of the linear regression model and the cross-task relatedness learning, thus playing major roles in optimizing the variables.

4.4 Parameter Setting

We exploit the grid searching method which is commonly used in multi-task clustering [Gu and Zhou, 2009] to identify the optimal parameters. Based on the parameter investigation, for MTCMRL, we set λ and α to search the grid {2-2 , 2-1 , 20 , 21 , 22 }, µ = 2-1 , β = 2-1 , and we choose cosine similarity to compute the similarity matrix. For CLR, the neighborhood size is searched from {10, 20, . . . , 100}. For LSSMTC, λ is set by searching the grid {0.1, 0.2, . . . , 0.9}, the dimensionality of the shared subspace is set by searching the grid {2, 4, 6, 8, 10}. For S-MBC and S-MKC, λ is set by searching the grid {0.1, 0.2, . . . , 1}. For DMTFC and DMTRC, λ1 and λ2 are both set by searching the grid {2-10 , 2-8 , . . . , 2-2 }. For MTSC, α and β are both searched from {2-2 , 2-1 , 20 , 21 , 22 }. For SAMTC, the number of nearest neighbors within each task and across the other tasks are both searched from {30, 60, 90, 120}, and we choose cosine similarity to compute the nearest neighbors.

4.5 Clustering Results

As CLR, DMTFC and DMTRC are convex optimization methods, we perform them once under each parameter setting, and show the clustering results under the best parameter setting. For the other methods, we repeat each method 10 times under each parameter setting, and show the mean clustering results and the standard deviations under the best parameter setting. As LSSMTC, DMTFC and DMTRC can only apply to the case that the cluster numbers of all the tasks are the same, we perform them on WebKB4 and Reuters. We report the clustering results in Table 2, Table 3 and Table 4, and it can be seen that:

1. MTCMRL performs better than the single-task clustering methods, since MTCMRL exploits the information from the related tasks, whereas the single-task clustering methods only utilize the information within each task. SNMF and SNMF+LR have the same clustering performance with MTCMRL on the Task 1 of Reuters, but they perform worse than MTCMRL on the Task 2 and Task 3 of Reuters. Because SNMF and SNMF+LR perform the same on the Task 1 of Reuters, showing that the linear regression model can not affect the clustering performance. Therefore, updating the model parameter by cross-task relatedness learning does not help improve the clustering performance on the Task 1 of Reuters.
2. LSSMTC performs worse than MTCMRL, because it assumes that there exists a shared subspace in which all the tasks share the same centroids, which is too idealized and restrictive. Moreover, LSSMTC may suffer from negative transfer when the tasks do not share the same categories like the tested data set Reuters.
3. S-MBC and S-MKC perform worse than MTCMRL, because they require the distributions of the tasks to be the same or similar, but the task distributions in the tested data sets do not show very high similarity.
4. DMTFC performs worse than MTCMRL in most cases, because DMTFC requires that the tasks share the same categories, which is seldom satisfied. As the tested data set WebKB4 meets this requirement and DMTFC is a convex optimization method, DMTFC performs the best on the Task 2 of WebKB4, whereas MTCMRL performs the second best.
5. Although DMTRC can automatically learn the task relatedness, its clustering performance is not so good as we expected. Because it assumes that the label marginal distribution in each task is evenly distributed, which is too strict.
6. MTSC performs worse than MTCMRL, except for the Task 1 of Reuters on which MTSC and MTCMRL show the same clustering performance. Because it manually controls the degree of sharing the lower dimensional feature space among the tasks, and it does not explicitly consider how to set the intertask correlation from the tasks themselves.
7. SAMTC performs worse than MTCMRL, because it only transfers the instance knowledge between each pair of subtasks, which may lose the useful instance knowledge not in the subtasks.
8. The reasons stated above are also why LSSMTC, S-MBC, S-MKC, DMTFC, DMTRC, MTSC and SAMTC sometimes even perform worse than the single-task clustering methods.
9. MTCMRL performs much better than the compared multi-task clustering methods in most cases, because MTCMRL can automatically learn the model parameter relatedness between each pair of tasks, which can exploit the positive relatedness among the tasks and avoid negative transfer. An exception is that for the Task 2 of WebKB4, MTCMRL performs the second best, whereas DMTFC performs the best, because DMTFC is a convex optimization method, and the WebKB4 data set happens to meet the requirement of DMTFC that the tasks share the same categories. Among the existing methods, DMTRC and SAMTC can also automatically learn the task relatedness. But DMTRC is based on a strict assumption that the label marginal distribution in each task distributes evenly and the tasks have the same cluster number, which is seldom satisfied. SAMTC only transfers the instance knowledge between each pair of subtasks, which may miss some potentially useful information that are not in the subtasks. These limitations will degrade their clustering performance.

Summarizing the above discussions, it can be concluded that MTCMRL performs the best among the compared methods.

4.6 Cluster Model Parameter Relatedness

For each data set with T tasks, there are T (T - 1) cluster model parameter relatedness matrices G(t,s) (t, s = 1, . . . , T , s ̸= t). We show the cluster model parameter relatedness matrix G(1,2) between Task 1 and Task 2 in the WebKB4, 20NewsGroups and Reuters data sets, respectively.

For WebKB4, the learnt cluster model parameter relatedness matrix G(1,2) is For 20NewsGroups, the learnt cluster model parameter relatedness matrix G(1,2) is

For Reuters, the learnt cluster model parameter relatedness matrix G(1,2) is

A larger G(1,2) ij means that cluster i in Task 1 and cluster j in Task 2 have a higher relatedness. Note that the cluster labels have been mapped to their true labels in the tested data sets through the permutation mapping function in the clustering accuracy performance measure [Gu et al., 2011]. Generally the clusters with the same label between Task 1 and Task 2 have the highest relatedness, e.g., for the 20NewsGroups data set, cluster 1 with the label "Comp.graphics" in Task 1 has the highest relatedness with cluster 1 with the label "Comp.os.ms-win.misc" in Task 2 (0.1730 is the biggest value in the first row of G(1,2) ). The same phenomenon can be observed for the clusters with the same label between the other pairs of tasks, we omitted these matrices due to space limitation.

4.7 Convergence Curves

The convergence criterion of the MTCMRL algorithm is to set a convergence threshold, which is 1 in the experiments. Once the objective value of Eq.(3) in the previous step minus that in the current step is smaller than 1, Algorithm 1 will converge. In the tested data sets, Algorithm 1 will converge after 99, 100 and 28 iterations for WebKB4, 20NewsGroups and Reuters, respectively. The convergence curves are shown in Figure 2.

4.8 Running Time of Cluster Model Parameter Relatedness Learning

In this section, we investigate the running time of learning the cluster model parameter relatedness G(t,s) under different cluster number settings. More specifically, we select Task 1 and Task 2 in 20NewsGroups as the tested data set, and set the cluster numbers h1 and h2 (h1 = h2 ) in Task 1 and Task 2 by searching the grid {5, 15, 25, 35, 45, 55, 65, 75, 85, 95}. Then we use the quadratic programming method and the FrankWolfe method to learn G(1,2) , respectively. The running time of learning G(1,2) under different cluster number settings is shown in Figure 3.

From Figure 3, it can be seen that the running time of learning G(1,2) by quadratic programming rises sharply when the cluster number is larger than 35. Whereas the running time of learning G(1,2) by the Frank-Wolfe method rises slowly with the increase of cluster number. Based on this observation, when the cluster number is smaller than 35, we can use quadratic programming to learn the cluster model parameter relatedness G(t,s) , otherwise we will use the Frank-Wolfe method to learn G(t,s) .

5 Conclusion

In this paper, we have proposed a multi-task clustering with model relation learning (MTCMRL) method, which automatically learns the model parameter relatedness between each pair of tasks. MTCMRL first introduces the linear regression model into symmetric nonnegative matrix factorization, then learns the model parameter relatedness between the clusters in each pair of tasks by constructing a similarity matrix. We further present an effective alternating algorithm to solve the non-convex optimization problem. Experimental results on several real data sets show the superiority of the proposed method over traditional single-task clustering methods and existing multi-task clustering methods.

Acknowledgments

This work was supported by National Science Foundation of China (No. 61632019).