Metadata-driven Task Relation Discovery for Multi-task Learning

1 Introduction

Task Relation Discovery (TRD), i.e., reveal the relation of tasks, has notable value: it is the key concept underlying Multi-task Learning (MTL) and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity [Zhang and Yang, 2017a; Zamir and others, 2018].

To automatically extract the task relation, historical training samples are commonly leveraged and heavily relied on [Liu and others, 2017; Zhang et al., 2016]. Nevertheless, in many real-world applications, training samples capture merely partial relation. For example, when inferring task relation for thermal comfort prediction (i.e., whether one is comfort, cold or hot), if similar tasks are clustered merely using samples of sensing data (e.g., temperature and humidity), it can miss some intrinsic domain information (e.g., climate, building information like whether the building is equipped with air-conditioner or not). In such cases, merely using samples for TRD easily leads to incorrect task relation construction and transferring knowledge between unrelated tasks degrades the MTL performance, which is referred to as the negative transfer [Pan and Yang, 2010]. That is especially critical in multi-task transfer learning scenarios where data samples of targeted tasks are usually insufficient due to sensing networking, or privacy issues, i.e., the data size is too small to automatically extract reliable and valuable intrinsic information without integrating additional domain knowledge. However, in real-world applications, TRD with domain knowledge as apriori information requires non-trivial human effort, bringing difficulties to the wide use of such approaches [Evgeniou and others, 2005; Kato et al., 2008; Han and others, 2014].

In this paper, we, for the first time, leverage metadata to handle the commonly existing negative-transfer issue in TRD literature. Metadata, i.e., the context attribute in the database of real-world systems [Balaji and others, 2016; Choi and others, 2017], is usually designed by domain experts for daily operations of the system and available before the learning and prediction process. Metadata usually contain intrinsic task information, such as the climate in the above example. Using metadata, we are able to extract task relation with domain knowledge (but at the same time) in an automatic manner and delivers a punch on the automatic TRD.

Metadata helps to avoid the negative transfer and reduce human effort. However, there are mainly three challenges to further release the potentials of metadata in MTL: (1) The first challenge lies in the representation generation. Originally, Metadata is developed by domain experts for the purpose of daily operations of modern systems, instead of designed by data scientists for the targeted prediction task. Due to the different objectives, simply leveraging all raw metadata in the database can lead to a negative transfer. (2) The second challenge lies in the different types and contents of metadata. In TRD, we need to quantify the task similarity with the introduced metadata representation. However, straightforward 1-hot encoding or Gower's coefficient methods on metadata can introduce false information when metadata are incomparable and cooperate for task relation discovery. (3) The final challenge is that directly adding metadata may not fully avoid negative transfer when additional constraints with metadata are not considered and properly integrated into the system.

In this paper, we propose a novel Metadata-driven TRD for MTL, which can automatically and effectively capture task relation through not only historical samples but also additional metadata. The contributions of the paper can be summarized as:

1. We are the first to introduce metadata into TRD. We propose the collection, and selection of metadata, which aims to automatically select useful metadata as task attributes to avoid negative transfer.
2. We leverage the common neighbor characteristic for distance measurement to tackle incomparable metadata. We discuss the interaction between metadata and data samples. We propose a two-phase clustering taking metadata as apriori constraints for regularization.
3. We evaluate the proposed method on five real-world datasets. We also discuss the advantages, limitations and application scenarios at the end of the paper.

We offer our approach as an attractive mechanism for MTL researchers and application developers who are on the lookout for automatic and effective approaches in MTL, especially for complicated systems with diverse metadata but insufficient data samples.

2 RelatedWork

Task Relation Discovery (TRD) is the key issue and the focus of recent works in transfer learning and multi-task learning [Zhang and Yang, 2017a], including mainly two types:

The first is to conduct TRD through apriori information with additional human effort. Task similarity graphs are given and leveraged to make the model parameters of similar tasks close to each other, based on domain knowledge [Evgeniou and others, 2005; Kato et al., 2008; Han and others, 2014]. However, in real-world applications, such apriori information constructed according to domain knowledge may not be easy to obtain, bringing difficulties to the wide use of such approaches. Few Shot Learning (FSL) [Larochelle et al., 2008; Isele et al., 2016] focus on transferring knowledge to tasks with little or no training sample, by merely leveraging task descriptors for TRD in MTL. However, human effort is still needed for data scientists to determine the specific task descriptors and their values. When it comes to TRD, FSL also ignores the interaction between training samples and task attributes, due to its assumption of little or no training sample.

Second, historical samples are used in MTL for automatic TRD which reduces the human effort [Zhang et al., 2016; Liu and others, 2017]. However, these methods merely rely on the training samples of the task for relation discovery, which can result in missing intrinsic information in complex systems, as reported in [de Roux and others, 2018].

Multi-task Learning (MTL) [Caruana, 1997] is developed to improve the performance of targeted tasks using information from the source tasks. In this sense, MTL is related to transfer learning [Pan and Yang, 2010], but the targeted tasks in MTL are learned simultaneously, whereas those tasks in transfer learning are learned independently. A survey is available in [Zhang and Yang, 2017b].

There are mainly two ways to transfer knowledge in MTL.

For targeted tasks, 1) Instance Transfer groups and reuses samples of the other tasks, e.g., Clustered MTL (CMTL) [Liu and others, 2017; Xu and others, 2015; Wang et al., 2009], while 2) Feature Representation Transfer groups and learns a common feature representation among the related tasks, e.g., Alternating Structure Optimization [Ando, 2006]. Particularly, our attention has been drawn to the study of instance transfer, i.e., CMTL, which naturally adapt to different feature engineering and learning models. But unlike traditional CMTL, we explicitly model the relations among tasks and extract meta-structure in an automatic manner. It is also possible to maintain a multi-task network.

3 Metadata-driven TRD Problem

In this section, we first give the notations and definitions used in the paper, especially the concept and collection of metadata. Then we formally define the problem of metadatadriven TRD for MTL.

Let xiv ∈ R denote the ith feature at index v, where i ∈ [1,m],m, ∈ N; Let yv denote the predicted label at index v. A sample sv of task j includes the corresponding feature and the predicted label, i.e., sv = (Xv, yv) ⊂ Rm × R, v ∈ Vj .

Definition of Task. A task is an abstraction read from raw data. Traditionally, each task t is associated with a set of training sample lt = [lv], ∀v ∈ Vti , and a learning function f Θ t (·) where Θ denotes the model parameter and the inferred result ˆ yv = f Θ t (Xv). We additionally associate each task with task attributes at. Thus, a task in our paper is a threetuple t = {at, lt , f Θ t (·)}.

3.1 Metadata

TRD is well-known as the key issue in MTL, which heavily relies on the task attribute. To extract the attributes, we introduce a new data source of metadata which helps to better identify, organize and describe the context of varying tasks.

Metadata is the descriptor of information about one or more aspects of a dataset, i.e., data about data [Wikipedia, Accessed 2 DEC 2018]. It is used to summarize basic information about data which can make tracking and working with specific data easier in a real-world system. Many distinct types of metadata exist, including descriptive metadata, structural metadata, administrative metadata, reference metadata, and statistical metadata [Riley, Accessed 5 DEC 2017].

Table 1 shows the collected metadata of thermal comfort tasks mentioned in the Introduction. It shows the season and building where historical samples of tasks are collected. Each metadata record consists of the identity (e.g., Task 11-1), the information type (e.g., Season) and value (e.g., Summer).

Note that metadata is designed to manage the dataset for various daily operations in modern systems. It is thus common that metadata is already collected and directly available in the database for a running system, serving as a hyper table used to manage other tables of samples. That is especially true for a public database with various datasets, e.g., an example of such a hyper table for the thermal comfort database of ARP-884 is in [The University of Sydney, 2018]. Thus, the first way to collect the metadata is to find and visit the hyper table for the database.

Another way is to collect metadata from those attributes in the samples. Table 2 shows the metadata in samples of thermal comfort tasks. It shows the season, duration of hours and the year when historical samples of tasks are collected. We see that the value of metadata for a specific task/ context is consistent regardless of time changes, e.g., a certain thermal comfort dataset of a task has a fixed value of the year of data collection. Accordingly, we observe a common characteristic that the metadata usually has duplicated values in a table of samples and merely a few values across all tables of samples. This characteristic can be used to detect and collect metadata, of which those samples are commonly regarded as informationless and even discarded in practice. For interested readers, more metadata extraction techniques can be found in [Liu and others, 2007].

3.2 ProblemDefinition

The output of our TRD problem is the Metadata-driven Task Mapping. It is a computationally found hypergraph with an emphasis on metadata of tasks. The metadata is used to infer the edge in the hypergraph. An edge between a group of source tasks and a target task represents a feasible transfer case and its weight is the prediction of its performance. We use these edges to estimate the globally optimal transfer policy to solve tasks. The final metadata-driven task relation produces a family of such graphs, parameterized by the chosen tasks, transfer orders, and transfer functions' expressiveness.

Formally, the task mapping is the match of transfer-able source and target tasks, which is defined as Θ = {T : S} where T is the set of tasks which we want to solve (target), and S is the set of tasks that can be trained (source).

Our problem of Metadata-driven TRD for MTL is inferring a task mapping Θ which maximizes the collective performance on a set T of tasks t ∈ T, given training samples and metadata of tasks.

It is critical to note the task mapping is meant to be a sampled set, not an exhaustive list, from a denser space of all conceivable tasks. This gives us a tractable way to sparsely model a dense space, and the hypothesis is that (subject to a proper sampling) the derived model should generalize to outof-mapping tasks. The more-regular-sampled the space, the better the generalization. To this end, our task mapping with metadata is, in fact, a regularization on the space which facilitates a more general transfer process. We show this in the Evaluation Section.

4 Methodology

The task mapping is built using a three-step process. In stage I, a metadata graph for each task in S is generated and effective metadata is selected for each task. In stage II, the task affinities acquired using metadata are computed and normalized, and in stage III, we synthesize a hypergraph by clustering which can gather similar tasks, use multiple inputs task to transfer to one target, and avoid negative transfer with both metadata and training samples.

4.1 Stage I: Task-specific Sampling

As the base of the task mapping, we establish a knowledge graph using metadata to organize tasks and task attributes, named metadata graph. Based on the existing work [Balaji and others, 2016] and [Xie et al., 2016], we denote the metadata as a triple {task, information type, information value}. For example, "Task 19-1 is conducted in the Season of Winter" can be denoted as {Task 19-1, Season, Winter}. Formally, let the nodes M, T ∈ U in the graph, i.e., G(U, E), denote the metadata and task, respectively, where U = M ∪ T ; the edge E in the graph represents the information type between nodes. Note that the edge weight w ∈ [0, 1] on the metadata graph refers to the relatedness between the task and metadata. E.g., it is 1 if nodes are related and otherwise 0, which can be determined using the performance improvement in the task attribute selection.

Raw metadata is not necessarily related to the targeted prediction task since it is designed by domain experts to serve various daily operations instead of our targeted prediction tasks. Both the objectives and approaches are significantly different when designing task attributes and metadata. We select metadata as a task attribute if it brings improvement of collective performance for tasks.

4.2 Stage II: AffinityMeasurement withMetadata

We want an affinity matrix of transfer abilities across tasks based on the metadata graph. We focus on discussing the distance measurement of metadata and omit that of samples in existing works due to page limitations. Between nodes of numerical and ordinal metadata, the distance can be computed using cosine similarity, with normalization to tackle the value in vastly different scales. An analytic hierarchy can be used to facilitate more effective normalization.

When it comes to nodes of categorical metadata, there are quantifying approaches including the straightforward 1-hot or 1-of-K encoding, the more-widely-used Gower's similarity coefficient [Gower, 1971] and its extensions [Legendre and Legendre, 2012; Podani, 1999]. However, these approaches basically treat categorical metadata as constant ordinal or numerical ones, which encodes a hidden ranking/ order relation across the metadata and brings noise to the final distance function. They also simply assume that different categories are independent and unrelated. Thus the above metadata processing methods are not suitable for MTL.

To capture the cooperation and avoid encoding a additional ranking relation, our idea is to leverage the number of common neighbors for distance measurement. The hypothesis is that a task should be more similar to another task, if it has higher numbers of shared metadata with another task. We compute Jaccard Distance [Jaccard, 1908], to compare the number of common categorical neighbors for task nodes a, b and jointly consider different metadata Na ∪ Nb: dinc(a, b) = 1 - |Na ∩ Nb|/|Na ∪ Nb|, where Na, Nb are categorical metadata and neighbors of a, b.

For all possible pair task nodes (a, b), the output is named prior distance matrix, which is computed using metadata, i.e., dmeta = [dinc(a, b)]. An example of the prior distance matrix is shown in Figure 1.

4.3 Stage III: Metadata-assisted TaskMapping

The final step is to infer task mapping based on the affinity matrix inferred from weighted metadata and training samples, e.g., d = w0dmeta + w1dsamp, where dsamp denotes the distance matrix computed with samples. However, when we introduce metadata, there is also a risk of introducing more negative transfers. For example, tasks with quite different values of metadata, e.g., thermal comfort tasks with completely different types of buildings and seasons, is likely to be inreasonable to transfer their samples to each other.

To avoid negative sample transfer between completely different tasks, we propose a two-phased clustering, i.e., prior clustering with metadata and the posterior clustering with samples. The prior clustering defines clusters using the metadata which limitting the variance of task attributes within it. The posterior clustering conducts further clustering with historical training samples based on the prior clusters, i.e.,

where Θ denotes the allocation of sample index for nc clusters, i.e., the matrix of V k j = {v|v ∈ cluster k for taskj}, k ∈ [0, nc], j ∈ [1, J]; c0(·) and c1(·) denote the prior and posterior clustering function. In this rest of this section, we focus on discussing the metadata clustering and omit the traditional clustering with merely samples due to page limitations.

As for the prior clustering, we propose to use Normalized Minimum Cut [Shi and Malik, 2000] for clustering given the distance matrix. The Normalized Minimum Cut can naturally adopt the case when the metadata graph is unconnected.

Consistent with our Thermal Comfort Dataset in previous sections, we also provide examples of prior-and-posterior phase clustering in Figure 2.

5 Evaluation

5.1 Experimental Setup

For the parameters, we apply grid searching [Gu and Zhou, 2009] to identify the optimal values. For all clustering algorithms, the number of nearest neighbors is set by searching the grid {⌈ ns 2×nc ⌉, ⌈ ns nc ⌉, min(⌈ 2×ns nc ⌉, ns)}, where ns and nc are the number of training samples and clusters. The number of clusters is set as the number of classes in each dataset. For SAMTL and our posterior-phase clustering, cosine similarity is used to compute nearest neighbors as [Zhang et al., 2016].

To capture the predictive capabilities of our approach, we use three metrics: Error Rate (ER), Root-mean-square error (RMSE) and Symmetric Mean Absolute Percent Error (SMAPE) for evaluation.

where N is the number of testing data; ˆ yn and yn are the estimation and the ground truth of the nth sample, respectively.

5.2 Datasets

The dataset is shown in Table 3. We use the following realworld datasets for our experiments on the baseline methods and the proposed method.

Hong Kong Industry Chiller Data (HK-BCOD) is used to predict the efficiency of different chillers, which are machines to generate cooling power for office buildings. The higher accuracy means more energy saving for cooling load allocation. For three office towers located in Hong Kong, chiller sensor data were collected from the building management system for four years at daily intervals.

ASHRAE RP-884 Thermal Comfort Data (ARP-884) is used to predict occupants' feeling of comfort, cold or hot. The data we use in this study is from a Public dataset of ASHRAE, which is a global professional association seeking to advance heating, ventilation, air conditioning, and refrigeration systems design and construction.

China Mobile Base Station Trace (CHM-BST) is collected to predict the number of mobile devices in Tianjin, China. It contains mobile traces collected from five base stations of China Mobile, a dominant carrier of the local mobile network, in Tianjin, China from 15th to 30th, August 2016.

IBMHKWeather Data (IBM-HWD) is used to predict the temperature in different locations of Hong Kong, China. We collected four-year meteorology data from the Public website of Weather Underground of IBM.

Hong Kong Traffic Sensing Data (HK-TSD) is used to predict the traffic speed in Hong Kong, China. We collected the four-month data in six-minute intervals from the Public government website of data.gov.hk.

5.3 Results on Clustering

To indicate the effect of metadata, Figure 3 compares the clustering result of Metadata Clustering (on both samples and metadata) with Sample Clustering (on mere training samples) in HK-BCOD, where tasks are conducted to predict the efficiency of chillers. We mark those chillers of two modelTypes, which are supposed to have different models, as 0 and 1. The modelType information is recorded in the metadata. In Sample Clustering, data samples of different modelTypes are mixed in the same cluster, which can incur negative transfer; whereas in Metadata Clustering, all samples in each cluster will be in the same modelType. That is because Metadata Clustering not only leverages additional metadata, but also treats the metadata as clustering constraints by using the Twophase Clustering; whereas Sample Clustering heavily relies on the historical training samples for clustering, which may result in the missing of some intrinsic information. Other datasets also reveal similar results and we omit the results due to page limitations.

5.4 Results onMTL

With the clustered samples for training tasks, we determine the learning model to maximize the accuracy of our given tasks. Let yj = [yv],Xj = [xiv ], v ∈ Vj denote label and feature vector of task j. Then, formally, we are to solve the following commonly-accepted MTL optimization problem [Caruana, 1997; Baxter and others, 2000; Evgeniou and Pontil, 2004]. Given a loss function l(·, ·), clustering parameter Θ, historical data of X and y, we are to infer W = argmin ∑j∈J 1/|Vj| ∑v∈Vj l(f Θ j (Xv,wv), yv) + P(W,Θ), where P(W,Θ) is the regularization term on parameters; Note that Θ, X and y are used to train parameters W, by merely using samples of tasks in the same cluster in the training process.

To evaluate the proposed method, we employ the following state-of-the-art methods as baselines. We compare our metadata-clustered MTL (mCMTL) with those without using metadata: (1) Single Task Learning (STL), which learns a single model by pooling together data from all tasks of each dataset; (2) Independent Multi-task Learning (IMTL), which learns each task independently without sharing any sample or knowledge; (3) the Self-Adapted Multi-task Learning (SAMTL), which learns the cluster of tasks merely with historical training samples as in [Zhang et al., 2016].

For all tasks, we use Support Vector Regression (SVR) model for the baseline prediction of all tasks since it is simple and allows to see the impact of different MTL methods. We chronologically order each dataset and use the first 1/2 for training and the remaining 1/4, 1/4 for evaluation and testing.

We use datasets to construct two typical cases in MTL: 1) The first case is that the metadata is not completely the same for all tasks. We use HK-BCOD, ARP-884, and CHM-BST to represent this case. 2) The second case is that the metadata are completely the same for all tasks. We use IBM-HWD, and HK-TSD to represent this case.

All our experiments are conducted in a private cloud with 16 cores of 2.6GHz CPU and 64G memory. The training time of all tasks is of seconds and the prediction time is even much less than one millisecond. Table 4 and Table 5 summarize the overall results of all the compared methods with respect to the three evaluation metrics. We have the following observations.

The Impact of Metadata. mCMTL generally outperforms SAMTL. This is because mCMTL considers metadata information for transfer which also captures different intrinsic characteristics of tasks, whereas SAMTL captures the related tasks merely by using the training samples, e.g., sensed from the outside environment.

The Impact of Multi-task Setting. mCMTL always performs better than Single Task Learning (STL) method, since mCMTL establishes different models for different tasks, whereas the single task learning assumes that tasks are of the same model and does not well capture the intrinsic characteristic and dynamic environmental difference among the tasks.

The Impact of Transfer. mCMTL always performs better than IMTL in HK-BCOD, ARP-884, and CHM-BST which contain less training data samples. This is because mCMTL exploits the information across the related tasks, whereas the IMTL only utilizes the information within each task.

The Scope of mCMTL. IMTL can outperform mCMTL and SAMTL in IBM-HWD and HK-TD. When training samples are sufficient, transfer between tasks may not be necessary and independent methods like IMTL can also work well. In addition, since tasks have the same metadata, mCMTL and SAMTL still keep transferring knowledge between different tasks which can incur a slight negative effect. mCMTL is more suitable for cases where samples are insufficient and the metadata is not completely the same.

6 Discussion

MetadataManagement. The proposed method uses metadata available from many modern systems. It does not require any major human effort because we focus on metadata from one system each time, where the semantics of metadata remain consistent. We leave metadata management from varying systems as future work. Interested readers can refer to specific metadata database, e.g., Project Brick [BETS Research Group, Accessed 1 DEC 2017], CANSIM [Dunstan and Humphrey, 2005] and METeOR [Australian Institute of Health and Welfare, 2018], and metadata standards from ISO and ANSI, e.g., ISO/IEC 11179 [ISO/ IEC JTC 1, 2018].

Semantics and Semantics Similarities. Semantics was developed to find the optimal context label and representation. Applications of semantics have benefitted the constant context adaptation in the domains such as semantic web and knowledge graph, as correctly pointed out by the reviewer. Our application differs from semantic web, etc, because, at least for the time being, metadata usually do not have strong semantics, and there lacks other information, e.g., an expert dictionary, to improve semantic analysis. We have tried to use multi-source semantic analysis for our metadata. We do observe that results can be slightly improved. Nevertheless, the improvement is light also because each task has a number of metadata, and the similarity of two tasks is more influenced by the number of similar metadata. We believe that we can find some applications where the semantics of metadata can have a stronger impact.

7 Conclusion

In this paper, we introduced metadata into TRD for MTL and proposed Metadata Clustered Multi-Task Learning (mCMTL). It leverages not only training samples, but also additional metadata, to automatically exploit the positive relatedness between tasks, which reduces human effort for data scientists in preparing the prior task information and avoids the negative transfer by identifying the reusable training samples between such related tasks. Experiments on many realworld datasets demonstrate the superiority of the proposed algorithm over existing MTL methods. In general, this study helps in automatic relation discovery among partially related tasks and sheds new light on the development of TRD in MTL through the use of metadata as apriori information.

Acknowledgements

This work was supported in part by CRF C5026-18G, in part by the Hong Kong ITF UIM/363, in part by Technical Innovation Department, Cloud BU, Huawei Technologies Co.Ltd. and in part by Fujian Nebula Big Data Application Service Co., Ltd. All opinions, findings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.