{"type":"ScholarlyArticle","title":"Adaptive Group Sparse Multi-task Learning via Trace Lasso","authors":[{"name":"Sulin Liu","slug":"sulin-liu","given":"Sulin","family":"Liu","affiliation":"Nanyang Technological University , Singapore"},{"name":"Sinno Jialin Pan","slug":"sinno-jialin-pan","given":"Sinno Jialin","family":"Pan","affiliation":"Nanyang Technological University , Singapore"}],"abstract":"In multi-task learning (MTL), tasks are learned jointly so that information among related tasks is shared and utilized to help improve generalization for each individual task. A major challenge in MTL is how to selectively choose what to share among tasks. Ideally, only related tasks should share information with each other. In this paper, we propose a new MTL method that can adaptively group correlated tasks into clusters and share information among the correlated tasks only. Our method is based on the assumption that each task parameter is a linear combination of other tasks' and the coefficients of the linear combination are active only if there is relatedness between the two tasks. Through introducing trace Lasso penalty on these coefficients, our method is able to adaptively select the subset of coefficients with respect to the tasks that are correlated to the task. Our model frees the process of determining task clustering structure as used in the literature. Efficient optimization methods based on alternating direction method of multipliers (ADMM) is developed to solve the problem. Experimental results on both synthetic and real-world datasets demonstrate the effectiveness of our method in terms of clustering related tasks and generalization performance.","doi":"10.24963/ijcai.2017/328","url":"https://ijcai.org/proceedings/2017/328","pdf":"https://ijcai.org/proceedings/2017/0328.pdf","markdown":"https://ijcai.org/proceedings/2017/328.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/328","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra, IIIA-CSIC","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":328,"pages":{"first":2358,"last":2364},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Ando and Zhang, 2005] Rie Kubota Ando and Tong Zhang. A framework for learning predictive structures from mul- tiple tasks and unlabeled data. JMLR, 6(Nov):1817-1853, 2005.","title":"A framework for learning predictive structures from multiple tasks and unlabeled data","authors":"Rie Kubota, Ando, Tong Zhang","year":"2005","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Argyriou et al., 2008a] Andreas Argyriou, Theodoros Ev- geniou, and Massimiliano Pontil. Convex multi-task fea- ture learning. Machine Learning, 73(3):243-272, 2008.","title":"Convex multi-task feature learning","authors":"Argyriou","year":"2008","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Argyriou et al., 2008b] Andreas Argyriou, Andreas Maurer, and Massimiliano Pontil. An algorithm for transfer learn- ing in a heterogeneous environment. In ECML-PKDD, pages 71-85. Springer, 2008.","title":"An algorithm for transfer learning in a heterogeneous environment","authors":"Argyriou","year":"2008","venue":"ECML-PKDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blitzer et al., 2007] John Blitzer, Mark Dredze, Fernando Pereira, et al. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL, pages 440-447, 2007.","title":"Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification","authors":"John Blitzer, Mark Dredze, Fernando Pereira","year":"2007","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2011] Jianhui Chen, Jiayu Zhou, and Jieping Ye. Integrating low-rank and group-sparse structures for robust multi-task learning. In KDD, pages 42-50. ACM, 2011.","title":"Integrating low-rank and group-sparse structures for robust multi-task learning","authors":"Jianhui Chen, Jiayu Zhou, Jieping Ye","year":"2011","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Evgeniou and Pontil, 2004] Theodoros Evgeniou and Mas- similiano Pontil. Regularized multi-task learning. In KDD, pages 109-117. ACM, 2004.","title":"Theodoros Evgeniou and Massimiliano Pontil. Regularized multi-task learning","authors":"Pontil Evgeniou","year":"2004","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Grave et al., 2011] Edouard Grave, Guillaume R Obozin- ski, and Francis R Bach. Trace lasso: a trace norm regular- ization for correlated designs. In NIPS, pages 2187-2195, 2011.","title":"Trace lasso: a trace norm regularization for correlated designs","authors":"Grave","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jacob et al., 2009] Laurent Jacob, Jean-philippe Vert, and Francis R Bach. Clustered multi-task learning: A convex formulation. In NIPS, pages 745-752, 2009.","title":"Clustered multi-task learning: A convex formulation","authors":"Laurent Jacob, Jean-Philippe Vert, Francis R Bach","year":"2009","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kang et al., 2011] Zhuoliang Kang, Kristen Grauman, and Fei Sha. Learning with whom to share in multi-task feature learning. In ICML, pages 521-528, 2011.","title":"Learning with whom to share in multi-task feature learning","authors":"Kang","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky, 2009] Alex Krizhevsky. Learning multiple lay- ers of features from tiny images. 2009.","title":"Learning multiple layers of features from tiny images","authors":"Alex Krizhevsky","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar and Daumé III, 2012] Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multi- task learning. In ICML, 2012.","title":"Abhishek Kumar and Hal Daumé III. Learning task grouping and overlap in multitask learning","authors":"Daumé Kumar, Iii","year":"2012","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lee et al., 2016] Giwoong Lee, Eunho Yang, et al. Asym- metric multi-task learning based on task relatedness and confidence. In ICML, pages 230-238, 2016.","title":"Asymmetric multi-task learning based on task relatedness and confidence","authors":"Lee","year":"2016","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2009] Jun Liu, Shuiwang Ji, and Jieping Ye. Multi-task feature learning via efficient ℓ 2,1 -norm mini- mization. In UAI, pages 339-348, 2009.","title":"Multi-task feature learning via efficient ℓ 2,1 -norm minimization","authors":"Liu","year":"2009","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE TKDE, 22(10):1345- 1359, October 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE TKDE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rai et al., 2012] Piyush Rai, Abhishek Kumar, and Hal Daume. Simultaneously leveraging output and task struc- tures for multiple-output regression. In NIPS, pages 3185- 3193, 2012.","title":"Simultaneously leveraging output and task structures for multiple-output regression","authors":"Piyush Rai, Abhishek Kumar, Hal Daume","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rosenstein et al., 2005] Michael T. Rosenstein, Zvika Marx, and Leslie Pack Kaelbling. To transfer or not to transfer. In NIPS-05 Workshop on Inductive Transfer: 10 Years Later, December 2005.","title":"To transfer or not to transfer","authors":"T Michael, Zvika Rosenstein, Leslie Marx, Kaelbling Pack","year":"2005","venue":"NIPS-05 Workshop on Inductive Transfer: 10 Years Later","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Zhang and Schneider, 2010] Yi Zhang and Jeff G Schnei- der. Learning multiple tasks with a sparse matrix-normal penalty. In NIPS, pages 2550-2558, 2010.","title":"Learning multiple tasks with a sparse matrix-normal penalty","authors":"Schneider Zhang, Yi Zhang, Jeff G Schneider","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Zhang and Yang, 2017] Yu Zhang and Qiang Yang. Learn- ing sparse task relations in multi-task learning. In AAAI, 2017.","title":"Learning sparse task relations in multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang and Yeung, 2010] Yu Zhang and Dit-Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In UAI, pages 733-442, 2010.","title":"A convex formulation for learning task relationships in multi-task learning","authors":"Yeung ; ] Yu Zhang, Dit-Yan Zhang, Yeung","year":"2010","venue":"UAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zhou and Zhao, 2016] Qiang Zhou and Qi Zhao. Flexi- ble clustered multi-task learning by learning representative tasks. IEEE TPAMI, 38(2):266-278, 2016.","title":"Flexible clustered multi-task learning by learning representative tasks","authors":"Qiang Zhou, Qi Zhao","year":"2016","venue":"IEEE TPAMI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhou et al., 2011] Jiayu Zhou, Jianhui Chen, and Jieping Ye. Clustered multi-task learning via alternating structure optimization. In NIPS, pages 702-710, 2011.","title":"Clustered multi-task learning via alternating structure optimization","authors":"Jiayu Zhou, Jianhui Chen, Jieping Ye","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}