{"type":"ScholarlyArticle","title":"A Principled Approach for Learning Task Similarity in Multitask Learning","authors":[{"name":"Changjian Shui","slug":"changjian-shui","given":"Changjian","family":"Shui","affiliation":"Université Laval"},{"name":"Mahdieh Abbasi","slug":"mahdieh-abbasi","given":"Mahdieh","family":"Abbasi","affiliation":"Université Laval"},{"name":"Louis-Émile Robitaille","slug":"louis-emile-robitaille","given":"Louis-Émile","family":"Robitaille","affiliation":"Université Laval"},{"name":"Boyu Wang","slug":"boyu-wang","given":"Boyu","family":"Wang","affiliation":"University of Pennsylvania {changjian. shui.1 ,"},{"name":"Christian Gagné","slug":"christian-gagne","given":"Christian","family":"Gagné","affiliation":"Université Laval"}],"abstract":"Multitask learning aims at solving a set of related tasks simultaneously, by exploiting the shared knowledge for improving the performance on individual tasks. Hence, an important aspect of multitask learning is to understand the similarities within a set of tasks. Previous works have incorporated this similarity information explicitly (e.g., weighted loss for each task) or implicitly (e.g., adversarial loss for feature adaptation), for achieving good empirical performances. However, the theoretical motivations for adding task similarity knowledge are often missing or incomplete. In this paper, we give a different perspective from a theoretical point of view to understand this practice. We first provide an upper bound on the generalization error of multitask learning, showing the benefit of explicit and implicit task similarity knowledge. We systematically derive the bounds based on two distinct task similarity metrics: H divergence and Wasserstein distance. From these theoretical results, we revisit the Adversarial Multi-task Neural Network, proposing a new training algorithm to learn the task relation coefficients and neural network parameters iteratively. We assess our new algorithm empirically on several benchmarks, showing not only that we find interesting and robust task relations, but that the proposed approach outperforms the baselines, reaffirming the benefits of theoretical insight in algorithm design.","doi":"10.24963/ijcai.2019/478","url":"https://ijcai.org/proceedings/2019/478","pdf":"https://ijcai.org/proceedings/2019/0478.pdf","markdown":"https://ijcai.org/proceedings/2019/478.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/478","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":478,"pages":{"first":3446,"last":3452},"keywords":["Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Learning Theory","Machine Learning: Adversarial Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Arjovsky et al., 2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint arXiv:1701.07875, 2017.","title":"Wasserstein gan","authors":"Martin Arjovsky, Soumith Chintala, Léon Bottou","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ben-David et al., 2010] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory of learning from dif- ferent domains. Machine learning, 79(1-2):151-175, 2010.","title":"A theory of learning from different domains","authors":"Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, Jennifer Wortman Vaughan","year":"2010","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information pro- cessing systems, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gulrajani et al., 2017] Ishaan Gulrajani, Faruk Ahmed, Mar- tin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.","title":"Improved training of wasserstein gans","authors":"Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Aaron C Vincent Dumoulin, Courville","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Janati et al., 2018] Hicham Janati, Marco Cuturi, and Alexandre Gramfort. Wasserstein regularization for sparse multi-task regression. arXiv preprint arXiv:1805.07833, 2018.","title":"Wasserstein regularization for sparse multi-task regression","authors":"Hicham Janati, Marco Cuturi, Alexandre Gramfort","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kendall et al., 2018] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7482-7491, 2018.","title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","authors":"Alex Kendall, Yarin Gal, Roberto Cipolla","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Konstantinov and Lampert, 2019] Nikola Konstantinov and Christoph Lampert. Robust learning from untrusted sources. arXiv preprint arXiv:1901.10310, 2019.","title":"Robust learning from untrusted sources","authors":"Lampert Konstantinov, Nikola Konstantinov, Christoph Lampert","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kremer et al., 2018] Jan Kremer, Lasse Borgholt, and Lars Maaløe. On the inductive bias of word-character-level multi-task learning for speech recognition. arXiv preprint arXiv:1812.02308, 2018.","title":"On the inductive bias of word-character-level multi-task learning for speech recognition","authors":"Jan Kremer, Lasse Borgholt, Lars Maaløe","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lee and Raginsky, 2018] Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances. In Advances in Neural Information Processing Systems, pages 2692-2701, 2018.","title":"Jaeho Lee and Maxim Raginsky. Minimax statistical learning with wasserstein distances","authors":"Raginsky Lee","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li et al., 2018] Yitong Li, David E Carlson, et al. Extracting relationships by multi-domain matching. In Advances in Neural Information Processing Systems, pages 6799-6810, 2018.","title":"Extracting relationships by multi-domain matching","authors":"Yitong Li, David E Carlson","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2017] Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. Adversarial multi-task learning for text classifi- cation. arXiv preprint arXiv:1704.05742, 2017.","title":"Adversarial multi-task learning for text classification","authors":"Pengfei Liu, Xipeng Qiu, Xuanjing Huang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Long et al., 2017] Mingsheng Long, Zhangjie Cao, Jianmin Wang, and S Yu Philip. Learning multiple tasks with multi- linear relationship networks. In Advances in Neural Infor- mation Processing Systems, pages 1594-1603, 2017.","title":"Learning multiple tasks with multilinear relationship networks","authors":"Mingsheng Long, Zhangjie Cao, Jianmin Wang, S Yu, Philip","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Maurer et al., 2016] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. The Journal of Machine Learning Research, 17(1):2853-2884, 2016.","title":"The benefit of multitask representation learning","authors":"Maurer","year":"2016","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Murugesan and Carbonell, 2017] Keerthiram Murugesan and Jaime Carbonell. Active learning from peers. In Advances in Neural Information Processing Systems, pages 7008-7017, 2017.","title":"Keerthiram Murugesan and Jaime Carbonell","authors":"Carbonell Murugesan","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Murugesan et al., 2016] Keerthiram Murugesan, Hanxiao Liu, Jaime Carbonell, and Yiming Yang. Adaptive smoothed online multi-task learning. In Advances in Neural Information Processing Systems, pages 4296-4304, 2016.","title":"Adaptive smoothed online multi-task learning","authors":"Murugesan","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pentina and Lampert, 2017] Anastasia Pentina and Christoph H Lampert. Multi-task learning with la- beled and unlabeled tasks. In International Conference on Machine Learning, pages 2807-2816, 2017.","title":"Multi-task learning with labeled and unlabeled tasks","authors":"Anastasia Pentina, Christoph H Lampert","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rakotomamonjy et al., 2018] Alain Rakotomamonjy, Abra- ham Traore, Maxime Berar, Rémi Flamary, and Nicolas Courty. Wasserstein distance measure machines. arXiv preprint arXiv:1803.00250, 2018.","title":"Wasserstein distance measure machines","authors":"Alain Rakotomamonjy, Abraham Traore, Maxime Berar, Rémi Flamary, Nicolas Courty","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Redko et al., 2017] Ievgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis of domain adaptation with optimal transport. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 737-753. Springer, 2017.","title":"Ievgen Redko, Amaury Habrard, and Marc Sebban. Theoretical analysis of domain adaptation with optimal transport","authors":"Redko","year":"2017","venue":"Joint European Conference on Machine Learning and Knowledge Discovery in Databases","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ruder, 2017] Sebastian Ruder. An overview of multi- task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.","title":"An overview of multitask learning in deep neural networks","authors":"Sebastian Ruder","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Urner and Ben-David, 2013] Ruth Urner and Shai Ben- David. Probabilistic lipschitzness a niceness assumption for deterministic labels. In In Learning Faster from Easy Data -Workshop @ NIPS, 2013.","title":"Probabilistic lipschitzness a niceness assumption for deterministic labels","authors":"Ben-David ; Urner, Ruth Urner, Shai Ben-David","year":"2013","venue":"Learning Faster from Easy Data -Workshop @ NIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang and Pineau, 2015] Boyu Wang and Joelle Pineau. On- line boosting algorithms for anytime transfer and multitask learning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Online boosting algorithms for anytime transfer and multitask learning","authors":"Wang, Pineau ; ] Boyu Wang, Joelle Pineau","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Yang, 2017] Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv:1707.08114, 2017.","title":"A survey on multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang and Yeung, 2010] Yu Zhang and Dit Yan Yeung. A convex formulation for learning task relationships in multi- task learning. In 26th Conference on Uncertainty in Arti- ficial Intelligence, UAI 2010, Catalina Island, CA, United States, 8-11 July 2010, Code 86680, 2010.","title":"A convex formulation for learning task relationships in multitask learning","authors":"Yu Zhang, Dit Yan, Yeung","year":"2010","venue":"26th Conference on Uncertainty in Artificial Intelligence, UAI 2010","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang, 2015] Yu Zhang. Multi-task learning and algorithmic stability. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.","title":"Multi-task learning and algorithmic stability","authors":"Yu Zhang","year":"2015","venue":"Twenty-Ninth AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}