{"type":"ScholarlyArticle","title":"Multi-Task Learning via Time-Aware Neural ODE","authors":[{"name":"Feiyang Ye","slug":"feiyang-ye","given":"Feiyang","family":"Ye","affiliation":"Department of Computer Science and Engineering , Southern University of Science and Technology; Australian Artificial Intelligence Institute , University of Technology Sydney; Centre for Frontier AI Research , A; Institute of High Performance Computing , A"},{"name":"Xuehao Wang","slug":"xuehao-wang","given":"Xuehao","family":"Wang","affiliation":"Department of Computer Science and Engineering , Southern University of Science and Technology"},{"name":"Yu Zhang","slug":"yu-zhang","given":"Yu","family":"Zhang","affiliation":"Department of Computer Science and Engineering , Southern University of Science and Technology; Peng Cheng Laboratory; * STAR"},{"name":"Ivor W. Tsang","slug":"ivor-w-tsang","given":"Ivor W.","family":"Tsang","affiliation":"Australian Artificial Intelligence Institute , University of Technology Sydney; Centre for Frontier AI Research , A; Institute of High Performance Computing , A"}],"abstract":"Multi-Task Learning (MTL) is a well-established paradigm for learning shared models for a diverse set of tasks. Moreover, MTL improves data efficiency by jointly training all tasks simultaneously. However, directly optimizing the losses of all the tasks may lead to imbalanced performance on all the tasks due to the competition among tasks for the shared parameters in MTL models. Many MTL methods try to mitigate this problem by dynamically weighting task losses or manipulating task gradients. Different from existing studies, in this paper, we propose a Neural Ordinal diffeRential equation based Multi-tAsk Learning (NORMAL) method to alleviate this issue by modeling task-specific feature transformations from the perspective of dynamic flows built on the Neural Ordinary Differential Equation (NODE). Specifically, the proposed NORMAL model designs a time-aware neural ODE block to learn task-specific time information, which determines task positions of feature transformations in the dynamic flow, in NODE automatically via gradient descent methods. In this way, the proposed NORMAL model handles the problem of competing shared parameters by learning task positions. Moreover, the learned task positions can be used to measure the relevance among different tasks. Extensive experiments show that the proposed NORMAL model outperforms state-of-the-art MTL models.","doi":"10.24963/ijcai.2023/500","url":"https://ijcai.org/proceedings/2023/500","pdf":"https://ijcai.org/proceedings/2023/0500.pdf","markdown":"https://ijcai.org/proceedings/2023/500.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/500","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Machine Learning","track":"Main Track","number":500,"pages":{"first":4495,"last":4503},"keywords":["Machine Learning: ML: Multi-task and transfer learning"],"videos":[],"references":[{"position":1,"raw":"[Bartlett and Mendelson, 2002] Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learn- ing Research, 3(Nov):463-482, 2002.","title":"Rademacher and gaussian complexities: Risk bounds and structural results","authors":"Mendelson ; Bartlett, L Peter, Shahar Bartlett, Mendelson","year":"2002","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Caruana, 1997] Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.","title":"Multitask learning","authors":"Caruana","year":"1997","venue":"Machine learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018a] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In Proceedings of the Euro- pean conference on computer vision (ECCV), pages 801- 818, 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Chen","year":"2018","venue":"Proceedings of the European conference on computer vision (ECCV)","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2018b] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary dif- ferential equations. Advances in neural information pro- cessing systems, 31, 2018.","title":"Neural ordinary differential equations","authors":"Chen","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2018c] Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradi- ent normalization for adaptive loss balancing in deep mul- titask networks. In International conference on machine learning, pages 794-803. PMLR, 2018.","title":"Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks","authors":"Chen","year":"2018","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[da Silva and Gazeau, 2020] André Belotto da Silva and Maxime Gazeau. A general system of differential equa- tions to model first-order adaptive algorithms. J. Mach. Learn. Res., 21, 2020.","title":"A general system of differential equations to model first-order adaptive algorithms","authors":"Silva Da, Gazeau ; André Belotto Da, Silva, Maxime Gazeau","year":"2020","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Dupont et al., 2019] Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neu- ral Information Processing Systems, 32, 2019.","title":"Augmented neural odes","authors":"Emilien Dupont, Arnaud Doucet, Yee Whye Teh","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Finlay et al., 2020] Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, and Adam Oberman. How to train your neural ode: the world of jacobian and kinetic regulariza- tion. In International conference on machine learning, pages 3154-3164. PMLR, 2020.","title":"How to train your neural ode: the world of jacobian and kinetic regularization","authors":"Chris Finlay, Jörn-Henrik Jacobsen, Levon Nurbekyan, Adam Oberman","year":"2020","venue":"International conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Fleischer and Tardos, 1998] Lisa Fleischer and Éva Tardos. Efficient continuous-time dynamic network flow algo- rithms. Operations Research Letters, 23(3-5):71-80, 1998.","title":"Efficient continuous-time dynamic network flow algorithms","authors":"Lisa Fleischer, Éva Tardos","year":"1998","venue":"Operations Research Letters","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Grathwohl et al., 2019] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, and David Duvenaud. Scalable re- versible generative models with free-form continuous dy- namics. In International Conference on Learning Repre- sentations, 2019.","title":"Scalable reversible generative models with free-form continuous dynamics","authors":"Grathwohl","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kendall et al., 2018] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7482-7491, 2018.","title":"Multi-task learning using uncertainty to weigh losses for scene geometry and semantics","authors":"Kendall","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar and Daume III, 2012] Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multi- task learning. arXiv preprint arXiv:1206.6417, 2012.","title":"Abhishek Kumar and Hal Daume III. Learning task grouping and overlap in multitask learning","authors":"Daume Kumar, Iii","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lechner and Hasani, 2020] Mathias Lechner and Ramin Hasani. Learning long-term dependencies in irregularly- sampled time series. arXiv preprint arXiv:2006.04418, 2020.","title":"Learning long-term dependencies in irregularlysampled time series","authors":"Hasani Lechner, Mathias Lechner, Ramin Hasani","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of the IEEE international conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2021a] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878-18890, 2021.","title":"Conflict-averse gradient descent for multi-task learning","authors":"Liu","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021b] Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. ICLR, 2021.","title":"Towards impartial multi-task learning","authors":"Liu","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Maninis et al., 2019] Kevis-Kokitsi Maninis, Ilija Ra- dosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 1851-1860, 2019.","title":"Attentive single-tasking of multiple tasks","authors":"Kevis-Kokitsi Maninis, Ilija Radosavovic, Iasonas Kokkinos","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Maurer et al., 2016] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Re- search, 17(81):1-32, 2016.","title":"The benefit of multitask representation learning","authors":"Andreas Maurer, Massimiliano Pontil, Bernardino Romera-Paredes","year":"2016","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Navon et al., 2022] Aviv Navon, Aviv Shamsian, Idan Achi- tuve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017, 2022.","title":"Multi-task learning as a bargaining game","authors":"Navon","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saenko et al., 2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference on computer vision, pages 213-226. Springer, 2010.","title":"Adapting visual category models to new domains","authors":"Saenko","year":"2010","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sener and Koltun, 2018] Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Ad- vances in neural information processing systems, 31, 2018.","title":"Multi-task learning as multi-objective optimization","authors":"Koltun ; ] Ozan Sener, Vladlen Sener, Koltun","year":"2018","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Silberman et al., 2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European conference on computer vision, pages 746-760. Springer, 2012.","title":"Indoor segmentation and support inference from rgbd images","authors":"Nathan Silberman, Derek Hoiem, Pushmeet Kohli, Rob Fergus","year":"2012","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Sun et al., 2021] Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Pa- tel, Dengxin Dai, and Luc Van Gool. Task switching network for multi-task learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8291-8300, 2021.","title":"Task switching network for multi-task learning","authors":"Sun","year":"2021","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Swersky et al., 2013] Kevin Swersky, Jasper Snoek, and Ryan P Adams. Multi-task Bayesian optimization. 2013.","title":"Multi-task Bayesian optimization","authors":"Kevin Swersky, Jasper Snoek, Ryan P Adams","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Venkateswara et al., 2017] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Pan- chanathan. Deep hashing network for unsupervised do- main adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5018- 5027, 2017.","title":"Deep hashing network for unsupervised domain adaptation","authors":"Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, Sethuraman Panchanathan","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xia et al., 2021] Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, and Bao Wang. Heavy ball neural ordinary differential equa- tions. Advances in Neural Information Processing Sys- tems, 34:18646-18659, 2021.","title":"Heavy ball neural ordinary differential equations","authors":"Hedi Xia, Vai Suliafu, Hangjie Ji, Tan Nguyen, Andrea Bertozzi, Stanley Osher, Bao Wang","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yao et al., 2019] Yaqiang Yao, Jie Cao, and Huanhuan Chen. Robust task grouping with representative tasks for clustered multi-task learning. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1408-1417, 2019.","title":"Robust task grouping with representative tasks for clustered multi-task learning","authors":"Yao","year":"2019","venue":"Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yildiz et al., 2019] Cagatay Yildiz, Markus Heinonen, and Harri Lahdesmaki. Ode2vae: Deep generative second or- der odes with bayesian neural networks. Advances in Neu- ral Information Processing Systems, 32, 2019.","title":"Ode2vae: Deep generative second order odes with bayesian neural networks","authors":"Cagatay Yildiz, Markus Heinonen, Harri Lahdesmaki","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2017] Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 472-480, 2017.","title":"Dilated residual networks","authors":"Fisher Yu, Vladlen Koltun, Thomas Funkhouser","year":"2017","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yu et al., 2020] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neu- ral Information Processing Systems, 33:5824-5836, 2020.","title":"Gradient surgery for multi-task learning","authors":"Yu","year":"2020","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zamir et al., 2018] Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learn- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3712-3722, 2018.","title":"Taskonomy: Disentangling task transfer learning","authors":"Alexander Amir R Zamir, William Sax, Leonidas J Shen, Jitendra Guibas, Silvio Malik, Savarese","year":"2018","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":"/proceedings/2019/871"},{"position":32,"raw":"[Zhang and Yang, 2022] Yu Zhang and Qiang Yang. A sur- vey on multi-task learning. IEEE Transactions on Knowl- edge and Data Engineering, 34(12):5586-5609, 2022.","title":"A survey on multi-task learning","authors":"Yu Zhang, Qiang Yang","year":"2022","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}