{"type":"ScholarlyArticle","title":"Dynamic Multi-Task Learning with Convolutional Neural Network","authors":[{"name":"Yuchun Fang","slug":"yuchun-fang","given":"Yuchun","family":"Fang","affiliation":"School of Computer Engineering and Science , Shanghai University"},{"name":"Zhengyan Ma","slug":"zhengyan-ma","given":"Zhengyan","family":"Ma","affiliation":"School of Computer Engineering and Science , Shanghai University"},{"name":"Zhaoxiang Zhang","slug":"zhaoxiang-zhang","given":"Zhaoxiang","family":"Zhang","affiliation":"Research Center for Brain-inspired Intelligence , CASIA; National Laboratory of Pattern Recognition , CASIA; CAS Center for Excellence in Brain Science and Intelligence Technology; University of Chinese Academy of Sciences"},{"name":"Xu-Yao Zhang","slug":"xu-yao-zhang","given":"Xu-Yao","family":"Zhang","affiliation":"National Laboratory of Pattern Recognition , CASIA"},{"name":"Xiang Bai","slug":"xiang-bai","given":"Xiang","family":"Bai","affiliation":"School of Electronic Information and Communication , Huazhong University of Science and Technology"}],"abstract":"Multi-task learning and deep convolutional neural network (CNN) have been successfully used in various fields. This paper considers the integration of CNN and multi-task learning in a novel way to further improve the performance of multiple related tasks. Existing multi-task CNN models usually empirically combine different tasks into a group which is then trained jointly with a strong assumption of model commonality. Furthermore, traditional approaches usually only consider small number of tasks with rigid structure, which is not suitable for large-scale applications. In light of this, we propose a dynamic multi-task CNN model to handle these problems. The proposed model directly learns the task relations from data instead of subjective task grouping. Due to its flexible structure, it supports task-wise incremental training, which is useful for efficient training of massive tasks. Specifically, we add a new task transfer connection (TTC) between the layers of each task. The learned TTC is able to reflect the correlation among different tasks guiding the model dynamically adjusting the multiplexing of the information among different tasks. With the help of TTC, multiple related tasks can further boost the whole performance for each other. Experiments demonstrate that the proposed dynamic multi-task CNN model outperforms traditional approaches.","doi":"10.24963/ijcai.2017/231","url":"https://ijcai.org/proceedings/2017/231","pdf":"https://ijcai.org/proceedings/2017/0231.pdf","markdown":"https://ijcai.org/proceedings/2017/231.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/231","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":231,"pages":{"first":1668,"last":1674},"keywords":["Machine Learning: Deep Learning","Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Boutell et al., 2004] Matthew R Boutell, Jiebo Luo, Xipeng Shen, and Christopher M Brown. Learning multi-label scene classification. Pattern recognition, 37(9):1757- 1771, 2004.","title":"Learning multi-label scene classification","authors":"Jiebo Matthew R Boutell, Xipeng Luo, Christopher M Shen, Brown","year":"2004","venue":"Pattern recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caruana, 1998] Rich Caruana. Multitask learning. In Learning to learn, pages 95-133. Springer, 1998.","title":"Multitask learning","authors":"Rich Caruana","year":"1998","venue":"Learning to learn","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Devries et al., 2014] Terrance Devries, Kumar Biswaran- jan, and Graham W. Taylor. Multi-task learning of facial landmarks and expression. In Computer and Robot Vision, pages 98-103, 2014.","title":"Multi-task learning of facial landmarks and expression","authors":"Terrance Devries, Kumar Biswaranjan, Graham W Taylor","year":"2014","venue":"Computer and Robot Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Fang et al., 2014] Ming Fang, Yuqi Xiao, Chongjun Wang, and Junyuan Xie. Multi-label classification: Dealing with imbalance by combining labels. In Tools with Artificial Intelligence (ICTAI), 2014 IEEE 26th International Con- ference on, pages 233-237. IEEE, 2014.","title":"Multi-label classification: Dealing with imbalance by combining labels","authors":"Ming Fang, Yuqi Xiao, Chongjun Wang, Junyuan Xie","year":"2014","venue":"Tools with Artificial Intelligence (ICTAI)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[He et al., 2009] Saike He, Taozheng Zhang, Xue Bai, Xi- aojie Wang, and Yuan Dong. Incorporating multi-task learning in conditional random fields for chunking in se- mantic role labeling. In Natural Language Processing and Knowledge Engineering, 2009. NLP-KE 2009. Inter- national Conference on, pages 1-5. IEEE, 2009.","title":"Incorporating multi-task learning in conditional random fields for chunking in semantic role labeling","authors":"He","year":"2009","venue":"Natural Language Processing and Knowledge Engineering","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Japkowicz and Stephen, 2002] Nathalie Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. IOS Press, 2002.","title":"The class imbalance problem: A systematic study","authors":"Stephen ; Japkowicz, Nathalie Japkowicz, Shaju Stephen","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural in- formation processing systems, pages 1097-1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[LeCun et al., 1989] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation ap- plied to handwritten zip code recognition. Neural compu- tation, 1(4):541-551, 1989.","title":"Backpropagation applied to handwritten zip code recognition","authors":"Yann Lecun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, Lawrence D Jackel","year":"1989","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pages 3730-3738, 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo, Xiaogang Wang, Xiaoou Tang","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Misra et al., 2016] Ishan Misra, Abhinav Shrivastava, Abhi- nav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 3994-4003, 2016.","title":"Cross-stitch networks for multi-task learning","authors":"Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, Martial Hebert","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Quattoni et al., 2008] Ariadna Quattoni, Michael Collins, and Trevor Darrell. Transfer learning for image classifi- cation with sparse prototype representations. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE, 2008.","title":"Transfer learning for image classification with sparse prototype representations","authors":"Ariadna Quattoni, Michael Collins, Trevor Darrell","year":"2008","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Redmon et al., 2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pages 779-788, 2016.","title":"You only look once: Unified, real-time object detection","authors":"Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schroff et al., 2015] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 815-823, 2015.","title":"Facenet: A unified embedding for face recognition and clustering","authors":"Schroff","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sun et al., 2014] Yi Sun, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation from predicting 10,000 classes. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1891-1898, 2014.","title":"Deep learning face representation from predicting 10,000 classes","authors":"Yi Sun, Xiaogang Wang, Xiaoou Tang","year":"2014","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Szegedy et al., 2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabi- novich. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.","title":"Going deeper with convolutions","authors":"Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Taigman et al., 2014] Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Computer Vision and Pattern Recognition, pages 1701-1708, 2014.","title":"Deepface: Closing the gap to human-level performance in face verification","authors":"Yaniv Taigman, Ming Yang, Marc'aurelio Ranzato, Lior Wolf","year":"2014","venue":"Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tishby and Zaslavsky, 2015] Naftali Tishby and Noga Za- slavsky. Deep learning and the information bottleneck principle. In Information Theory Workshop (ITW), 2015 IEEE, pages 1-5. IEEE, 2015.","title":"Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle","authors":"Zaslavsky Tishby","year":"2015","venue":"Information Theory Workshop (ITW)","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Yang et al., 2015] Shuo Yang, Ping Luo, Chen-Change Loy, and Xiaoou Tang. From facial parts responses to face de- tection: A deep learning approach. In Proceedings of the IEEE International Conference on Computer Vision, pages 3676-3684, 2015.","title":"From facial parts responses to face detection: A deep learning approach","authors":"Shuo Yang, Ping Luo, Chen-Change Loy, Xiaoou Tang","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang and Zhang, 2014] Cha Zhang and Zhengyou Zhang. Improving multiview face detection with multi-task deep convolutional neural networks. In Applications of Com- puter Vision (WACV), 2014 IEEE Winter Conference on, pages 1036-1041. IEEE, 2014.","title":"Improving multiview face detection with multi-task deep convolutional neural networks","authors":"Zhang ; ] Cha Zhang, Zhengyou Zhang, Zhang","year":"2014","venue":"Applications of Computer Vision (WACV), 2014 IEEE Winter Conference on","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zhang et al., 2014] Zhanpeng Zhang, Ping Luo, Change Loy Chen, and Xiaoou Tang. Facial land- mark detection by deep multi-task learning. In European Conference on Computer Vision, pages 94-108, 2014.","title":"Facial landmark detection by deep multi-task learning","authors":"Zhang","year":"2014","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}