{"type":"ScholarlyArticle","title":"Learning Shared Knowledge for Deep Lifelong Learning using Deconvolutional Networks","authors":[{"name":"Seungwon Lee","slug":"seungwon-lee","given":"Seungwon","family":"Lee","affiliation":"University of Pennsylvania , Philadelphia , PA , USA"},{"name":"James Stokes","slug":"james-stokes","given":"James","family":"Stokes","affiliation":"University of Pennsylvania , Philadelphia , PA , USA; Flatiron Institute , New York , NY , USA"},{"name":"Eric Eaton","slug":"eric-eaton","given":"Eric","family":"Eaton","affiliation":"University of Pennsylvania , Philadelphia , PA , USA"}],"abstract":"Current mechanisms for knowledge transfer in deep networks tend to either share the lower layers between tasks, or build upon representations trained on other tasks. However, existing work in non-deep multi-task and lifelong learning has shown success with using factorized representations of the model parameter space for transfer, permitting more flexible construction of task models. Inspired by this idea, we introduce a novel architecture for sharing latent factorized representations in convolutional neural networks (CNNs). The proposed approach, called a deconvolutional factorized CNN, uses a combination of deconvolutional factorization and tensor contraction to perform flexible transfer between tasks. Experiments on two computer vision data sets show that the DF-CNN achieves superior performance in challenging lifelong learning settings, resists catastrophic forgetting, and exhibits reverse transfer to improve previously learned tasks from subsequent experience without retraining.","doi":"10.24963/ijcai.2019/393","url":"https://ijcai.org/proceedings/2019/393","pdf":"https://ijcai.org/proceedings/2019/0393.pdf","markdown":"https://ijcai.org/proceedings/2019/393.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/393","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":393,"pages":{"first":2837,"last":2844},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Baxter, 2000] J. Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12(1):149-198, February 2000.","title":"A model of inductive bias learning","authors":"J Baxter","year":"2000","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bell and Renals, 2015] P. Bell and S. Renals. Regular- ization of context-dependent deep neural networks with context-independent multi-task training. In Proc. of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4290-4294, 2015.","title":"Regularization of context-dependent deep neural networks with context-independent multi-task training","authors":"P Bell, S Renals","year":"2015","venue":"Proc. of the 2015 IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Caruana, 1993] R. Caruana. Multitask learning: A knowledge-based source of inductive bias. In Proc. of the 10th International Conference on Machine Learning, pages 41-48, 1993.","title":"Multitask learning: A knowledge-based source of inductive bias","authors":"R Caruana","year":"1993","venue":"Proc. of the 10th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen and Liu, 2016] Z. Chen and B. Liu. Lifelong Machine Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Morgan & Claypool Publishers, 2016.","title":"Lifelong Machine Learning","authors":"Chen, Liu, Z Chen, B Liu","year":"2016","venue":"Synthesis Lectures on Artificial Intelligence and Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2017] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, and T. Chua. SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning","authors":"L Chen, H Zhang, J Xiao, L Nie, J Shao, T Chua","year":"2017","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gao et al., 2019] Y. Gao, Q. She, J. Ma, M. Zhao, W. Liu, and A. L. Yuille. NDDR-CNN: Layer-wise feature fusing in multi-task CNN by neural discriminative dimensionality reduction. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3205-3214, 2019.","title":"NDDR-CNN: Layer-wise feature fusing in multi-task CNN by neural discriminative dimensionality reduction","authors":"Y Gao, Q She, J Ma, M Zhao, W Liu, A L Yuille","year":"2019","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ha et al., 2017] D. Ha, A. M. Dai, and Q. V. Le. Hypernet- works. In Proc. of the International Conference on Learn- ing Representations, 2017.","title":"Hypernetworks","authors":"D Ha, A M Dai, Q V Le","year":"2017","venue":"Proc. of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2016] K. He, X. Zhang, and S. Ren. Deep resid- ual learning for image recognition. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"K He, X Zhang, S Ren","year":"2016","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hu et al., 2018] Jie Hu, Li Shen, and Gang Sun. Squeeze- and-excitation networks. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Squeezeand-excitation networks","authors":"Jie Hu, Li Shen, Gang Sun","year":"2018","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Huang et al., 2013] J. Huang, J. Li, D. Yu, L. Deng, and Y. Gong. Cross-language knowledge transfer using mul- tilingual deep neural network with shared hidden layers. In Proc. of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 7304- 7308, 2013.","title":"Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers","authors":"J Huang, J Li, D Yu, L Deng, Y Gong","year":"2013","venue":"Proc. of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Jia et al., 2016] X. Jia, B. D. Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. Advances in Neural Information Processing Systems 29:667-675, 2016.","title":"Dynamic filter networks","authors":"Jia","year":"2016","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirkpatrick et al., 2017] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, and A. A. Rusu, et al. Overcoming catastrophic forgetting in neural net- works. Proc. of the National Academy of Sciences, 114(13):3521-3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"J Kirkpatrick, R Pascanu, N C Rabinowitz, J Veness, G Desjardins, A A Rusu","year":"2017","venue":"Proc. of the National Academy of Sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Krizhevsky and Hinton, 2009] A. Krizhevsky and G. Hin- ton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.","title":"Learning multiple layers of features from tiny images","authors":"A Krizhevsky, G Hinton","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Krizhevsky et al., 2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convo- lutional neural networks. Advances in Neural Information Processing Systems 25:1097-1105, 2012.","title":"ImageNet classification with deep convolutional neural networks","authors":"A Krizhevsky, I Sutskever, G E Hinton","year":"2012","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kumar and Daume, 2012] A. Kumar and H. Daume. Learn- ing task grouping and overlap in multi-task learning. In Proc. of the 29th International Conference on Machine Learning, pages 1383-1390, 2012.","title":"Learning task grouping and overlap in multi-task learning","authors":"A Kumar, H Daume","year":"2012","venue":"Proc. of the 29th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[LeCun et al., 1998] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE, 86(11):2278-2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Y Lecun, L Bottou, Y Bengio, P Haffner","year":"1998","venue":"Proc. of the IEEE","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Liu et al., 2017a] H. Liu, J. Lu, J. Feng, and J. Zhou. Learning deep sharable and structural detectors for face alignment. IEEE Transactions on Image Processing, 26(4):1666-1678, 2017.","title":"Learning deep sharable and structural detectors for face alignment","authors":"Liu","year":"2017","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liu et al., 2017b] P. Liu, X. Qiu, and X. Huang. Adversarial multi-task learning for text classification. In Proc. of the 55th Annual Meeting of the Association for Computational Linguistics, 2017.","title":"Adversarial multi-task learning for text classification","authors":"Liu","year":"2017","venue":"Proc. of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Long et al., 2017] M. Long, Z. Cao, J. Wang, and P. S. Yu. Learning multiple tasks with multilinear relationship net- works. Advances in Neural Information Processing Sys- tems 30:1594-1603, 2017.","title":"Learning multiple tasks with multilinear relationship networks","authors":"M Long, Z Cao, J Wang, P S Yu","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2017] Y. Lu, A. Kumar, S. Zhai, Y. Cheng, T. Ja- vidi, and R. Feris. Fully-adaptive feature sharing in multi- task networks with applications in person attribute classi- fication. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.","title":"Fully-adaptive feature sharing in multitask networks with applications in person attribute classification","authors":"Y Lu, A Kumar, S Zhai, Y Cheng, T Javidi, R Feris","year":"2017","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Maurer et al., 2013] A. Maurer, M. Pontil, and B. Romera- Paredes. Sparse coding for multitask and transfer learning. In Proc. of the 30th International Conference on Machine Learning, pages 343-351, 2013.","title":"Sparse coding for multitask and transfer learning","authors":"A Maurer, M Pontil, B Romera-Paredes","year":"2013","venue":"Proc. of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Misra et al., 2016] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learning. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3994-4003, 2016.","title":"Cross-stitch networks for multi-task learning","authors":"Misra","year":"2016","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Noh et al., 2015] H. Noh, S. Hong, and B. Han. Learn- ing deconvolution network for semantic segmentation. In Proc. of the 2015 IEEE International Conference on Com- puter Vision, pages 1520-1528, 2015.","title":"Learning deconvolution network for semantic segmentation","authors":"H Noh, S Hong, B Han","year":"2015","venue":"Proc. of the 2015 IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Pinto and Gupta, 2017] L. Pinto and A. Gupta. Learning to push by grasping: Using multiple tasks for effective learn- ing. In Proc. of the 2017 IEEE International Conference on Robotics and Automation, pages 2161-2168, 2017.","title":"Learning to push by grasping: Using multiple tasks for effective learning","authors":"L Pinto, A Gupta","year":"2017","venue":"Proc. of the 2017 IEEE International Conference on Robotics and Automation","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ranjan et al., 2017] R. Ranjan, V. Patel, and R. Chellappa. Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2017.","title":"Hyperface: A deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition","authors":"R Ranjan, V Patel, R Chellappa","year":"2017","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Rusu et al., 2016] A. A. Rusu, N. C. Rabinowitz, G. Des- jardins, H. Soyer, J. Kirkpatrick, and K. Kavukcuoglu, et al. Progressive neural networks. CoRR, abs/1606.04671, 2016.","title":"Progressive neural networks","authors":"A A Rusu, N C Rabinowitz, G Desjardins, H Soyer, J Kirkpatrick, K Kavukcuoglu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Ruvolo and Eaton, 2013] P. Ruvolo and E. Eaton. ELLA: An efficient lifelong learning algorithm. In Proc. of the 30th International Conference on Machine Learning, pages 507-515, 2013.","title":"ELLA: An efficient lifelong learning algorithm","authors":"P Ruvolo, E Eaton","year":"2013","venue":"Proc. of the 30th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. In Proc. of the International Confer- ence on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"Proc. of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Venkateswara et al., 2017] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing net- work for unsupervised domain adaptation. In Proc. of the IEEE Conference on Computer Vision and Pattern Recog- nition, 2017.","title":"Deep hashing network for unsupervised domain adaptation","authors":"H Venkateswara, J Eusebio, S Chakraborty, S Panchanathan","year":"2017","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Yang and Hospedales, 2017] Y. Yang and T. Hospedales. Deep multi-task representation learning: a tensor factori- sation approach. In Proc. of the International Conference on Learning Representations, 2017.","title":"Deep multi-task representation learning: a tensor factorisation approach","authors":"Y Yang, T Hospedales","year":"2017","venue":"Proc. of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Yim et al., 2015] J. Yim, H. Jung, B. Yoo, C. Choi, D. Park, and J. Kim. Rotating your face using multi-task deep neu- ral network. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, pages 676-684, 2015.","title":"Rotating your face using multi-task deep neural network","authors":"J Yim, H Jung, B Yoo, C Choi, D Park, J Kim","year":"2015","venue":"Proc. of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yoon et al., 2018] J. Yoon, E. Yang, and S. Hwang. Life- long learning with dynamically expandable networks. In Proc. of the International Conference on Learning Repre- sentations, 2018.","title":"Lifelong learning with dynamically expandable networks","authors":"J Yoon, E Yang, S Hwang","year":"2018","venue":"Proc. of the International Conference on Learning Representations","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}