{"type":"ScholarlyArticle","title":"Taskonomy: Disentangling Task Transfer Learning","authors":[{"name":"Amir Zamir","slug":"amir-zamir","given":"Amir","family":"Zamir","affiliation":"Stanford University; University of California , Berkeley http :// taskonomy.vision /"},{"name":"Alexander Sax","slug":"alexander-sax","given":"Alexander","family":"Sax","affiliation":"Stanford University"},{"name":"William Shen","slug":"william-shen","given":"William","family":"Shen","affiliation":"Stanford University"},{"name":"Leonidas Guibas","slug":"leonidas-guibas","given":"Leonidas","family":"Guibas","affiliation":"Stanford University"},{"name":"Jitendra Malik","slug":"jitendra-malik","given":"Jitendra","family":"Malik","affiliation":"University of California , Berkeley http :// taskonomy.vision /"},{"name":"Silvio Savarese","slug":"silvio-savarese","given":"Silvio","family":"Savarese","affiliation":"Stanford University"}],"abstract":"Do visual tasks have relationships, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a certain structure among visual tasks. Knowing this structure has notable values; it provides a principled way for identifying relationships across tasks, for instance, in order to reuse supervision among tasks with redundancies or solve many tasks in one system without piling up the complexity. \r\n\t\r\nWe propose a fully computational approach for modeling the transfer learning structure of the space of visual tasks. This is done via finding transfer learning dependencies across tasks in a dictionary of twenty-six 2D, 2.5D, 3D, and semantic tasks. The product is a computational taxonomic map among tasks for transfer learning, and we exploit it to reduce the demand for labeled data. For example, we show that the total number of labeled datapoints needed for solving a set of 10 tasks can be reduced by roughly 2/3 (compared to training independently) while keeping the performance nearly the same. We provide a set of tools for computing and visualizing this taxonomical structure at http://taskonomy.vision.","doi":"10.24963/ijcai.2019/871","url":"https://ijcai.org/proceedings/2019/871","pdf":"https://ijcai.org/proceedings/2019/0871.pdf","markdown":"https://ijcai.org/proceedings/2019/871.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/871","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Best Sister Conferences","track":"Best Sister Conferences","number":871,"pages":{"first":6241,"last":6245},"keywords":["Computer Vision: Computer Vision","Computer Vision: 2D and 3D Computer Vision","Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[],"references":[{"position":1,"raw":"[Armeni et al., 2017] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene un- derstanding. arXiv preprint arXiv:1702.01105, 2017.","title":"Joint 2d-3d-semantic data for indoor scene understanding","authors":"Iro Armeni, Sasha Sax, Silvio Amir R Zamir, Savarese","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspec- tives. IEEE transactions on pattern analysis and machine intelli- gence, 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen and Liu, 2016] Zhiyuan Chen and Bing Liu. Lifelong Ma- chine Learning. Morgan & Claypool Publishers, 2016.","title":"Lifelong Machine Learning","authors":"Zhiyuan Chen, Bing Liu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Doersch et al., 2015] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In International Conference on Computer Vision, 2015.","title":"Unsupervised visual representation learning by context prediction","authors":"Carl Doersch, Abhinav Gupta, Alexei A Efros","year":"2015","venue":"International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Erhan et al., 2010] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 2010.","title":"Why does unsupervised pre-training help deep learning","authors":"Yoshua Dumitru Erhan, Aaron Bengio, Pierre-Antoine Courville, Pascal Manzagol, Samy Vincent, Bengio","year":"2010","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ge, 2013] Rong Ge. Provable algorithms for machine learning problems. PhD thesis, Princeton University, 2013.","title":"Provable algorithms for machine learning problems","authors":"Rong Ge","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Geman et al., 2002] Stuart Geman, Daniel F Potter, and Zhiyi Chi. Composition systems. Quarterly of Applied Mathematics, 60(4):707-736, 2002.","title":"Composition systems","authors":"Stuart Geman, Daniel F Potter, Zhiyi Chi","year":"2002","venue":"Quarterly of Applied Mathematics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gopnik et al., 1999] Alison Gopnik, Andrew N Meltzoff, and Pa- tricia K Kuhl. The scientist in the crib: Minds, brains, and how children learn. William Morrow & Co, 1999.","title":"The scientist in the crib: Minds, brains, and how children learn","authors":"Alison Gopnik, Andrew N Meltzoff, Patricia K Kuhl","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Gurobi Optimization, 2016] Inc. Gurobi Optimization. Gurobi op- timizer reference manual, 2016.","title":"Inc. Gurobi Optimization. Gurobi optimizer reference manual","authors":null,"year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Henry, 2008] Kevin Henry. The theory and applications of homo- morphic cryptography. 2008.","title":"The theory and applications of homomorphic cryptography","authors":"Kevin Henry","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hoshen and Peleg, 2015] Yedid Hoshen and Shmuel Peleg. Visual learning of arithmetic operations. CoRR, 2015.","title":"Yedid Hoshen and Shmuel Peleg","authors":"Peleg Hoshen","year":"2015","venue":"Visual learning of arithmetic operations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kokkinos, 2016] Iasonas Kokkinos. Ubernet: Training auniver- sal'convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. arXiv, 2016.","title":"Iasonas Kokkinos. Ubernet: Training auniversal'convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory","authors":"Kokkinos","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lake et al., 2016] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pages 1-101, 2016.","title":"Building machines that learn and think like people","authors":"Brenden M Lake, Joshua B Tomer D Ullman, Samuel J Tenenbaum, Gershman","year":"2016","venue":"Behavioral and Brain Sciences","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Mccloskey and Cohen, 1989] Michael Mccloskey and Neil J. Co- hen. Catastrophic interference in connectionist networks: The sequential learning problem. The Psychology of Learning and Motivation, 24, 1989.","title":"Catastrophic interference in connectionist networks: The sequential learning problem","authors":"Michael Mccloskey, Neil J Cohen","year":"1989","venue":"The Psychology of Learning and Motivation","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pentina and Lampert, 2017] Anastasia Pentina and Christoph H Lampert. Multi-task learning with labeled and unlabeled tasks. stat, 1050:1, 2017.","title":"Anastasia Pentina and Christoph H Lampert. Multi-task learning with labeled and unlabeled tasks","authors":"Lampert Pentina","year":"2017","venue":"stat","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Piaget and Cook, 1952] Jean Piaget and Margaret Cook. The ori- gins of intelligence in children, volume 8. International Univer- sities Press New York, 1952.","title":"The origins of intelligence in children","authors":"Cook Piaget, Jean Piaget, Margaret Cook","year":"1952","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Pratt, 1993] Lorien Y Pratt. Discriminability-based transfer be- tween neural networks. In Advances in neural information pro- cessing systems, pages 204-211, 1993.","title":"Discriminability-based transfer between neural networks","authors":"Y Lorien, Pratt","year":"1993","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Russakovsky and others, 2015] Olga Russakovsky et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky","year":"2015","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Saaty, 1987] R. W. Saaty. The analytic hierarchy process -what it is and how it is used. Mathematical Modeling, 1987.","title":"The analytic hierarchy process -what it is and how it is used","authors":"R W Saaty","year":"1987","venue":"Mathematical Modeling","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Salakhutdinov et al., 2012] Ruslan Salakhutdinov, Joshua Tenen- baum, and Antonio Torralba. One-shot learning with a hierar- chical nonparametric bayesian model. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, 2012.","title":"One-shot learning with a hierarchical nonparametric bayesian model","authors":"Salakhutdinov","year":"2012","venue":"Proceedings of ICML Workshop on Unsupervised and Transfer Learning","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sax et al., 2018] Alexander Sax, Bradley Emi, Amir Zamir, Leonidas Guibas, Silvio Savarese, and Jitendra Malik. Mid-level visual representations improve generalization and sample effi- ciency for learning visuomotor policies. arXiv preprint, 2018.","title":"Mid-level visual representations improve generalization and sample efficiency for learning visuomotor policies","authors":"Alexander Sax, Bradley Emi, Amir Zamir, Leonidas Guibas, Silvio Savarese, Jitendra Malik","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sharif Razavian et al., 2014] Ali Sharif Razavian, Hossein Az- izpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off-the-shelf: an astounding baseline for recognition. In Con- ference on computer vision and pattern recognition workshops, 2014.","title":"Cnn features off-the-shelf: an astounding baseline for recognition","authors":"Sharif Razavian","year":"2014","venue":"Conference on computer vision and pattern recognition workshops","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Silver et al., 2013] Daniel L. Silver, Qiang Yang, and Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In in AAAI Spring Symposium Series, 2013.","title":"Lifelong machine learning systems: Beyond learning algorithms","authors":"L Daniel, Qiang Silver, Lianghao Yang, Li","year":"2013","venue":"AAAI Spring Symposium Series","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Socher et al., 2013] Richard Socher, Milind Ganjoo, Christo- pher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. In NIPS, pages 935-943, 2013.","title":"Zero-shot learning through cross-modal transfer","authors":"Richard Socher, Milind Ganjoo, Christopher D Manning, Andrew Ng","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Standley et al., 2019] Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? arXiv preprint arXiv:1905.07553, 2019.","title":"Which tasks should be learned together in multi-task learning","authors":"Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, Silvio Savarese","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Thrun and Pratt, 2012] Sebastian Thrun and Lorien Pratt. Learn- ing to learn. Springer Science & Business Media, 2012.","title":"Sebastian Thrun and Lorien Pratt","authors":"Pratt Thrun","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Turing, 1950] Alan M Turing. Computing machinery and intelli- gence. Mind, 59(236):433-460, 1950.","title":"Computing machinery and intelligence","authors":"Alan M Turing","year":"1950","venue":"Mind","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zamir et al., 2018] Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Taskon- omy: Disentangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.","title":"Taskonomy: Disentangling task transfer learning","authors":"Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, Silvio Savarese","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhou et al., 2014] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages 487-495, 2014.","title":"Learning deep features for scene recognition using places database","authors":"Zhou","year":"2014","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}