{"type":"ScholarlyArticle","title":"Consistent MetaReg: Alleviating Intra-task Discrepancy for Better Meta-knowledge","authors":[{"name":"Pinzhuo Tian","slug":"pinzhuo-tian","given":"Pinzhuo","family":"Tian","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Lei Qi","slug":"lei-qi","given":"Lei","family":"Qi","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Shaokang Dong","slug":"shaokang-dong","given":"Shaokang","family":"Dong","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Yinghuan Shi","slug":"yinghuan-shi","given":"Yinghuan","family":"Shi","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"},{"name":"Yang Gao","slug":"yang-gao","given":"Yang","family":"Gao","affiliation":"National Key Laboratory for Novel Software Technology , Nanjing University , China"}],"abstract":"In the few-shot learning scenario, the data-distribution discrepancy between training data and test data in a task usually exists due to the limited data. However, most existing meta-learning approaches seldom consider this intra-task discrepancy in the meta-training phase which might deteriorate the performance. To overcome this limitation, we develop a new consistent meta-regularization method to reduce the intra-task data-distribution discrepancy. Moreover, the proposed meta-regularization method could be readily inserted into existing optimization-based meta-learning models to learn better meta-knowledge. Particularly, we provide the theoretical analysis to prove that using the proposed meta-regularization, the conventional gradient-based meta-learning method can reach the lower regret bound. The extensive experiments also demonstrate the effectiveness of our method, which indeed improves the performances of the state-of-the-art gradient-based meta-learning models in the few-shot classification task.","doi":"10.24963/ijcai.2020/377","url":"https://ijcai.org/proceedings/2020/377","pdf":"https://ijcai.org/proceedings/2020/0377.pdf","markdown":"https://ijcai.org/proceedings/2020/377.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/377","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":377,"pages":{"first":2718,"last":2724},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24540/file_24540.m3u8","length_sec":291,"provider":"hls"}],"references":[{"position":1,"raw":"[Al-Shedivat et al., 2018] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In ICLR, 2018.","title":"Continuous adaptation via meta-learning in nonstationary and competitive environments","authors":"Al-Shedivat","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amit and Meir, 2018] Ron Amit and Ron Meir. Meta- learning by adjusting priors based on extended pac-bayes theory. In ICML, pages 205-214, 2018.","title":"Ron Amit and Ron Meir. Metalearning by adjusting priors based on extended pac-bayes theory","authors":"Meir Amit","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 1992] Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2, 1992.","title":"On the optimization of a synaptic learning rule","authors":"Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, Jan Gecsei","year":"1992","venue":"Preprints Conf. Optimality in Artificial and Biological Neural Networks","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bertinetto et al., 2019] Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In ICLR, 2019.","title":"Meta-learning with differentiable closed-form solvers","authors":"Luca Bertinetto, João F Henriques, H S Philip, Andrea Torr, Vedaldi","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2019] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2019.","title":"A closer look at few-shot classification","authors":"Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank, Wang, Jia-Bin Huang","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Finn and Levine, 2018] Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018.","title":"Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm","authors":"Levine Finn, Chelsea Finn, Sergey Levine","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.","title":"Model-agnostic meta-learning for fast adaptation of deep networks","authors":"Finn","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Franceschi et al., 2018] Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pon- til. Bilevel programming for hyperparameter optimization and meta-learning. In ICML, pages 1563-1572, 2018.","title":"Bilevel programming for hyperparameter optimization and meta-learning","authors":"Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, Massimiliano Pontil","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hazan et al., 2007] Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex op- timization. ML, 69(2-3):169-192, 2007.","title":"Logarithmic regret algorithms for online convex optimization","authors":"Elad Hazan, Amit Agarwal, Satyen Kale","year":"2007","venue":"ML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Javed and White, 2019] Khurram Javed and Martha White. Meta-learning representations for continual learning. In NeurIPS, pages 1818-1828, 2019.","title":"Meta-learning representations for continual learning","authors":"Khurram Javed, Martha White","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Lee et al., 2019] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with dif- ferentiable convex optimization. In CVPR, pages 10657- 10665, 2019.","title":"Meta-learning with differentiable convex optimization","authors":"Lee","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint arXiv:1707.09835, 2017.","title":"Meta-sgd: Learning to learn quickly for few shot learning","authors":"Zhenguo Li, Fengwei Zhou, Fei Chen, Hang Li","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Munkhdalai and Yu, 2017] Tsendsuren Munkhdalai and Hong Yu. Meta networks. In ICML, pages 2554-2563, 2017.","title":"Tsendsuren Munkhdalai and Hong Yu. Meta networks","authors":"Yu Munkhdalai","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Na et al., 2019] Donghyun Na, Haebeom Lee, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for im- balanced and out-of-distribution tasks. arXiv preprint arXiv:1905.12917, 2019.","title":"Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks","authors":"Donghyun Na, Haebeom Lee, Saehoon Kim, Minseop Park, Eunho Yang, Sung Ju Hwang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rajeswaran et al., 2019] Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta- learning with implicit gradients. In NeurIPS, pages 113-124, 2019.","title":"Metalearning with implicit gradients","authors":"Aravind Rajeswaran, Chelsea Finn, M Sham, Sergey Kakade, Levine","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Rakelly et al., 2019] Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off- policy meta-reinforcement learning via probabilistic con- text variables. In ICML, pages 5331-5340, 2019.","title":"Efficient offpolicy meta-reinforcement learning via probabilistic context variables","authors":"Rakelly","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Ravi and Larochelle, 2017] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In ICLR, 2017.","title":"Optimization as a model for few-shot learning","authors":"Larochelle ; Ravi, Sachin Ravi, Hugo Larochelle","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ren et al., 2018] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR, 2018.","title":"Meta-learning for semi-supervised few-shot classification","authors":"Ren","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S Bernstein, Alexander C Berg, Fei-Fei Li","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Saenko et al., 2010] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, pages 213-226, 2010.","title":"Adapting visual category models to new domains","authors":"Saenko","year":"2010","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Schmidhuber, 1987] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook. PhD thesis, Technische Universität München, 1987.","title":"Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta","authors":"Jürgen Schmidhuber","year":"1987","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Snell et al., 2017] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NIPS, pages 4077-4087, 2017.","title":"Prototypical networks for few-shot learning","authors":"Jake Snell, Kevin Swersky, Richard Zemel","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Thrun and Pratt, 1998] Sebastian Thrun and Lorien Y. Pratt, editors. Learning to Learn. Springer, 1998.","title":"Learning to Learn","authors":null,"year":"1998","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Tian et al., 2019] Pinzhuo Tian, Zhangkai Wu, Lei Qi, Lei Wang, Yinghuan Shi, and Yang Gao. Differentiable meta- learning model for few-shot semantic segmentation. arXiv preprint arXiv:1911.10371, 2019.","title":"Differentiable metalearning model for few-shot semantic segmentation","authors":"Tian","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Utgoff, 1986] Paul E Utgoff. Shift of bias for inductive con- cept learning. Machine learning: An artificial intelligence approach, 1986.","title":"Shift of bias for inductive concept learning","authors":"Paul E Utgoff","year":"1986","venue":"Machine learning: An artificial intelligence approach","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vilalta and Drissi, 2002] Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.","title":"Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning","authors":"Drissi Vilalta","year":"2002","venue":"Artificial intelligence review","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim- othy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In NIPS, pages 3630-3638, 2016.","title":"Matching networks for one shot learning","authors":"Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Xie et al., 2019] Yujia Xie, Haoming Jiang, Feng Liu, Tuo Zhao, and Hongyuan Zha. Meta learning with relational information for short sequences. In NeurIPS, pages 9901- 9912, 2019.","title":"Meta learning with relational information for short sequences","authors":"Yujia Xie, Haoming Jiang, Feng Liu, Tuo Zhao, Hongyuan Zha","year":"2019","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zinkevich, 2003] Martin Zinkevich. Online convex pro- gramming and generalized infinitesimal gradient ascent. In ICML, pages 928-936, 2003.","title":"Online convex programming and generalized infinitesimal gradient ascent","authors":"Martin Zinkevich","year":"2003","venue":"ICML","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}