{"type":"ScholarlyArticle","title":"Self-paced Convolutional Neural Networks","authors":[{"name":"Hao Li","slug":"hao-li","given":"Hao","family":"Li","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education Xidian University , Xi'an , China"},{"name":"Maoguo Gong","slug":"maoguo-gong","given":"Maoguo","family":"Gong","affiliation":"Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education Xidian University , Xi'an , China"}],"abstract":"Convolutional neural networks (CNNs) have achieved breakthrough performance in many pattern recognition tasks. In order to distinguish the reliable data from the noisy and confusing data, we improve CNNs with self-paced learning (SPL) for enhancing the learning robustness of CNNs. In the proposed self-paced convolutional network (SPCN), each sample is assigned to a weight to reflect the easiness of the sample. Then a dynamic self-paced function is incorporated into the leaning objective of CNN to jointly learn the parameters of CNN and the latent weight variable. SPCN learns the samples from easy to complex and the sample weights can dynamically control the learning rates for converging to better values. To gain more insights of SPCN, theoretical studies are conducted to show that SPCN converges to a stationary solution and is robust to the noisy and confusing data. Experimental results on MNIST and rectangles datasets demonstrate that the proposed method outperforms baseline methods.","doi":"10.24963/ijcai.2017/293","url":"https://ijcai.org/proceedings/2017/293","pdf":"https://ijcai.org/proceedings/2017/0293.pdf","markdown":"https://ijcai.org/proceedings/2017/293.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/293","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":293,"pages":{"first":2110,"last":2116},"keywords":["Machine Learning: Machine Learning","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Basu and Christensen, 2013] Sumit Basu and Janara Chris- tensen. Teaching classification boundaries to humans. In AAAI, pages 109-115, 2013.","title":"Sumit Basu and Janara Christensen. Teaching classification boundaries to humans","authors":"Christensen Basu","year":"2013","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pages 41-48, 2009.","title":"Curriculum learning","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bengio et al., 2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1798-1828, 2013.","title":"Representation learning: A review and new perspectives","authors":"Yoshua Bengio, Aaron Courville, Pascal Vincent","year":"2013","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Girshick et al., 2014] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, pages 580-587, 2014.","title":"Rich feature hierarchies for accurate object detection and semantic segmentation","authors":"Ross Girshick, Jeff Donahue, Trevor Darrell, Jitendra Malik","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Jiang et al., 2014a] Lu Jiang, Deyu Meng, Teruko Mitamu- ra, and Alexander G Hauptmann. Easy samples first: Self- paced reranking for zero-example multimedia search. In ACM MM, pages 547-556, 2014.","title":"Easy samples first: Selfpaced reranking for zero-example multimedia search","authors":"Jiang","year":"2014","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Jiang et al., 2014b] Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Haupt- mann. Self-paced learning with diversity. In NIPS, pages 2078-2086, 2014.","title":"Self-paced learning with diversity","authors":"Jiang","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2015] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander G Hauptmann. Self-paced curriculum learning. In AAAI, pages 2694-2700, 2015.","title":"Self-paced curriculum learning","authors":"Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander G Hauptmann","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Karpathy et al., 2014] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolu- tional neural networks. In CVPR, pages 1725-1732, 2014.","title":"Large-scale video classification with convolutional neural networks","authors":"Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, Li Fei-Fei","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Khan et al., 2011] Faisal Khan, Bilge Mutlu, and Xiaojin Zhu. How do humans teach: On curriculum learning and teaching dimension. In NIPS, pages 1449-1457, 2011.","title":"How do humans teach: On curriculum learning and teaching dimension","authors":"Faisal Khan, Bilge Mutlu, Xiaojin Zhu","year":"2011","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In NIPS, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"Benjamin M Pawan Kumar, Daphne Packer, Koller","year":"2010","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kumar et al., 2011] M Pawan Kumar, Haithem Turki, Dan Preston, and Daphne Koller. Learning specific-class seg- mentation from diverse data. In ICCV, pages 1800-1807, 2011.","title":"Learning specific-class segmentation from diverse data","authors":"Haithem M Pawan Kumar, Dan Turki, Daphne Preston, Koller","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Larochelle et al., 2007] Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, and Yoshua Bengio. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, pages 473-480, 2007.","title":"An empirical evaluation of deep architectures on problems with many factors of variation","authors":"Hugo Larochelle, Dumitru Erhan, Aaron Courville, James Bergstra, Yoshua Bengio","year":"2007","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning ap- plied to document recognition. Proc. IEEE, 86(11):2278- 2324, 1998.","title":"Gradient-based learning applied to document recognition","authors":"Yann Lecun, Léon Bottou, Yoshua Bengio, Patrick Haffner","year":"1998","venue":"Proc. IEEE","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Ge- offrey Hinton. Deep learning. Nature, 521(7553):436- 444, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2016] Hao Li, Maoguo Gong, Deyu Meng, and Qiguang Miao. Multi-objective self-paced learning. In AAAI, pages 1802-1808, 2016.","title":"Multi-objective self-paced learning","authors":"Hao Li, Maoguo Gong, Deyu Meng, Qiguang Miao","year":"2016","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Meng and Zhao, 2015] Deyu Meng and Qian Zhao. What objective does self-paced learning indeed optimize? arXiv preprint arXiv:1511.06049, 2015.","title":"What objective does self-paced learning indeed optimize","authors":"Zhao ; ] Deyu Meng, Qian Meng, Zhao","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Pi et al., 2016] Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, and Yueting Zhuang. Self-paced boost learning for classification. In IJCAI, pages 1932-1938, 2016.","title":"Self-paced boost learning for classification","authors":"Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, Yueting Zhuang","year":"2016","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2016/276"},{"position":19,"raw":"[Rifai et al., 2011] Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto-encoders: Explicit invariance during feature extrac- tion. In ICML, pages 833-840, 2011.","title":"Contractive auto-encoders: Explicit invariance during feature extraction","authors":"Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, Yoshua Bengio","year":"2011","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Supančič and Ramanan, 2013] James Steven Supančič and Deva Ramanan. Self-paced learning for long-term track- ing. In CVPR, pages 2379-2386, 2013.","title":"Self-paced learning for long-term tracking","authors":"Ramanan ; Supančič, Steven James, Deva Supančič, Ramanan","year":"2013","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Taigman et al., 2014] Yaniv Taigman, Ming Yang, Mar- c'Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In CVPR, pages 1701-1708, 2014.","title":"Deepface: Closing the gap to human-level performance in face verification","authors":"Yaniv Taigman, Ming Yang, Marc'aurelio Ranzato, Lior Wolf","year":"2014","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Tang et al., 2012a] Kevin Tang, Vignesh Ramanathan, Li Fei-Fei, and Daphne Koller. Shifting weights: Adapt- ing object detectors from image to video. In NIPS, pages 638-646, 2012.","title":"Shifting weights: Adapting object detectors from image to video","authors":"Tang","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Tang et al., 2012b] Ye Tang, Yu-Bin Yang, and Yang Gao. Self-paced dictionary learning for image classification. In ACM MM, pages 833-836, 2012.","title":"Self-paced dictionary learning for image classification","authors":"Tang","year":"2012","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Vincent et al., 2008] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Ex- tracting and composing robust features with denoising autoencoders. In ICML, pages 1096-1103, 2008.","title":"Extracting and composing robust features with denoising autoencoders","authors":"Pascal Vincent, Hugo Larochelle, Yoshua Bengio, Pierre-Antoine Manzagol","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2015] Chang Xu, Dacheng Tao, and Chao Xu. Multi-view self-paced learning for clustering. In IJCAI, pages 3974-3980, 2015.","title":"Multi-view self-paced learning for clustering","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/558"},{"position":26,"raw":"[Zhang et al., 2015] Dingwen Zhang, Deyu Meng, Chao Li, Lu Jiang, Qian Zhao, and Junwei Han. A self-paced multiple-instance learning framework for co-saliency de- tection. In ICCV, pages 594-602, 2015.","title":"A self-paced multiple-instance learning framework for co-saliency detection","authors":"Dingwen Zhang, Deyu Meng, Chao Li, Lu Jiang, Qian Zhao, Junwei Han","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhao et al., 2015] Qian Zhao, Deyu Meng, Lu Jiang, Qi X- ie, Zongben Xu, and Alexander G Hauptmann. Self-paced learning for matrix factorization. In AAAI, pages 3196- 3202, 2015.","title":"Self-paced learning for matrix factorization","authors":"Qian Zhao, Deyu Meng, Lu Jiang, Qi Xie, Zongben Xu, Alexander G Hauptmann","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zöhrer and Pernkopf, 2014] Matthias Zöhrer and Franz Pernkopf. General stochastic networks for classification. In NIPS, pages 2015-2023, 2014.","title":"General stochastic networks for classification","authors":"Pernkopf Zöhrer, Matthias Zöhrer, Franz Pernkopf","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}