{"type":"ScholarlyArticle","title":"Distributed Self-Paced Learning in Alternating Direction Method of Multipliers","authors":[{"name":"Xuchao Zhang","slug":"xuchao-zhang","given":"Xuchao","family":"Zhang","affiliation":"Discovery Analytics Center , Virginia Tech , Falls Church , VA , USA"},{"name":"Liang Zhao","slug":"liang-zhao","given":"Liang","family":"Zhao","affiliation":"George Mason University , Fairfax , VA , USA"},{"name":"Zhiqian Chen","slug":"zhiqian-chen","given":"Zhiqian","family":"Chen","affiliation":"Discovery Analytics Center , Virginia Tech , Falls Church , VA , USA"},{"name":"Chang-Tien Lu","slug":"chang-tien-lu","given":"Chang-Tien","family":"Lu","affiliation":"Discovery Analytics Center , Virginia Tech , Falls Church , VA , USA"}],"abstract":"Self-paced learning (SPL) mimics the cognitive process of humans, who generally learn from easy samples to hard ones. One key issue in SPL is the training process required for each instance weight depends on the other samples and thus cannot easily be run in a distributed manner in a large-scale dataset. In this paper, we reformulate the self-paced learning problem into a distributed setting and propose a novel Distributed Self-Paced Learning method (DSPL) to handle large scale datasets. Specifically, both the model and instance weights can be optimized in parallel for each batch based on a consensus alternating direction method of multipliers. We also prove the convergence of our algorithm under mild conditions. Extensive experiments on both synthetic and real datasets demonstrate that our approach is superior to those of existing methods.","doi":"10.24963/ijcai.2018/437","url":"https://ijcai.org/proceedings/2018/437","pdf":"https://ijcai.org/proceedings/2018/0437.pdf","markdown":"https://ijcai.org/proceedings/2018/437.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/437","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Machine Learning","track":"Main track","number":437,"pages":{"first":3148,"last":3154},"keywords":["Machine Learning: Machine Learning","Heuristic Search and Game Playing: Distributed Search","Constraints and SAT: Constraint Optimisation"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2009] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learn- ing. In Proceedings of the 26th annual international con- ference on machine learning, pages 41-48. ACM, 2009.","title":"Curriculum learn-ing","authors":"Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston","year":"2009","venue":"Proceedings of the 26th annual international conference on machine learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bhatia et al., 2015] Kush Bhatia, Prateek Jain, and Pu- rushottam Kar. Robust regression via hard thresholding. In Advances in Neural Information Processing Systems, pages 721-729, 2015.","title":"Robust regression via hard thresholding","authors":"Bhatia","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Boyd et al., 2011] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. Distributed opti- mization and statistical learning via the alternating direc- tion method of multipliers. Foundations and Trends R in Machine Learning, 3(1):1-122, 2011.","title":"Distributed optimization and statistical learning via the alternating direction method of multipliers","authors":"Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein","year":"2011","venue":"Foundations and Trends R in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Fan et al., 2017] Yanbo Fan, Ran He, Jian Liang, and Bao- Gang Hu. Self-paced learning: an implicit regularization perspective. In AAAI, 2017.","title":"Self-paced learning: an implicit regularization perspective","authors":"Yanbo Fan, Ran He, Jian Liang, Bao-Gang Hu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gorski et al., 2007] Jochen Gorski, Frank Pfeuffer, and Kathrin Klamroth. Biconvex sets and optimization with bi- convex functions: a survey and extensions. Mathematical Methods of Operations Research, 66(3):373-407, 2007.","title":"Biconvex sets and optimization with biconvex functions: a survey and extensions","authors":"Gorski","year":"2007","venue":"Mathematical Methods of Operations Research","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hong et al., 2016] Mingyi Hong, Zhi-Quan Luo, and Meisam Razaviyayn. Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems. SIAM Journal on Optimization, 26(1):337-364, 2016.","title":"Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems","authors":"Mingyi Hong, Zhi-Quan Luo, Meisam Razaviyayn","year":"2016","venue":"SIAM Journal on Optimization","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2014a] Lu Jiang, Deyu Meng, Teruko Mita- mura, and Alexander G Hauptmann. Easy samples first: Self-paced reranking for zero-example multimedia search. In Proceedings of the 22nd ACM international conference on Multimedia, pages 547-556. ACM, 2014.","title":"Easy samples first: Self-paced reranking for zero-example multimedia search","authors":"Jiang","year":"2014","venue":"Proceedings of the 22nd ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jiang et al., 2014b] Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Haupt- mann. Self-paced learning with diversity. In Advances in Neural Information Processing Systems, pages 2078- 2086, 2014.","title":"Self-paced learning with diversity","authors":"Jiang","year":"2014","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jiang et al., 2015] Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, and Alexander Hauptmann. Self-paced curriculum learning, 2015.","title":"Self-paced curriculum learning","authors":"Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander Hauptmann","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Keerthiram Murugesan, 2017] Jaime Carbonell Keerthi- ram Murugesan. Self-paced multitask learning with shared knowledge. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2522-2528, 2017.","title":"Self-paced multitask learning with shared knowledge","authors":"Jaime Carbonell, Keerthiram Murugesan","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17","doi":null,"url":null,"internal_url":"/proceedings/2017/351"},{"position":11,"raw":"[Kumar et al., 2010] M Pawan Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems, pages 1189-1197, 2010.","title":"Self-paced learning for latent variable models","authors":"Benjamin M Pawan Kumar, Daphne Packer, Koller","year":"2010","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2017] Changsheng Li, Junchi Yan, Fan Wei, Weis- han Dong, Qingshan Liu, and Hongyuan Zha. Self-paced multi-task learning. In AAAI, pages 2175-2181, 2017.","title":"Self-paced multi-task learning","authors":"Changsheng Li, Junchi Yan, Fan Wei, Weishan Dong, Qingshan Liu, Hongyuan Zha","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ma et al., 2017a] Fan Ma, Deyu Meng, Qi Xie, Zina Li, and Xuanyi Dong. Self-paced co-training. In Inter- national Conference on Machine Learning, pages 2275- 2284, 2017.","title":"Self-paced co-training","authors":"Ma","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2017b] Zilu Ma, Shiqi Liu, and Deyu Meng. On convergence property of implicit self-paced objective. arXiv preprint arXiv:1703.09923, 2017.","title":"On convergence property of implicit self-paced objective","authors":"Ma","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Meng et al., 2015] Deyu Meng, Qian Zhao, and Lu Jiang. What objective does self-paced learning indeed optimize? arXiv preprint arXiv:1511.06049, 2015.","title":"What objective does self-paced learning indeed optimize","authors":"Meng","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pi et al., 2016] Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, and Yueting Zhuang. Self-paced boost learning for classification. In Proceedings of the Twenty- Fifth International Joint Conference on Artificial Intelli- gence, IJCAI'16, pages 1932-1938. AAAI Press, 2016.","title":"Self-paced boost learning for classification","authors":"Te Pi, Xi Li, Zhongfei Zhang, Deyu Meng, Fei Wu, Jun Xiao, Yueting Zhuang","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16","doi":null,"url":null,"internal_url":"/proceedings/2016/276"},{"position":17,"raw":"[Supancic and Ramanan, 2013] James S Supancic and Deva Ramanan. Self-paced learning for long-term tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2379-2386, 2013.","title":"Self-paced learning for long-term tracking","authors":"S James, Deva Supancic, Ramanan","year":"2013","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Xu et al., 2015] Chang Xu, Dacheng Tao, and Chao Xu. Multi-view self-paced learning for clustering. In IJCAI, pages 3974-3980, 2015.","title":"Multi-view self-paced learning for clustering","authors":"Chang Xu, Dacheng Tao, Chao Xu","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/558"},{"position":19,"raw":"[Zhang et al., 2016] Dingwen Zhang, Deyu Meng, Long Zhao, and Junwei Han. Bridging saliency detection to weakly supervised object detection based on self-paced curriculum learning. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16, pages 3538-3544. AAAI Press, 2016.","title":"Bridging saliency detection to weakly supervised object detection based on self-paced curriculum learning","authors":"Zhang","year":"2016","venue":"Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI'16","doi":null,"url":null,"internal_url":"/proceedings/2016/499"},{"position":20,"raw":"[Zhang et al., 2017a] Xuchao Zhang, Liang Zhao, Arnold P. Boedihardjo, and Chang-Tien Lu. Online and distributed robust regressions under adversarial data corruption. 2017 IEEE International Conference on Data Mining (ICDM), pages 625-634, 2017.","title":"Online and distributed robust regressions under adversarial data corruption","authors":"Zhang","year":"2017","venue":"IEEE International Conference on Data Mining (ICDM)","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Zhang et al., 2017b] Xuchao Zhang, Liang Zhao, Arnold P. Boedihardjo, and Chang-Tien Lu. Robust regression via heuristic hard thresholding. In Proceedings of the Twenty- Sixth International Joint Conference on Artificial Intelli- gence, IJCAI'17. AAAI Press, 2017.","title":"Robust regression via heuristic hard thresholding","authors":"Zhang","year":"2017","venue":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI'17","doi":null,"url":null,"internal_url":"/proceedings/2017/480"},{"position":22,"raw":"[Zhang et al., 2017c] Xuchao Zhang, Liang Zhao, Arnold P. Boedihardjo, Chang-Tien Lu, and Naren Ramakrishnan. Spatiotemporal event forecasting from incomplete hyper- local price data. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17, pages 507-516, New York, NY, USA, 2017. ACM.","title":"Spatiotemporal event forecasting from incomplete hyperlocal price data","authors":"Zhang","year":"2017","venue":"Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM '17","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}