{"type":"ScholarlyArticle","title":"Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm","authors":[{"name":"Mengdi Huai","slug":"mengdi-huai","given":"Mengdi","family":"Huai","affiliation":"Department of Computer Science , University of Virginia , VA , USA"},{"name":"Hongfei Xue","slug":"hongfei-xue","given":"Hongfei","family":"Xue","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA"},{"name":"Chenglin Miao","slug":"chenglin-miao","given":"Chenglin","family":"Miao","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA"},{"name":"Liuyi Yao","slug":"liuyi-yao","given":"Liuyi","family":"Yao","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA"},{"name":"Lu Su","slug":"lu-su","given":"Lu","family":"Su","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA"},{"name":"Changyou Chen","slug":"changyou-chen","given":"Changyou","family":"Chen","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA"},{"name":"Aidong Zhang","slug":"aidong-zhang","given":"Aidong","family":"Zhang","affiliation":"Department of Computer Science , University of Virginia , VA , USA"}],"abstract":"As an effective way to learn a distance metric between pairs of samples, deep metric learning (DML) has drawn significant attention in recent years. The key idea of DML is to learn a set of hierarchical nonlinear mappings using deep neural networks, and then project the data samples into a new feature space for comparing or matching. Although DML has achieved practical success in many applications, there is no existing work that theoretically analyzes the generalization error bound for DML, which can measure how good a learned DML model is able to perform on unseen data. In this paper, we try to fill up this research gap and derive the generalization error bound for DML. Additionally, based on the derived generalization bound, we propose a novel DML method (called ADroDML), which can adaptively learn the retention rates for the DML models with dropout in a theoretically justified way. Compared with existing DML works that require predefined retention rates, ADroDML can learn the retention rates in an optimal way and achieve better performance. We also conduct experiments on real-world datasets to verify the findings derived from the generalization error bound and demonstrate the effectiveness of the proposed adaptive DML method.","doi":"10.24963/ijcai.2019/352","url":"https://ijcai.org/proceedings/2019/352","pdf":"https://ijcai.org/proceedings/2019/0352.pdf","markdown":"https://ijcai.org/proceedings/2019/352.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/352","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":352,"pages":{"first":2535,"last":2541},"keywords":["Machine Learning: Learning Theory","Machine Learning: Data Mining"],"videos":[],"references":[{"position":1,"raw":"[Bertsekas, 1999] Dimitri P Bertsekas. Nonlinear program- ming. Athena scientific Belmont, 1999.","title":"Nonlinear programming","authors":"Dimitri P Bertsekas","year":"1999","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2016] Qiong Cao, Zheng-Chu Guo, and Yiming Ying. Generalization bounds for metric and similarity learn- ing. Machine Learning, 102(1):115-132, 2016.","title":"Generalization bounds for metric and similarity learning","authors":"Qiong Cao, Zheng-Chu Guo, Yiming Ying","year":"2016","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Clémenc ¸on et al., 2005] Stéphan Clémenc ¸on, Gábor Lugosi, and Nicolas Vayatis. Ranking and scoring using empirical risk minimization. In Proceedings of COLT, pages 1-15, 2005.","title":"Ranking and scoring using empirical risk minimization","authors":"Stéphan Clémenc ¸on, Gábor Lugosi, Nicolas Vayatis","year":"2005","venue":"Proceedings of COLT","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Gouk et al., 2015] Henry Gouk, Bernhard Pfahringer, and Michael Cree. Fast metric learning for deep neural net- works. arXiv preprint arXiv:1511.06442, 2015.","title":"Fast metric learning for deep neural networks","authors":"Henry Gouk, Bernhard Pfahringer, Michael Cree","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Hoffer and Ailon, 2015] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop on Similarity-Based Pattern Recognition, 2015.","title":"Deep metric learning using triplet network","authors":"Elad Hoffer, Nir Ailon","year":"2015","venue":"International Workshop on Similarity-Based Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Huai et al., 2018a] Mengdi Huai, Chenglin Miao, Yaliang Li, Qiuling Suo, Lu Su, and Aidong Zhang. Metric learning from probabilistic labels. In Proceedings of SIGKDD, pages 1541-1550, 2018.","title":"Metric learning from probabilistic labels","authors":"Huai","year":"2018","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Huai et al., 2018b] Mengdi Huai, Chenglin Miao, Qiuling Suo, Yaliang Li, Jing Gao, and Aidong Zhang. Uncorrelated patient similarity learning. In Proceedings of SDM, pages 270-278, 2018.","title":"Uncorrelated patient similarity learning","authors":"Huai","year":"2018","venue":"Proceedings of SDM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2015] Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, and Xilin Chen. Log-euclidean metric learning on symmetric positive definite manifold with appli- cation to image set classification. In Proceedings of ICML, pages 720-729, 2015.","title":"Log-euclidean metric learning on symmetric positive definite manifold with application to image set classification","authors":"Zhiwu Huang, Ruiping Wang, Shiguang Shan, Xianqiu Li, Xilin Chen","year":"2015","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2016] Chen Huang, Chen Change Loy, and Xi- aoou Tang. Local similarity-aware deep feature embedding. In Proceedings of NeurIPS, pages 1262-1270, 2016.","title":"Local similarity-aware deep feature embedding","authors":"Chen Huang, Chen Change Loy, Xiaoou Tang","year":"2016","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Jin et al., 2009] Rong Jin, Shijun Wang, and Yang Zhou. Regularized distance metric learning: Theory and algo- rithm. In Proceedings of NeurIPS, pages 862-870, 2009.","title":"Regularized distance metric learning: Theory and algorithm","authors":"Rong Jin, Shijun Wang, Yang Zhou","year":"2009","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Law et al., 2017] Marc T Law, Raquel Urtasun, and Richard S Zemel. Deep spectral clustering learning. In Proceedings of ICML, pages 1985-1994, 2017.","title":"Deep spectral clustering learning","authors":"Raquel Marc T Law, Richard S Urtasun, Zemel","year":"2017","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Ma et al., 2016] Xuezhe Ma, Yingkai Gao, Zhiting Hu, Yao- liang Yu, Yuntian Deng, and Eduard Hovy. Dropout with expectation-linear regularization. arXiv preprint arXiv:1609.08017, 2016.","title":"Dropout with expectation-linear regularization","authors":"Ma","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mason et al., 2017] Blake Mason, Lalit Jain, and Robert Nowak. Learning low-dimensional metrics. In Proceedings of NeurIPS, pages 4139-4147, 2017.","title":"Learning low-dimensional metrics","authors":"Blake Mason, Lalit Jain, Robert Nowak","year":"2017","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ni et al., 2017] Jiazhi Ni, Jie Liu, Chenxin Zhang, Dan Ye, and Zhirou Ma. Fine-grained patient similarity measuring using deep metric learning. In Proceedings of CIKM, pages 1189-1198, 2017.","title":"Fine-grained patient similarity measuring using deep metric learning","authors":"Jiazhi Ni, Jie Liu, Chenxin Zhang, Dan Ye, Zhirou Ma","year":"2017","venue":"Proceedings of CIKM","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Qian et al., 2014] Qi Qian, Juhua Hu, Rong Jin, Jian Pei, and Shenghuo Zhu. Distance metric learning using dropout: a structured regularization approach. In Proceedings of SIGKDD, pages 323-332, 2014.","title":"Distance metric learning using dropout: a structured regularization approach","authors":"Juhua Qi Qian, Rong Hu, Jian Jin, Shenghuo Pei, Zhu","year":"2014","venue":"Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Raghunathan et al., 2018] Aditi Raghunathan, Jacob Stein- hardt, and Percy Liang. Certified defenses against adversar- ial examples. In Proceedings of ICML, 2018.","title":"Certified defenses against adversarial examples","authors":"Aditi Raghunathan, Jacob Steinhardt, Percy Liang","year":"2018","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shalev-Shwartz and Ben-David, 2014] Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.","title":"Understanding machine learning: From theory to algorithms","authors":"Shalev-Shwartz, Ben-David ; Shai Shalev-Shwartz, Shai Ben-David","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sohn, 2016] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In Proceedings of NeurIPS, pages 1857-1865, 2016.","title":"Improved deep metric learning with multi-class n-pair loss objective","authors":"Kihyuk Sohn, Sohn","year":"2016","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Song et al., 2016] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of CVPR, pages 4004-4012, 2016.","title":"Deep metric learning via lifted structured feature embedding","authors":"Hyun Oh Song, Yu Xiang, Stefanie Jegelka, Silvio Savarese","year":"2016","venue":"Proceedings of CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[St Amand and Huan, 2017] Joseph St Amand and Jun Huan. Sparse compositional local metric learning. In Proceedings of Proceedings of SIGKDD, pages 1097-1104, 2017.","title":"Sparse compositional local metric learning","authors":"St Amand, Joseph Huan, Amand St, Jun Huan","year":"2017","venue":"Proceedings of Proceedings of SIGKDD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sun et al., 2014] Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. Deep learning face representation by joint identification-verification. In Proceedings of NeurIPS, pages 1988-1996, 2014.","title":"Deep learning face representation by joint identification-verification","authors":"Yi Sun, Yuheng Chen, Xiaogang Wang, Xiaoou Tang","year":"2014","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Suo et al., 2018] Qiuling Suo, Weida Zhong, Fenglong Ma, Yuan Ye, Mengdi Huai, and Aidong Zhang. Multi-task sparse metric learning for monitoring patient similarity progression. In Proceedings of ICDM, pages 477-486, 2018.","title":"Multi-task sparse metric learning for monitoring patient similarity progression","authors":"Suo","year":"2018","venue":"Proceedings of ICDM","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wan et al., 2013] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neu- ral networks using dropconnect. In Proceedings of ICML, pages 1058-1066, 2013.","title":"Regularization of neural networks using dropconnect","authors":"Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, Rob Fergus","year":"2013","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wang et al., 2017] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. arXiv preprint arXiv:1708.01682, 2017.","title":"Deep metric learning with angular loss","authors":"Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, Yuanqing Lin","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Weinberger et al., 2006] Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. In Proceedings of NeurIPS, pages 1473-1480, 2006.","title":"Distance metric learning for large margin nearest neighbor classification","authors":"John Kilian Q Weinberger, Lawrence K Blitzer, Saul","year":"2006","venue":"Proceedings of NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zadeh et al., 2016] Pourya Zadeh, Reshad Hosseini, and Su- vrit Sra. Geometric mean metric learning. In Proceedings of ICML, pages 2464-2471, 2016.","title":"Geometric mean metric learning","authors":"Zadeh","year":"2016","venue":"Proceedings of ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhai and Wang, 2018] Ke Zhai and Huan Wang. Adaptive dropout with rademacher complexity regularization. In Proceedings of ICLR, 2018.","title":"Adaptive dropout with rademacher complexity regularization","authors":"Wang ; ] Ke Zhai, Huan Zhai, Wang","year":"2018","venue":"Proceedings of ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}