{"type":"ScholarlyArticle","title":"Exemplar-centered Supervised Shallow Parametric Data Embedding","authors":[{"name":"Martin Renqiang Min","slug":"martin-renqiang-min","given":"Martin Renqiang","family":"Min","affiliation":"NEC Labs America Princeton , NJ 08540"},{"name":"Hongyu Guo","slug":"hongyu-guo","given":"Hongyu","family":"Guo","affiliation":"National Research Council Canada Ottawa , ON K1A 0R6"},{"name":"Dongjin Song","slug":"dongjin-song","given":"Dongjin","family":"Song","affiliation":"NEC Labs America Princeton , NJ 08540"}],"abstract":"Metric learning methods for dimensionality reduction in combination  with k-Nearest Neighbors (kNN) have been extensively deployed in many classification, data embedding, and information retrieval applications. However, most of these approaches involve pairwise training data comparisons, and thus have quadratic computational complexity with respect to the size of training set, preventing them from scaling to fairly big datasets. Moreover, during testing, comparing test data against all the training data points is also expensive in terms of both computational cost and resources required. Furthermore, previous metrics are either too constrained or too expressive to be well learned. To effectively solve these issues, we present an exemplar-centered supervised shallow parametric data embedding model, using a Maximally Collapsing Metric Learning (MCML) objective. Our strategy learns a shallow high-order parametric embedding function and compares training/test data only with learned or precomputed exemplars, resulting in a cost function with linear computational complexity for both training and testing. We also empirically demonstrate, using several benchmark datasets, that for classification in two-dimensional embedding space, our approach not only gains speedup of kNN by hundreds of times, but also outperforms state-of-the-art supervised embedding approaches.","doi":"10.24963/ijcai.2017/345","url":"https://ijcai.org/proceedings/2017/345","pdf":"https://ijcai.org/proceedings/2017/0345.pdf","markdown":"https://ijcai.org/proceedings/2017/345.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/345","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":345,"pages":{"first":2479,"last":2485},"keywords":["Machine Learning: Neural Networks","Machine Learning: Data Mining","Machine Learning: Machine Learning"],"videos":[],"references":[{"position":1,"raw":"[Bellet et al., 2013] Aurelien Bellet, Amaury Habrard, and Marc Sebban. A survey on metric learning for feature vectors and structured data. CoRR, abs/1306.6709, 2013.","title":"A survey on metric learning for feature vectors and structured data","authors":"Aurelien Bellet, Amaury Habrard, Marc Sebban","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Globerson and Roweis, 2006] Amir Globerson and Sam Roweis. Metric learning by collapsing classes. In Pro- ceedings of Advances in Neural Information Processing Systems 21, pages 451-458. MIT Press, Cambridge, MA, 2006.","title":"Metric learning by collapsing classes","authors":"Roweis Globerson, Amir Globerson, Sam Roweis","year":"2006","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Goldberger et al., 2004] Jacob Goldberger, Sam T. Roweis, Geoffrey E. Hinton, and Ruslan Salakhutdinov. Neighbour- hood components analysis. In Proceedings of Advances in Neural Information Processing Systems 19, pages 513-520. 2004.","title":"Neighbourhood components analysis","authors":"Jacob Goldberger, Sam T Roweis, Geoffrey E Hinton, Ruslan Salakhutdinov","year":"2004","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Guo et al., 2015] Hongyu Guo, Xiaodan Zhu, and Mar- tin Renqiang Min. A deep learning model for structured outputs with high-order interaction. CoRR, abs/1504.08022, 2015.","title":"A deep learning model for structured outputs with high-order interaction","authors":"Hongyu Guo, Xiaodan Zhu, Martin Renqiang, Min","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He and Niyogi, 2003] Xiaofei He and Partha Niyogi. Local- ity preserving projections. In Proceedings of Advances in Neural Information Processing Systems 16, 2003.","title":"Locality preserving projections","authors":"Xiaofei He, Partha Niyogi","year":"2003","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Kedem et al., 2012] Dor Kedem, Stephen Tyree, Fei Sha, Gert R. Lanckriet, and Kilian Q Weinberger. Non-linear metric learning. In Proceedings of Advances in Neural In- formation Processing Systems 25, pages 2573-2581. 2012.","title":"Non-linear metric learning","authors":"Dor Kedem, Stephen Tyree, Fei Sha, Gert R Lanckriet, Kilian Q Weinberger","year":"2012","venue":"Proceedings of Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Kuksa et al., 2015] Pavel P. Kuksa, Martin R. Min, Rishabh Dugar, and Mark Gerstein. High-order neural networks and kernel methods for peptide-MHC binding prediction. Bioinformatics, 31(22):3600-3607, 2015.","title":"High-order neural networks and kernel methods for peptide-MHC binding prediction","authors":"Pavel P Kuksa, Martin R Min, Rishabh Dugar, Mark Gerstein","year":"2015","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kulis, 2013] Brian Kulis. Metric learning: A survey. Foun- dations and Trends in Machine Learning, 5(4):287-364, 2013.","title":"Metric learning: A survey","authors":"Brian Kulis","year":"2013","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kusner et al., 2014] Matt J. Kusner, Stephen Tyree, Kilian Q. Weinberger, and Kunal Agrawal. Stochastic neighbor com- pression. In Proceedings of the 31st International Confer- ence on Machine Learning, pages 622-630, 2014.","title":"Stochastic neighbor compression","authors":"Matt J Kusner, Stephen Tyree, Kilian Q Weinberger, Kunal Agrawal","year":"2014","venue":"Proceedings of the 31st International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Memisevic, 2011] Roland Memisevic. Gradient-based learn- ing of higher-order image features. In ICCV, pages 1591- 1598, 2011.","title":"Roland Memisevic. Gradient-based learning of higher-order image features","authors":"Memisevic","year":"2011","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Min et al., 2009] Renqiang Min, David A Stanley, Zineng Yuan, Anthony Bonner, and Zhaolei Zhang. A deep non- linear feature mapping for large-margin knn classification. In ICDM, pages 357-366. IEEE, 2009.","title":"A deep nonlinear feature mapping for large-margin knn classification","authors":"Min","year":"2009","venue":"ICDM","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Min et al., 2010] Martin Renqiang Min, Laurens van der Maaten, Zineng Yuan, Anthony J. Bonner, and Zhaolei Zhang. Deep supervised t-distributed embedding. In Pro- ceedings of the 27th International Conference on Machine Learning, pages 791-798, 2010.","title":"Deep supervised t-distributed embedding","authors":"Martin Renqiang, Min, Laurens Van Der Maaten, Zineng Yuan, Anthony J Bonner, Zhaolei Zhang","year":"2010","venue":"Proceedings of the 27th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Min et al., 2014a] Martin Renqiang Min, Salim Chowdhury, Yanjun Qi, Alex Stewart, and Rachel Ostroff. An integrated approach to blood-based cancer diagnosis and biomarker discovery. In Pacific Symposium on Biocomputing (PSB), pages 87-98, 2014.","title":"An integrated approach to blood-based cancer diagnosis and biomarker discovery","authors":"Min","year":"2014","venue":"Pacific Symposium on Biocomputing (PSB)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Min et al., 2014b] Martin Renqiang Min, Xia Ning, Chao Cheng, and Mark Gerstein. Interpretable sparse high-order boltzmann machines. In Proceedings of the Seventeenth In- ternational Conference on Artificial Intelligence and Statis- tics, pages 614-622, 2014.","title":"Interpretable sparse high-order boltzmann machines","authors":"Min","year":"2014","venue":"Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Min, 2005] Martin Renqiang Min. A non-linear dimension- ality reduction method for improving nearest neighbour classification. In Master Thesis. Department of Computer Science, University of Toronto, 2005.","title":"A non-linear dimensionality reduction method for improving nearest neighbour classification","authors":"Martin Renqiang, Min","year":"2005","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Perrot and Habrard, 2015] Michaël Perrot and Amaury Habrard. Regressive virtual metric learning. In NIPS15, Montreal, Quebec, Canada, pages 1810-1818, 2015.","title":"Regressive virtual metric learning","authors":"Michaël Perrot, Amaury Habrard","year":"2015","venue":"NIPS15","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Purushotham et al., 2014] S. Purushotham, M. R. Min, C- C. Jay Kuo, and R. Ostroff. Factorized sparse learning models with interpretable high order feature interactions. In KDD, New York, USA, 2014.","title":"Factorized sparse learning models with interpretable high order feature interactions","authors":"S Purushotham, M R Min, C-C. Jay Kuo, R Ostroff","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Ranzato and Hinton, 2010] Marc'Aurelio Ranzato and Ge- offrey E. Hinton. Modeling pixel means and covariances using factorized third-order boltzmann machines. In CVPR, 2010.","title":"Marc'Aurelio Ranzato and Geoffrey E. Hinton. Modeling pixel means and covariances using factorized third-order boltzmann machines","authors":"Hinton Ranzato","year":"2010","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ranzato et al., 2010] Marc'Aurelio Ranzato, Alex Krizhevsky, and Geoffrey E. Hinton. Factored 3- way restricted boltzmann machines for modeling natural images. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, pages 621-628, 2010.","title":"Factored 3way restricted boltzmann machines for modeling natural images","authors":"Ranzato","year":"2010","venue":"Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Simonyan and Zisserman, 2015] K. Simonyan and A. Zis- serman. Very deep convolutional networks for large-scale image recognition. In Proceedings of International Confer- ence on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"K Simonyan, A Zisserman","year":"2015","venue":"Proceedings of International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[van der Maaten and Hinton, 2008] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. The Journal of Machine Learning Research, 9:2579-2605, 2008.","title":"Visualizing data using t-sne","authors":"Van Der Maaten, Laurens Hinton, Geoffrey Van Der Maaten, Hinton","year":"2008","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[van der Maaten, 2009] Laurens van der Maaten. Learning a parametric embedding by preserving local structure. In Pro- ceedings of the 12th International Conference on Artificial Intelligence and Statistics, pages 384-391, 2009.","title":"Learning a parametric embedding by preserving local structure","authors":"Van Der Maaten ; Laurens Van Der Maaten","year":"2009","venue":"Proceedings of the 12th International Conference on Artificial Intelligence and Statistics","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Weinberger and Saul, 2009] Kilian Q. Weinberger and Lawrence K. Saul. Distance metric learning for large mar- gin nearest neighbor classification. J. Mach. Learn. Res., 10:207-244, June 2009.","title":"Distance metric learning for large margin nearest neighbor classification","authors":"Q Kilian, Lawrence K Weinberger, Saul","year":"2009","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}