{"type":"ScholarlyArticle","title":"Representation Learning with Weighted Inner Product for Universal Approximation of General Similarities","authors":[{"name":"Geewook Kim","slug":"geewook-kim","given":"Geewook","family":"Kim","affiliation":"Graduate School of Informatics , Kyoto University RIKEN Center for Advanced Intelligence Project"},{"name":"Akifumi Okuno","slug":"akifumi-okuno","given":"Akifumi","family":"Okuno","affiliation":"Graduate School of Informatics , Kyoto University RIKEN Center for Advanced Intelligence Project"},{"name":"Kazuki Fukui","slug":"kazuki-fukui","given":"Kazuki","family":"Fukui","affiliation":"Graduate School of Informatics , Kyoto University RIKEN Center for Advanced Intelligence Project"},{"name":"Hidetoshi Shimodaira","slug":"hidetoshi-shimodaira","given":"Hidetoshi","family":"Shimodaira","affiliation":"Graduate School of Informatics , Kyoto University RIKEN Center for Advanced Intelligence Project"}],"abstract":"We propose weighted inner product similarity (WIPS) for neural network-based graph embedding. In addition to the parameters of neural networks, we optimize the weights of the inner product by allowing positive and negative values. Despite its simplicity, WIPS can approximate arbitrary general similarities including positive definite, conditionally positive definite, and indefinite kernels. WIPS is free from similarity model selection, since it can learn any similarity models such as cosine similarity, negative Poincaré distance and negative Wasserstein distance. Our experiments show that the proposed method can learn high-quality distributed representations of nodes from real datasets, leading to an accurate approximation of similarities as well as high performance in inductive tasks.","doi":"10.24963/ijcai.2019/699","url":"https://ijcai.org/proceedings/2019/699","pdf":"https://ijcai.org/proceedings/2019/0699.pdf","markdown":"https://ijcai.org/proceedings/2019/699.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/699","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Natural Language Processing","track":"Main track","number":699,"pages":{"first":5031,"last":5038},"keywords":["Natural Language Processing: Embeddings","Machine Learning: Unsupervised Learning","Machine Learning Applications: Networks","Machine Learning: Dimensionality Reduction and Manifold Learning","Machine Learning Applications: Applications of Unsupervised Learning","Machine Learning: Learning Theory"],"videos":[],"references":[{"position":1,"raw":"[Agirre et al., 2009] Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches. In Proceedings of Hu- man Language Technologies: The 2009 Annual Confer- ence of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 19- 27. Association for Computational Linguistics, 2009.","title":"A Study on Similarity and Relatedness Using Distributional and WordNet-based Approaches","authors":"Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Pasca, Aitor Soroa","year":"2009","venue":"Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bellet et al., 2013] Aurélien Bellet, Amaury Habrard, and Marc Sebban. A survey on metric learning for feature vec- tors and structured data. arXiv:1306.6709, 2013.","title":"A survey on metric learning for feature vectors and structured data","authors":"Bellet","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Böttcher and Lancaster, 1996] A. Böttcher and P. Lancaster. Lectures on Operator Theory and Its Applications. Fields Institute monographs. American Mathematical Society, 1996.","title":"Lectures on Operator Theory and Its Applications","authors":"A Böttcher, P Lancaster","year":"1996","venue":"Fields Institute monographs","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bromley et al., 1994] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a \"siamese\" time delay neural network. In Advances in Neural Information Processing Systems 6, pages 737-744.","title":"Signature verification using a \"siamese\" time delay neural network","authors":"Jane Bromley, Isabelle Guyon, Yann Lecun, Eduard Säckinger, Roopak Shah","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chechik et al., 2009] Gal Chechik, Uri Shalit, Varun Sharma, and Samy Bengio. An Online Algorithm for Large Scale Image Similarity Learning. In Advances in Neural Information Processing Systems 22, pages 306- 314.","title":"An Online Algorithm for Large Scale Image Similarity Learning","authors":"Chechik","year":"2009","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"Curran Associates, Inc., 2009.","title":null,"authors":"Curran Associates, Inc","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cybenko, 1989] G. Cybenko. Approximation by superpo- sitions of a sigmoidal function. Mathematics of Control, Signals, and Systems (MCSS), 2(4):303-314, 1989.","title":"Approximation by superpositions of a sigmoidal function","authors":"G Cybenko","year":"1989","venue":"Mathematics of Control, Signals, and Systems (MCSS)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Funahashi, 1989] Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 2(3):183-192, 1989.","title":"On the approximate realization of continuous mappings by neural networks","authors":"Ken-Ichi Funahashi","year":"1989","venue":"Neural Networks","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goldberg, 2016] Yoav Goldberg. A primer on neural net- work models for natural language processing. Journal of Artificial Intelligence Research, 57(1):345-420, 2016.","title":"A primer on neural network models for natural language processing","authors":"Yoav Goldberg","year":"2016","venue":"Journal of Artificial Intelligence Research","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goldberger et al., 2005] Jacob Goldberger, Geoffrey E Hin- ton, Sam T. Roweis, and Ruslan R Salakhutdinov. Neigh- bourhood Components Analysis. In Advances in Neural Information Processing Systems 17, pages 513-520. MIT Press, 2005.","title":"Neighbourhood Components Analysis","authors":"Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, Ruslan R Salakhutdinov","year":"2005","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goldfarb, 1984] Lev Goldfarb. A unified approach to pat- tern recognition. Pattern Recognition, 17(5):575-582, 1984.","title":"Lev Goldfarb. A unified approach to pattern recognition","authors":"Goldfarb","year":"1984","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Greub, 1975] Werner Greub. Linear Algebra. Graduate texts in mathematics. Springer New York, NY, 1975.","title":"Linear Algebra","authors":"Werner Greub","year":"1975","venue":"Graduate texts in mathematics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hamilton et al., 2017] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems 30, pages 1024-1034.","title":"Inductive Representation Learning on Large Graphs","authors":"Hamilton","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Hill et al., 2015] Felix Hill, Roi Reichart, and Anna Korho- nen. Simlex-999: Evaluating Semantic Models with Gen- uine Similarity Estimation. Computational Linguistics, 41(4):665-695, 2015.","title":"Simlex-999: Evaluating Semantic Models with Genuine Similarity Estimation","authors":"Felix Hill, Roi Reichart, Anna Korhonen","year":"2015","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kolouri et al., 2016] S. Kolouri, Y. Zou, and G. K. Rohde. Sliced Wasserstein Kernels for Probability Distributions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5258-5267. IEEE, 2016.","title":"Sliced Wasserstein Kernels for Probability Distributions","authors":"S Kolouri, Y Zou, G K Rohde","year":"2016","venue":"IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Kulis, 2013] Brian Kulis. Metric Learning: A Survey. Foun- dations and Trends in Machine Learning, 5(4):287-364, 2013.","title":"Metric Learning: A Survey","authors":"Brian Kulis","year":"2013","venue":"Foundations and Trends in Machine Learning","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Kung, 2014] Sun Yuan Kung. Kernel Methods and Machine Learning. Cambridge University Press, 2014.","title":"Kernel Methods and Machine Learning","authors":"Sun Yuan, Kung","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Leimeister and Wilson, 2018] Matthias Leimeister and Benjamin J. Wilson. Skip-gram word embeddings in hyperbolic space. arXiv:1809.01498, 2018.","title":"Skip-gram word embeddings in hyperbolic space","authors":"Matthias Leimeister, Benjamin J Wilson","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Repre- sentations of Words and Phrases and Their Compositional- ity. In Advances in Neural Information Processing Systems 26, pages 3111-3119.","title":"Distributed Representations of Words and Phrases and Their Compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Minh et al., 2006] Ha Quang Minh, Partha Niyogi, and Yuan Yao. Mercer's Theorem, Feature Maps, and Smooth- ing. In Proceedings of the 19th Annual Conference on Learning Theory, pages 154-168. Springer, 2006.","title":"Mercer's Theorem, Feature Maps, and Smoothing","authors":"Ha Quang, Minh, Partha Niyogi, Yuan Yao","year":"2006","venue":"Proceedings of the 19th Annual Conference on Learning Theory","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Nickel and Kiela, 2017] Maximillian Nickel and Douwe Kiela. Poincaré Embeddings for Learning Hierarchical Representations. In Advances in Neural Information Pro- cessing Systems 30, pages 6338-6347.","title":"Maximillian Nickel and Douwe Kiela. Poincaré Embeddings for Learning Hierarchical Representations","authors":"Kiela Nickel","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nickel and Kiela, 2018] Maximillian Nickel and Douwe Kiela. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry. In Proceedings of the 35th International Conference on Machine Learning (ICML), pages 3779-3788. PMLR, 2018.","title":"Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry","authors":"Kiela Nickel, Maximillian Nickel, Douwe Kiela","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Oglic and Gaertner, 2018] Dino Oglic and Thomas Gaert- ner. Learning in Reproducing Kernel Kre ȋn Spaces. In Proceedings of the 35th International Conference on Ma- chine Learning (ICML), pages 3859-3867. PMLR, 2018.","title":"Dino Oglic and Thomas Gaertner. Learning in Reproducing Kernel Kre ȋn Spaces","authors":"Gaertner Oglic","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Okuno et al., 2018] Akifumi Okuno, Tetsuya Hada, and Hidetoshi Shimodaira. A probabilistic framework for multi-view feature learning with many-to-many associa- tions via neural networks. In Proceedings of the 35th Inter- national Conference on Machine Learning (ICML), pages 3885-3894. PMLR, 2018.","title":"A probabilistic framework for multi-view feature learning with many-to-many associations via neural networks","authors":"Akifumi Okuno, Tetsuya Hada, Hidetoshi Shimodaira","year":"2018","venue":"Proceedings of the 35th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Okuno et al., 2019] Akifumi Okuno, Geewook Kim, and Hidetoshi Shimodaira. Graph Embedding with Shifted In- ner Product Similarity and Its Improved Approximation Capability. In Proceedings of the 22nd International Con- ference on Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, pages 644-653. PMLR, 2019.","title":"Graph Embedding with Shifted Inner Product Similarity and Its Improved Approximation Capability","authors":"Akifumi Okuno, Geewook Kim, Hidetoshi Shimodaira","year":"2019","venue":"Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS)","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ong et al., 2004] CS. Ong, X. Mary, S. Canu, and AJ. Smola. Learning with Non-Positive Kernels. In Pro- ceedings of the 21st International Conference on Machine Learning (ICML), pages 81-81. ACM Press, 2004.","title":"Learning with Non-Positive Kernels","authors":"Ong","year":"2004","venue":"Proceedings of the 21st International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics, 2014.","title":"GloVe: Global Vectors for Word Representation","authors":"Jeffrey Pennington, Richard Socher, Christopher Manning","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pages 701-710. ACM, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Prado et al., 2013] Adriana Prado, Marc Plantevit, Céline Robardet, and Jean-Francois Boulicaut. Mining graph topological patterns: Finding covariations among vertex descriptors. IEEE Transactions on Knowledge and Data Engineering, 25(9):2090-2104, 2013.","title":"Mining graph topological patterns: Finding covariations among vertex descriptors","authors":"Prado","year":"2013","venue":"IEEE Transactions on Knowledge and Data Engineering","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Schleif and Tino, 2015] Frank-Michael Schleif and Peter Tino. Indefinite Proximity Learning: A Review. Neural Computation, 27(10):2039-2096, 2015.","title":"Indefinite Proximity Learning: A Review","authors":"Tino ; Frank-Michael Schleif, Peter Schleif, Tino","year":"2015","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Schölkopf, 2001] Bernhard Schölkopf. The kernel trick for distances. In Advances in Neural Information Processing Systems 13, pages 301-307. MIT Press, 2001.","title":"The kernel trick for distances","authors":"Bernhard Schölkopf","year":"2001","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Tang et al., 2015] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. LINE: Large- scale Information Network Embedding. In Proceedings of the 24th International Conference on World Wide Web (WWW), pages 1067-1077. International World Wide Web Conferences Steering Committee, 2015.","title":"LINE: Largescale Information Network Embedding","authors":"Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, Qiaozhu Mei","year":"2015","venue":"Proceedings of the 24th International Conference on World Wide Web (WWW)","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Telgarsky, 2017] Matus Telgarsky. Neural Networks and Rational Functions. In Proceedings of the 34th Interna- tional Conference on Machine Learning (ICML), pages 3387-3393. PMLR, 2017.","title":"Neural Networks and Rational Functions","authors":"Matus Telgarsky","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning (ICML)","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Wang et al., 2016] Daixin Wang, Peng Cui, and Wenwu Zhu. Structural Deep Network Embedding. In Proceed- ings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1225- 1234. ACM, 2016.","title":"Structural Deep Network Embedding","authors":"Wang","year":"2016","venue":"Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Xu et al., 2018] Hongteng Xu, Wenlin Wang, Wei Liu, and Lawrence Carin. Distilled Wasserstein Learning for Word Embedding and Topic Modeling. In Advances in Neu- ral Information Processing Systems 31, pages 1723-1732.","title":"Distilled Wasserstein Learning for Word Embedding and Topic Modeling","authors":"Xu","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":38,"raw":"[Yan et al., 2007] Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, and Stephen Lin. Graph embedding and extensions: A general framework for di- mensionality reduction. IEEE Trans. Pattern Anal. Mach. Intell., 29(1):40-51, 2007.","title":"Graph embedding and extensions: A general framework for dimensionality reduction","authors":"Shuicheng Yan, Dong Xu, Benyu Zhang, Hong-Jiang Zhang, Qiang Yang, Stephen Lin","year":"2007","venue":"IEEE Trans. Pattern Anal. Mach. Intell","doi":null,"url":null,"internal_url":null},{"position":39,"raw":"[Yang and Powers, 2005] Dongqiang Yang and David M. W. Powers. Measuring Semantic Similarity in the Taxonomy of WordNet. In Proceedings of the 28th Australasian Con- ference on Computer Science -Volume 38, pages 315-322. Australian Computer Society, Inc., 2005.","title":"Measuring Semantic Similarity in the Taxonomy of WordNet","authors":"Yang, Powers, Dongqiang Yang, David M W Powers","year":"2005","venue":"Proceedings of the 28th Australasian Conference on Computer Science","doi":null,"url":null,"internal_url":null},{"position":40,"raw":"[Yarotsky, 2017] Dmitry Yarotsky. Error bounds for approx- imations with deep ReLU networks. Neural Networks, 94:103-114, 2017.","title":"Error bounds for approximations with deep ReLU networks","authors":"Dmitry Yarotsky","year":"2017","venue":"Neural Networks","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}