{"type":"ScholarlyArticle","title":"Ornstein Auto-Encoders","authors":[{"name":"Youngwon Choi","slug":"youngwon-choi","given":"Youngwon","family":"Choi","affiliation":"Department of Statistics , Seoul National University , Republic of Korea"},{"name":"Joong-Ho Won","slug":"joong-ho-won","given":"Joong-Ho","family":"Won","affiliation":"Department of Statistics , Seoul National University , Republic of Korea"}],"abstract":"We propose the Ornstein auto-encoder (OAE), a representation learning model for correlated data. In many interesting applications, data have nested structures. Examples include the VGGFace and MNIST datasets. We view such data consist of i.i.d. copies of a stationary random process, and seek a latent space representation of the observed sequences. This viewpoint necessitates a distance measure between two random processes. We propose to use Orstein's d-bar distance, a process extension of Wasserstein's distance. We first show that the theorem by Bousquet et al. (2017) for Wasserstein auto-encoders extends to stationary random processes. This result, however, requires both encoder and decoder to map an entire sequence to another. We then show that, when exchangeability within a process, valid for VGGFace and MNIST, is assumed, these maps reduce to univariate ones, resulting in a much simpler, tractable optimization problem. Our experiments show that OAEs successfully separate individual sequences in the latent space, and can generate new variations of unknown, as well as known, identity. The latter has not been possible with other existing methods.","doi":"10.24963/ijcai.2019/301","url":"https://ijcai.org/proceedings/2019/301","pdf":"https://ijcai.org/proceedings/2019/0301.pdf","markdown":"https://ijcai.org/proceedings/2019/301.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/301","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":301,"pages":{"first":2172,"last":2178},"keywords":["Machine Learning: Learning Generative Models","Computer Vision: Biometrics, Face and Gesture Recognition","Machine Learning: Deep Learning","Machine Learning: Clustering","Machine Learning: Dimensionality Reduction and Manifold Learning"],"videos":[],"references":[{"position":1,"raw":"[Bousquet et al., 2017] Olivier Bousquet, Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Simon-Gabriel, and Bernhard Schoelkopf. From optimal transport to generative modeling: the VEGAN cookbook. arXiv preprint arXiv:1705.07642, 2017.","title":"From optimal transport to generative modeling: the VEGAN cookbook","authors":"Olivier Bousquet, Sylvain Gelly, Ilya Tolstikhin, Carl-Johann Simon-Gabriel, Bernhard Schoelkopf","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Cao et al., 2018] Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. VGGFace2: A dataset for recognising faces across pose and age. In 13th IEEE Inter- national Conference on Automatic Face Gesture Recogni- tion (FG 2018), pages 67-74, 2018.","title":"VGGFace2: A dataset for recognising faces across pose and age","authors":"Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, Andrew Zisserman","year":"2018","venue":"13th IEEE International Conference on Automatic Face Gesture Recognition (FG 2018)","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Diggle et al., 2002] Peter Diggle, Peter J Diggle, Patrick Heagerty, Patrick J Heagerty, Kung-Yee Liang, Scott Zeger, et al. Analysis of longitudinal data. Oxford Uni- versity Press, 2002.","title":"Analysis of longitudinal data","authors":"Peter Diggle, J Peter, Patrick Diggle, Patrick J Heagerty, Kung-Yee Heagerty, Scott Liang, Zeger","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dundar et al., 2007] Murat Dundar, Balaji Krishnapuram, Jinbo Bi, and R. Bharat Rao. Learning classifiers when the training data is not IID. In Proceedings of the 20th Inter- national Joint Conference on Artifical Intelligence, pages 756-761.","title":"Learning classifiers when the training data is not IID","authors":"Balaji Murat Dundar, Jinbo Krishnapuram, R Bharat Bi, Rao","year":null,"venue":"Proceedings of the 20th International Joint Conference on Artifical Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2007/121"},{"position":5,"raw":"[Fitzmaurice et al., 2012] Garrett M Fitzmaurice, Nan M Laird, and James H Ware. Applied longitudinal analysis, volume 998. John Wiley & Sons, 2012.","title":"Applied longitudinal analysis","authors":"Nan M Garrett M Fitzmaurice, James H Laird, Ware","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gray et al., 1975] Robert M Gray, David L Neuhoff, and Paul C Shields. A generalization of Ornstein's d distance with applications to information theory. The Annals of Probability, pages 315-328, 1975.","title":"A generalization of Ornstein's d distance with applications to information theory","authors":"David L Robert M Gray, Paul C Neuhoff, Shields","year":"1975","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gretton et al., 2012] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773, 2012.","title":"A kernel two-sample test","authors":"Arthur Gretton, Karsten M Borgwardt, J Malte, Bernhard Rasch, Alexander Schölkopf, Smola","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems 30, pages 6626-6637.","title":"GANs trained by a two time-scale update rule converge to a local Nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, Sepp Hochreiter","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"Curran Associates, Inc., 2017.","title":null,"authors":"Curran Associates, Inc","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceed- ings of the 32nd International Conference on Machine Learning, volume 37, pages 448-456. PMLR, 2015.","title":"Batch normalization: Accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proceedings of the 32nd International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Interna- tional Conference on Learning Representations, 12 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maaten and Hinton, 2008] Laurens van der Maaten and Ge- offrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605, 2008.","title":"Laurens van der Maaten and Geoffrey Hinton","authors":"Hinton Maaten","year":"2008","venue":"Journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Makhzani et al., 2016] Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, and Ian Goodfellow. Adversarial autoen- coders. In International Conference on Learning Repre- sentations, 2016.","title":"Adversarial autoencoders","authors":"Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow","year":"2016","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Odena et al., 2017] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier GANs. In Proceedings of the 34th Inter- national Conference on Machine Learning -Volume 70, pages 2642-2651. JMLR.org, 2017.","title":"Conditional image synthesis with auxil-iary classifier GANs","authors":"Odena","year":"2017","venue":"Proceedings of the 34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ornstein, 1973] Donald S Ornstein. An application of er- godic theory to probability theory. The Annals of Proba- bility, 1(1):43-58, 1973.","title":"An application of ergodic theory to probability theory","authors":"Donald S Ornstein","year":"1973","venue":"The Annals of Probability","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Parkhi et al., 2015] Omkar M Parkhi, Andrea Vedaldi, An- drew Zisserman, et al. Deep face recognition. In British Machine Vision Conference, volume 1, page 6, 2015.","title":"Deep face recognition","authors":"Andrea Omkar M Parkhi, Andrew Vedaldi, Zisserman","year":"2015","venue":"British Machine Vision Conference","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rubenstein et al., 2018] Paul K. Rubenstein, Bernhard Schölkopf, and Ilya Tolstikhin. Wasserstein auto- encoders: Latent dimensionality and random encoders. In Workshop at the 6th International Conference on Learning Representations (ICLR), May 2018.","title":"Wasserstein autoencoders: Latent dimensionality and random encoders","authors":"Paul K Rubenstein, Bernhard Schölkopf, Ilya Tolstikhin","year":"2018","venue":"Workshop at the 6th International Conference on Learning Representations (ICLR)","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tolstikhin et al., 2018] Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. In International Conference on Learning Representations, 2018.","title":"Wasserstein auto-encoders","authors":"Tolstikhin","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Villani, 2008] Cédric Villani. Optimal Transport: Old and new, volume 338. Springer Science & Business Media, 2008.","title":"Optimal Transport: Old and new","authors":"Cédric Villani","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2004] Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. Image quality assess- ment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.","title":"Image quality assessment: from error visibility to structural similarity","authors":"Zhou Wang, Alan C Bovik, Eero P Hamid R Sheikh, Simoncelli","year":"2004","venue":"IEEE Transactions on Image Processing","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}