{"type":"ScholarlyArticle","title":"Stabilizing Adversarial Invariance Induction from Divergence Minimization Perspective","authors":[{"name":"Yusuke Iwasawa","slug":"yusuke-iwasawa","given":"Yusuke","family":"Iwasawa","affiliation":"The University of Tokyo , Japan"},{"name":"Kei Akuzawa","slug":"kei-akuzawa","given":"Kei","family":"Akuzawa","affiliation":"The University of Tokyo , Japan"},{"name":"Yutaka Matsuo","slug":"yutaka-matsuo","given":"Yutaka","family":"Matsuo","affiliation":"The University of Tokyo , Japan"}],"abstract":"Adversarial invariance induction (AII) is a generic and powerful framework for enforcing an invariance to nuisance attributes into neural network representations. However, its optimization is often unstable and little is known about its practical behavior. This paper presents an analysis of the reasons for the optimization difficulties and provides a better optimization procedure by rethinking AII from a divergence minimization perspective. Interestingly, this perspective indicates a cause of the optimization difficulties: it does not ensure proper divergence minimization, which is a requirement of the invariant representations. We then propose a simple variant of AII, called invariance induction by discriminator matching, which takes into account the divergence minimization interpretation of the invariant representations. Our method consistently achieves near-optimal invariance in toy datasets with various configurations in which the original AII is catastrophically unstable. Extentive experiments on four real-world datasets also support the superior performance of the proposed method, leading to improved user anonymization and domain generalization.","doi":"10.24963/ijcai.2020/271","url":"https://ijcai.org/proceedings/2020/271","pdf":"https://ijcai.org/proceedings/2020/0271.pdf","markdown":"https://ijcai.org/proceedings/2020/271.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/271","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Machine Learning","track":"Main track","number":271,"pages":{"first":1955,"last":1962},"keywords":["Machine Learning: Transfer, Adaptation, Multi-task Learning","Machine Learning: Deep Learning"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25167/file_25167.m3u8","length_sec":297,"provider":"hls"}],"references":[{"position":1,"raw":"[Ajakan et al., 2014] Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, and Mario Marchand. Domain-adversarial neural networks. In NIPS Workshop on Transfer and Multi-Task Learning: Theory meets Prac- tice, 2014.","title":"Domain-adversarial neural networks","authors":"Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Marchand","year":"2014","venue":"NIPS Workshop on Transfer and Multi-Task Learning: Theory meets Practice","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Barber et al., 2018] David Barber, Mingtian Zhang, Raza Habib, and Thomas Bird. Spread divergences. arXiv preprint arXiv:1811.08968, 2018.","title":"Spread divergences","authors":"David Barber, Mingtian Zhang, Raza Habib, Thomas Bird","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Behrmann et al., 2018] Jens Behrmann, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. arXiv preprint arXiv:1811.00995, 2018.","title":"Invertible residual networks","authors":"Behrmann","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Edwards and Storkey, 2016] Harrison Edwards and Amos Storkey. Censoring representations with an adversary. In ICLR, 2016.","title":"Censoring representations with an adversary","authors":"Storkey Edwards, Harrison Edwards, Amos Storkey","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Gan et al., 2016] Chuang Gan, Tianbao Yang, and Boqing Gong. Learning attributes equals multi-source domain generalization. In CVPR, 2016.","title":"Learning attributes equals multi-source domain generalization","authors":"Chuang Gan, Tianbao Yang, Boqing Gong","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Ganin et al., 2016] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMRL, 17(1):2096-2030, 2016.","title":"Domain-adversarial training of neural networks","authors":"Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Mario Franc ¸ois Laviolette, Victor Marchand, Lempitsky","year":"2016","venue":"JMRL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Gerchinovitz et al., 2017] Sebastien Gerchinovitz, Pierre Ménard, and Gilles Stoltz. Fano's inequality for random variables. arXiv preprint arXiv:1702.05985, 2017.","title":"Fano's inequality for random variables","authors":"Sebastien Gerchinovitz, Pierre Ménard, Gilles Stoltz","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ghifary et al., 2015] Muhammad Ghifary, W Basti- aan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In ICCV, pages 2551-2559, 2015.","title":"Domain generalization for object recognition with multi-task autoencoders","authors":"Ghifary","year":"2015","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial nets. In NIPS, pages 2672-2680, 2014.","title":"Generative adversarial nets","authors":"Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio","year":"2014","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Iwasawa et al., 2017] Yusuke Iwasawa, Kotaro Nakayama, Ikuko Eguchi Yairi, and Yutaka Matsuo. Privacy issues regarding the application of dnns to activity-recognition using wearables and its countermeasures by use of adver- sarial training. In IJCAI, pages 1930-1936, 2017.","title":"Privacy issues regarding the application of dnns to activity-recognition using wearables and its countermeasures by use of adversarial training","authors":"Yusuke Iwasawa, Kotaro Nakayama, Ikuko Eguchi Yairi, Yutaka Matsuo","year":"2017","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2017/268"},{"position":11,"raw":"[Jaiswal et al., 2019] Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan. Invariant representations through adversarial forgetting. In AAAI, 2019.","title":"Invariant representations through adversarial forgetting","authors":"Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael Abdalmageed, Premkumar Natarajan","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Krizhevsky et al., 2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, pages 1097- 1105, 2012.","title":"Imagenet classification with deep convolutional neural networks","authors":"Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton","year":"2012","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2017] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier do- main generalization. In ICCV, pages 5543-5551. IEEE, 2017.","title":"Deeper, broader and artier domain generalization","authors":"Da Li, Yongxin Yang, Yi-Zhe Song, Timothy M Hospedales","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2018a] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- learning for domain generalization. In AAAI, 2018.","title":"Learning to generalize: Metalearning for domain generalization","authors":"Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Li et al., 2018b] Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In AAAI, 2018.","title":"Domain generalization via conditional invariant representations","authors":"Li","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Li et al., 2018c] Ya Li, Xinmei Tian, Mingming Gong, Ya- jing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant ad- versarial networks. In ECCV, pages 624-639, 2018.","title":"Deep domain generalization via conditional invariant adversarial networks","authors":"Li","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Louizos et al., 2016] Christos Louizos, Kevin Swersky, Yu- jia Li, Max Welling, and Richard Zemel. The variational fair auto encoder. In ICLR, 2016.","title":"The variational fair auto encoder","authors":"Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, Richard Zemel","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mescheder et al., 2018] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, pages 3478-3487, 2018.","title":"Which training methods for gans do actually converge?","authors":"Mescheder","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Motiian et al., 2017] Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-Shot Adversar- ial Domain Adaptation. In NIPS, pages 6673-6683.","title":"Few-Shot Adversarial Domain Adaptation","authors":"Saeid Motiian, Quinn Jones, Seyed Iranmanesh, Gianfranco Doretto","year":null,"venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Moyer et al., 2018] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. Invari- ant representations without adversarial training. In NIPS, 2018.","title":"Invariant representations without adversarial training","authors":"Daniel Moyer, Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, Aram Galstyan","year":"2018","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Muandet et al., 2013] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via in- variant feature representation. In ICML, pages 10-18, 2013.","title":"Domain generalization via invariant feature representation","authors":"Muandet","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sagha et al., 2011] Hesam Sagha, Sundara Tejaswi Digu- marti, José del R Millán, Ricardo Chavarriaga, Alberto Ca- latroni, Daniel Roggen, and Gerhard Tröster. Benchmark- ing classification techniques using the opportunity human activity dataset. In INSS, 2011.","title":"Benchmarking classification techniques using the opportunity human activity dataset","authors":"Hesam Sagha, Tejaswi Sundara, José Digumarti, Ricardo Del R Millán, Alberto Chavarriaga, Daniel Calatroni, Gerhard Roggen, Tröster","year":"2011","venue":"INSS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shankar et al., 2018] Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross- gradient training. In ICLR, 2018.","title":"Generalizing across domains via crossgradient training","authors":"Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, Sunita Sarawagi","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wang et al., 2018] Ye Wang, Toshiaki Koike-Akino, and Deniz Erdogmus. Invariant representations from ad- versarially censored autoencoders. arXiv preprint arXiv:1805.08097, 2018.","title":"Invariant representations from adversarially censored autoencoders","authors":"Ye Wang, Toshiaki Koike-Akino, Deniz Erdogmus","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xie et al., 2017] Qizhe Xie, Zihang Dai, Yulun Du, Ed- uard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In NIPS, 2017.","title":"Controllable invariance through adversarial feature learning","authors":"Xie","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Yang et al., 2015] Jian Bo Yang, Minh Nhut Nguyen, Phyo Phyo San, Xiao Li Li, and Shonali Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In IJCAI, pages 3995-4001, 2015.","title":"Deep convolutional neural networks on multichannel time series for human activity recognition","authors":"Bo Jian, Minh Nhut Yang, Phyo Nguyen, Xiao Li Phyo San, Shonali Li, Krishnaswamy","year":"2015","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2015/561"},{"position":27,"raw":"[Zemel et al., 2013] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair represen- tations. In ICML, pages 325-333, 2013.","title":"Learning fair representations","authors":"Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, Cynthia Dwork","year":"2013","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhang and Sawchuk, 2012] Mi Zhang and Alexander A Sawchuk. USC-HAD: a daily activity dataset for ubiq- uitous activity recognition using wearable sensors. In Ubi- Comp, pages 1036-1043. ACM, 2012.","title":"USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors","authors":"Mi Zhang, Alexander A Sawchuk","year":"2012","venue":"Ubi-Comp","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}