{"type":"ScholarlyArticle","title":"Multiband VAE: Latent Space Alignment for Knowledge Consolidation in Continual Learning","authors":[{"name":"Kamil Deja","slug":"kamil-deja","given":"Kamil","family":"Deja","affiliation":"Warsaw University of Technology ,"},{"name":"Paweł Wawrzyński","slug":"pawe-wawrzynski","given":"Paweł","family":"Wawrzyński","affiliation":"Warsaw University of Technology ,"},{"name":"Wojciech Masarczyk","slug":"wojciech-masarczyk","given":"Wojciech","family":"Masarczyk","affiliation":"Warsaw University of Technology ,"},{"name":"Daniel Marczak","slug":"daniel-marczak","given":"Daniel","family":"Marczak","affiliation":"Warsaw University of Technology ,"},{"name":"Tomasz Trzciński","slug":"tomasz-trzcinski","given":"Tomasz","family":"Trzcinski","affiliation":"Warsaw University of Technology ,; Jagiellonian University ,; Tooploox"}],"abstract":"We propose a new method for unsupervised generative continual learning through realignment of Variational Autoencoder's latent space. Deep generative models suffer from catastrophic forgetting in the same way as other neural structures. Recent generative continual learning works approach this problem and try to learn from new data without forgetting previous knowledge.  However, those methods usually focus on artificial scenarios where examples share almost no similarity between subsequent portions of data - an assumption not realistic in the real-life applications of continual learning. In this work, we identify this limitation and posit the goal of generative continual learning as a knowledge accumulation task. We solve it by continuously aligning latent representations of new data that we call bands in additional latent space where examples are encoded independently of their source task. In addition, we introduce a method for controlled forgetting of past data that simplifies this process. On top of the standard continual learning benchmarks, we propose a novel challenging knowledge consolidation scenario and show that the proposed approach outperforms state-of-the-art by up to twofold across all experiments and additional real-life evaluation. To our knowledge, Multiband VAE is the first method to show forward and backward knowledge transfer in generative continual learning.","doi":"10.24963/ijcai.2022/402","url":"https://ijcai.org/proceedings/2022/402","pdf":"https://ijcai.org/proceedings/2022/0402.pdf","markdown":"https://ijcai.org/proceedings/2022/402.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/402","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":402,"pages":{"first":2902,"last":2908},"keywords":["Computer Vision: Neural generative models, auto encoders, GANs","Machine Learning: Incremental Learning"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986507","length_sec":83,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984650","length_sec":363,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Achille et al., 2018] Alessandro Achille, Tom Eccles, et al. Life-long disentangled representation learning with cross- domain latent homologies. Advances in Neural Informa- tion Processing Systems, 31, 2018.","title":"Life-long disentangled representation learning with crossdomain latent homologies","authors":"Alessandro Achille, Tom Eccles","year":"2018","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bińkowski et al., 2018] Mikołaj Bińkowski, Danica J Sutherland, et al. Demystifying mmd gans. In ICLR, 2018.","title":"Demystifying mmd gans","authors":"Mikołaj Bińkowski, Danica J Sutherland","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Deja et al., 2020] Kamil Deja, Jan Dubiński, et al. End-to- end sinkhorn autoencoder with noise generator. IEEE Ac- cess, 9:7211-7219, 2020.","title":"End-toend sinkhorn autoencoder with noise generator","authors":"Kamil Deja, Jan Dubiński","year":"2020","venue":"IEEE Access","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Egorov et al., 2021] Evgenii Egorov, Anna Kuzina, et al. Boovae: Boosting approach for continual learning of vae. Advances in Neural Information Processing Systems, 34, 2021.","title":"Boovae: Boosting approach for continual learning of vae","authors":"Evgenii Egorov, Anna Kuzina","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Goodfellow et al., 2014] Ian Goodfellow, Jean Pouget- Abadie, et al. Generative Adversarial Networks. In NeurIPS, 2014.","title":"Generative Adversarial Networks","authors":"Ian Goodfellow, Jean Pouget-Abadie","year":"2014","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Heusel et al., 2017] Martin Heusel, Hubert Ramsauer, et al. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in neural information processing systems, pages 6626-6637, 2017.","title":"Gans trained by a two time-scale update rule converge to a local nash equilibrium","authors":"Martin Heusel, Hubert Ramsauer","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hsu et al., 2019] Tzu-Ming Harry Hsu, Hang Qi, et al. Measuring the effects of non-identical data distribu- tion for federated visual classification. arXiv preprint arXiv:1909.06335, 2019.","title":"Measuring the effects of non-identical data distribution for federated visual classification","authors":"Hsu","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Jang et al., 2016] Eric Jang, Shixiang Gu, et al. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016.","title":"Categorical reparameterization with gumbel-softmax","authors":"Eric Jang, Shixiang Gu","year":"2016","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kansal et al., 2021] Raghav Kansal, Javier Duarte, et al. Particle cloud generation with message passing genera- tive adversarial networks. Advances in Neural Information Processing Systems, 34, 2021.","title":"Particle cloud generation with message passing generative adversarial networks","authors":"Raghav Kansal, Javier Duarte","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ke et al., 2021] Zixuan Ke, Bing Liu, et al. Achieving for- getting prevention and knowledge transfer in continual learning. Advances in Neural Information Processing Sys- tems, 34, 2021.","title":"Achieving forgetting prevention and knowledge transfer in continual learning","authors":"Ke","year":"2021","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kingma and Welling, 2014] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In ICLR, 2014.","title":"Auto-Encoding Variational Bayes","authors":"P Diederik, Max Kingma, Welling","year":"2014","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, et al. Overcoming catastrophic forgetting in neu- ral networks. Proceedings of the national academy of sci- ences, 114(13):3521-3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu","year":"2017","venue":"Proceedings of the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Lake et al., 2015] Brenden M Lake, Ruslan Salakhutdinov, et al. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-1338, 2015.","title":"Human-level concept learning through probabilistic program induction","authors":"Ruslan Brenden M Lake, Salakhutdinov","year":"2015","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lesort et al., 2019] Timothée Lesort, Hugo Caselles-Dupré, et al. Generative models from the perspective of contin- ual learning. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2019.","title":"Generative models from the perspective of continual learning","authors":"Lesort","year":"2019","venue":"International Joint Conference on Neural Networks (IJCNN)","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2015] Ziwei Liu, Ping Luo, et al. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.","title":"Deep learning face attributes in the wild","authors":"Ziwei Liu, Ping Luo","year":"2015","venue":"Proceedings of International Conference on Computer Vision (ICCV)","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lopez-Paz and Ranzato, 2017] David Lopez-Paz and Marc'Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.","title":"Gradient episodic memory for continual learning","authors":"David Lopez, - Paz, Marc'aurelio Ranzato","year":"2017","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Nguyen et al., 2018] Cuong V Nguyen, Yingzhen Li, et al. Variational continual learning. In International Confer- ence on Learning Representations, 2018.","title":"Variational continual learning","authors":"Yingzhen Cuong V Nguyen, Li","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Paganini et al., 2018] Michela Paganini, Luke de Oliveira, et al. Calogan: Simulating 3d high energy particle show- ers in multilayer electromagnetic calorimeters. Physical Review D, 97(1):014021, 2018.","title":"Calogan: Simulating 3d high energy particle showers in multilayer electromagnetic calorimeters","authors":"Michela Paganini, Luke De Oliveira","year":"2018","venue":"Physical Review D","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ramapuram et al., 2020] Jason Ramapuram, Magda Gre- gorova, et al. Lifelong generative modeling. Neurocom- puting, 404:381-400, 2020.","title":"Jason Ramapuram, Magda Gregorova, et al. Lifelong generative modeling","authors":"Ramapuram","year":"2020","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rao et al., 2019] Dushyant Rao, Francesco Visin, et al. Continual unsupervised representation learning. Advances in Neural Information Processing Systems, 32, 2019.","title":"Continual unsupervised representation learning","authors":"Dushyant Rao, Francesco Visin","year":"2019","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rebuffi et al., 2017] S. Rebuffi, A. Kolesnikov, et al. iCaRL: Incremental Classifier and Representation Learn- ing. In CVPR, 2017.","title":"iCaRL: Incremental Classifier and Representation Learning","authors":"S Rebuffi, A Kolesnikov","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sajjadi et al., 2018] Mehdi SM Sajjadi, Olivier Bachem, et al. Assessing generative models via precision and re- call. arXiv preprint arXiv:1806.00035, 2018.","title":"Assessing generative models via precision and recall","authors":"S M Mehdi, Olivier Sajjadi, Bachem","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Shin et al., 2017] Hanul Shin, Jung Kwon Lee, et al. Con- tinual Learning with Deep Generative Replay. In NeurIPS, 2017.","title":"Continual Learning with Deep Generative Replay","authors":"Hanul Shin, Jung Kwon Lee","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Van de Ven and Tolias, 2019] Gido M Van de Ven and An- dreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734, 2019.","title":"Three scenarios for continual learning","authors":"M Gido, Andreas S Van De Ven, Tolias","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[von Oswald et al., 2019] Johannes von Oswald, Christian Henning, et al. Continual learning with hypernetworks. In International Conference on Learning Representations, 2019.","title":"Continual learning with hypernetworks","authors":"Oswald Von","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wu et al., 2018] Chenshen Wu, Luis Herranz, et al. Mem- ory replay gans: Learning to generate new categories with- out forgetting. In NeurIPS, 2018.","title":"Memory replay gans: Learning to generate new categories without forgetting","authors":"Wu","year":"2018","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xiao et al., 2017] Han Xiao, Kashif Rasul, et al. Fashion- MNIST: a Novel Image Dataset for Benchmarking Ma- chine Learning Algorithms, 2017. arXiv:1708.07747.","title":"Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms","authors":"Han Xiao, Kashif Rasul","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ye and Bors, 2020] Fei Ye and Adrian G Bors. Learning latent representations across multiple data domains using lifelong vaegan. In European Conference on Computer Vision, pages 777-795. Springer, 2020.","title":"Learning latent representations across multiple data domains using lifelong vaegan","authors":"Bors ; ] Fei Ye, Adrian G Ye, Bors","year":"2020","venue":"European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zenke et al., 2017] Friedemann Zenke, Ben Poole, et al. Continual learning through synaptic intelligence. In Inter- national Conference on Machine Learning, pages 3987- 3995. PMLR, 2017.","title":"Continual learning through synaptic intelligence","authors":"Friedemann Zenke, Ben Poole","year":"2017","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}