{"type":"ScholarlyArticle","title":"Continual Semantic Segmentation Leveraging Image-level Labels and Rehearsal","authors":[{"name":"Mathieu Pagé Fortin","slug":"mathieu-page-fortin","given":"Mathieu","family":"Pagé Fortin","affiliation":"Laval University , Québec , Canada"},{"name":"Brahim Chaib-draa","slug":"brahim-chaib-draa","given":"Brahim","family":"Chaib-draa","affiliation":"Laval University , Québec , Canada"}],"abstract":"Despite the remarkable progress of deep learning models for semantic segmentation, the success of these models is strongly limited by the following aspects: 1) large datasets with pixel-level annotations must be available and 2) training must be performed with all classes simultaneously. Indeed, in incremental learning scenarios, where new classes are added to an existing framework, these models are prone to catastrophic forgetting of previous classes. To address these two limitations, we propose a weakly-supervised mechanism for continual semantic segmentation that can leverage cheap image-level annotations and a novel rehearsal strategy that intertwines the learning of past and new classes. Specifically, we explore two rehearsal technique variants: 1) imprinting past objects on new images and 2) transferring past representations in intermediate features maps. We conduct extensive experiments on Pascal-VOC by varying the proportion of fully- and weakly-supervised data in various setups and show that our contributions consistently improve the mIoU on both past and novel classes. Interestingly, we also observe that models trained with less data in incremental steps sometimes outperform the same architectures trained with more data. We discuss the significance of these results and propose some hypotheses regarding the dynamics between forgetting and learning.","doi":"10.24963/ijcai.2022/177","url":"https://ijcai.org/proceedings/2022/177","pdf":"https://ijcai.org/proceedings/2022/0177.pdf","markdown":"https://ijcai.org/proceedings/2022/177.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/177","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":177,"pages":{"first":1268,"last":1275},"keywords":["Machine Learning: Weakly Supervised Learning","Machine Learning: Incremental Learning","Computer Vision: Segmentation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38987125","length_sec":95,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38985268","length_sec":676,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Cermelli et al., 2020] Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, and Barbara Ca- puto. Modeling the background for incremental learning in semantic segmentation. In CVPR, pages 9233-9242, 2020.","title":"Modeling the background for incremental learning in semantic segmentation","authors":"Fabio Cermelli, Massimiliano Mancini, Samuel Rota Bulo, Elisa Ricci, Barbara Caputo","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834-848, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for se- mantic image segmentation. In ECCV, pages 801-818, 2018.","title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","authors":"Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, Hartwig Adam","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Delange et al., 2021] Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning sur- vey: Defying forgetting in classification tasks. TPAMI, 2021.","title":"A continual learning survey: Defying forgetting in classification tasks","authors":"Matthias Delange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, Tinne Tuytelaars","year":"2021","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Douillard et al., 2021] Arthur Douillard, Yifu Chen, Ar- naud Dapogny, and Matthieu Cord. Plop: Learning with- out forgetting for continual semantic segmentation. In CVPR, pages 4040-4050, 2021.","title":"Plop: Learning without forgetting for continual semantic segmentation","authors":"Arthur Douillard, Yifu Chen, Arnaud Dapogny, Matthieu Cord","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Everingham et al., 2015] Mark Everingham, SM Ali Es- lami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1):98-136, 2015.","title":"The pascal visual object classes challenge: A retrospective","authors":"Mark Everingham, Ali Sm, Luc Eslami, Van Gool, K I Christopher, John Williams, Andrew Winn, Zisserman","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ghiasi et al., 2021] Golnaz Ghiasi, Yin Cui, Aravind Srini- vas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is a strong data aug- mentation method for instance segmentation. In CVPR, pages 2918-2928, 2021.","title":"Simple copy-paste is a strong data augmentation method for instance segmentation","authors":"Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D Cubuk, Quoc V Le, Barret Zoph","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Hayes et al., 2020] Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Re- mind your neural network to prevent catastrophic forget- ting. In ECCV, pages 466-483. Springer, 2020.","title":"Remind your neural network to prevent catastrophic forgetting","authors":"Kushal Tyler L Hayes, Robik Kafle, Manoj Shrestha, Christopher Acharya, Kanan","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kirkpatrick et al., 2017] James Kirkpatrick, Razvan Pas- canu, Neil Rabinowitz, Joel Veness, Guillaume Des- jardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcom- ing catastrophic forgetting in neural networks. Proc. the national academy of sciences, 114(13):3521-3526, 2017.","title":"Overcoming catastrophic forgetting in neural networks","authors":"James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska","year":"2017","venue":"Proc. the national academy of sciences","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Li and Hoiem, 2017] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):2935-2947, 2017.","title":"Learning without forgetting","authors":"Zhizhong Li, Derek Hoiem","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Long et al., 2015] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for seman- tic segmentation. In CVPR, pages 3431-3440, 2015.","title":"Fully convolutional networks for semantic segmentation","authors":"Jonathan Long, Evan Shelhamer, Trevor Darrell","year":"2015","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Maracani et al., 2021] Andrea Maracani, Umberto Michieli, Marco Toldo, and Pietro Zanuttigh. Re- call: Replay-based continual learning in semantic segmentation. In ICCV, pages 7026-7035, 2021.","title":"Recall: Replay-based continual learning in semantic segmentation","authors":"Andrea Maracani, Umberto Michieli, Marco Toldo, Pietro Zanuttigh","year":"2021","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Michieli and Zanuttigh, 2019] Umberto Michieli and Pietro Zanuttigh. Incremental learning techniques for semantic segmentation. In ICCVW, pages 3205-3212, 2019.","title":"Incremental learning techniques for semantic segmentation","authors":"Zanuttigh Michieli, Umberto Michieli, Pietro Zanuttigh","year":"2019","venue":"ICCVW","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Michieli and Zanuttigh, 2021] Umberto Michieli and Pietro Zanuttigh. Continual semantic segmentation via repulsion- attraction of sparse and disentangled latent representa- tions. In CVPR, pages 1114-1124, 2021.","title":"Continual semantic segmentation via repulsionattraction of sparse and disentangled latent representations","authors":"Zanuttigh Michieli, Umberto Michieli, Pietro Zanuttigh","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Prabhu et al., 2020] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that ques- tions our progress in continual learning. In ECCV, pages 524-540. Springer, 2020.","title":"Gdumb: A simple approach that questions our progress in continual learning","authors":"Ameya Prabhu, Puneet K Philip Hs Torr, Dokania","year":"2020","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Rebuffi et al., 2017] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001-2010, 2017.","title":"icarl: Incremental classifier and representation learning","authors":"Sylvestre-Alvise, Alexander Rebuffi, Georg Kolesnikov, Christoph H Sperl, Lampert","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Russakovsky et al., 2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-252, 2015.","title":"Imagenet large scale visual recognition challenge","authors":"Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein","year":"2015","venue":"IJCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Selvaraju et al., 2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618-626, 2017.","title":"Grad-cam: Visual explanations from deep networks via gradient-based localization","authors":"Michael Ramprasaath R Selvaraju, Abhishek Cogswell, Ramakrishna Das, Devi Vedantam, Dhruv Parikh, Batra","year":"2017","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Serra et al., 2018] Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In ICML, pages 4548-4557. PMLR, 2018.","title":"Overcoming catastrophic forgetting with hard attention to the task","authors":"Joan Serra, Didac Suris, Marius Miron, Alexandros Karatzoglou","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Walker and Stickgold, 2004] Matthew P Walker and Robert Stickgold. Sleep-dependent learning and memory consol- idation. Neuron, 44(1):121-133, 2004.","title":"Sleep-dependent learning and memory consolidation","authors":"P Matthew, Robert Walker, Stickgold","year":"2004","venue":"Neuron","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In CVPR, pages 374- 382, 2019.","title":"Large scale incremental learning","authors":"Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, Yun Fu","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yu et al., 2020] Lu Yu, Xialei Liu, and Joost van de Weijer. Self-training for class-incremental semantic segmentation. arXiv preprint arXiv:2012.03362, 2020.","title":"Self-training for class-incremental semantic segmentation","authors":"Lu Yu, Xialei Liu, Joost Van De Weijer","year":"2020","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zenke et al., 2017] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic in- telligence. In ICML, pages 3987-3995. PMLR, 2017.","title":"Continual learning through synaptic intelligence","authors":"Friedemann Zenke, Ben Poole, Surya Ganguli","year":"2017","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhou et al., 2016] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pages 2921-2929, 2016.","title":"Learning deep features for discriminative localization","authors":"Zhou","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zou et al., 2021] Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for seman- tic segmentation. In ICLR, 2021.","title":"Pseudoseg: Designing pseudo labels for semantic segmentation","authors":"Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, Tomas Pfister","year":"2021","venue":"ICLR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}