{"type":"ScholarlyArticle","title":"Unsupervised Scene Adaptation with Memory Regularization in vivo","authors":[{"name":"Zhedong Zheng","slug":"zhedong-zheng","given":"Zhedong","family":"Zheng","affiliation":"ReLER , University of Technology Sydney , Australia; Baidu Research , China"},{"name":"Yi Yang","slug":"yi-yang","given":"Yi","family":"Yang","affiliation":"ReLER , University of Technology Sydney , Australia"}],"abstract":"This work focuses on the unsupervised scene adaptation problem of learning from both labeled source data and unlabeled target data. Existing approaches focus on minoring the inter-domain gap between the source and target domains. However, the intra-domain knowledge and inherent uncertainty learned by the network are under-explored. In this paper, we propose an orthogonal method, called memory regularization in vivo, to exploit the intra-domain knowledge and regularize the model training. Specifically, we refer to the segmentation model itself as the memory module, and minor the discrepancy of the two classifiers, i.e., the primary classifier and the auxiliary classifier, to reduce the prediction inconsistency. Without extra parameters, the proposed method is complementary to most existing domain adaptation methods and could generally improve the performance of existing methods. Albeit simple, we verify the effectiveness of memory regularization on two synthetic-to-real benchmarks: GTA5 → Cityscapes and SYNTHIA → Cityscapes, yielding +11.1% and +11.3% mIoU improvement over the baseline model, respectively. Besides, a similar +12.0% mIoU improvement is observed on the cross-city benchmark: Cityscapes → Oxford RobotCar.","doi":"10.24963/ijcai.2020/150","url":"https://ijcai.org/proceedings/2020/150","pdf":"https://ijcai.org/proceedings/2020/0150.pdf","markdown":"https://ijcai.org/proceedings/2020/150.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/150","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Computer Vision","track":"Main track","number":150,"pages":{"first":1076,"last":1082},"keywords":["Computer Vision: Recognition: Detection, Categorization, Indexing, Matching, Retrieval, Semantic Interpretation","Computer Vision: Big Data and Large Scale Methods"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/24315/file_24315.m3u8","length_sec":298,"provider":"hls"}],"references":[{"position":1,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convo- lutional nets, atrous convolution, and fully connected crfs. TPAMI, 40(4):834-848, 2017.","title":"Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs","authors":"Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, Alan L Yuille","year":"2017","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Yanbei Chen, Xiatian Zhu, and Shaogang Gong. Semi-supervised deep learning with memory. In ECCV, 2018.","title":"Semi-supervised deep learning with memory","authors":"Chen","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Cordts et al., 2016] Marius Cordts, Mohamed Omran, Se- bastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene under- standing. In CVPR, 2016.","title":"The cityscapes dataset for semantic urban scene understanding","authors":"Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, Bernt Schiele","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[French et al., 2017] Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adap- tation. arXiv:1706.05208, 2017.","title":"Self-ensembling for visual domain adaptation","authors":"Geoffrey French, Michal Mackiewicz, Mark Fisher","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey Hinton, Oriol Vinyals, Jeff Dean","year":"2015","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hoffman et al., 2018] Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, and Trevor Darrell. Cycada: Cycle-consistent ad- versarial domain adaptation. In ICML, 2018.","title":"Cycada: Cycle-consistent adversarial domain adaptation","authors":"Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A Efros, Trevor Darrell","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Isola et al., 2017] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with con- ditional adversarial networks. In CVPR, 2017.","title":"Image-to-image translation with conditional adversarial networks","authors":"Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A Efros","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Laine and Aila, 2016] Samuli Laine and Timo Aila. Tem- poral ensembling for semi-supervised learning. In ICLR, 2016.","title":"Temporal ensembling for semi-supervised learning","authors":"Laine, Aila ;, Samuli Laine, Timo Aila","year":"2016","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Luo et al., 2019a] Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang. Significance-aware information bot- tleneck for domain adaptive semantic segmentation. In ICCV, 2019.","title":"Significance-aware information bottleneck for domain adaptive semantic segmentation","authors":"Luo","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Luo et al., 2019b] Yawei Luo, Liang Zheng, Tao Guan, Jun- qing Yu, and Yi Yang. Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR, 2019.","title":"Taking a closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation","authors":"Luo","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Maddern et al., 2017] Will Maddern, Geoff Pascoe, Chris Linegar, and Paul Newman. 1 Year, 1000km: The Oxford RobotCar Dataset. IJRR, 36(1):3-15, 2017.","title":"1 Year, 1000km: The Oxford RobotCar Dataset","authors":"Maddern","year":"2017","venue":"IJRR","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Richter et al., 2016] Stephan R Richter, Vibhav Vineet, Ste- fan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.","title":"Playing for data: Ground truth from computer games","authors":"Vibhav Stephan R Richter, Stefan Vineet, Vladlen Roth, Koltun","year":"2016","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ros et al., 2016] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.","title":"The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes","authors":"Ros","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Saito et al., 2018] Kuniaki Saito, Kohei Watanabe, Yoshi- taka Ushiku, and Tatsuya Harada. Maximum classifier dis- crepancy for unsupervised domain adaptation. In CVPR, 2018.","title":"Maximum classifier discrepancy for unsupervised domain adaptation","authors":"Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, Tatsuya Harada","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Tarvainen and Valpola, 2017] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight- averaged consistency targets improve semi-supervised deep learning results. In NeurlPS, 2017.","title":"Mean teachers are better role models: Weightaveraged consistency targets improve semi-supervised deep learning results","authors":"Valpola Tarvainen, Antti Tarvainen, Harri Valpola","year":"2017","venue":"NeurlPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Tsai et al., 2018] Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, and Manmo- han Chandraker. Learning to adapt structured output space for semantic segmentation. In CVPR, 2018.","title":"Learning to adapt structured output space for semantic segmentation","authors":"Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, Manmohan Chandraker","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tsai et al., 2019] Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker. Domain adaptation for structured output via discriminative representations. In ICCV, 2019.","title":"Domain adaptation for structured output via discriminative representations","authors":"Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, Manmohan Chandraker","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vu et al., 2019] Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Ad- versarial entropy minimization for domain adaptation in semantic segmentation. In CVPR, 2019.","title":"Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation","authors":"Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, Patrick Pérez","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2019] S Wang, L Zhang, W Zuo, and B Zhang. Class-specific reconstruction transfer learning for visual recognition across domains. TIP, 29(1):2424-2438, 2019.","title":"Class-specific reconstruction transfer learning for visual recognition across domains","authors":"L Wang, W Zhang, Zuo, Zhang","year":"2019","venue":"TIP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Weston et al., 2014] Jason Weston, Sumit Chopra, and An- toine Bordes. Memory networks. arXiv:1410.3916, 2014.","title":"Memory networks","authors":"Jason Weston, Sumit Chopra","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wu et al., 2018] Zuxuan Wu, Xintong Han, Yen-Liang Lin, Mustafa Gokhan Uzunbas, Tom Goldstein, Ser Nam Lim, and Larry S Davis. Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation. In ECCV, 2018.","title":"Dcan: Dual channel-wise alignment networks for unsupervised scene adaptation","authors":"Wu","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wu et al., 2019] Zuxuan Wu, Xin Wang, Joseph E Gonza- lez, Tom Goldstein, and Larry S Davis. Ace: Adapting to changing environments for semantic segmentation. In ICCV, 2019.","title":"Ace: Adapting to changing environments for semantic segmentation","authors":"Wu","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Yue et al., 2019] Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, and Bo- qing Gong. Domain randomization and pyramid consis- tency: Simulation-to-real generalization without accessing target domain data. In ICCV, 2019.","title":"Domain randomization and pyramid consistency: Simulation-to-real generalization without accessing target domain data","authors":"Xiangyu Yue, Yang Zhang, Sicheng Zhao, Alberto Sangiovanni-Vincentelli, Kurt Keutzer, Boqing Gong","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhang et al., 2018a] Yiheng Zhang, Zhaofan Qiu, Ting Yao, Dong Liu, and Tao Mei. Fully convolutional adaptation networks for semantic segmentation. In CVPR, 2018.","title":"Fully convolutional adaptation networks for semantic segmentation","authors":"Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2018b] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, 2018.","title":"Deep mutual learning","authors":"Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019] Li Zhang, Xiangtai Li, Anurag Arnab, Kuiyuan Yang, Yunhai Tong, and Philip HS Torr. Dual graph convolutional network for semantic segmentation. In BMVC, 2019.","title":"Dual graph convolutional network for semantic segmentation","authors":"Li Zhang, Xiangtai Li, Anurag Arnab, Kuiyuan Yang, Yunhai Tong, Philip Hs Torr","year":"2019","venue":"BMVC","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Zhao et al., 2017] Hengshuang Zhao, Jianping Shi, Xiao- juan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.","title":"Pyramid scene parsing network","authors":"Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, Jiaya Jia","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zou et al., 2018] Yang Zou, Zhiding Yu, BVK Vijaya Ku- mar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In ECCV, 2018.","title":"Unsupervised domain adaptation for semantic segmentation via class-balanced self-training","authors":"Yang Zou, Zhiding Yu, Bvk Vijaya Kumar, Jinsong Wang","year":"2018","venue":"ECCV","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zou et al., 2019] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In ICCV, 2019.","title":"Confidence regularized self-training","authors":"Yang Zou, Zhiding Yu, Xiaofeng Liu, Jinsong Kumar, Wang","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}