{"type":"ScholarlyArticle","title":"Robust Image Ordinal Regression with Controllable Image Generation","authors":[{"name":"Yi Cheng","slug":"yi-cheng","given":"Yi","family":"Cheng","affiliation":"School of Software Technology , Zhejiang Univerisity"},{"name":"Haochao Ying","slug":"haochao-ying","given":"Haochao","family":"Ying","affiliation":"School of Public Health , Zhejiang University"},{"name":"Renjun Hu","slug":"renjun-hu","given":"Renjun","family":"Hu","affiliation":"Alibaba Group"},{"name":"Jinhong Wang","slug":"jinhong-wang","given":"Jinhong","family":"Wang","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Wenhao Zheng","slug":"wenhao-zheng","given":"Wenhao","family":"Zheng","affiliation":"College of Computer Science and Technology , Zhejiang University"},{"name":"Xiao Zhang","slug":"xiao-zhang","given":"Xiao","family":"Zhang","affiliation":"School of Computer Science and Technology , Shandong University"},{"name":"Danny Chen","slug":"danny-chen","given":"Danny","family":"Chen","affiliation":"Department of Computer Science and Engineering , University of Notre Dame"},{"name":"Jian Wu","slug":"jian-wu","given":"Jian","family":"Wu","affiliation":"School of Public Health , Zhejiang University; Second Affiliated Hospital School of Medicine {chengy1 , haochaoying , wangjinhong , zhengwenhao ,"}],"abstract":"Image ordinal regression has been mainly studied along the line of exploiting the order of categories. However, the issues of class imbalance and category overlap that are very common in ordinal regression were largely overlooked. As a result, the performance on minority categories is often unsatisfactory. In this paper, we propose a novel framework called CIG based on controllable image generation to directly tackle these two issues. Our main idea is to generate extra training samples with specific labels near category boundaries, and the sample generation is biased toward the less-represented categories. To achieve controllable image generation, we seek to separate structural and categorical information of images based on structural similarity, categorical similarity, and reconstruction constraints. We evaluate the effectiveness of our new CIG approach in three different image ordinal regression scenarios. The results demonstrate that CIG can be flexibly integrated with off-the-shelf image encoders or ordinal regression models to achieve improvement, and further, the improvement is more significant for minority categories.","doi":"10.24963/ijcai.2023/70","url":"https://ijcai.org/proceedings/2023/70","pdf":"https://ijcai.org/proceedings/2023/0070.pdf","markdown":"https://ijcai.org/proceedings/2023/70.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/70","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Computer Vision","track":"Main Track","number":70,"pages":{"first":627,"last":635},"keywords":["Computer Vision: CV: Recognition (object detection, categorization)"],"videos":[],"references":[{"position":1,"raw":"[Beckham and Pal, 2017] Christopher Beckham and Christopher J. Pal. Unimodal probability distributions for deep ordinal classification. In Doina Precup and Yee Whye Teh, editors, 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 411-419. PMLR, 2017.","title":"Unimodal probability distributions for deep ordinal classification","authors":"Christopher Beckham, Christopher J Pal","year":"2017","venue":"34th International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bulten et al., 2022] Wouter Bulten, Kimmo Kartasalo, Po- Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Ku- nal Nagpal, Yuannan Cai, David F Steiner, Hester van Boven, Robert Vink, et al. Artificial intelligence for diag- nosis and Gleason grading of prostate cancer: The PANDA challenge. Nature Medicine, 28(1):154-163, 2022.","title":"Artificial intelligence for diagnosis and Gleason grading of prostate cancer: The PANDA challenge","authors":"Bulten","year":"2022","venue":"Nature Medicine","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2022] Tingting Chen, Yi Cheng, Jinhong Wang, Zhaoxia Yang, Wenhao Zheng, Danny Z. Chen, and Jian Wu. Automating blastocyst formation and quality predic- tion in time-lapse imaging with adaptive key frame selec- tion. In Linwei Wang, Qi Dou, P. Thomas Fletcher, Ste- fanie Speidel, and Shuo Li, editors, Medical Image Com- puting and Computer Assisted Intervention, Part IV, vol- ume 13434 of Lecture Notes in Computer Science, pages 445-455. Springer, 2022.","title":"Automating blastocyst formation and quality prediction in time-lapse imaging with adaptive key frame selection","authors":"Tingting Chen, Yi Cheng, Jinhong Wang, Zhaoxia Yang, Wenhao Zheng, Danny Z Chen, Jian Wu","year":"2022","venue":"Lecture Notes in Computer Science","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, Li- Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer So- ciety Conference on Computer Vision and Pattern Recog- nition, pages 248-255. IEEE Computer Society, 2009.","title":"ImageNet: A large-scale hierarchical image database","authors":"Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, Li Fei-Fei","year":"2009","venue":"IEEE Computer Society Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Diaz and Marathe, 2019] Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In IEEE Conference on Com- puter Vision and Pattern Recognition, pages 4738-4747. Computer Vision Foundation / IEEE, 2019.","title":"Raul Diaz and Amit Marathe. Soft labels for ordinal regression","authors":"Marathe Diaz","year":"2019","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Fu and Huang, 2008] Yun Fu and Thomas S. Huang. Hu- man age estimation with regression on discriminative ag- ing manifold. IEEE Trans. Multim., 10(4):578-584, 2008.","title":"Human age estimation with regression on discriminative aging manifold","authors":"Yun Fu, Thomas S Huang","year":"2008","venue":"IEEE Trans. Multim","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Fu et al., 2018] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 2002-2011. Computer Vision Founda- tion / IEEE Computer Society, 2018.","title":"Deep ordinal regression network for monocular depth estimation","authors":"Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, Dacheng Tao","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Geiger et al., 2013] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 32(11):1231-1237, 2013.","title":"Vision meets robotics: The KITTI dataset","authors":"Andreas Geiger, Philip Lenz, Christoph Stiller, Raquel Urtasun","year":"2013","venue":"Int. J. Robotics Res","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Guo and Mu, 2013] Guodong Guo and Guowang Mu. Joint estimation of age, gender and ethnicity: CCA vs. PLS. In 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pages 1-6. IEEE Computer Society, 2013.","title":"Joint estimation of age, gender and ethnicity: CCA vs","authors":"Mu Guo, Guodong Guo, Guowang Mu","year":"2013","venue":"PLS. In 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He et al., 2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15979-15988. IEEE, 2022.","title":"Masked autoencoders are scalable vision learners","authors":"He","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Levi and Hassner, 2015] Gil Levi and Tal Hassner. Age and gender classification using convolutional neural networks. In 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 34-42. IEEE Computer So- ciety, 2015.","title":"Age and gender classification using convolutional neural networks","authors":"Hassner Levi, Gil Levi, Tal Hassner","year":"2015","venue":"IEEE Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Li et al., 2021] Wanhua Li, Xiaoke Huang, Jiwen Lu, Jian- jiang Feng, and Jie Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In IEEE Conference on Computer Vision and Pattern Recognition, pages 13896-13905. Computer Vision Foundation / IEEE, 2021.","title":"Learning probabilistic ordinal embeddings for uncertainty-aware regression","authors":"Wanhua Li, Xiaoke Huang, Jiwen Lu, Jianjiang Feng, Jie Zhou","year":"2021","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2018a] Xiaofeng Liu, Yang Zou, Yuhang Song, Chao Yang, Jane You, and B. V. K. Vijaya Kumar. Ordinal regression with neuron stick-breaking for medical diagno- sis. In Laura Leal-Taixé and Stefan Roth, editors, ECCV Workshops, Proceedings, Part VI, volume 11134 of Lec- ture Notes in Computer Science, pages 335-344. Springer, 2018.","title":"Ordinal regression with neuron stick-breaking for medical diagnosis","authors":"Liu","year":"2018","venue":"ECCV Workshops, Proceedings, Part VI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2018b] Yanzhu Liu, Adams Wai-Kin Kong, and Chi Keong Goh. A constrained deep neural network for ordinal regression. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, pages 831-839. Computer Vision Foundation / IEEE Computer Society, 2018.","title":"A constrained deep neural network for ordinal regression","authors":"Liu","year":"2018","venue":"IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Yanzhu Liu, Fan Wang, and Adams Wai- Kin Kong. Probabilistic deep ordinal regression based on Gaussian processes. In 2019 IEEE/CVF International Conference on Computer Vision, pages 5300-5308. IEEE, 2019.","title":"Probabilistic deep ordinal regression based on Gaussian processes","authors":"Liu","year":"2019","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Liu et al., 2021] Lina Liu, Xibin Song, Mengmeng Wang, Yong Liu, and Liangjun Zhang. Self-supervised monocu- lar depth estimation for all day images using domain sep- aration. In 2021 IEEE/CVF International Conference on Computer Vision, pages 12717-12726. IEEE, 2021.","title":"Self-supervised monocular depth estimation for all day images using domain separation","authors":"Lina Liu, Xibin Song, Mengmeng Wang, Yong Liu, Liangjun Zhang","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lukyanenko et al., 2021] Stanislav Lukyanenko, Won- Dong Jang, Donglai Wei, Robbert Struyven, Yoon Kim, Brian D. Leahy, Helen Y. Yang, Alexander M. Rush, Dalit Ben-Yosef, Daniel Needleman, and Hanspeter Pfister. Developmental stage classification of embryos using two-stream neural network with linear-chain conditional random field. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert, editors, Medical Image Com- puting and Computer Assisted Intervention, Part VIII, volume 12908 of Lecture Notes in Computer Science, pages 363-372. Springer, 2021.","title":"Developmental stage classification of embryos using two-stream neural network with linear-chain conditional random field","authors":"Stanislav Lukyanenko, Won-Dong Jang, Donglai Wei, Robbert Struyven, Yoon Kim, Brian D Leahy, Helen Y Yang, Alexander M Rush, Dalit Ben-Yosef, Daniel Needleman, Hanspeter Pfister","year":"2021","venue":"Lecture Notes in Computer Science","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Niu et al., 2016] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple out- put CNN for age estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4920-4928, 2016.","title":"Ordinal regression with multiple output CNN for age estimation","authors":"Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, Gang Hua","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Paszke et al., 2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 8024-8035.","title":"PyTorch: An imperative style, highperformance deep learning library","authors":"Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"Curran Associates, Inc., 2019.","title":null,"authors":"Curran Associates, Inc","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Ratner et al., 2018] Vadim Ratner, Yoel Shoshan, and Tal Kachman. Learning multiple non-mutually-exclusive tasks for improved classification of inherently ordered la- bels. CoRR, abs/1805.11837, 2018.","title":"Learning multiple non-mutually-exclusive tasks for improved classification of inherently ordered labels","authors":"Yoel Vadim Ratner, Tal Shoshan, Kachman","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Ronneberger et al., 2015] Olaf Ronneberger, Philipp Fis- cher, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells III, and Alejan- dro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention, Part III, volume 9351 of Lecture Notes in Computer Science, pages 234-241. Springer, 2015.","title":"Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation","authors":"Ronneberger","year":"2015","venue":"Medical Image Computing and Computer-Assisted Intervention, Part III","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Schifanella et al., 2015] Rossano Schifanella, Miriam Redi, and Luca Maria Aiello. An image is worth more than a thousand favorites: Surfacing the hidden beauty of Flickr pictures. In Meeyoung Cha, Cecilia Mascolo, and Chris- tian Sandvig, editors, 9th International Conference on Web and Social Media, pages 397-406. AAAI Press, 2015.","title":"An image is worth more than a thousand favorites: Surfacing the hidden beauty of Flickr pictures","authors":"Rossano Schifanella, Miriam Redi, Luca Maria Aiello","year":"2015","venue":"9th International Conference on Web and Social Media","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Shin et al., 2022] Nyeong-Ho Shin, Seon-Ho Lee, and Chang-Su Kim. Moving window regression: A novel ap- proach to ordinal regression. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18739- 18748. IEEE, 2022.","title":"Moving window regression: A novel approach to ordinal regression","authors":"Shin","year":"2022","venue":"IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Simonyan and Zisserman, 2015] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, 2015.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2015","venue":"3rd International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Wang et al., 2021] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision Transformer: A ver- satile backbone for dense prediction without convolutions. In 2021 IEEE/CVF International Conference on Computer Vision, pages 548-558. IEEE, 2021.","title":"Pyramid vision Transformer: A versatile backbone for dense prediction without convolutions","authors":"Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao","year":"2021","venue":"IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE In- ternational Conference on Computer Vision, pages 2242- 2251. IEEE Computer Society, 2017.","title":"Unpaired image-to-image translation using cycle-consistent adversarial networks","authors":"Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A Efros","year":"2017","venue":"IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}