{"type":"ScholarlyArticle","title":"SGDCL: Semantic-Guided Dynamic Correlation Learning for Explainable Autonomous Driving","authors":[{"name":"Chengtai Cao","slug":"chengtai-cao","given":"Chengtai","family":"Cao","affiliation":"City University of Hong Kong; City University of Hong Kong Shenzhen Futian Research Institute"},{"name":"Xinhong Chen","slug":"xinhong-chen","given":"Xinhong","family":"Chen","affiliation":"City University of Hong Kong; City University of Hong Kong Shenzhen Futian Research Institute"},{"name":"Jianping Wang","slug":"jianping-wang","given":"Jianping","family":"Wang","affiliation":"City University of Hong Kong; City University of Hong Kong Shenzhen Futian Research Institute"},{"name":"Qun Song","slug":"qun-song","given":"Qun","family":"Song","affiliation":"Delft University of Technology"},{"name":"Rui Tan","slug":"rui-tan","given":"Rui","family":"Tan","affiliation":"Nanyang Technological University"},{"name":"Yung-Hui Li","slug":"yung-hui-li","given":"Yung-Hui","family":"Li","affiliation":"Foxconn Research"}],"abstract":"By learning expressive representations, deep learning (DL) has revolutionized autonomous driving (AD). Despite significant advancements, the inherent opacity of DL models engenders public distrust, impeding their widespread adoption. For explainable autonomous driving, current studies primarily concentrate on extracting features from input scenes to predict driving actions and their corresponding explanations. However, these methods underutilize semantics and correlation information within actions and explanations (collectively called categories in this work), leading to suboptimal performance. To address this issue, we propose Semantic-Guided Dynamic Correlation Learning (SGDCL), a novel approach that effectively exploits semantic richness and dynamic interactions intrinsic to categories. SGDCL employs a semantic-guided learning module to obtain category-specific representations and a dynamic correlation learning module to adaptively capture intricate correlations among categories. Additionally, we introduce an innovative loss term to leverage fine-grained co-occurrence statistics of categories for refined regularization. We extensively evaluate SGDCL on two well-established benchmarks, demonstrating its superiority over seven state-of-the-art baselines and a large vision-language model. SGDCL significantly promotes explainable autonomous driving with up to 15.3% performance improvement and interpretable attention scores, bolstering public trust in AD.","doi":"10.24963/ijcai.2024/66","url":"https://ijcai.org/proceedings/2024/66","pdf":"https://ijcai.org/proceedings/2024/0066.pdf","markdown":"https://ijcai.org/proceedings/2024/66.md","bibtex":"https://ijcai.org/proceedings/2024/bibtex/66","volume":{"slug":"2024","title":"Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2024","url":"https://ijcai.org/proceedings/2024/","year":2024,"location":"Jeju, Korea","dates":"3-9 August 2024","editors":"Kate Larson","isbn":"978-1-956792-04-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2024"},"section":"Computer Vision","track":"Main Track","number":66,"pages":{"first":596,"last":604},"keywords":["Computer Vision: CV: Interpretability and transparency","Computer Vision: CV: Machine learning for vision","Machine Learning: ML: Explainable/Interpretable machine learning","Machine Learning: ML: Classification"],"videos":[],"references":[{"position":1,"raw":"[Atakishiyev et al., 2021] Shahin Atakishiyev, Mohammad Salameh, Hengshuai Yao, and Randy Goebel. To- wards safe, explainable, and regulated autonomous driv- ing. arXiv preprint arXiv:2111.10518, 2021.","title":"Towards safe, explainable, and regulated autonomous driving","authors":"Shahin Atakishiyev, Mohammad Salameh, Hengshuai Yao, Randy Goebel","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2017] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587, 2017.","title":"Rethinking atrous convolution for semantic image segmentation","authors":"Liang-Chieh Chen, George Papandreou, Florian Schroff, Hartwig Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2019a] Tianshui Chen, Muxin Xu, Xiaolu Hui, Hefeng Wu, and Liang Lin. Learning semantic-specific graph representation for multi-label image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 522-531, 2019.","title":"Learning semantic-specific graph representation for multi-label image recognition","authors":"Chen","year":"2019","venue":"Proceedings of the IEEE/CVF International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chen et al., 2019b] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5177-5186, 2019.","title":"Multi-label image recognition with graph convolutional networks","authors":"Chen","year":"2019","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2021a] Tina Chen, Taotao Jing, Renran Tian, Yaobin Chen, Joshua Domeyer, Heishiro Toyoda, Rini Sherony, and Zhengming Ding. Psi: A pedestrian behav- ior dataset for socially intelligent autonomous car. arXiv preprint arXiv:2112.02604, 2021.","title":"Psi: A pedestrian behavior dataset for socially intelligent autonomous car","authors":"Chen","year":"2021","venue":null,"doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021b] Zhaomin Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Learning graph convolutional networks for multi-label recognition and applications. IEEE Transactions on Pattern Analysis and Machine In- telligence, 45(6):6969-6983, 2021.","title":"Learning graph convolutional networks for multi-label recognition and applications","authors":"Chen","year":"2021","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Cui et al., 2023] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on mul- timodal large language models for autonomous driving. arXiv preprint arXiv:2311.12320, 2023.","title":"A survey on multimodal large language models for autonomous driving","authors":"Cui","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cultrera et al., 2020] Luca Cultrera, Lorenzo Seidenari, Federico Becattini, Pietro Pala, and Alberto Del Bimbo. Explaining autonomous driving by learning end-to-end vi- sual attention. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition Work- shops, pages 340-341, 2020.","title":"Explaining autonomous driving by learning end-to-end visual attention","authors":"Luca Cultrera, Lorenzo Seidenari, Federico Becattini, Pietro Pala, Alberto Del Bimbo","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Dong et al., 2023] Jiqian Dong, Sikai Chen, Mohammad Miralinaghi, Tiantian Chen, Pei Li, and Samuel Labi. Why did the ai make that decision? towards an explainable ar- tificial intelligence (xai) for autonomous driving systems. Transportation Research Part C: Emerging Technologies, 156:104358, 2023.","title":"Why did the ai make that decision? towards an explainable artificial intelligence (xai) for autonomous driving systems","authors":"Dong","year":"2023","venue":"Transportation Research Part C: Emerging Technologies","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Feng et al., 2023] Yuchao Feng, Wei Hua, and Yuxiang Sun. Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding. IEEE Transactions on Intelligent Trans- portation Systems, 24(9):9780-9791, 2023.","title":"Nle-dm: Natural-language explanations for decision making of autonomous driving based on semantic scene understanding","authors":"Feng","year":"2023","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Goldman and Bustin, 2022] Claudia V Goldman and Ronit Bustin. Trusting explainable autonomous driving: Sim- ulated studies. In IEEE Intelligent Vehicles Symposium, pages 1255-1260, 2022.","title":"Trusting explainable autonomous driving: Simulated studies","authors":"V Claudia, Ronit Goldman, Bustin","year":"2022","venue":"IEEE Intelligent Vehicles Symposium","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.","title":"Deep residual learning for image recognition","authors":"He","year":"2016","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jing et al., 2022] Taotao Jing, Haifeng Xia, Renran Tian, Haoran Ding, Xiao Luo, Joshua Domeyer, Rini Sherony, and Zhengming Ding. Inaction: Interpretable action de- cision making for autonomous driving. In Proceedings of the European Conference on Computer Vision, pages 370- 387, 2022.","title":"Inaction: Interpretable action decision making for autonomous driving","authors":"Taotao Jing, Haifeng Xia, Renran Tian, Haoran Ding, Xiao Luo, Joshua Domeyer, Rini Sherony, Zhengming Ding","year":"2022","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kipf and Welling, 2017] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Rep- resentations, 2017.","title":"Semi-supervised classification with graph convolutional networks","authors":"N Thomas, Max Kipf, Welling","year":"2017","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Koh et al., 2020] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Inter- national Conference on Machine Learning, pages 5338- 5348, 2020.","title":"Concept bottleneck models","authors":"Pang Wei Koh, Thao Nguyen, Siang Yew, Stephen Tang, Emma Mussmann, Been Pierson, Percy Kim, Liang","year":"2020","venue":"International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Madhav and Tyagi, 2022] AV Shreyas Madhav and Amit Kumar Tyagi. Explainable artificial intelligence (xai): connecting artificial decision-making and human trust in autonomous vehicles. In Proceedings of Interna- tional Conference on Computing, Communications, and Cyber-Security, pages 123-136, 2022.","title":"Explainable artificial intelligence (xai): connecting artificial decision-making and human trust in autonomous vehicles","authors":"Tyagi ; Av Shreyas Madhav, Amit Madhav, Tyagi Kumar","year":"2022","venue":"Proceedings of International Conference on Computing, Communications, and Cyber-Security","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Omeiza et al., 2021] Daniel Omeiza, Helena Webb, Marina Jirotka, and Lars Kunze. Explanations in autonomous driving: A survey. IEEE Transactions on Intelligent Trans- portation Systems, 23(8):10142-10162, 2021.","title":"Explanations in autonomous driving: A survey","authors":"Daniel Omeiza, Helena Webb, Marina Jirotka, Lars Kunze","year":"2021","venue":"IEEE Transactions on Intelligent Transportation Systems","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[OpenAI, 2023] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.","title":null,"authors":"Openai","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Reimers and Gurevych, 2019] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the Conference on Empirical Methods in Natural Language Process- ing and the International Joint Conference on Natural Language Processing, pages 3982-3992, 2019.","title":"Sentence-bert: Sentence embeddings using siamese bert-networks","authors":"Gurevych Reimers, Nils Reimers, Iryna Gurevych","year":"2019","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ren et al., 2015] Shaoqing Ren, Kaiming He, Ross Gir- shick, and Jian Sun. Faster r-cnn: Towards real-time ob- ject detection with region proposal networks. In Advances in Neural Information Processing Systems, pages 91-99, 2015.","title":"Faster r-cnn: Towards real-time object detection with region proposal networks","authors":"Ren","year":"2015","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Sys- tems, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, Illia Polosukhin","year":"2017","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In Interna- tional Conference on Learning Representations, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio","year":"2018","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2020] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen. Multi- label classification with label graph superimposing. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 12265-12272, 2020.","title":"Multilabel classification with label graph superimposing","authors":"Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, Shilei Wen","year":"2020","venue":"Proceedings of the AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wen et al., 2023] Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingx- uan Li, Linran Xu, Dengke Shang, et al. On the road with gpt-4v (ision): Early explorations of visual- language model on autonomous driving. arXiv preprint arXiv:2311.05332, 2023.","title":"On the road with gpt-4v (ision): Early explorations of visuallanguage model on autonomous driving","authors":"Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2020] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1):4-24, 2020.","title":"A comprehensive survey on graph neural networks","authors":"Wu","year":"2020","venue":"IEEE Transactions on Neural Networks and Learning Systems","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural net- works? In International Conference on Learning Rep- resentations, 2019.","title":"How powerful are graph neural networks","authors":"Xu","year":"2019","venue":"International Conference on Learning Representations","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Xu et al., 2020] Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, and Nuno Vasconcelos. Explainable object-induced action decision for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9523-9532, 2020.","title":"Explainable object-induced action decision for autonomous vehicles","authors":"Yiran Xu, Xiaoyin Yang, Lihang Gong, Hsuan-Chu Lin, Tz-Ying Wu, Yunsheng Li, Nuno Vasconcelos","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Ye et al., 2020] Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, and Yu Qiao. Attention-driven dynamic graph con- volutional network for multi-label image recognition. In Proceedings of the European Conference on Computer Vi- sion, pages 649-665, 2020.","title":"Attention-driven dynamic graph convolutional network for multi-label image recognition","authors":"Jin Ye, Junjun He, Xiaojiang Peng, Wenhao Wu, Yu Qiao","year":"2020","venue":"Proceedings of the European Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Yu et al., 2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driv- ing dataset for heterogeneous multitask learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2636-2645, 2020.","title":"Bdd100k: A diverse driving dataset for heterogeneous multitask learning","authors":"Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, Trevor Darrell","year":"2020","venue":"Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Zablocki et al., 2022] Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision-based autonomous driving systems: Review and challenges. International Journal of Computer Vision, 130(10):2425-2452, 2022.","title":"Explainability of deep vision-based autonomous driving systems: Review and challenges","authors":"Zablocki","year":"2022","venue":"International Journal of Computer Vision","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Zhang et al., 2022] Zhengming Zhang, Renran Tian, Rini Sherony, Joshua Domeyer, and Zhengming Ding. Attention-based interrelation modeling for explainable au- tomated driving. IEEE Transactions on Intelligent Vehi- cles, 8(2):1564-1573, 2022.","title":"Attention-based interrelation modeling for explainable automated driving","authors":"Zhengming Zhang, Renran Tian, Rini Sherony, Joshua Domeyer, Zhengming Ding","year":"2022","venue":"IEEE Transactions on Intelligent Vehicles","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}