{"type":"ScholarlyArticle","title":"Human-Centred Multimodal Deep Learning Models for Chest X-Ray Diagnosis","authors":[{"name":"Chihcheng Hsieh","slug":"chihcheng-hsieh","given":"Chihcheng","family":"Hsieh","affiliation":"School of Information Systems , Queensland University of Technology , Brisbane , Australia Centre for Data Science ,; Queensland University of Technology , Brisbane , Australia"}],"abstract":"My thesis consists of investigating how chest X-ray images, radiologists' eye movements and patients' clinical data can be used to teach a machine how radiologists read and classify images with the goal of creating human-centric AI architectures that can (1) capture radiologists' search behavioural patterns using their eye-movements in order to improve classification in DL systems, and (2) automatically detect lesions in medical images using clinical data and eye tracking data. Heterogeneous data sources such as chest X-rays, radiologists' eye movements, and patients' clinical data can contribute to novel multimodal DL architectures that, instead of learning directly from images' pixels, will learn human classification patterns encoded in both the eye movements of the images' regions and patients' medical history. In addition to a quantitative evaluation, I plan to conduct questionnaires with expert radiologists to understand the effectiveness of the proposed multimodal DL architecture.","doi":"10.24963/ijcai.2023/817","url":"https://ijcai.org/proceedings/2023/817","pdf":"https://ijcai.org/proceedings/2023/0817.pdf","markdown":"https://ijcai.org/proceedings/2023/817.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/817","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Doctoral Consortium","track":"Doctoral Consortium","number":817,"pages":{"first":7085,"last":7086},"keywords":["Humans and AI: HAI: Human-AI collaboration","Computer Vision: CV: Recognition (object detection, categorization)","Computer Vision: CV: Biomedical image analysis"],"videos":[],"references":[{"position":1,"raw":"[Castillo et al., 2021] C. Chelsea et al. The effect of clinical information on radiology reporting: A systematic review. J. of Medical Radiation Sciences, 68:60-74, 2021.","title":"The effect of clinical information on radiology reporting: A systematic review","authors":"C Chelsea","year":"2021","venue":"J. of Medical Radiation Sciences","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Ganesan et al., 2018] A. Ganesan et al. A review of factors influencing radiologists' visual search behaviour. J. of Med- ical Imaging & Radiation Oncology, 62:747-757, 2018.","title":"A review of factors influencing radiologists' visual search behaviour","authors":"A Ganesan","year":"2018","venue":"J. of Medical Imaging & Radiation Oncology","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Hsieh et al., 2023] C. Hsieh et al.Mdf-net: Multimodal dual- fusion network for abnormality detection using cxr images and clinical data. arXiv:2302.13390, 2023.","title":"Mdf-net: Multimodal dualfusion network for abnormality detection using cxr images and clinical data","authors":"C Hsieh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Hsieh et al., 2023] C. Hsieh et al. MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multi- modal Deep Learning Applications. PhysioNet, 2023.","title":"MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications","authors":"C Hsieh","year":"2023","venue":null,"doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Karargyris et al., 2021] A. Karargyris et al.Creation and val- idation of a chest x-ray dataset with eye-tracking and report dictation for AI development. Scientific Data, 8, 2021.","title":"Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for AI development","authors":"A Karargyris","year":"2021","venue":"Scientific Data","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Lanfredi et al., 2022] R. Lanfredi et al.REFLACX, a dataset of reports and eye-tracking data for localization of abnor- malities in chest x-rays. Scientific Data, 9, 2022.","title":"REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays","authors":"R Lanfredi","year":"2022","venue":"Scientific Data","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Luís et al., 2023] A. Luís et al.Integrating eye-gaze data into CXR DL approaches: A preliminary study, IEEE Conf. on VR and 3D User Interfaces Abstracts and Workshops, 2023 [","title":"Integrating eye-gaze data into CXR DL approaches: A preliminary study","authors":"A Luís","year":"2023","venue":"IEEE Conf. on VR and 3D User Interfaces Abstracts and Workshops","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Saporta et al., 2022] A. Saporta et al.Benchmarking saliency methods for chest X-ray interpretation. Nature Machine Intelligence, 4:867-878, 2022.","title":"Benchmarking saliency methods for chest X-ray interpretation","authors":"A Saporta","year":"2022","venue":"Nature Machine Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}