{"type":"ScholarlyArticle","title":"MNN: Multimodal Attentional Neural Networks for Diagnosis Prediction","authors":[{"name":"Zhi Qiao","slug":"zhi-qiao","given":"Zhi","family":"Qiao","affiliation":"Tencent Medical AI Lab"},{"name":"Xian Wu","slug":"xian-wu","given":"Xian","family":"Wu","affiliation":"Tencent Medical AI Lab"},{"name":"Shen Ge","slug":"shen-ge","given":"Shen","family":"Ge","affiliation":"Tencent Medical AI Lab"},{"name":"Wei Fan","slug":"wei-fan","given":"Wei","family":"Fan","affiliation":"Tencent Medical AI Lab"}],"abstract":"Diagnosis prediction plays a key role in clinical decision supporting process, which attracted extensive research attention recently. Existing studies mainly utilize discrete medical codes (e.g., the ICD codes and procedure codes) as the primary features in prediction. However, in real clinical settings, such medical codes could be either incomplete or erroneous. For example, missed diagnosis will neglect some codes which should be included, mis-diagnosis will generate incorrect medical codes. To increase the robustness towards noisy data, we introduce textual clinical notes in addition to medical codes. Combining information from both sides will lead to improved understanding towards clinical health conditions. To accommodate both the textual notes and discrete medical codes in the same framework, we propose Multimodal Attentional Neural Networks (MNN), which integrates multi-modal data in a collaborative manner. Experimental results on real world EHR datasets demonstrate the advantages of MNN in terms of both robustness and accuracy.","doi":"10.24963/ijcai.2019/823","url":"https://ijcai.org/proceedings/2019/823","pdf":"https://ijcai.org/proceedings/2019/0823.pdf","markdown":"https://ijcai.org/proceedings/2019/823.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/823","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"AI for Improving Human Well-being","track":"AI for Improving Human Well-being","number":823,"pages":{"first":5937,"last":5943},"keywords":["Special Track on AI for Improving Human-Well Being: Human wellbeing (Special Track on AI and Human Wellbeing)","Special Track on AI for Improving Human-Well Being: Health applications (Special Track on AI and Human Wellbeing)"],"videos":[],"references":[{"position":1,"raw":"[Abadi et al., 2015] Martín Abadi et al. TensorFlow: Large- Scale Machine Learning on Heterogeneous Systems. 2015. Software available from tensorflow.org.","title":"TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems","authors":"Abadi","year":"2015","venue":"Software available from tensorflow","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015.","title":"Neural Machine Translation by Jointly Learning to Align and Translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Baytas et al., 2017] Inci M. Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K. Jain, and Jiayu Zhou. Patient Subtyping via Time-Aware LSTM Networks. In KDD, pages 65-74, 2017.","title":"Patient Subtyping via Time-Aware LSTM Networks","authors":"Inci M Baytas, Cao Xiao, Xi Zhang, Fei Wang, Anil K Jain, Jiayu Zhou","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Cheng et al., 2016] Yu Cheng, Fei Wang, Ping Zhang, and Jianying Hu. Risk Prediction with Electronic Health Records: A Deep Learning Approach. In SDM, pages 432-440, 2016.","title":"Risk Prediction with Electronic Health Records: A Deep Learning Approach","authors":"Yu Cheng, Fei Wang, Ping Zhang, Jianying Hu","year":"2016","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merrienboer, C ¸aglar Gülc ¸ehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Transla- tion. In EMNLP, pages 1724-1734, 2014.","title":"Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2016a] Edward Choi, Mohammad Taha Ba- hadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Doctor AI: Predicting Clinical Events via Recurrent Neu- ral Networks. In MLHC, pages 301-318, 2016.","title":"Doctor AI: Predicting Clinical Events via Recurrent Neural Networks","authors":"Choi","year":"2016","venue":"MLHC","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi et al., 2016b] Edward Choi, Mohammad Taha Ba- hadori, Jimeng Sun, Joshua Kulas, Andy Schuetz, and Walter Stewart. RETAIN: An Interpretable Predic- tive Model for Healthcare using Reverse Time Attention Mechanism. In NIPS, pages 3504-3512, 2016.","title":"RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism","authors":"Choi","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Choi et al., 2017] Edward Choi, Andy Schuetz, Walter F Stewart, and Jimeng Sun. Using recurrent neural net- work models for early detection of heart failure onset. Journal of the American Medical Informatics Association, 24(2):361-370, 2017.","title":"Using recurrent neural network models for early detection of heart failure onset","authors":"Edward Choi, Andy Schuetz, Walter F Stewart, Jimeng Sun","year":"2017","venue":"Journal of the American Medical Informatics Association","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Farhan et al., 2016] Wael Farhan, Zhimu Wang, Yingxiang Huang, Shuang Wang, Fei Wang, and Xiaoqian Jiang. A Predictive Model for Medical Events Based on Contextual Embedding of Temporal Sequences. JMIR Medical Infor- matics, 4(4):e39, 2016.","title":"A Predictive Model for Medical Events Based on Contextual Embedding of Temporal Sequences","authors":"Wael Farhan, Zhimu Wang, Yingxiang Huang, Shuang Wang, Fei Wang, Xiaoqian Jiang","year":"2016","venue":"JMIR Medical Informatics","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Goldberger et al., 2000] Ary Goldberger, Luis Amaral, Leon Glass, Jeffrey Hausdorff, Plamen Ivanov, Roger Mark, Joseph Mietus, George Moody, Chung-Kang Peng, and H. Eugene Stanley. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation, 101(23):e215- e220, 2000.","title":"PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals","authors":"Ary Goldberger, Luis Amaral, Leon Glass, Jeffrey Hausdorff, Plamen Ivanov, Roger Mark, Joseph Mietus, George Moody, Chung-Kang Peng, H Eugene Stanley","year":"2000","venue":"Circulation","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Kim, 2014] Yoon Kim. Convolutional Neural Networks for Sentence Classification. In EMNLP, pages 1746-1751, 2014.","title":"Convolutional Neural Networks for Sentence Classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lian, 2018] Jianxun Lian. xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Sys- tems. In KDD, pages 1754-1763, 2018.","title":"xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems","authors":"Lian ;, Jianxun Lian","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2015] Chuanren Liu, Fei Wang, Jianying Hu, and Hui Xiong. Temporal Phenotyping from Longitudinal Electronic Health Records: A Graph Based Framework. In KDD, pages 705-714, 2015.","title":"Temporal Phenotyping from Longitudinal Electronic Health Records: A Graph Based Framework","authors":"Chuanren Liu, Fei Wang, Jianying Hu, Hui Xiong","year":"2015","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Ma et al., 2017] Fenglong Ma, Radha Chitta, Jing Zhou, Quanzeng You, Tong Sun, and Jing Gao. Dipole: Diagno- sis Prediction in Healthcare via Attention-based Bidirec- tional Recurrent Neural Networks. In KDD, pages 1903- 1911, 2017.","title":"Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks","authors":"Ma","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Ma et al., 2018a] Feanglong Ma, Quanzeng You, Houping Xia, and Jing Gao. KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare. In KDD, pages 743-752, 2018.","title":"KAME: Knowledge-based Attention Model for Diagnosis Prediction in Healthcare","authors":"Ma","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Ma et al., 2018b] Tengfei Ma, Cao Xiao, and Fei Wang. Health-ATM: A Deep Architecture for Multifaceted Pa- tient Health Record Representation and Risk Prediction. In SDM, pages 261-269, 2018.","title":"Health-ATM: A Deep Architecture for Multifaceted Patient Health Record Representation and Risk Prediction","authors":"Ma","year":"2018","venue":"SDM","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Miotto et al., 2016] Riccardo Miotto, Li Li, Brian Kidd, and Joel T. Dudley. Deep Patient: An Unsupervised Represen- tation to Predict the Future of Patients from the Electronic Health Records. Scientific Reports, 6:26094, 05 2016.","title":"Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records","authors":"Riccardo Miotto, Li Li, Brian Kidd, Joel T Dudley","year":null,"venue":"Scientific Reports","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Qiao et al., 2018a] Zhi Qiao, Ning Sun, Xiang Li, Eryu Xia, Shiwan Zhao, and Yong Qin. Using Machine Learning Approaches for Emergency Room Visit Prediction Based on Electronic Health Record Data. Studies in health tech- nology and informatics, 247:111-115, 01 2018.","title":"Using Machine Learning Approaches for Emergency Room Visit Prediction Based on Electronic Health Record Data","authors":"Qiao","year":"2018","venue":"Studies in health technology and informatics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Qiao et al., 2018b] Zhi Qiao, Shiwan Zhao, Cao Xiao, Xi- ang Li, Fei Wang, and Yong Qin. Pairwise-Ranking based Collaborative Recurrent Neural Networks for Clini- cal Event Prediction. In IJCAI, pages 3520-3526, 2018.","title":"Pairwise-Ranking based Collaborative Recurrent Neural Networks for Clinical Event Prediction","authors":"Qiao","year":"2018","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2018/489"},{"position":20,"raw":"[Schuster and Paliwal, 1997] Mike Schuster and Kuldip K. Paliwal. Bidirectional Recurrent Neural Networks. IEEE Trans. Signal Processing, 45(11):2673-2681, 1997.","title":"Bidirectional Recurrent Neural Networks","authors":"Mike Schuster, Kuldip K Paliwal","year":"1997","venue":"IEEE Trans. Signal Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wu et al., 2010] Jionglin Wu, Jason Roy, and Walter F Stewart. Prediction modeling using EHR data: chal- lenges, strategies, and a comparison of machine learning approaches. Medical care, 48(6):S106-S113, 2010.","title":"Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches","authors":"Wu","year":"2010","venue":"Medical care","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Xu and Sun, 2018] Yanbo Xu and Jimeng Sun. RAIM Re- current Attentive and Intensive Model of Multimodal Pa- tient Monitoring Data. In KDD, pages 2565-2573, 2018.","title":"RAIM Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data","authors":"Yanbo Xu, Jimeng Sun","year":"2018","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhang et al., 2018] Changqing Zhang, Ehsan Adeli, Tao Zhou, and Xiaobo Chen. Multi-Layer Multi-View Classifi- cation for Alzheimer's Disease Diagnosis. In AAAI, pages 4406-4413, 2018.","title":"Multi-Layer Multi-View Classification for Alzheimer's Disease Diagnosis","authors":"Changqing Zhang, Ehsan Adeli, Tao Zhou, Xiaobo Chen","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}