{"type":"ScholarlyArticle","title":"Medical Concept Embedding with Time-Aware Attention","authors":[{"name":"Xiangrui Cai","slug":"xiangrui-cai","given":"Xiangrui","family":"Cai","affiliation":"Nankai University , China"},{"name":"Jinyang Gao","slug":"jinyang-gao","given":"Jinyang","family":"Gao","affiliation":"National University of Singapore , Singapore; Naitional University of Singapore."},{"name":"Kee Yuan Ngiam","slug":"kee-yuan-ngiam","given":"Kee Yuan","family":"Ngiam","affiliation":"National University Health System , Singapore"},{"name":"Beng Chin Ooi","slug":"beng-chin-ooi","given":"Beng Chin","family":"Ooi","affiliation":"National University of Singapore , Singapore"},{"name":"Ying Zhang","slug":"ying-zhang","given":"Ying","family":"Zhang","affiliation":"Nankai University , China"},{"name":"Xiaojie Yuan","slug":"xiaojie-yuan","given":"Xiaojie","family":"Yuan","affiliation":"Nankai University , China"}],"abstract":"Embeddings of medical concepts such as medication, procedure and diagnosis codes in Electronic Medical Records (EMRs) are central to healthcare analytics. Previous work on medical concept embedding takes medical concepts and EMRs as words and documents respectively. Nevertheless, such models miss out the temporal nature of EMR data. On the one hand, two consecutive medical concepts do not indicate they are temporally close, but the correlations between them can be revealed by the time gap. On the other hand, the temporal scopes of medical concepts often vary greatly (e.g., common cold and diabetes). In this paper, we propose to incorporate the temporal information to embed medical codes. Based on the Continuous Bag-of-Words model, we employ the attention mechanism to learn a ``soft'' time-aware context window for each medical concept. Experiments on public and proprietary datasets through clustering and nearest neighbour search tasks demonstrate the effectiveness of our model, showing that it outperforms five state-of-the-art baselines.","doi":"10.24963/ijcai.2018/554","url":"https://ijcai.org/proceedings/2018/554","pdf":"https://ijcai.org/proceedings/2018/0554.pdf","markdown":"https://ijcai.org/proceedings/2018/554.md","bibtex":"https://ijcai.org/proceedings/2018/bibtex/554","volume":{"slug":"2018","title":"Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2018","url":"https://ijcai.org/proceedings/2018/","year":2018,"location":"Stockholm","dates":"13-19 July 2018","editors":"Jérôme Lang","isbn":"978-0-9992411-2-7","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2018"},"section":"Natural Language Processing","track":"Main track","number":554,"pages":{"first":3984,"last":3990},"keywords":["Natural Language Processing: Embeddings","Machine Learning Applications: Bio;Medicine"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. JMLR, 3(2):1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bojanowski et al., 2017] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vec- tors with subword information. ACL, 2017.","title":"Enriching word vectors with subword information","authors":"Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov","year":"2017","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Che et al., 2017] Zhengping Che, Yu Cheng, Zhaonan Sun, and Yan Liu. Exploiting convolutional neural network for risk prediction with medical feature embedding. NIPS- ML4HC, 2017.","title":"Exploiting convolutional neural network for risk prediction with medical feature embedding","authors":"Zhengping Che, Yu Cheng, Zhaonan Sun, Yan Liu","year":"2017","venue":"NIPS-ML4HC","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chiang et al., 2014] Jane L Chiang, M Sue Kirkman, Lori MB Laffel, and Anne L Peters. Type 1 diabetes through the life span: a position statement of the ameri- can diabetes association. Diabetes care, 37(7):2034-2054, 2014.","title":"Type 1 diabetes through the life span: a position statement of the american diabetes association","authors":"Jane L Chiang, M Sue Kirkman, Lori Mb Laffel, Anne L Peters","year":"2014","venue":"Diabetes care","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Choi et al., 2016] Edward Choi, Mohammad Taha Ba- hadori, Elizabeth Searles, Catherine Coffey, and Jimeng Sun. Multi-layer representation learning for medical con- cepts. In SIGKDD, pages 1495-1504, 2016.","title":"Multi-layer representation learning for medical concepts","authors":"Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Jimeng Sun","year":"2016","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Choi et al., 2017] Edward Choi, Mohammad Taha Ba- hadori, Le Song, Walter F Stewart, and Jimeng Sun. Gram: Graph-based attention model for healthcare representation learning. In SIGKDD, pages 787-795. ACM, 2017.","title":"Gram: Graph-based attention model for healthcare representation learning","authors":"Edward Choi, Mohammad Taha Bahadori, Le Song, Walter F Stewart, Jimeng Sun","year":"2017","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Choi, 2016] Youngduck Choi. Learning low-dimensional representations of medical concepts. In Proceedings of the AMIA Summit on Clinical Research Informatics(CRI), pages 41-50, 2016.","title":"Learning low-dimensional representations of medical concepts","authors":"Youngduck Choi, Choi","year":"2016","venue":"Proceedings of the AMIA Summit on Clinical Research Informatics(CRI)","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In ICML, pages 160-167. ACM, 2008.","title":"A unified architecture for natural language processing: Deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[De Vine et al., 2014] Lance De Vine, Guido Zuccon, Bevan Koopman, Laurianne Sitbon, and Peter Bruza. Medical se- mantic similarity with a neural language model. In CIKM, pages 1819-1822. ACM, 2014.","title":"Medical semantic similarity with a neural language model","authors":"Lance De Vine, Guido Zuccon, Bevan Koopman, Laurianne Sitbon, Peter Bruza","year":"2014","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Ghassemi et al., 2014] Marzyeh Ghassemi, Tristan Nau- mann, Finale Doshi-Velez, Nicole Brimmer, Rohit Joshi, Anna Rumshisky, and Peter Szolovits. Unfolding physio- logical state: Mortality modelling in intensive care units. In SIGKDD, pages 75-84, 2014.","title":"Unfolding physiological state: Mortality modelling in intensive care units","authors":"Ghassemi","year":"2014","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Harris, 1954] Zellig S Harris. Distributional structure. Word, 10(2-3):146-162, 1954.","title":"Distributional structure","authors":"S Zellig, Harris","year":"1954","venue":"Word","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Hillestad et al., 2005] Richard Hillestad, James Bigelow, Anthony Bower, Federico Girosi, Robin Meili, Richard Scoville, and Roger Taylor. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health affairs, 24(5):1103-1117, 2005.","title":"Can electronic medical record systems transform health care? potential health benefits, savings, and costs","authors":"Richard Hillestad, James Bigelow, Anthony Bower, Federico Girosi, Robin Meili, Richard Scoville, Roger Taylor","year":"2005","venue":"Health affairs","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Ling et al., 2015] Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, and Silvio Amir. Not all contexts are created equal: Better word representations with variable attention. In EMNLP, pages 1367-1372, 2015.","title":"Not all contexts are created equal: Better word representations with variable attention","authors":"Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, Silvio Amir","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Liu et al., 2017] Liping Liu, Francisco Ruiz, Susan Athey, and David Blei. Context selection for embedding models. In NIPS, pages 4819-4828, 2017.","title":"Context selection for embedding models","authors":"Liu","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Melamud et al., 2016] Oren Melamud, David McClosky, Siddharth Patwardhan, and Mohit Bansal. The role of context types and dimensionality in learning word embed- dings. In NAACL, pages 1030-1040, 2016.","title":"The role of context types and dimensionality in learning word embeddings","authors":"Oren Melamud, David Mcclosky, Siddharth Patwardhan, Mohit Bansal","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mikolov et al., 2013a] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. 2013.","title":"Efficient estimation of word representations in vector space","authors":"Mikolov","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013b] Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their composi- tionality. In NIPS, pages 3111-3119, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Minarro-Giménez et al., 2014] José Antonio Minarro- Giménez, Oscar Marin-Alonso, and Matthias Samwald. Exploring the application of deep learning techniques on medical text corpora. Studies in health technology and informatics, 205:584-588, 2014.","title":"José Antonio Minarro-Giménez, Oscar Marin-Alonso, and Matthias Samwald. Exploring the application of deep learning techniques on medical text corpora","authors":"Minarro-Giménez","year":"2014","venue":"Studies in health technology and informatics","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Nguyen et al., 2017] Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, and Svetha Venkatesh. Deepr: A convo- lutional net for medical records. IEEE journal of biomed- ical and health informatics, 21(1):22-30, 2017.","title":"Deepr: A convolutional net for medical records","authors":"Phuoc Nguyen, Truyen Tran, Nilmini Wickramasinghe, Svetha Venkatesh","year":"2017","venue":"IEEE journal of biomedical and health informatics","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Pham et al., 2016] Trang Pham, Truyen Tran, Dinh Phung, and Svetha Venkatesh. Deepcare: A deep dynamic mem- ory model for predictive medicine. In PAKDD, pages 30- 41. Springer, 2016.","title":"Deepcare: A deep dynamic memory model for predictive medicine","authors":"Trang Pham, Truyen Tran, Dinh Phung, Svetha Venkatesh","year":"2016","venue":"PAKDD","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Rudolph et al., 2016] Maja Rudolph, Francisco Ruiz, Stephan Mandt, and David Blei. Exponential family embeddings. In NIPS, pages 478-486, 2016.","title":"Exponential family embeddings","authors":"Maja Rudolph, Francisco Ruiz, Stephan Mandt, David Blei","year":"2016","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Rumelhart et al., 1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. Cognitive modeling, 5(3):1, 1988.","title":"Learning representations by back-propagating errors","authors":"Geoffrey E David E Rumelhart, Ronald J Hinton, Williams","year":"1988","venue":"Cognitive modeling","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sun et al., 2012] Jimeng Sun, Fei Wang, Jianying Hu, and Shahram Edabollahi. Supervised patient similarity mea- sure of heterogeneous patient records. ACM SIGKDD Ex- plorations Newsletter, 14(1):16-24, 2012.","title":"Supervised patient similarity measure of heterogeneous patient records","authors":"Jimeng Sun, Fei Wang, Jianying Hu, Shahram Edabollahi","year":"2012","venue":"ACM SIGKDD Explorations Newsletter","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Suo et al., 2017] Qiuling Suo, Fenglong Ma, Ye Yuan, Mengdi Huai, Weida Zhong, Aidong Zhang, and Jing Gao. Personalized disease prediction using a cnn-based similar- ity learning method. In BIBM, 2017.","title":"Personalized disease prediction using a cnn-based similarity learning method","authors":"Suo","year":"2017","venue":"BIBM","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Tran et al., 2015] Truyen Tran, Tu Dinh Nguyen, Dinh Phung, and Svetha Venkatesh. Learning vector repre- sentation of medical objects via EMR-driven nonnega- tive restricted boltzmann machines (eNRBM). Journal of biomedical informatics, 54:96-105, 2015.","title":"Learning vector representation of medical objects via EMR-driven nonnegative restricted boltzmann machines (eNRBM)","authors":"Truyen Tran, Tu Dinh Nguyen, Dinh Phung, Svetha Venkatesh","year":"2015","venue":"Journal of biomedical informatics","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhu et al., 2016] Zihao Zhu, Changchang Yin, Buyue Qian, Yu Cheng, Jishang Wei, and Fei Wang. Measuring patient similarities via a deep architecture with medical concept embedding. ICDM, pages 749-758, 2016.","title":"Measuring patient similarities via a deep architecture with medical concept embedding","authors":"Zihao Zhu, Changchang Yin, Buyue Qian, Yu Cheng, Jishang Wei, Fei Wang","year":"2016","venue":"ICDM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}