Automatic Emergency Diagnosis with Knowledge-Based Tree Decoding

1 Introduction

The clinical note, an essential part of Electronic Health Record (EHR), generally contains a patient's past medical history, chief complaints and current symptoms. The physicians need to study and be on probation for years before they can give diagnosis individually, but the diagnosis is still timeconsuming and error-prone.

To address existing drawbacks of human diagnosis, researchers started to study automatic diagnosis [Xiao et al., 2018]. Automatic diagnosis takes raw texts of clinical notes as input, and gives the codes of diseases according to the ICD coding system [WHO, 1978], which is adopted in hospitals world-wide. The ICD codes are naturally organized as a tree structure. The tree starts from a virtual root node, goes deeper through intermediate nodes and finally reaches diseases in leaf nodes. Only leaf nodes correspond to a specific disease with its ICD code. Intermediate nodes represent a medical concept or a range of diseases.

∗Contact Author

Automatic diagnosis has become a popular research field recently [Perotte et al., 2013; Wang et al., 2016; Subotin and Davis, 2016], and the majority of existing works formalize it as a plain text classification task. For example, [Lipton et al., 2016] and [Li et al., 2018] use Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) to predict diseases. To give support from clinical notes for final predictions, attention mechanisms are used in [Sha and Wang, 2017] and [Mullenbach et al., 2018].

Although existing models have made progress on the accuracy of disease diagnosis by a considerable margin, automatic diagnosis is still confronted with two major problems.

Negligence of the Medical Relationships among Diseases.

Existing models generally treat diseases as mutually independent. However, from a medical perspective, diseases are interconnected and they are organized hierarchically in the ICD tree. The constraints of hierarchical structure can prevent the predictions from being too far away from the ground truth. For example, without hierarchical restrictions, a patient with acute myocardial infarction (disease of the circulatory system) can be diagnosed with acute gastroenteritis (disease of the digestive system) by mistake, and this may bring about life loss and huge economic compensation.

Low Practicality of Support. Existing attention-based models can already tag words that have deep impact on final predictions. However, this one-step attention cannot reveal a transparent reasoning process, thus lacking in the practicality of assisting doctors in making decisions. In general, diagnosis is a step-by-step procedure, where the physician first locates the diseased organ and then uses the knowledge he has learned and the information from patients to stepwisely reach final results. Each step inside the diagnosis procedure requires different support from clinical notes.

To address these problems, we propose a knowledge-based tree decoding model named K-BTD, consisting of Clinical Notes Encoder, Knowledge Encoder, Judge Net and Fusion Net. K-BTD takes in raw clinical notes as input, and the inference procedure is a top-down decoding process of the ICD tree. Each node inside the tree is equipped with external medical knowledge extracted fromWikipedia or Baidupedia. At each decoding step, the Judge Net decides whether to expand the children of the current node, and the Fusion Net aggregates information frommultiple resources to promote further decoding. The whole process is repeated recursively until there are no more children to expand. The stepwise decoding process imitates the diagnosis procedure of a human doctor, and at each step our model can give support for its decision, which visualizes the reasoning process and provides human doctors with better references.

We conduct experiments on real-world data from the emergency department of Beijing Tsinghua Changgung Hospital, a large-scale hospital in China. Experimental results indicate that our model achieves significant improvements over other state-of-the-art models in micro-F1, macro-F1 and semantic distance. We also show the superiority of our model in terms of interpretability. Ablation analysis and error analysis are conducted to verify the internal mechanism. To the best of our knowledge, this is the first empirical study to inference diagnosis with knowledge-based tree decoding.

2 RelatedWork

2.1 Automatic Diagnosis

Automatic Diagnosis is a long-standing task in the field of medical informatics. Early works utilize machine learning models such as hierarchical Support Vector Machine (SVM) [Perotte et al., 2013]. With the rapid development of deep learning technologies, researchers start to formalize it as a text classification task. Long Short Term Memory (LSTM) [Lipton et al., 2016] and CNN [Li et al., 2018] are used to extract semantic features from textual content. Bag-of-words and disease correlation graph are explored in [Wang et al., 2016]. However, these methods can only extract shallow features and cannot give support for final predictions, which block it from practical application.

With the widespread of attention mechanism, researchers begin to predict diseases and their support by incorporating deep learning models with it. [Mullenbach et al., 2018] adopts per-label attention mechanism and allow the model to learn distinct representations for each disease label.

2.2 Tree-basedMulti-label Classification

Tree-based multi-label classification is a branch of multilabel classification, and it is applicable when the predictor has a hierarchical structure. To be specific, the label space is a tree where nodes represent nested semantic concepts, and the specificity of them increases with depth. Its successful implementation can reduce a large discrete sample space to only a small number of candidate labels.

Some researchers focus on inducing tree structure label space to improve inference efficiency [Beygelzimer et al., 2009; Daum´e III et al., 2017]. Some researchers employ the already existed label space structure. For example, the natural tree structure of Medical Subject Heading (MeSH) is utilized by [Singh et al., 2018] in MeSH tagging task.

Some researchers have explored the tree structure of ICD coding system. [Perotte et al., 2013] adopts hierarchical SVM

to model the inclusion and exclusion relationships of diseases. [Kamkar et al., 2015] reaches stable and better feature selection based on the ICD tree structure. [Xie and Xing, 2018] applies tree-of-sequences LSTM to model the latent representation of each node in the ICD tree with textual descriptions of the ICD codes and their hierarchical structure. However, none of them formalizes automatic diagnosis as a tree-decoding procedure along the ICD tree.

3 Model

3.1 ProblemFormulation

Considering that a patient can be diagnosed with more than one disease, we treat automatic diagnosis as a multi-label classification task over ICD-9 codes. ICD-9 is a standard version of the ICD coding system, which contains over 15,000 codes in its taxonomy 1 . In our study, we only consider high-frequent ICD-9 codes such as top-100 and top-150 codes, which takes up more than 90% of all appeared ICD-9 codes. Each node in the ICD-9 tree is equipped with a piece of text representing the external knowledge extracted from Wikipedia or Baidupedia.

The input of our model, the clinical notes of a patient, is a word sequence X = {x1, x2, . . . , xN}, where N is the length of sequence X. Let the ICD-9 codes for diseases to be the label space L, and the labeling task is to determine yl ∈ {0, 1} for all l ∈ L.

3.2 Model Overview

The clinical note from patient is first encoded by Clinical Notes Encoder to get feature sequence T = {t1, t2, . . . , tN} and flowing vector Froot for root node. We name Fn as flowing vector for node n because it contains all information flowing from the root node to current node n. Next, the tree decoding process starts from the root node of the ICD-9 tree.

Each step of the decoding process will be conducted on a particular node n in the tree (e.g., root). The Knowledge Encoder first encodes the external knowledge of each child of the current node n. Next, the Judge Net enumerates each child node and decides whether to expand this child node for further decoding. If a child node m is chosen to be expanded, the Fusion Net will aggregate flowing vector Fn and the external knowledge of node m to get flowing vector Fm for child node m. The decoding process will repeat recursively until there are no more nodes to expand. The expanded leaf nodes will be chosen as final predictions. The overall framework is shown in Figure 1.

3.3 Clinical Notes Encoder

Taking the word sequence X of clinical notes as input, the Clinical Notes Encoder computes the latent text description through two layers, i.e., embedding layer and RNN layer.

We first convert each word xi to a vector with pretrained Tencent AI Lab Embedding Corpus [Song et al., 2018], which features its strength in the medical domain. We have tried building word embeddings with the dataset we adopt, and the performance is not as good. After the embedding layer, an RNN is used to extract the contextual information from clinical notes. It is noticeable that the RNN can be in any form such as Gated Recurrent Unit (GRU), LSTM and BERT [Devlin et al., 2018]. After the RNN layer, we can get clinical notes feature sequence T = {t1, t2, . . . , tN} ∈ RN×q , where q is the dimension of the hidden state. The flowing vector for the root node is set as Froot = tN.

1 The ICD-9 tree structure can be found at https://bioportal. bioontology.org/ontologies/ICD9CM.

3.4 Knowledge Encoder

For Knowledge Encoder, we choose CNN due to its high efficiency. The input of Knowledge Encoder from node m is a word sequence of external medical knowledge Gm = {gm,1, gm,2, . . . , gm,M}, where M is the sequence length. The same embedding method is applied on the sequence to get ˜Gm = {˜ gm,1, ˜ gm,2, . . . , ˜ gm,M }.

The convolution operation is done with a convolution matrix W ∈ Ru×(h×k) , where u is the number of filters, h is the length of the sliding window and k is the dimension of word embeddings. The convolutional feature matrix Cm ∈ RM×u is calculated by:

where ˜ gm,i:i+h-1 is the concatenation of word embeddings within the i-th sliding window, and b ∈ Ru is a bias vector.

To integrate the information from convolutional feature matrix Cm, we adopt self attention and calculate Am, the feature vector of external knowledge for node m as follows:

where Wa is a multi-layer perceptron.

3.5 Judge Net

Suppose the decoding process is now conducted on node n, and the network is deciding whether to expand child node m for further decoding. The Judge Net calculates the possibility that child node m will be expanded with two measurements, namely semantic-level score and word-level score.

Semantic-level score. We use semantic-level score s1 to represent the semantic-level interactions between external knowledge feature vector Am and the aggregated information Fn that has flown to current node n:

where W1 is a multi-layer perceptron and ◦ indicates element-wise multiplication.

Word-level score. Semantic-level score only considers the interactions between holistic feature vectors. Here, we use word-level score s2 to introduce reactions at the word-level between clinical notes feature sequence T = {t1, t2, . . . , tN} and convolutional feature sequence {Cm,i}.

First, we calculate the similarity matrix S with {ti} and {Cm,j} as follows:

where W2 ∈ Rq×u is a parameter matrix. Then, row-wise and column-wise softmax are separately applied on S to get γij and δij:

we use ξi = ∑Mj=1 δij to measure the importance of the ith word in clinical notes. It is noticeable that ξi is only calculated for interpretability, and is not involved in subsequent calculations.

Next, we generate intermediate representation pi ∈ Ru by attentively aggregate {Cm,j}:

Then, we apply a GRU to process the generated intermediate representation P = {p1, p2, . . . , pN} ∈ RN×u :

where dp is the dimension of the hidden state. Next, we project P to a compressed vector by max-pooling along the column axis to get the word-level score s2:

where W3 is a multi-layer perceptron, and T is a shortcut connection to facilitate the training process.

Overall score. To consider both scores simultaneously, we formulate the overall score s as a weighted summation of semantic-level score and word-level score:

where λ ∈ [0, 1] is a hyper-parameter. If s is greater than a preset threshold ϵ, the child node m will be expanded. If s is smaller than ϵ, the Judge Net will check the next child node of current node n. ϵ is set to be 0.45 in our experiment to balance precision and recall.

3.6 Fusion Net

The Fusion Net comes into operation only if a child node m is chosen to be expanded by Judge Net. Its purpose is to generate Fm by fusing the information flowing to parent node n and the external knowledge of child node m.

First, the weight βi for each position i in the convolutional feature sequence {Cm,i} is calculated via dot-production with flowing vector Fn and subsequent softmax operation:

Next, the flowing vector of child node m is the weighted summation of Fn, Am and the weighted average of {Cm,i}:

The different weight for Am and ∑Mi=1 βiCm,i is designed to introduce more information from child node m. If s1 is smaller than s2, the model needs to pay more attention to the word-level information. If s1 is greater than s2, the model should absorb more information from the semantic-level.

3.7 Training

We use layerwise multi-label cross entropy to train our network. The loss function is as follows:

where MaxDepth is the depth of the ICD-9 tree, L is binary cross entropy loss, pd is the prediction of Judge Net at layer d and yd is the ground truth label at layer d. The exponential part is added to penalize more for early error and penalize less for late error.

4 Experiments

4.1 Dataset Description

The data from the emergency department of Beijing Tsinghua Changgung Hospital, a large-scale hospital in China, is collected from 2015 to 2017, and it is the first large automatic diagnosis dataset in Chinese. The dataset contains unstructured clinical notes including chief complaints, history of recent illness, past medical history and structured data such as auxiliary examination results.To keep track with previous work, we only utilize the unstructured part.

Each admission record is tagged with one or more ICD-9 codes by licensed physicians, denoting the identified diseases. A summary of the dataset statistics is provided in Table 1. We choose the top-100 and top-150 most frequent codes to conduct two separate experiments. When doing experiments on top-k frequent codes, we filter the dataset down to instances that have at least one of the top-k frequent codes. In experiment, we conduct random five fold cross-validation to examine the performance of our model as well as baselines.

For external medical knowledge, we use Wikipedia and Baidupedia as resources. [Trevena, 2011] has demonstrated the reliability of medical articles in Wikipedia, and medical terms in Baidupedia are under the supervision of National Health Care Commission of China. The extraction of external medical knowledge is divided into three steps. For each node in the ICD-9 tree, we first search the corresponding Wikipedia page. If the Wikipedia page does not exist, we use the corresponding Baidupedia page to make up for it. If the corresponding Baidupedia page also does not exist, we will consider highly related candidates in Baidupedia. We rank related candidates by the similarity between candidates and query terms and select the one with highest similarity. For Wikipedia pages, we use the content in the section of Signs and Symptoms. For Baidupedia pages, we extract the materials in sections of Clinical Manifestations and Cause of Diseases. An illustration of external medical knowledge is shown in Table 2.

4.2 Baselines

For comparison, we reproduce two major categories of baselines. The first category utilizes only clinical notes and the second employs external medical knowledge as well.

As for the first category baselines, we first choose TFIDF+SVM, TextCNN [Kim, 2014] and BERT [Devlin et al., 2018], which are classic models for text classification. Besides, three classic automatic diagnosis models DeepLabeler [Li et al., 2018], AIC [Xie and Xing, 2018] and CAML [Mullenbach et al., 2018] are also adopted for comparison.

As for the second category baselines, we adopt CMemNNs [Prakash et al., 2017], which uses multi-hop memory networks to inference diagnosis. As there are few works in automatic diagnosis that utilize external knowledge, we adopt a classic reading comprehension model Fact-Law Attention Model [Luo et al., 2017] from the legal judgement domain for comprehensive comparison. All baselines in type #2, as well as the proposed model use the same external knowledge to assure fair comparison.

To better demonstrate the advantage of our model, we use LSTM, GRU and BERT to replace the RNN in Clinical Notes Encoder and conduct three different experiments. The code for our model is publicly available at https://github.com/ kaisadadi/K-BTD.

4.3 EvaluationMetrics

The distribution of diseases is highly imbalanced so we evaluate our model with both micro-F1 and macro-F1. Besides, we propose another metrics named semantic distance (SD), which is modified from [Singh et al., 2018]. SD is defined as:

where Y is the set of target codes and ˆ Y is the set of predicted codes. D is a distance function that measures the shortest distance between two nodes in the ICD-9 tree. If ˆ Y is empty, SD is set to be the maximum distance between two nodes in the ICD-9 tree. The semantic distance measures the distance between predictions and labels from a medical view. Mismatch with small semantic distance is tolerable, while a large semantic distance might result in completely wrong operations or even death.

4.4 Experimental Settings

In experiment, we set λ to 0.5, ϵ to 0.45 and dp to 256. We adopt Adam [Kingma and Ba, 2015] for optimization. The size of the mini-batch is 64, and the learning rate is 10-5 for BERT-related models and 10-3 otherwise. Dropout is set to 0.5, and weight decay is 10-5 .

Clinical Note: The patient got upper abdominal pain and paroxysmal colic 12 hours ago with no obvious inducement. The symptom exacerbated after 6 hours, accompanied with acid reflux, nausea and multiple vomiting, which is yellow watery. The patient also has chills, sore throat, and fatigue all over. The patient takes ibuprofen and the symptom is not relieved ....... Past history: allergy history (-), last menstruation: 2017-*-**. Menstruation is delayed.

Diagnosis: Gastroenteritis(ICD-9: 558.9), Reflux esophagitis (ICD-9: 530.1)

4.5 Experimental Results

We evaluate our model on real-world data from the emergency department of Beijing Tsinghua Changgung Hospital, and experimental results are shown in Table 3. From the results we can see that:

1. K-BTD exceeds all baselines in both experiments, which demonstrates the effectiveness and robustness of our model.
2. K-BTD surpasses baselines in macro-F1 by a considerable margin, and this indicates that the utilization of tree structure enables the model to make the right decisions on diseases that are not common to appear.
3. K-BTD reduces semantic distance significantly compared with baselines. This indicates that the results of our model are closer to the ground truth than baselines and can provide better references for human doctors.
4. K-BTD can benefit from the research progress in text encoder. K-BTD is actually a framework, where the components inside can be updated easily. We are confident that our model can reach higher performance with better text encoders in the future.

4.6 Evaluation of Interpretability

Because our model employs tree decoding architecture, it can give support for decision at each step while conventional models can only provide support for final predictions. Here we demonstrate the interpretability of our model with an actual clinical note from the dataset, and compare it with another explainable model CAML [Mullenbach et al., 2018].

For K-BTD, each step's support is chosen from words with value ξi greater than a preset threshold. For CAML, we follow the original settings. The results are shown in Figure 2.

We can see from results that support from both CAML and our model catch similar important words such as "vomiting" and "abdominal pain". However, without stepwise inference, CAML misdiagnoses upper respiratory infections because of the shared symptoms between diseases (e.g. sore throat). Besides, we can see that our model's decision at each step has different support. For example, the support for diseases of the digestive system are "abdominal" and "vomiting", and the support for diseases of esophagus, stomach and duodenum are "acid reflux", "vomiting" and "chills". This observation demonstrates that our model pays attention to different parts of the clinical notes at different positions in the decoding process. The stepwise support brings more interpretability to final predictions and can provide human doctors with better assistance in diagnosis inference.

4.7 Ablation Analysis

External medical knowledge is a critical part of our model. To further explore the role that external medical knowledge plays in the whole architecture, we conduct three experiments:

Experiment #1. Randomly mask from 10% to 50% of the knowledge texts for all nodes in the ICD-9 tree.

Experiment #2. Randomly shuffle the knowledge for 10% to 50% nodes in the ICD-9 tree.

Experiment #3. Randomly replace the knowledge for 10% to 50% nodes in the ICD-9 tree with totally irrelevant materials.

We repeat each experiment five times to reduce the effect of randomness, and use the mean value as final results. For all three experiments, the model takes more epochs to converge and the performance on both micro-F1 and macro-F1 of the initial three to five epochs decreases a lot. The results are shown in Figure 3.

We observe that the model performance decreases on all three experiments, and the degree of decline increases from experiment #1 to experiment #3. It's clear that noise to external medical knowledge can decrease model performance. The severer the noise is, the more the model performance drops, which indicates that our model relies heavily on the quality of external medical knowledge. We can safely predict that external knowledge with better quality can bring further improvement to model performance.

4.8 Error Analysis

To give better insight into out model's performance, we calculate the ratio of first-occurred errors that appear at different depth in the ICD-9 tree. We define the depth of the root node to be 0, and the maximum depth is 4. The results are shown in Table 4. For baseline models without tree decoding, we use the ICD-9 tree to locate the depth of the first-occurred error.

It's clear that K-BTD make mistakes in deeper positions in the ICD-9 tree compared with baselines. The deeper the first error occurs in the decoding process, the closer the distance between predictions and ground truth is, which again proves the effectiveness of our model. The reason of low error ratio in depth 4 is that the number of children of nodes in depth 3 is quite small.

5 Conclusion

In this paper, we propose K-BTD, a knowledge-based tree decoding model for automatic emergency diagnosis. To be specific, we utilize the medically significant ICD-9 tree structure and formulate this task as a stepwise top-down decoding procedure. External knowledge is extracted to assist the decoding process, and we have demonstrated its importance in our model. The top-down decoding process enables our model to give different support for each decision, which adds to the interpretability and practicality of our model compared with existing single-step models. Experimental results on real-world data show that our model outperforms baselines in all metrics, which proves the effectiveness and robustness of our model.

In the future, we will focus on increasing the diagnosis accuracy of infrequent diseases.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (grants 61872218, 61721003, 61673241 and 61906105), National Key R&D Program of China (2019YFB1404804), Tsinghua-Fuzhou Institute of Digital Technology, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua UniversityPeking Union Medical College Hospital Initiative Scientific Research Program, and Tsinghua University Big Data Research Center. The funders had no roles in study design, data collection and analysis, the decision to publish, and preparation of the manuscript. Besides, the author would like to thank Haoxi Zhong, Yudong Chen and Zhijing Wu for their sincere and helpful suggestions on paper writing.