CISA: Chinese Information Structure Analysis for Scientific Writing with Cross-lingual Adversarial Learning

1 Introduction

This work presents CISA, a system for Chinese scientific writing advising based on information structure analysis. We extend the scope of our prior system [Huang and Chen, 2017], which addresses English writing advising, to the second largest language in the world. In recent years, a variety of Chinese writing assistant models have been proposed for grammatical error detection and correction [Lee et al., 2016; Shiue et al., 2017]. Beyond the word level and the syntax level, our system focuses on providing the writing advice at the discourse level and the knowledge level.

The core of CISA is a neural network for information structure identification. Once a user submits a scientific article, the function of each sentence will be recognized into five types, including Background, Purpose, Method, Results, and Conclusion. Based on the analysis of writing structure and organization, our system makes writing suggestions and retrieves the related information from a knowledge base (KB). Previous work on information structure identification, or argumentative zoning, has been shown effective in document summarization [Contractor et al., 2012], citation indexing [Teufel, 2006], and literature review [Guo et al., 2014]. However, most of them are limited to English. To the best of our knowledge, there is still no study on Chinese information structure identification, and the corpus has also yet to construct.

Due to the lack of training data, we propose a cross-lingual adversarial network that is trained on the labeled English data and is capable of dealing with Chinese instances. Crosslingual transfer learning with deep neural network has been shown effective in sentiment analysis [Chen et al., 2016], POS tagging [Kim et al., 2017], and question-answering [Joty et al., 2017]. In our model, cross-lingual information including the Universal POS tags 1 and the bilingual word embedding [Gouws et al., 2015; Vuli´c and Moens, 2015] are used to represent the sentences, and the sentence representation is further enhanced by using language-adversarial training.

The contributions of this work are three-fold. (1) This work shows the first study on Chinese information structure identification and demonstrates its application for Chinese writing assistant. (2) We propose a novel method that achieves a promising performance without the need of the labeled Chinese data. (3) Our system provides related knowledge for a submitted article with cross-lingual knowledge retrieval. The demonstration system is available online. 2

2 Method

Figure 1 shows an overview of our model for information structure identification with cross-lingual adversarial learning. The main task is to classify a sentence into one of five types of information structure. The adversarial learning task is to discriminate the language of a sentence. As denoted in Figure 1, the layers in gray are shared between the main network and the discriminator. Our goal is to train a languageneutral sentence representation and force the main network to learn without depending on language-specific information.

The input layer of the sentence representation is the word embeddings initialized with pre-trained bilingual word vectors. We do not update the word embedding layer during training because its cross-lingual capability will be violated when only English data are seen. The sentence representation is encoded by a GRU [Cho et al., 2014] layer.

In addition to the word sequence, the sequence of POS tags, and location features such as the position of the sentence in the abstract are taken into account. Both the Chinese and the English POS tags are converted to the Universal POS tags for the cross-lingual purpose. Note that language-specific features such as bag of words are excluded although some of them have been shown useful in the monolingual task.

1 http://universaldependencies.org/u/pos/

2 http://nlg18.csie.ntu.edu.tw/cisa

3 Evaluation

NTHU Academic Writing Database, 3 which consists of 597 labeled English abstracts in the EECS domain, is adopted as the training data. For evaluation, we manually annotate 100 Chinese abstracts in the EECS domain as the ground-truth. The statistics of both datasets are given in Table 1. From UMCorpus, a large-scale English-Chinese parallel corpus [Tian et al., 2014], we select 30,000 English and Chinese sentences in the Thesis domain for training the language discriminator.

As shown in Table 2, we evaluate our method in different settings. The main network for information structure identification is denoted as MAIN, the main network that is co-trained with the language discriminator is denoted as MAIN+ADV, and SEQ denotes that sequence modeling on the top of the model is employed to maximize the probability of the whole abstract. The Viterbi algorithm is performed to find the most likely sequence. En/En and Ch/Ch denote the monolingual performances by 10-fold cross-validation on the English and the Chinese datasets, respectively. En/Ch denotes the crosslingual performances, where the model is trained on the English data and tested on the Chinese data.

The cross-lingual MAIN model achieves an F-score of 58.17%, which is slightly superior to that of its monolingual counterpart trained on the small Chinese dataset. This result shows that the basic model, which regards the information from bilingual word embeddings and the Universal POS tags, is capable of cross-lingual generalization. Sequential modeling is useful for both monolingual and cross-lingual models.

3 http://writcent.nthu.edu.tw/writcent/

The language-adversarial learning strategy further improves the MAIN and MAIN+SEQ models. Co-training with a language discriminator successfully reduces the performance gap between the source and the target languages. The adversarial model with sequence labeling achieves a promising F-score of 71.93%, making a significantly improvement over all other models for the target language (p < 0.01).

4 Cross-lingual Knowledge Retrieval

We construct a KB with the results extracted by information structure identification. For each sentence in a collection of EECS abstracts written in both English and Chinese, we label its information structure type and insert it into the KB as a node. The relation between a node pair in the KB is denoted by the tuple of their types. For example, 〈Purpose,Method〉 denotes a relation between nodes a and b, where b is the method for the research goal a.

For an abstract submitted to CISA, each sentence will be labeled with its information structure type and linked to the related nodes with the same type in the KB. Various types of knowledge stored in the KB will be retrieved by using graph traversal. Finally, CISA will list the feasible approaches to user's research goal, the potential applications of the proposed method, and so on.

For the linking of cross-lingual information, the sentences in both English and Chinese are represented by using the bilingual word embeddings. Once all the nodes in the KB and the sentences in the submitted abstract are encoded as vectors in the same space, their relatedness can be computed by using cosine similarity. An evaluation on 100 abstracts shows our sentence representation achieves an MRR of 0.70 in cross-lingual sentence retrieval.

5 Conclusion

This work presents CISA, the first system for Chinese information structure analysis. We propose a model for Chinese information structure identification without the need of labeled Chinese data. The multi-label, finer-grained information structure analysis will be explored in the future.

Acknowledgements

This research was partially supported by Ministry of Science and Technology, Taiwan, under grants MOST-105-2221E-002-154-MY3, MOST-106-2923-E-002-012-MY3 and MOST-107-2634-F-002-011-. We thank the Writing Center at National Tsing Hua University for providing us the corpus.