Towards Generating Summaries for Lexically Confusing Code through Code Erosion

1 Introduction

Code summaries, also called code comments that summarize code functionality in natural language, are essential software components assisting in program comprehension. Due to developing time constraints, developers cannot write useful summaries for every code snippet, which leads to strong demand for automatic source code summarization.

As a hot topic in software engineering, many research works have been conducted for automatic code summarization. Conventional approaches mainly rely on manually crafted rules [Moreno et al., 2013] and information retrieval (IR) [Wong et al., 2013]. With the development of neural machine translation, more researchers try to model code summarization as a translation from code to natural language. CodeNN [Iyer et al., 2016] and CodeAttention [Zheng et al., 2019] utilize attention LSTM for code encoding and generating summaries. Approaches like SBT [Hu et al., 2018] attempt to extract structural semantics by taking abstract syntax tree as input. The aforementioned approaches, although effective, have a reliance on the correlation between code identifier tokens and summary words. To generate meaningful summaries, meaningful tokens must be observed in the code, which requires the code to be lexically precise.

∗Ming Li is the corresponding author.

However, code is not always lexically precise with welldefined identifier names. Firstly, source code files in the real-world are often obfuscated in which identifier names are replaced with short, opaque, and meaningless symbols (e.g. Figure 1(a)). Such obfuscation maintains the program semantics but diminishes code readability to hide business logic and prevent software plagiarism [Bavishi et al., 2018; Tran et al., 2019]. Secondly, it is nontrivial for developers, especially inexperienced developers, to choose appropriate identifier names (e.g. Figure 1(b)). What is worse, identifier names are subject to decay during software evolution [Deissenboeck and Pizka, 2006]. The concept they refer to may be altered or abandoned during the software evolution, which may mislead the summarization model to extract incorrect semantics. Such lexically confusing code places significant challenges for current approaches since the correlation between code tokens and summary words becomes implicit. What makes the challenge bigger is that lexically confusing code has few useful human-written summaries [Deissenboeck and Pizka, 2006], which causes a lack of data to learn to adapt to those circumstances.

One question arises here: how can we generate highquality summaries for lexically confusing code? One straightforward way is to replace all the identifier names with a single '<OTHER>' token and learns to map the replaced code to summaries [LeClair et al., 2019]. Such an approach can reduce the model's dependence on user-defined identifier names. However, collapsing all the identifiers to only one token changes the functional semantics without alleviating the difficulty in essence, which yields poor evaluation performance. Intuitively, every lexically confusing code snippet is like an eroded version of a lexically precise one, in which the functional semantics remains the same but the lexical semantics are eroded. Suppose that the model can learn the erosion process explicitly and restore the eroded semantics in a latent feature space, it is possible to generate reliable summaries even without precise lexical information.

This paper proposes a novel code summarization framework called VECOS (Variational Eroded COde Summarizapublic static int method1745(byte[] var0, int var1, CharSequence var2) {

int var3 = var2.length();

int var4 = var1;

for(int var5 = 0; var5 < var3; ++var5) {

char var6 = var2.charAt(var5);

………

(a) An obfuscated code snippet from a reverse engineer-

ing project whose identifiers are replaced with mean-

ingless symbols.

……… while(tmp1 != null && flag){ if(tmp2.getX() == tmp1.getX() && tmp2.getY() == tmp1.getY()){ tmp1.setNum(tmp1.getNum()+tmp2.getNum()); flag = false; ………

(b) Poorly written loop code with inappropriate identifier names whose lexical semantics is deficient.

Figure 1: Lexically confusing code examples.

tion) for lexically confusing code summarization. VECOS first simulates the lexical erosion process by converting every code snippet to a unified eroded version to adapt to various confusing identifier naming styles, in which the original identifiers are replaced with some pre-defined symbols. To learn the erosion process explicitly, we force the learned semantics representation of the eroded code to align with its lexically precise counterpart by variational reference. By doing that, VECOS learns how to "de-erode" code by restoring the eroded lexical semantics in a latent feature space. Experimental results show that VECOS can generate high-quality summaries for various lexically confusing circumstances.

The main contribution of this paper lies in two folds:

• We propose an erosion mechanism for code summarization to conquer the model's reliance on precise lexical information, and hence the problem of summarizing lexically confusing code can be formalized as learning from lexically precise code with an erosion process.
• We design a novel generative code summarization framework named VECOS for lexically confusing code summarization.

The rest of this paper is organized as follows: the second section explains our approach VECOS in detail. The third section shows the experiments together with some analysis. The fourth section introduces some related work. Finally, we conclude the whole paper.

2 Our Approach: VECOS

Let D = {(c1, y1), (c2, y2), ..., (cn, yn)} denotes a lexically precise code-summary corpus, where ci stands for a code snippet whose identifiers are well specified and yi stands for the related summary. Let M = (τ, ψ) represents our summarization framework VECOS, where τ is a rule-based erosion function which maps ci to its unified eroded version: ˜ci = τ(ci), and ψ is a tree-based neural summarization model generating code summaries ˜y′i = ψ(˜ci), where the generated summary is expected to be similar to the humanwritten one: ˜y′i ≃ yi. The training goal is, for any test case c′j , no matter how confusing the lexical information of c′j is, the generated summary ˜y′j is always reliable.

Unlike existing approaches, we take a code erosion process τ as the first step for code summarization. τ replaces userdefined identifiers with pre-defined symbols without changing functionality: ˜ci = τ(ci). For the training phase, such replacement forces the summarization model to learn semantics from the eroded code, cutting off the dependence on userdefined identifier tokens. For the testing phase, τ acts as a process of data standardization by converting various lexically confusing code to a unified eroded version, which is beneficial to improve the generalization ability.

We take the abstract syntax tree of eroded code ˜ci as the input for ψ's encoder. We assume there exists a continuous latent feature space Z for code functionanlity representation, from which ψ infers the latent variable zi with given ˜ci, i.e. pψ(zi|˜ci). Then zi together with ˜ci will be used to guide the summary decoding, i.e. p(yi|zi, ˜ci). To learn the erosion process explicitly, we leverage the origin lexically precise code ci and corresponding summary yi to pre-train an auxiliary model φ. Since ci and ˜ci share the same functionality, we force the distribution pψ(zi|˜ci) inferred by ψ to assimilate pφ(zi|ci) inferred by auxiliary model φ using variational inference. By doing that, the eroded semantics is expected to be restored in the latent feature space Z.

The overall framework of VECOS is shown in Figure 3. An auxiliary model φ is first trained with the translation cross entropy loss. Since the input for φ is lexically precise, we take pφ(zi|ci) as the lexical enriched prior of pψ(zi|˜ci). Our summarization model ψ is trained by exploiting both the translation supervision and the divergence between the prior and posterior distribution via variational inference.

2.1 Code Erosion

The code erosion part τ is responsible for converting arbitrary code snippets to a unified eroded version by replacing userdefined identifiers with ordered symbols. We introduce code obfuscation techniques to complete the erosion process τ . An example is provided in Figure 2. With function granularity, user-defined identifiers (the referenced external API names are not included) can be divided into function names, parameter names, and local variables. We take 'func', 'para', and 'var' together with an order suffix to make the replacement. Our method preserves complete functional semantics and part of the identifier type information, which avoids ambiguity in functionality representation.

We design τ for two reasons. Firstly, most code snippets of the current code-summary corpus have well-defined identifiers correlated to code functionality. For example, for the Java code-summary corpus in [Hu et al., 2018] and [Husain et al., 2019], the identifier tokens can be found in the summary directly for more than 65% code-summary pairs. To prevent the model from relying on such lexical correlation, we introduce τ to conduct the erosion process to avoid exposing sufficient lexical information directly to the model. Secondly, the identifier naming of lexically confusing code is quite flexible since identifiers can be arbitrarily chosen and elude automated analysis. Various confusing naming styles can be unified as a single eroded version with the erosion process, which alleviates the summarization model's learning burden and enhances generalization towards unseen data.

2.2 Code Summarization

We propose a generative tree-based model ψ for code summarization. Previous discriminative neural summarization models usually rely on the attention mechanism to identify semantics alignment between code fragments and summary tokens. The functional semantics is learned implicitly. However, such alignment may deviate when the lexical correlation between the code and the summary is weak, which is common under lexically confusing circumstances. To tackle this problem, we introduce a latent variable z to represent code functional semantics explicitly. z is inferred from the given eroded code ˜ci, and used as a global semantics indicator for summarization. The encoder of ψ learns to approximate pψ(z|˜ci) while the decoder learns to approximate pψ(yi|z,˜ci). The generative process can be formulated as Equation 1:

Tree-based Encoder with Latent Variable

The tree-based encoder takes the abstract syntax tree (AST) of code as input. As shown in previous works [Hu et al., 2018; Wan et al., 2018; LeClair et al., 2019], AST is capable of representing complex structural semantics of source code. We build our encoder based on Tree-LSTM [Tai et al., 2015], which is a generalization of LSTM to tree-structured network topologies, traversing a single tree from leaf nodes to the root node. [Tai et al., 2015] propose two kinds of Tree-LSTM: N-ary and Child-Sum. Considering the order of child nodes is necessary, we choose N-ary and set N as 2, in which N is the number of child nodes. Since traditional AST does not restrict the number of child nodes, we transform the original AST to a binary left-child-right-sibling tree.

Tree-LSTM outputs encoding for every node of AST, in which the encoding of the root node her is the synthesis of the whole tree. We parameterize the approximate posterior as pψ(z|˜ci) = N(µ, σ2 ), in which the mean and variance vector is calculated from the encoding of the root node her :

In Equation 2, we introduce tanh as the activation function. For one thing, tanh increases the non-linearity of the model to improve the representation capability. For another, tanh's lower bound prevents the value of σ from being so small that the model degenerates into a discriminative model. To ensure the gradient can be transferred to the encoder, z needs to be reparameterized using µ and σ:

Equation 3 is the Reparameterize Technique, in which ⊙ refers to the element-wise product and ϵ is a variable sampled from standard Gaussian distribution: ϵ ∼ N(0, 1).

Sequential Decoder with Attention

We take the vanilla sequential LSTM with the dot attention mechanism to decode the encoded information to code summaries. The decoding process can be formulated as:

In Equation 4, hdi , sdi are the hidden state and cell state of LSTM at time step i. oi is the output word distribution of next generated summary token. Square brackets expression like [hdi , z] means concatenation of two vectors. As shown in the last equation of 4, latent variable z is feed into the output layer together with current decoding hidden state hdi at every time step for generating summary tokens. ci-1 is the attentional weighted sum of encoders outputs: ci-1 = ∑Nj=1 αT ij hej , in which αij is the attention score of previous decoding hidden state hdi-1 and encoding output hej :

2.3 Training via Variational Alignment

Since the user-defined lexical information is eroded, it is nontrivial for the summarization model ψ to learn a well-formed semantics distribution of z for the eroded code ˜ci. Thus we propose to pre-train an auxiliary summarization model φ. φ shares the same network structure as ψ and is trained with the original lexically precise code corpus D = {(ci, yi)}:

We train the auxiliary model φ by minimizing the crossentropy loss Lφ. For one thing, φ can learn the underlying semantic distribution more effectively with ci since it contains more lexical information than ˜ci: pφ(z|ci) ≃ p(z|ci). For another, the erosion process τ does not change functionality. Thus the distribution of z should be the same given ˜ci or given ci: p(z|˜ci) = p(z|ci). As a result, pφ(z|ci) can be treated as a lexical-enriched prior guiding the learning of pψ(z|˜ci).

Inspired by stochastic gradient variational Bayes [Kingma and Welling, 2013], we propose a variational training framework to align pψ(z|˜ci) with pφ(z|ci) by approximating maximum likelihood estimation of log pψ(yi|˜ci):

Equation 6 is the ELBO (Evidence Lower BOund) of the log likelihood. The encoder and decoder of ψ are used to model pψ(z|˜ci) and pψ(yi|z,˜ci) respectively. The first term of Equation 6 is the negative cross-entropy of generating target sequences yi. The second second term is the Kullback-Leibler(KL) divergence between learned pψ(z|˜ci) and its prior p(z|˜ci), which is assigned as pφ(z|ci). For clarity, we use L(i) ce and L(i) kl to represent cross entropy and KL

divergence for a given pair (˜ci, yi):

The overall training objective for ψ is to minimize:

In Equation 9, we introduce a λkl to balance the translation loss and KL loss as a hyperparameter. Since variational encoder-decoder framework is tricky with RNN models, λkl is used to anneal the KL divergence to balance the value of the two terms [Bowman et al., 2015; Bahuleyan et al., 2018]. The training process of ψ can be regarded as a combination of a translation task by reducing cross entropy and distribution aligment by reducing KL divergence.

3 Experiment

To evaluate the effectiveness of VECOS, we conduct experiments on a benchmark dataset and compare it with several state-of-the-art approaches for code summarization.

3.1 Dataset Preparation

We take the same dataset as [Hu et al., 2018], in which java code snippets are provided in functional granularity with corresponding summaries. For roughly 65% code summary pairs in this dataset, the identifier tokens can be found in the summary directly, which indicates the original identifiers are lexically precise to reveal code functionality. We do some filtering to this dataset. The class constructors, setters, getters, tester methods, and code snippets whose summaries are shorter than four words, are dropped. While our summarization model takes the tree-based data structure as input for the encoder, we cannot truncate it to a fixed size. So we filter out some extremely long code to avoid large zero padding. Code summaries vary in length. We only take the first sentence of a single summary and fix the length as 10. All the summary tokens are stemmed. Consequently, we obtain 274,302 code-summary pairs. We randomly split the dataset as 8:1:1 to construct datasets for training, validating, and testing. We provide two ways to replace original identifiers in the test set to simulate lexically confusing circumstances for testing:

• Symbolic replacement: well-defined code identifiers are replaced with ordered symbols such as 'var0', 'var1', which simulates obfuscated circumstances.
• Synonym replacement: sub-tokens of identifiers are replaced with synonyms. For example, 'upStreaming' being replaced as 'upFlow', in which 'streaming' and 'flow' are similar in general context but different in professional domains. Synonym replacement simulates inappropriate and flexible variable naming styles. The synonym is chosen by calculating the neighborhoods of word embeddings according to pre-trained GloVe embedding [Pennington et al., 2014].

We replace identifiers of each test case according to varied ratios (0%, 25%, 50%, 75%, 100%), where 0%means no replacement so the code is still lexically precise and 100% means replacing every identifier. Both symbolic and synonym replacement will not change the functional semantics.

3.2 Experimental Settings

We use average BLEU [Papineni et al., 2002] and Ribes [Isozaki et al., 2010] as evaluation metrics to evaluate our approach measuring the correspondence between a generated summary and several output references. BLEU calculates the average precision over the n-gram mechanism with a penalty for short sequences, whose value varies from 0 to 100 as a percentage. Ribes takes into account the rank correlation coefficients with word precision, varying from 0 to 1. Higher BLEU and Ribes scores indicate better evaluation performance.

We choose eight approaches as baselines:

• CodeNN, AttnGRU, and Transformer [Iyer et al., 2016; Cho et al., 2014; Vaswani et al., 2017]: Sequential neural approaches which treat code as plain text. CodeNN is the first neural-network-based approach based on LSTM for source code summarization. AttnGRU is a bi-directional GRU model with an attention mechanism. Transformer is a famous self-attention model proposed for neural machine translation.
• SBT [Hu et al., 2018]: SBT is short for Structure-Based Traversal, a new way of converting an abstract syntax tree into a node sequence. An attention-based multilayer LSTM processes the SBT node sequences to generate code summaries.
• AST-AttendGRU [LeClair et al., 2019]: A structural GRU-based model specially designed for code with zero internal documentation. It replaces identifier names with a single '<OTHER>' token and takes the SBT sequence of the replaced AST as input.
• Tree-LSTM [Tai et al., 2015]: A generalization of LSTM to tree-structured network topologies, which is used as the encoder to process AST. The decoder is vanilla attentional LSTM.
• Tree-COS, ECOS: Two variants of VECOS. Tree-COS refers to the auxiliary model φ described in Section 2.3 without erosion part τ. ECOS shares the same model structure with VECOS but is trained without variational semantics alignment.

For our generative summarization model ψ of VECOS, both the input size and hidden size of LSTM are set as 256. The size of the latent variable z is 100. Both the internal LSTM of the encoder and the sequential LSTM for the decoder has only one layer. The initial learning rate is 0.01, and it will decay on learning plateaus with a decaying speed of 0.5. We take the sigmoid annealing strategy to adjust the coefficient λkl dynamically from 1.0 to 0.01 [Bowman et al., 2015]. The weight decay factor for L2-regularization is set as 0.00001. The batch size for the training set is set as 100. The training process will stop when the performance of the validation set stops increasing for three epochs. For a fair comparison, the embedding size and hidden size of baselines are all set as 256. All the experiments are conducted with a machine equipped with one Intel Xeon E5-2620 CPU, 32GB RAM, a Nvidia Titan X graphic card with 12GB graphic storage. The operating system is Ubuntu 16.04.

3.3 Results and Analysis

Let δ denotes the replacement ratio. Table 1 and Figure 4 show the BLEU and Ribes scores of different models on test data with different δ and replacement approaches. From the statistics, VECOS significantly outperforms other approaches for various lexically confusing circumstances (δ > 0 ). When code is lexically precise (δ=0), VECOS's performance is still comparable with previous approaches. The following is a detailed comparative and ablation analysis.

Comparative Analysis

Without any replacement (δ=0), the auxiliary model (TreeCOS) achieves the best performance. Compared with TreeLSTM, Tree-COS improves 2.65 BLEU points, indicating the effectiveness of introducing latent variable z to represent functional semantics explicitly. Other approaches that considering structural information (Tree-LSTM, SBT) performs better than approaches that treat code as plain text (CodeNN, AttnGRU, Transformer). Although equipped with the erosion part, VECOS can still achieve comparable performance with most of the baselines for lexically precise code.

When we replace parts of the precisely defined identifiers (δ > 0), the performance of sequential approaches (CodeNN, AttnGRU, Transformer) drops sharply. Even 25% replacement will result in half of the performance loss. When δ reaches 100%, their BLEU scores go below 10. Although SBT considers structural information by taking code's flatten AST as input, it still performs poorly when δ > 0. We think this is because the sequential representation of AST is susceptible to disturbance. It is hard for neural models to adapt to such unseen data disturbance. Approaches that retain real tree-structured network topologies have better generalization performance under various lexically confusing circumstances, like Tree-LSTM and Tree-COS. However, they still suffer a decline when δ rises.

VECOS, ECOS, and AST-AttendGRU have a stable performance for different δ. This is because the replacement mechanism of AST-AttendGRU and the erosion process τ of VECOS and ECOS replace identifier names with pre-defined symbols aforehand. As δ increases, those approaches' advantage over other baselines gets more prominent. For the evaluation results, VECOS yields much better performance than AST-AttenGRU for both BLEU and Ribes. We think there are two main reasons. Firstly, AST-AttendGRU replaces all the identifiers with a single '<OTHER>' token, which changes the original code's functional semantics. Secondly, ASTAttendGRU attempts to learn the mapping from the replaced code to summaries directly, which is quite challenging because of the lack of lexical information and the changed functionality.

Ablation Analysis

We compare the performance of Tree-COS, ECOS, and VECOS to verify the effectiveness of each component of our approach. Taking away the erosion process τ, Tree-COS can achieve the highest BLEU score of lexically precise code (δ = 0) but suffers decline when δ goes up. Equipped with the erosion part τ but without variational alignment training, ECOS can achieve a stable BLEU score of 23.91. With the variational alignment training, VECOS can improve the BLEU score to 26.28, which is a 10 percent increase over ECOS. In summary, the combination of erosion process and variational alignment ensures VECOS's excellent generalization towards various lexically confusing circumstances.

4 RelatedWork

Code Summarization

Automatic source code summarization is a hot topic in software engineering. Early studies in this field usually utilize techniques of text summarization or information retrieval for summary generation [Haiduc et al., 2010]. Approaches like AutoComment [Wong et al., 2013] try to leverage codedescription mappings collect from StackOverflow and generate comments via code similarity. These approaches are usually evaluated by experienced programmers manually.

Since neural networks have achieved remarkable success in machine translation, researchers start to model this problem as a translation from programming language to natural language. CodeNN [Iyer et al., 2016] and CodeAttention [Zheng et al., 2019] utilize attention LSTM to generate summaries sequentially. To extract structural information, [Hu et al., 2018] propose SBT as a new way to traverse abstract syntax tree so that sequential models can encode tree-structural data. Both AST-AttendGRU [LeClair et al., 2019] and HybridDRL [Wan et al., 2018] try to encode structural and textual information independently and fuse for later comment decoding. Hybrid-DRL [Wan et al., 2018] introduces reinforcement learning into this field for decoding to avoid exposure bias problem. AST-AttendGRU [LeClair et al., 2019] proposes to generate code summaries without internal documentation.

Variational Neural Model

Variational AutoEncoder (VAE) is the first variational neural model proposed by [Kingma and Welling, 2013] for image generation, in which the prior of the latent variable is assigned as a standard Gaussian. The following works like [Zhang et al., 2016; Serban et al., 2017] extend it to a general variational encoder-decoder (VED) form for various NLP tasks like machine translation, sentence generation, and dialog systems. With the help of latent variables, those approaches usually yield better performances than vanilla attentional RNN models. On their basis, [Bahuleyan et al., 2018] take a step forward by modeling attention mechanism via variational inferences to avoid bypassing phenomenon.

5 Conclusions

This paper proposes a novel framework called VECOS to generate reliable summaries for lexically confusing code. VECOS introduces an erosion process to map arbitrary source code to a unified eroded version to fit various realworld circumstances. With the help of variational alignment training, VECOS learns to extract functional semantics in a latent space Z by explicitly modeling the erosion process. The evaluation results show that VECOS is reliable to generate coherent summaries for various lexically confusing code. An interesting future work can be applying variational semantics alignment to more general scenarios such as code representation learning with data augmentation.

Acknowledgements

This research was supported by NSFC (62076121) and National Key Research and Development Program (2017YFB1001903).