Teacher-Student Networks with Multiple Decoders for Solving Math Word Problem

1 Introduction

Introducing the math word problem (MWP) aims to develop machine learning models, i.e., automated solvers, to answer mathematical questions given in texts. It has been an interesting natural language understanding task for more than a half decade [Bobrow, 1964] and attracted much attention in recent years [Koncel-Kedziorski et al., 2015; Shi et al., 2015; Huang et al., 2017; Wang et al., 2018b; Liu et al., 2019; Xie and Sun, 2019]. The MWP solving task is challenging due to three reasons. (1) Math problems are often stated in diverse word scenarios. (2) There are strong semantic gaps between language logic and math expression. (3) It is not obvious how to model the mathematical laws, e.g., commutative law of summation and multiplication, in machine learning paradigms.

* Corresponding Author

1 Code and datasets: https://github.com/2003pro/TSN-MD

Typically, an MWP is composed of a problem description text, a solution expression in math and a final answer as a number. An example is given in Figure 1. With such training examples, MWP solvers should have two objectives to optimize, i.e., the generation loss of answers, as well as the loss of solution expressions which indirectly minimizes the answer loss. While, most existing approaches use only the latter objective. In specific, their solvers are required to identify the relevant numbers and compose solution expression from the inferred relations between/among those numbers [Wang et al., 2017; Wang et al., 2018a]. The state-ofthe-art framework is based on the deep generative neural network composed by an encoder and a decoder [Xie and Sun, 2019]. The encoder encodes the problem text into latent embedding which is then fed into the decoder to generate the solution expression. The solution expression is represented by either a sequential [Wang et al., 2017; Wang et al., 2018a; Chiang and Chen, 2019] or a tree structure [Liu et al., 2019; Xie and Sun, 2019]. Most of such frameworks deploy single decoder and fit to the only one solution expression given on datasets. It is, therefore, hard for them to generate diverse but equivalent solution expressions, limiting the generalization to various word scenarios.

In order to overcome these limitations, we propose a novel and effective MWP solver which we call Teacher-Student Networks with Multiple Decoders (TSN-MD). We pre-train the teacher network to guide the learning behavior of the student network with the same training data. The intuition is that the teacher observed through all solutions including those with equivalent solution expressions under mathematical laws. It gains the knowledge of regularizing the learning behaviours of the student network. More intuitively, when student predicts an equivalent solution expression to the ground-truth but causes an undesirable loss, the teacher network will help to amend this mistake. In addition, to encourage high-diversity predictions (i.e., solution expressions) from the student network, we use multiple decoders, e.g., one decoder is conventional and the other one has its input embedding perturbed by a Gaussian noise.

Contributions. Our main contributions are three-fold. (i) A novel and effective teacher-student approach that addresses the key limitations of solution expressions existing on the MWP data. (ii) A multiple-decoder architecture that works for our teacher-student networks to improve the diversity of generated expressions. (iii) Extensive experiments and ablative studies on two large-scale MWP benchmarks - Math23k [Wang et al., 2017] and MAWPS [KoncelKedziorski et al., 2016], showing the superiority of the proposed approach over related works. In addition, the visualized results show that our model is really generating diverse and equivalent expressions of the solution, explicitly revealing the reason behind our superiority.

2 RelatedWork

Math Word Problems. In the context of math word problems (MWP), the algorithms are designed to calculate the results of mathematical questions given in a natural language. Therefore, the deep learning community tends to regard MWP as one sort of natural language process (NLP) task. Most of these existing methods adopt an encoder-decoder framework, where the encoder is given the representations of problem text and the decoder generates the corresponding solution expression. For instance, Wang et al. [2017] proposed a large-scale MWP dataset and applied a vanilla sequence-tosequence (Seq2Seq) model to translate the problem text to a solution expression.

The later works took more exclusive characteristics of the problem into account progressively. For instance, considering that the composition of the equations is based on some clear regulars, Wang et al. [2018a] introduced an equation normalization method to normalize the duplicated equations. Inspired by the stack mechanism in the data structure, Chiang and Chen [2019] introduced the copy mechanism with a stack. Li et al. [2019] introduced MWP specific priors into the intermediate representations.

The recent studies have attempted to improve the generation of solution expressions using sequence-to-tree (Seq2Tree) models [Wang et al., 2018a; Chiang and Chen,

2019; Wang et al., 2019; Liu et al., 2019; Xie and Sun, 2019]. For example, the Goal-driven Tree Structure model (GTS) [Xie and Sun, 2019] achieved the state-of-the-art performance by guiding the model to generate specific solution equations using a tree structure.

However, all these methods suffer from same limitation. In the beginning, researchers [Wang et al., 2018a] suggest that the equation search space in the decoding process is large with much ambiguity in solution expression. Therefore, these models are optimized to generate a fixed target solution expression instead of finding the correct answers. This constraint to fit an MWP to single solution expression ignores the possibility that an MWP can be solved by multiple solution expressions. In many cases, the model will falsely penalize the correct generated solution expression, which introduces undesired bias.

Knowledge Distillation. Knowledge distillation (KD) was first introduced by Hinton et al. [2015]. Aiming at compressing neural networks, it transfers knowledge from a complicated high-performance teacher model to a simple student model without losing too much performance. To this end, Hinton et al. [2015] reshaped the training procedure, where they regarded the predictions of the teacher model as "soft labels" and trained the student model to fit the soft labels. Romero et al. [2015] trained the student model to mimic not only the final outputs of the teacher model but also the intermediate feature map of the teacher model. This can help to narrow the performance gap between teacher and student models.

While knowledge distillation was firstly proposed for model compression, the intuition behind the soft labels was applied widely in different tasks and domains. Furlanello et al. [2018] and Zhang et al. [2018] found that applying the soft labels as the training target can help the student model achieve better performance. Zagoruyko and Komodakis [2017] showed that the teacher-student learning framework managed to distill the attention map from a strong teacher model to the student model.

Some works also explored applying knowledge distillation in natural language processing tasks. Kim and Rush [2016] introduced a sequence-level distillation framework to distill the structure loss for neural machine translation. Knowledge distillation was also applied in other tasks such as text classification [Zagoruyko and Komodakis, 2017], parsing [Kuncoro et al., 2016] and machine reading comprehension [Hu et al., 2018]. To the best of our knowledge, this is the first work that explores the usage of knowledge distillation on MWP task.

Multiple-Decoder Networks. Multiple-decoder networks are mainly applied in machine translation and image captioning tasks to generate diverse outputs [Jacobs et al., 1991; Eigen et al., 2014]. For instance, Yang et al. [2018] proposed to use multiple softmax heads after the encoding by recurrent neural networks. Shen et al. [2019] and He et al. [2018] used multiple decoders with uniform mixing coefficient to improve diversity in machine translation. In this paper, we adapt multiple decoders to generate diverse solution expressions for a given MWP.

3 Teacher-Student Networks withMultiple Decoders (TSN-MD)

As illustrated in Figure 2, the proposed TSN-MD is composed of a teacher network and a student network. The student network includes an encoder and two decoders. Teacher network is pre-trained (in the baseline way) and it regularizes the learning behaviour of the student network by reducing the loss when student predicts solution expressions that are mathematically equivalent to the ground-truth. Here, we note that student network has one more decoder than teacher network in order to output more diverse solution expressions.

Notations. We denote a math word problem X by a set of n words {xi}ni=1 , whose word embedding vectors {ei}ni=1 are obtained through a linear projection of word embedding matrixE, i.e., ei = E xXi , and its corresponding ground-truth solution expression by Y .

Considering choosing important numbers from problem text, we follow Wang et al. [2017] which proposed to mapping numbers into special tokens following these two rules: 1) All numbers appeared in the problem text are determined if they are significant numbers, which means they will be used in the solution expression. This is the so-called Significant Number Identify (SNI) mechanism [Wang et al., 2017]; 2) All the recognized significant numbers in a problem X are mapped to a list of mapped position symbols {n1, ..., nl} according to their appearing orders in the problem text. Simultaneously, numbers in solution expression are then mapped as the position symbols {n1, ..., nl}. For example, the position symbols and mapped solution expression of the problem in Figure 1 are {n1 = 20, n2 = 10, n3 = 200, n4 = 12} and (n3-n4×n2)÷n1 respectively. Then, solution expressions are normalized to prefix expression following the pre-defined rules.

Our baseline model is the goal driven tree structure solver (GTS) [Xie and Sun, 2019]. Based on its architecture, we introduce how to build our TSN-MD - the teacher-student networks with one conventional decoder as well as one perturbed decoder.

3.1 Teacher-Student Networks

It is often the case that there exists correct solution expressions different from the annotation solution expression. Previous models will falsely penalize these correct solutions as the training objective only fits the fixed solution expression.

For example, the label solution expression of the problem in Figure 1 is (200 - 12 × 10) ÷ 20, although 200 ÷ 20 - 12 × 10 ÷ 20 is also a correct solution expression and can compute the right answer. The previous training objective will undoubtedly penalize this kind of generation results.

In the empirical analysis on GTS's generated solution expression, we found that the number of exactly matched generated solution expressions is lower than generated solution expressions that can get the correct answer. That is to say, GTS model can find some correct solution expressions different from annotated ones. This also demonstrates the existence of the undesired bias in training objective.

A simplest solution is to improve solution annotations and add extra training examples. However, the current datasets all only give the single solution expression annotation. Here, based on the interesting fact that GTS model can find some correct new solution expressions, we propose to use teacherstudent learning, which is also known as knowledge distillation, to transfer these kinds of multi-solution knowledge to the student model.

More concretely, consider the classical MWP solving setting where we have training set consisting of tuples of problem texts and solution expressions (x, y) with possible classes V . We aim at obtaining the function f(x) : X ↦→ Y , which can achieve good results in unseen data. Commonly, the function f(x) is parametrized as a neural network f(x, θ). θ represents parameter of this neural network. The common training criteria is to minimize negative log-likelihood (NLL) loss for each example from the training data:

where 1 is the indicator function and p is the output distribution from the model with parameter θ. From the viewpoint of cross entropy, this loss function can be regarded as the cross entropy between the degenerate data distribution (with all the probabilities focusing on one class) and the model output distribution.

In teacher-student network, we assume to have access to a well-trained teacher network parametrized with θT . More precisely, we choose GTS [Xie and Sun, 2019] as our teacher network. The output distribution q(y|x; θT ) of teacher network can provide extra source of training information and this has been proved in previous work. Thus, we tried to minimize the cross entropy between our network's output distribution and teacher network's output distribution:

The student network should not only aim at fitting the teacher network's output, which has the risk of being limited by the teacher's performance, but also learn from the original label. Thus, we interpolate between the two loss functions:

where α is the interpolation coefficient.

3.2 Multiple Decoders

If only simply borrow the idea of teacher-student learning, we may suffer from false guidance given by teacher model. After all, teacher model still predicts false results in test set. In other words, if the teacher model falls into some local optimization point, the student model possibly also falls into the similar local optimization points. In addition, we suggest teacher-student learning is not enough to overcome the issue. In order to alleviate this extra undesired bias given by teacher model, we propose to use multiple decoder structure to further increase the possibility of getting diverse solutions.

More precisely, this novel multiple decoder structure is composed with one conventional decoder and many perturbed decoders. This can improve both prediction accuracy and diversity.

The Overall Multiple Decoder Architecture

Our TSN-MD model naturally incorporates diversity in the architecture both during training and inference. Furthermore, we try to encourage the diversity by feeding these decoders with different latent variable inputs.

Perturbations to Get Multiple Decoders. In the student network, all of the decoders share the encoder, i.e., this model requires processing the input sentence only once. Aiming at getting different outputs from different decoders, we try to feed latent variables to different decoders. Here, except for the conventional decoder which is directly fed with encoder output, we introduce perturbed decoder. In this perturbed decoder, perturbation masks Maskp are added to the output of encoder H to get different input Hl for perturbed decoder:

where ⊗ represents element-wise production. In order to get Maskp, we first define the mask rate Pmask. Next, we sample Pmask percent region in H according to Gaussian distribution. we define a zero matrix Maskzero with the same shape of H and assign one to positions that locate in the region.

Diversity Regularization. After getting the masked latent variable set H = {h1, h2, · · · }, we feed hl to the l-th tree decoder and get outputs yl. In order to encourage different decoders to generate different results, we introduce the diverse regularization item. More specifically, we adopt the cosine similarity to quantify the difference of different outputs generated by different decoders. Our intuition is consistent with promoting the diversity of generation. For two output solution expressions (yl, yl+1) , if they are far apart, the regularization item should give larger penalty:

where t represents the position of word, and l and l1 mean two outputs from two different decoders. The regularization loss on the completed sequence is:

where T means the length of generated sequence.

Separate Beam Search per Component. When it comes the inference, separate beam search is executed per mixture component. As different components tend to focus on different information and grab different patterns, performing independent beam search encourages the diversity of generation. Note that we have |H| different components and the beam size as b. Overall, this requires processing |H|×b candidates at each step.

It is worth noticing that the solver can only output one solution expression. Therefore, we vote to get the output solution expression according to the prediction score given by different decoders.

3.3 The Overall Training Objectives

In the stage of training teacher network, we follow GTS training process [Xie and Sun, 2019] and save the output classification probability distribution of training set as the soft label. In the stage of training student network, we feed both soft label and one-hot hard label to the model. Also, the diversity regularization term and teacher-student learning objectives are combined. Consequently, the overall objective function is formed as:

where β is the weight of the regularization term and N is the number of decoders.

4 Experiments

In this section, we conduct extensive ablation studies to validate the effectiveness of our contributing components - teacher-student networks and multiple decoders. We also demonstrate a case study to show the effects by using our proposed methods.

Datasets. Two large-scale MWP benchmarks, i.e., Math23K [Wang et al., 2017] and MAWPS [KoncelKedziorski et al., 2016], are used in our experiments.

Baselines. We compare TSN-MD to following baseline and state-of-the-art models: DNS [Wang et al., 2017] uses a vanilla seq2seq model to generate expressions. MathEN [Wang et al., 2018a] proposes an equation normalization to reduce target space. T-RNN [Wang et al., 2019] applies recursive neural networks over the predicted tree-structure templates. S-Aligned [Chiang and Chen, 2019] introduces a stack structure decoder to track the semantic meanings of operands. GROUP-ATT [Li et al., 2019] applies the idea of multi-head attentions from Transformer [Vaswani et al., 2017]. AST-Dec [Liu et al., 2019] proposes to generate expression tree by a tree LSTM decoder. GTS [Xie and Sun, 2019] develops a tree structured neural network in a goaldriven manner.

Implementation Details and Evaluation Metric In TSNMD, we use a word embedding with 128 units. The dimension of the hidden state for all the other layers are set to 512. Besides, we use two heads decoder in TSN-MD. Our model is trained for 80 epochs. Mini-batch size and dropout rate are set to 64 and 0.45, respectively. For optimizer, we use Adam with learning rate set to 0.001, β1 = 0.94 and β2 = 0.99, and the learning rate will be halved every 20 epochs. Also, We use a beam size of 5 in beam search.

4.1 Overall Results

The main results are shown in Table 1. For evaluation with test set on Math23k, the single model performance achieved by our proposed TSN-MD achieves the state-of-the-art. As the code for GTS is made available 2 , we implemented GTS and tested it on all dataset settings. The empirical results show the effectiveness of teacher-student learning and mixture of decoder structure. On all the benchmark datasets, our model achieves the new state-of-the-art performance.

4.2 Solution Expression Prediction Analysis

We report the accuracy of solution expression prediction by the sequence-to-sequence model (Math-EN), sequenceto-tree model (GTS) and our TSN-MD in Table 2. If the predicted solution expression exactly matches the annotated solution, we consider it a positive example. Here, we can compute the number of alternative solution that each model can provide, which can partly tell the degree of diversity of a model. It can be observed that the accuracy of solution expression generation is lower than the final accuracy of problem solving. This reflects that these MWP solvers have spontaneously learned to adapt the situation that a certain problem may have multiple solutions.

4.3 Ablation Study and Parameter Analysis

Effect of Different Components

To better understand the performance contributed by different components adopted in this paper, we perform a series of ablation tests. All the performance in ablation study is evaluated by the test set on Math23k.

We investigate the effects from different modules. The results of different MWP solver variants on Math23K are shown in Table 3. Taking GTS as the base model, we try to verify the effectiveness of proposed techniques. First, we evaluate the vanilla teacher-student network by removing multiple decoder structure. The results show that the teacher-student network contributes a lot to improve the performance. Next, we observe that it is beneficial to use multiple decoders. Using 2 decoders with diversity regularization item, the accuracy improved by approximately 0.6%. We also notice that it is necessary to include the diversity regularization techniques. We ablate the regulation item both for the full model and the model with only multiple decoders structure. The results in Table 3 show performance degradation to different extents. In addition, we have verified that "GTS+Teacher-Student Learning+Multi-Decoder" and "GTS+Teacher-Student Learning" in Table 3 solve different parts of MWPs while sharing accuracy. It is not fair to infer that "Multi-Decoder" is unnecessary from this comparison. Compared with single decoder, the model can get more different solutions with multiple decoders. In this case, diversity regularization item can have larger influence. Therefore, although the impact on the multiple decoders without teacherstudent network is relatively small, the influence on the full model is large.

2 https://github.com/ShichaoSun/math Seq2Tree

Parameter Analysis

Mask Rate. The percentage of mask rate is a tune-able hyperparameter in our TSN-MD. Thus, we investigate the effect of the percentage of mask rate on our model's performance. Table 4 suggests 5% achieves the best performance.

Dynamic or Static. We explore whether to fix the mask in the beginning (static) or dynamically change it during training (dynamic). The results show that in most cases, the static strategy can get better results. We argue that as the dynamic mask keeps changing the decoder inputs, it is harder to get effective information.

Number of Decoders. At last, we investigate the effect of increasing number of decoders. As adding decoders will bring a lot of extra computation cost, we only examine the effect of adding to 3 decoders. The experiment results show that the 2 decoder is the best configuration.

Besides, Table 4 has also shown the importance of random perturbation.

4.4 Case Study

We present several generation examples sampled from the Math23k dataset in Table 5. In the first example, the problem is asking the proportion of the fired ones. However, GTS falsely predicts the solution expression to compute the proportion of the remaining members and Decoder1 gets the same error. Our model gives the right solution with the help of Decoder2 exploring one more possibility. For the second example, GTS generates a new solution different from the ground-truth. Two decoders also give diverse correct solutions. For the third example, there are key words in the problem text like "quotient" and "remainder" while the solution actually needs "×" and "+". This case asks for model to handle the confusing descriptions. Our TSN-MD can better deal with this situation.

5 Conclusions

In this paper, we propose a novel approach Teacher-Student Networks with Multiple Decoders (TSN-MD) to tackle the MWP problems. The key components in our approach, i.e., teacher-student regularization and multiple decoders in student network, are validated to be of high effectiveness. The new state-of-the-art results are achieved on the Math23k and MAWPS benchmarks. Furthermore, our approach is actually generic and should be easy to plug in other WMP solvers. Exploring such possibility can be an interesting direction for our future work.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61832001 and No. 61672133), Sichuan Science and Technology Program (No. 2019YFG0535 and No. 2018GZDZX0032), the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centres in Singapore Funding Initiative. We thank all reviewers for their valuable comments.