{"type":"ScholarlyArticle","title":"Teacher-Student Networks with Multiple Decoders for Solving Math Word Problem","authors":[{"name":"Jipeng Zhang","slug":"jipeng-zhang","given":"Jipeng","family":"Zhang","affiliation":"Center for Future Media , University of Electronic Science and Technology of China; School of Information Systems , Singapore Management University"},{"name":"Roy Ka-Wei Lee","slug":"roy-ka-wei-lee","given":"Roy Ka-Wei","family":"Lee","affiliation":"Department of Computer Science , University of Saskatchewan"},{"name":"Ee-Peng Lim","slug":"ee-peng-lim","given":"Ee-Peng","family":"Lim","affiliation":"School of Information Systems , Singapore Management University"},{"name":"Wei Qin","slug":"wei-qin","given":"Wei","family":"Qin","affiliation":"School of Information Systems , Singapore Management University; School of Computer Science and Information Engineering , Hefei University of Technology"},{"name":"Lei Wang","slug":"lei-wang","given":"Lei","family":"Wang","affiliation":"School of Information Systems , Singapore Management University"},{"name":"Jie Shao","slug":"jie-shao","given":"Jie","family":"Shao","affiliation":"Center for Future Media , University of Electronic Science and Technology of China; Sichuan Artificial Intelligence Research Institute"},{"name":"Qianru Sun","slug":"qianru-sun","given":"Qianru","family":"Sun","affiliation":"School of Information Systems , Singapore Management University"}],"abstract":"Math word problem (MWP) is challenging due to the limitation in training data where only one “standard” solution is available. MWP models often simply fit this solution rather than truly understand or solve the problem. The generalization of models (to diverse word scenarios) is thus limited. To address this problem, this paper proposes a novel approach, TSN-MD, by leveraging the teacher network to integrate the knowledge of equivalent solution expressions and then to regularize the learning behavior of the student network. In addition, we introduce the multiple-decoder student network to generate multiple candidate solution expressions by which the final answer is voted. In experiments, we conduct extensive comparisons and ablative studies on two large-scale MWP benchmarks, and show that using TSN-MD can surpass the state-of-the-art works by a large margin. More intriguingly, the visualization results demonstrate that TSN-MD not only produces correct final answers but also generates diverse equivalent expressions of the solution.","doi":"10.24963/ijcai.2020/555","url":"https://ijcai.org/proceedings/2020/555","pdf":"https://ijcai.org/proceedings/2020/0555.pdf","markdown":"https://ijcai.org/proceedings/2020/555.md","bibtex":"https://ijcai.org/proceedings/2020/bibtex/555","volume":{"slug":"2020","title":"Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2020","url":"https://ijcai.org/proceedings/2020/","year":2020,"location":"Yokohama","dates":"January 2021 (postponed from 11-17 July 2020)","editors":"Christian Bessiere","isbn":"978-0-9992411-6-5","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2020"},"section":"Natural Language Processing","track":"Main track","number":555,"pages":{"first":4011,"last":4017},"keywords":["Natural Language Processing: Question Answering","Natural Language Processing: Natural Language Generation"],"videos":[{"label":"Video","url":"https://videos.ijcai20.org/25431/file_25431.m3u8","length_sec":281,"provider":"hls"}],"references":[{"position":1,"raw":"[Bobrow, 1964] Daniel G. Bobrow. Natural language input for a computer problem solving system. pages 146-226, 1964.","title":"Natural language input for a computer problem solving system","authors":"G Daniel, Bobrow","year":"1964","venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chiang and Chen, 2019] Ting-Rui Chiang and Yun-Nung Chen. Semantically-aligned equation generation for solv- ing and reasoning math word problems. In NAACL-HLT, pages 2656-2668, 2019.","title":"Semantically-aligned equation generation for solving and reasoning math word problems","authors":"Ting-Rui Chiang, Yun-Nung Chen","year":"2019","venue":"NAACL-HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Eigen et al., 2014] David Eigen, Marc'Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. In ICLR Workshop Track Pro- ceedings, 2014.","title":"Learning factored representations in a deep mixture of experts","authors":"David Eigen, Marc'aurelio Ranzato, Ilya Sutskever","year":"2014","venue":"ICLR Workshop Track Proceedings","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Furlanello et al., 2018] Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born-again neural net- works. In ICML, pages 1602-1611, 2018.","title":"Born-again neural networks","authors":"Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, Anima Anandkumar","year":"2018","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2018] Xuanli He, Gholamreza Haffari, and Mo- hammad Norouzi. Sequence to sequence mixture model for diverse machine translation. In CoNLL, pages 583- 592, 2018.","title":"Xuanli He, Gholamreza Haffari, and Mohammad Norouzi. Sequence to sequence mixture model for diverse machine translation","authors":"He","year":"2018","venue":"CoNLL","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hinton et al., 2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural net- work. CoRR, abs/1503.02531, 2015.","title":"Distilling the knowledge in a neural network","authors":"Geoffrey E Hinton, Oriol Vinyals, Jeffrey Dean","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Hu et al., 2018] Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. In EMNLP, pages 2077-2086, 2018.","title":"Attention-guided answer distillation for machine reading comprehension","authors":"Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, Ming Zhou","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Huang et al., 2017] Danqing Huang, Shuming Shi, Chin- Yew Lin, and Jian Yin. Learning fine-grained expressions to solve math word problems. In EMNLP, pages 805-814, 2017.","title":"Learning fine-grained expressions to solve math word problems","authors":"Huang","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jacobs et al., 1991] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mix- tures of local experts. Neural Computation, 3(1):79-87, 1991.","title":"Adaptive mixtures of local experts","authors":"Robert A Jacobs, Michael I Jordan, Steven J Nowlan, Geoffrey E Hinton","year":"1991","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kim and Rush, 2016] Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In EMNLP, pages 1317-1327, 2016.","title":"Sequence-level knowledge distillation","authors":"Yoon Kim, Alexander M Rush","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Koncel-Kedziorski et al., 2015] Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. TACL, 3:585-597, 2015.","title":"Parsing algebraic word problems into equations","authors":"Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, Siena Dumas, Ang","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Koncel-Kedziorski et al., 2016] Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In NAACL, pages 1152-1157, 2016.","title":"MAWPS: A math word problem repository","authors":"Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, Hannaneh Hajishirzi","year":"2016","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kuncoro et al., 2016] Adhiguna Kuncoro, Miguel Balles- teros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. Distilling an ensemble of greedy dependency parsers into one MST parser. In EMNLP, pages 1744-1753, 2016.","title":"Distilling an ensemble of greedy dependency parsers into one MST parser","authors":"Kuncoro","year":"2016","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2019] Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. Modeling intra-relation in math word problems with different func- tional multi-head attentions. In ACL, pages 6162-6167, 2019.","title":"Modeling intra-relation in math word problems with different functional multi-head attentions","authors":"Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, Dongxiang Zhang","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2019] Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. Tree-structured decoding for solving math word problems. In EMNLP-IJCNLP, pages 2370- 2379, November 2019.","title":"Tree-structured decoding for solving math word problems","authors":"Liu","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Romero et al., 2015] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.","title":"Fitnets: Hints for thin deep nets","authors":"Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Shen et al., 2019] Tianxiao Shen, Myle Ott, Michael Auli, and Marc'Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In ICML, pages 5719-5728, 2019.","title":"Mixture models for diverse machine translation: Tricks of the trade","authors":"Shen","year":"2019","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Shi et al., 2015] Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. Automatically solving number word problems by semantic parsing and reason- ing. In EMNLP, pages 1132-1142, 2015.","title":"Automatically solving number word problems by semantic parsing and reasoning","authors":"Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, Yong Rui","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pages 5998-6008, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Wang et al., 2017] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In EMNLP, pages 845-854, 2017.","title":"Deep neural solver for math word problems","authors":"Yan Wang, Xiaojiang Liu, Shuming Shi","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Wang et al., 2018a] Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. Translating a math word problem to a expression tree. In EMNLP, pages 1064-1069, 2018.","title":"Translating a math word problem to a expression tree","authors":"Wang","year":"2018","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2018b] Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. Mathdqn: Solving arithmetic word problems via deep re- inforcement learning. In AAAI, pages 5545-5552, 2018.","title":"Mathdqn: Solving arithmetic word problems via deep reinforcement learning","authors":"Wang","year":"2018","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019] Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. Template-based math word problem solvers with re- cursive neural networks. In AAAI, 2019.","title":"Template-based math word problem solvers with recursive neural networks","authors":"Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, Heng Tao Shen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie and Sun, 2019] Zhipeng Xie and Shichao Sun. A goal- driven tree-structured neural model for math word prob- lems. In IJCAI, pages 5299-5305, 7 2019.","title":"A goaldriven tree-structured neural model for math word problems","authors":"Zhipeng Xie, Shichao Sun","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Yang et al., 2018] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. In ICLR, 2018.","title":"Breaking the softmax bottleneck: A high-rank RNN language model","authors":"Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, William W Cohen","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zagoruyko and Komodakis, 2017] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Im- proving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.","title":"Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer","authors":"Komodakis Zagoruyko, Sergey Zagoruyko, Nikos Komodakis","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2018] Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. In CVPR, pages 4320-4328, 2018.","title":"Deep mutual learning","authors":"Ying Zhang, Tao Xiang, Timothy M Hospedales, Huchuan Lu","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}