{"type":"ScholarlyArticle","title":"A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram","authors":[{"name":"Ming-Liang Zhang","slug":"ming-liang-zhang","given":"Ming-Liang","family":"Zhang","affiliation":"MAIS , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Fei yin","slug":"fei-yin","given":"Fei","family":"Yin","affiliation":"MAIS , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Cheng-Lin Liu","slug":"cheng-lin-liu","given":"Cheng-Lin","family":"Liu","affiliation":"MAIS , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"}],"abstract":"Geometry problem solving (GPS) is a high-level mathematical reasoning requiring the capacities of multi-modal fusion and geometric knowledge application. Recently, neural solvers have shown great potential in GPS but still be short in diagram presentation and modal fusion. In this work, we convert diagrams into basic textual clauses to describe diagram features effectively, and propose a new neural solver called PGPSNet to fuse multi-modal information efficiently. Combining structural and semantic pre-training, data augmentation and self-limited decoding, PGPSNet is endowed with rich knowledge of geometry theorems and geometric representation, and therefore promotes geometric understanding and reasoning. In addition, to facilitate the research of GPS, we build a new large-scale and fine-annotated GPS dataset named PGPS9K, labeled with both fine-grained diagram annotation and interpretable solution program. Experiments on PGPS9K and an existing dataset Geometry3K validate the superiority of our method over the state-of-the-art neural solvers. Our code, dataset and appendix material are available at \\url{https://github.com/mingliangzhang2018/PGPS}.","doi":"10.24963/ijcai.2023/376","url":"https://ijcai.org/proceedings/2023/376","pdf":"https://ijcai.org/proceedings/2023/0376.pdf","markdown":"https://ijcai.org/proceedings/2023/376.md","bibtex":"https://ijcai.org/proceedings/2023/bibtex/376","volume":{"slug":"2023","title":"Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2023","url":"https://ijcai.org/proceedings/2023/","year":2023,"location":"Macao, SAR","dates":"19-25 August 2023","editors":"Edith Elkind","isbn":"978-1-956792-03-4","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2023"},"section":"Knowledge Representation and Reasoning","track":"Main Track","number":376,"pages":{"first":3374,"last":3382},"keywords":["Knowledge Representation and Reasoning: KRR: Qualitative, geometric, spatial, and temporal reasoning","Machine Learning: ML: Multi-modal learning","Multidisciplinary Topics and Applications: MDA: Education"],"videos":[],"references":[{"position":1,"raw":"[Amini et al., 2019] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In NAACL, 2019.","title":"MathQA: Towards interpretable math word problem solving with operation-based formalisms","authors":"Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, Hannaneh Hajishirzi","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Anderson et al., 2018] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for im- age captioning and visual question answering. In CVPR, 2018.","title":"Bottom-up and top-down attention for image captioning and visual question answering","authors":"Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bahdanau et al., 2015] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.","title":"Neural machine translation by jointly learning to align and translate","authors":"Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio","year":"2015","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Brown et al., 2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and Pra- fulla Dhariwal et al. Language models are few-shot learn- ers. In NeurIPS, 2020.","title":"Language models are few-shot learners","authors":"B Tom, Benjamin Brown, Nick Mann, Melanie Ryder, Jared Subbiah, Prafulla Kaplan, Dhariwal","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cao and Xiao, 2022] Jie Cao and Jing Xiao. An aug- mented benchmark dataset for geometric question answer- ing through dual parallel text encoding. In COLING, 2022.","title":"An augmented benchmark dataset for geometric question answering through dual parallel text encoding","authors":"Xiao ; ] Jie Cao, Jing Cao, Xiao","year":"2022","venue":"COLING","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen et al., 2021] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark to- wards multimodal numerical reasoning. In Findings of ACL, 2021.","title":"GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning","authors":"Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, Liang Lin","year":"2021","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2022] Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Uni- geo: Unifying geometry logical reasoning via reformulat- ing mathematical expression. In EMNLP, 2022.","title":"Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression","authors":"Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, Xiaodan Liang","year":"2022","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Cho et al., 2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In EMNLP, 2014.","title":"Learning phrase representations using RNN encoder-decoder for statistical machine translation","authors":"Cho","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Cho et al., 2021] Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text gen- eration. In ICML, 2021.","title":"Unifying vision-and-language tasks via text generation","authors":"Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Chou et al., 1996] Shang Ching Chou, Xiao Shan Gao, and Jing Zhong Zhang. Automated generation of readable proofs with geometric invariants. Journal of Automated Reasoning, 17:349-370, 1996.","title":"Automated generation of readable proofs with geometric invariants","authors":"Ching Shang, Xiao Shan Chou, Jing Zhong Gao, Zhang","year":"1996","venue":"Journal of Automated Reasoning","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"BERT: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Gan et al., 2019] Wenbin Gan, Xinguo Yu, Ting Zhang, and Mingshu Wang. Automatically proving plane geome- try theorems stated by text and diagram. International Journal of Pattern Recognition and Artificial Intelligence, 33(07):1940003, 2019.","title":"Automatically proving plane geometry theorems stated by text and diagram","authors":"Wenbin Gan, Xinguo Yu, Ting Zhang, Mingshu Wang","year":"2019","venue":"International Journal of Pattern Recognition and Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Hao et al., 2022] Yihan Hao, Mingliang Zhang, Fei Yin, and Linlin Huang. PGDP5K: A diagram parsing dataset for plane geometry problems. In ICPR, 2022.","title":"PGDP5K: A diagram parsing dataset for plane geometry problems","authors":"Yihan Hao, Mingliang Zhang, Fei Yin, Linlin Huang","year":"2022","venue":"ICPR","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Khashabi et al., 2020] Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In EMNLP, 2020.","title":"UNIFIEDQA: Crossing format boundaries with a single QA system","authors":"Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kim et al., 2021] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without con- volution or region supervision. In ICML, 2021.","title":"Vilt: Vision-and-language transformer without convolution or region supervision","authors":"Kim","year":"2021","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lewis et al., 2020] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2020.","title":"BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension","authors":"Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer","year":"2020","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Liang et al., 2022] Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang. MWP-BERT: Numeracy-augmented pre-training for math word problem solving. In Findings of NAACL, July 2022.","title":"MWP-BERT: Numeracy-augmented pre-training for math word problem solving","authors":"Liang","year":"2022","venue":"Findings of NAACL","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Loshchilov and Hutter, 2017] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2017.","title":"Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization","authors":"Hutter Loshchilov","year":"2017","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Lu et al., 2021] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL, 2021.","title":"Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning","authors":"Lu","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Lu et al., 2022] Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. ArXiv, abs/2212.10535, 2022.","title":"A survey of deep learning for mathematical reasoning","authors":"Lu","year":"2022","venue":null,"doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Lu et al., 2023] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reason- ing. In ICLR, 2023.","title":"Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning","authors":"Lu","year":"2023","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Manjunatha et al., 2019] Varun Manjunatha, Nirat Saini, and Larry S. Davis. Explicit bias discovery in visual ques- tion answering models. In CVPR, 2019.","title":"Explicit bias discovery in visual question answering models","authors":"Varun Manjunatha, Nirat Saini, Larry S Davis","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Minaee et al., 2021] Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, and Jianfeng Gao. Deep learning-based text classification: A comprehensive review. ACM Comput. Surv., 54(3), apr 2021.","title":"Deep learning-based text classification: A comprehensive review","authors":"Shervin Minaee, Nal Kalchbrenner, Erik Cambria, Narjes Nikzad, Meysam Chenaghlu, Jianfeng Gao","year":"2021","venue":"ACM Comput. Surv","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Patel et al., 2021] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In NAACL, 2021.","title":"Are NLP models really able to solve simple math word problems?","authors":"Patel","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sachan and Xing, 2017] Mrinmaya Sachan and Eric Xing. Learning to solve geometry problems from natural lan- guage demonstrations in textbooks. In SEM, 2017.","title":"Learning to solve geometry problems from natural language demonstrations in textbooks","authors":"Xing Sachan, Mrinmaya Sachan, Eric Xing","year":"2017","venue":"SEM","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Sachan et al., 2017] Mrinmaya Sachan, Avinava Dubey, and Eric P. Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In EMNLP, 2017.","title":"From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems","authors":"Sachan","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Seo et al., 2015] Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geom- etry problems: Combining text and diagram interpretation. In EMNLP, 2015.","title":"Solving geometry problems: Combining text and diagram interpretation","authors":"Seo","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Tito et al., 2021] Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question an- swering. In ICDAR, 2021.","title":"Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. Document collection visual question answering","authors":"Tito","year":"2021","venue":"ICDAR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Tsai et al., 2021] Shih-hung Tsai, Chao-Chun Liang, Hsin- Min Wang, and Keh-Yih Su. Sequence to general tree: Knowledge-guided geometry word problem solving. In ACL, 2021.","title":"Sequence to general tree: Knowledge-guided geometry word problem solving","authors":"Shih-Hung Tsai, Chao-Chun Liang, Hsin-Min Wang, Keh-Yih Su","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Vaswani et al., 2017] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.","title":"Attention is all you need","authors":"Ashish Vaswani, Noam M Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, Illia Polosukhin","year":"2017","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Wang et al., 2017] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In EMNLP, 2017.","title":"Deep neural solver for math word problems","authors":"Yan Wang, Xiaojiang Liu, Shuming Shi","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":33,"raw":"[Xie and Sun, 2019] Zhipeng Xie and Shichao Sun. A goal- driven tree-structured neural model for math word prob- lems. In IJCAI, 2019.","title":"A goaldriven tree-structured neural model for math word problems","authors":"Zhipeng Xie, Shichao Sun","year":"2019","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":34,"raw":"[Xu et al., 2020] Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In SIGKDD, 2020.","title":"Layoutlm: Pre-training of text and layout for document image understanding","authors":"Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou","year":"2020","venue":"SIGKDD","doi":null,"url":null,"internal_url":null},{"position":35,"raw":"[Yu et al., 2019] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for vi- sual question answering. In CVPR, 2019.","title":"Deep modular co-attention networks for visual question answering","authors":"Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, Qi Tian","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":36,"raw":"[Zhang and Moshfeghi, 2022] Jiaxin Zhang and Yashar Moshfeghi. ELASTIC: Numerical reasoning with adaptive symbolic compiler. In NeurIPS, 2022.","title":"ELASTIC: Numerical reasoning with adaptive symbolic compiler","authors":"Moshfeghi Zhang, Jiaxin Zhang, Yashar Moshfeghi","year":"2022","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":37,"raw":"[Zhang et al., 2022] Ming-Liang Zhang, Fei Yin, Yi-Han Hao, and Cheng-Lin Liu. Plane geometry diagram pars- ing. In IJCAI, 2022.","title":"Plane geometry diagram parsing","authors":"Ming-Liang Zhang, Fei Yin, Yi-Han Hao, Cheng-Lin Liu","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/228"},{"position":38,"raw":"[Zhu et al., 2021] Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. TAT-QA: A question answer- ing benchmark on a hybrid of tabular and textual content in finance. In ACL, 2021.","title":"TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance","authors":"Zhu","year":"2021","venue":"ACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}