{"type":"ScholarlyArticle","title":"Plane Geometry Diagram Parsing","authors":[{"name":"Ming-Liang Zhang","slug":"ming-liang-zhang","given":"Ming-Liang","family":"Zhang","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Fei Yin","slug":"fei-yin","given":"Fei","family":"Yin","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"},{"name":"Yi-Han Hao","slug":"yi-han-hao","given":"Yi-Han","family":"Hao","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation of Chinese Academy of Sciences; School of Electronic Information Engineering , Beijing Jiaotong University"},{"name":"Cheng-Lin Liu","slug":"cheng-lin-liu","given":"Cheng-Lin","family":"Liu","affiliation":"National Laboratory of Pattern Recognition , Institute of Automation of Chinese Academy of Sciences; School of Artificial Intelligence , University of Chinese Academy of Sciences"}],"abstract":"Geometry diagram parsing plays a key role in geometry problem solving, wherein the primitive extraction and relation parsing remain challenging due to the complex layout and between-primitive relationship. In this paper, we propose a powerful diagram parser based on deep learning and graph reasoning. Specifically, a modified instance segmentation method is proposed to extract geometric primitives, and the graph neural network (GNN) is leveraged to realize relation parsing and primitive classification incorporating geometric features and prior knowledge. All the modules are integrated into an end-to-end model called PGDPNet to perform all the sub-tasks simultaneously. In addition, we build a new large-scale geometry diagram dataset named PGDP5K with primitive level annotations. Experiments on PGDP5K and an existing dataset IMP-Geometry3K show that our model outperforms state-of-the-art methods in four sub-tasks remarkably. Our code, dataset and appendix material are available at https://github.com/mingliangzhang2018/PGDP.","doi":"10.24963/ijcai.2022/228","url":"https://ijcai.org/proceedings/2022/228","pdf":"https://ijcai.org/proceedings/2022/0228.pdf","markdown":"https://ijcai.org/proceedings/2022/228.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/228","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Computer Vision","track":"Main Track","number":228,"pages":{"first":1636,"last":1643},"keywords":["Computer Vision: Scene analysis and understanding","Computer Vision: Segmentation","Computer Vision: Recognition (object detection, categorization)","Multidisciplinary Topics and Applications: Education","Computer Vision: Visual reasoning and symbolic representation"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986455","length_sec":80,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984598","length_sec":339,"provider":"slideslive"}],"references":[{"position":1,"raw":"IMP-Geometry3K PGDP5K All Likely Same 73.71 / 99.17 / 99.33 65.70 / 98.40 / 99.00 Almost Same 50.08 / 95.51 / 98.50 44.40 / 93.10 / 96.60 Perfect Recall 45.26 / 81.03 / 92.18 40.00 / 79.70 / 86.20 Totally Same 34.28 / 80.53 / 91.51 27.30 / 78.20 / 84.70 Geo2Geo Likely Same 69.88 / 99.67 / 99.50 63.90 / 99.10 / 99.00 Almost Same 56.24 / 99.50 / 99.00 49.40 / 97.30 / 97.10 Perfect Recall 74.71 / 99.33 / 99.17 78.70 / 96.90 / 97.40 Totally Same 47.59 / 98.84 / 98.33 40.80 / 93.60 / 94.50 Non-geo 2Geo Likely Same 77.04 / 96.01 / 99.00 67.30 / 95.80 / 98.00 Almost Same 59.07 / 89.35 / 96.01 49.80 / 88.20 / 94.90 Perfect Recall 50.92 / 81.20 / 92.85 45.70 / 81.30 / 87.00 Totally Same 48.59 / 80.87 / 92.85 40.50 / 80.60 / 86.40","title":"IMP-Geometry3K PGDP5K All Likely Same","authors":null,"year":null,"venue":null,"doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Amini et al., 2019] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In NAACL HLT, 2019.","title":"MathQA: Towards interpretable math word problem solving with operation-based formalisms","authors":"Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, Hannaneh Hajishirzi","year":"2019","venue":"NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Chen et al., 2021] Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, and Liang Lin. GeoQA: A geometric question answering benchmark to- wards multimodal numerical reasoning. In Findings of ACL, 2021.","title":"GeoQA: A geometric question answering benchmark towards multimodal numerical reasoning","authors":"Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing, Liang Lin","year":"2021","venue":"Findings of ACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Chou et al., 1996] Shang Ching Chou, Xiao Shan Gao, and Jing Zhong Zhang. Automated generation of readable proofs with geometric invariants: II. theorem proving with full-angles. Journal of Automated Reasoning, 17:349-370, 1996.","title":"Automated generation of readable proofs with geometric invariants: II. theorem proving with full-angles","authors":"Ching Shang, Xiao Shan Chou, Jing Zhong Gao, Zhang","year":"1996","venue":"Journal of Automated Reasoning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[De Brabandere et al., 2017] Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmenta- tion with a discriminative loss function. In CoRR, volume abs/1708.0, 2017.","title":"Semantic instance segmentation with a discriminative loss function","authors":"De Brabandere","year":"2017","venue":"CoRR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Gan et al., 2018] Wenbin Gan, Xinguo Yu, Chao Sun, Bin He, and Mingshu Wang. Understanding plane geome- try problems by integrating relations extracted from text and diagram. PSIVT 2017: Image and Video Technology, LNCS, 10749:366-381, 2018.","title":"Understanding plane geometry problems by integrating relations extracted from text and diagram","authors":"Wenbin Gan, Xinguo Yu, Chao Sun, Bin He, Mingshu Wang","year":"2018","venue":"PSIVT 2017: Image and Video Technology","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Guo et al., 2021] Yunfei Guo, Wei Feng, Fei Yin, Tao Xue, Shuqi Mei, and Cheng-Lin Liu. Learning to understand traffic signs. In ACM MM, 2021.","title":"Learning to understand traffic signs","authors":"Yunfei Guo, Wei Feng, Fei Yin, Tao Xue, Shuqi Mei, Cheng-Lin Liu","year":"2021","venue":"ACM MM","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[He et al., 2018] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:386- 397, 2018.","title":"Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn","authors":"He","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Huang et al., 2018] Kun Huang, Yifan Wang, Zihan Zhou, Tianjiao Ding, Shenghua Gao, and Yi Ma. Learning to parse wireframes in images of man-made environments. In CVPR, 2018.","title":"Learning to parse wireframes in images of man-made environments","authors":"Kun Huang, Yifan Wang, Zihan Zhou, Tianjiao Ding, Shenghua Gao, Yi Ma","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2017] Tsung Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Fea- ture pyramid networks for object detection. In CVPR, 2017.","title":"Feature pyramid networks for object detection","authors":"Tsung Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2021] Hengyue Liu, Ning Yan, Masood Mor- tazavi, and Bir Bhanu. Fully convolutional scene graph generation. In CVPR, 2021.","title":"Fully convolutional scene graph generation","authors":"Liu","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lu et al., 2021] Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL-IJCNLP, 2021.","title":"Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning","authors":"Lu","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Neven et al., 2018] Davy Neven, Bert De Brabandere, Sta- matios Georgoulis, Marc Proesmans, and Luc Van Gool. Towards end-to-end lane detection: An instance segmen- tation approach. Proceedings of IEEE Intelligent Vehicles Symposium, pages 286-291, 2018.","title":"Towards end-to-end lane detection: An instance segmentation approach","authors":"Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool","year":"2018","venue":"Proceedings of IEEE Intelligent Vehicles Symposium","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Neven et al., 2019] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool. Instance segmentation by jointly optimizing spatial embeddings and clustering band- width. In CVPR, 2019.","title":"Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth","authors":"Davy Neven, Bert De Brabandere, Marc Proesmans, Luc Van Gool","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pratt, 2007] William K Pratt. Digital Image Processing, Fourth Edition. Wiley, 2007.","title":"Digital Image Processing","authors":"K William, Pratt","year":"2007","venue":null,"doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Sachan et al., 2017] Mrinmaya Sachan, Avinava Dubey, and Eric P. Xing. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In EMNLP, 2017.","title":"From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems","authors":"Sachan","year":"2017","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Sachan et al., 2020] Mrinmaya Sachan, Avinava Dubey, Eduard H. Hovy, Tom M. Mitchell, Dan Roth, and Eric P. Xing. Discourse in multimedia: A case study in extract- ing geometry knowledge from textbooks. Computational Linguistics, 45:627-665, 2020.","title":"Discourse in multimedia: A case study in extracting geometry knowledge from textbooks","authors":"Sachan","year":"2020","venue":"Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Men- glong Zhu, Andrey Zhmoginov, and Liang Chieh Chen. MobileNetV2: Inverted residuals and linear bottlenecks. In CVPR, 2018.","title":"MobileNetV2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang Chieh, Chen","year":"2018","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Seo et al., 2014] Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. Diagram understanding in ge- ometry questions. In AAAI, 2014.","title":"Diagram understanding in geometry questions","authors":"Seo","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Seo et al., 2015] Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geom- etry problems: Combining text and diagram interpretation. In EMNLP, 2015.","title":"Solving geometry problems: Combining text and diagram interpretation","authors":"Seo","year":"2015","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Tian et al., 2020] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: A simple and strong anchor-free object detector. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 8828:1-1, 2020.","title":"FCOS: A simple and strong anchor-free object detector","authors":"Chunhua Zhi Tian, Hao Shen, Tong Chen, He","year":"2020","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Veličković et al., 2018] Petar Veličković, Arantxa Casanova, Pietro Liò, Guillem Cucurull, Adriana Romero, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Arantxa Casanova, Pietro Liò, Guillem Cucurull, Adriana Romero, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Xu et al., 2017] Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In CVPR, 2017.","title":"Scene graph generation by iterative message passing","authors":"Danfei Xu, Yuke Zhu, Christopher B Choy, Li Fei-Fei","year":"2017","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Ye et al., 2020] Jun-Yu Ye, Yan-Ming Zhang, Qing Yang, and Cheng-Lin Liu. Contextual stroke classification in on- line handwritten documents with edge graph attention net- works. SN Computer Science, 1(3), 2020.","title":"Contextual stroke classification in online handwritten documents with edge graph attention networks","authors":"Ye","year":"2020","venue":"SN Computer Science","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Ying et al., 2021] Hui Ying, Zhaojin Huang, Shu Liu, Tian- jia Shao, and Kun Zhou. Embedmask: Embedding cou- pling for one-stage instance segmentation. In IJCAI, 2021.","title":"Embedmask: Embedding coupling for one-stage instance segmentation","authors":"Hui Ying, Zhaojin Huang, Shu Liu, Tianjia Shao, Kun Zhou","year":"2021","venue":"IJCAI","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhou et al., 2019] Yichao Zhou, Haozhi Qi, and Yi Ma. End-to-end wireframe parsing. In ICCV, 2019.","title":"End-to-end wireframe parsing","authors":"Zhou","year":"2019","venue":"ICCV","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}