{"type":"ScholarlyArticle","title":"Deterministic Routing between Layout Abstractions for Multi-Scale Classification of Visually Rich Documents","authors":[{"name":"Ritesh Sarkhel","slug":"ritesh-sarkhel","given":"Ritesh","family":"Sarkhel","affiliation":"Department of Computer Science & Engineering , The Ohio State University"},{"name":"Arnab Nandi","slug":"arnab-nandi","given":"Arnab","family":"Nandi","affiliation":"Department of Computer Science & Engineering , The Ohio State University"}],"abstract":"Classifying heterogeneous visually rich documents is a challenging task. Difficulty of this task increases even more if the maximum allowed inference turnaround time is constrained by a threshold. The increased overhead in inference cost, compared to the limited gain in classification capabilities make current multi-scale approaches infeasible in such scenarios. There are two major contributions of this work. First, we propose a spatial pyramid model to extract highly discriminative multi-scale feature descriptors from a visually rich document by leveraging the inherent hierarchy of its layout. Second, we propose a deterministic routing scheme for accelerating end-to-end inference by utilizing the spatial pyramid model. A depth-wise separable multi-column convolutional network is developed to enable our method. We evaluated the proposed approach on four publicly available, benchmark datasets of visually rich documents. Results suggest that our proposed approach demonstrates robust performance compared to the state-of-the-art methods in both classification accuracy and total inference turnaround.","doi":"10.24963/ijcai.2019/466","url":"https://ijcai.org/proceedings/2019/466","pdf":"https://ijcai.org/proceedings/2019/0466.pdf","markdown":"https://ijcai.org/proceedings/2019/466.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/466","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":466,"pages":{"first":3360,"last":3366},"keywords":["Machine Learning: Deep Learning","Machine Learning Applications: Applications of Supervised Learning","Computer Vision: Language and Vision","Machine Learning: Classification"],"videos":[],"references":[{"position":1,"raw":"[Afzal et al., 2015] Muhammad Zeshan Afzal, Samuele Capo- bianco, Muhammad Imran Malik, Simone Marinai, Thomas M. Breuel, Andreas Dengel, and Marcus Liwicki. Deepdocclassi- fier: Document classification with deep convolutional neural net- work. In 2015 13th International Conference on Document Anal- ysis and Recognition (ICDAR), pages 1111-1115, Aug 2015.","title":"Deepdocclassifier: Document classification with deep convolutional neural network","authors":"Afzal","year":"2015","venue":"2015 13th International Conference on Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bagdanov and Worring, 2001] Andrew D. Bagdanov and Marcel Worring. Fine-grained document genre classification using first order random graphs. In Proceedings of Sixth International Con- ference on Document Analysis and Recognition, pages 79-83, 2001.","title":"Fine-grained document genre classification using first order random graphs","authors":"Andrew D Bagdanov, Marcel Worring","year":"2001","venue":"Proceedings of Sixth International Conference on Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Bay et al., 2006] Herbert Bay, Tinne Tuytelaars, and Luc Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pages 404-417. Springer, 2006.","title":"Surf: Speeded up robust features","authors":"Herbert Bay, Tinne Tuytelaars, Luc Van Gool","year":"2006","venue":"European conference on computer vision","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Breuel, 2007] Thomas M. Breuel. The hocr microformat for ocr workflow and results. In Ninth International Conference on Doc- ument Analysis and Recognition (ICDAR 2007), volume 2, pages 1063-1067, Sept 2007.","title":"The hocr microformat for ocr workflow and results","authors":"M Thomas, Breuel","year":"2007","venue":"Ninth International Conference on Document Analysis and Recognition (ICDAR 2007)","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Cesarini et al., 2001] Francesca Cesarini, Marco Lastri, Simone Marinai, and Giovanni Soda. Encoding of modified x-y trees for document classification. In Proceedings of Sixth International Conference on Document Analysis and Recognition, pages 1131- 1136. IEEE, 2001.","title":"Encoding of modified x-y trees for document classification","authors":"Francesca Cesarini, Marco Lastri, Simone Marinai, Giovanni Soda","year":"2001","venue":"Proceedings of Sixth International Conference on Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Das et al., 2018] Arindam Das, Saikat Roy, and Ujjwal Bhat- tacharya. Document image classification with intra-domain trans- fer learning and stacked generalization of deep convolutional neural networks. arXiv preprint arXiv:1801.09321, 2018.","title":"Document image classification with intra-domain transfer learning and stacked generalization of deep convolutional neural networks","authors":"Arindam Das, Saikat Roy, Ujjwal Bhattacharya","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Forbus et al., 1995] Kenneth D Forbus, Dedre Gentner, and Keith Law. Mac/fac: A model of similarity-based retrieval. Cognitive science, 19(2):141-205, 1995.","title":"Mac/fac: A model of similarity-based retrieval","authors":"Dedre Kenneth D Forbus, Keith Gentner, Law","year":"1995","venue":"Cognitive science","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gordo et al., 2013] Albert Gordo, Florent Perronnin, and Ernest Valveny. Large-scale document image retrieval and classifica- tion with runlength histograms and binary embeddings. Pattern Recognition, 46(7):1898 -1905, 2013.","title":"Large-scale document image retrieval and classification with runlength histograms and binary embeddings","authors":"Gordo","year":"2013","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Harley et al., 2015] Adam W Harley, Alex Ufkes, and Konstanti- nos G Derpanis. Evaluation of deep convolutional nets for doc- ument image classification and retrieval. In Document Analysis and Recognition (ICDAR), 2015 13th International Conference on, pages 991-995. IEEE, 2015.","title":"Evaluation of deep convolutional nets for document image classification and retrieval","authors":"Adam W Harley, Alex Ufkes, Konstantinos G Derpanis","year":"2015","venue":"Document Analysis and Recognition (ICDAR)","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[He and Schomaker, 2017] Sheng He and Lambert Schomaker. Be- yond ocr: Multi-faceted understanding of handwritten document characteristics. Pattern Recognition, 63:321-333, 2017.","title":"Beyond ocr: Multi-faceted understanding of handwritten document characteristics","authors":"Schomaker He, Sheng He, Lambert Schomaker","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Howard et al., 2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.","title":"Mobilenets: Efficient convolutional neural networks for mobile vision applications","authors":"Menglong Andrew G Howard, Bo Zhu, Dmitry Chen, Weijun Kalenichenko, Tobias Wang, Marco Weyand, Hartwig Andreetto, Adam","year":"2017","venue":null,"doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Jiang et al., 2018] Lilong Jiang, Protiva Rahman, and Arnab Nandi. Evaluating interactive data systems: Workloads, metrics, and guidelines. In Proceedings of the 2018 International Confer- ence on Management of Data, pages 1637-1644. ACM, 2018.","title":"Evaluating interactive data systems: Workloads, metrics, and guidelines","authors":"Lilong Jiang, Protiva Rahman, Arnab Nandi","year":"2018","venue":"Proceedings of the 2018 International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Kumar et al., 2014] Jayant Kumar, Peng Ye, and David Doermann. Structural similarity for document image classification and re- trieval. Pattern Recognition Letters, 43:119-126, 2014.","title":"Structural similarity for document image classification and retrieval","authors":"Jayant Kumar, Peng Ye, David Doermann","year":"2014","venue":"Pattern Recognition Letters","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Lazebnik et al., 2006] Svetlana Lazebnik, Cordelia Schmid, and Jean Ponce. Beyond bags of features: Spatial pyramid match- ing for recognizing natural scene categories. In Computer vision and pattern recognition, 2006 IEEE computer society conference on, volume 2, pages 2169-2178. IEEE, 2006.","title":"Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories","authors":"Svetlana Lazebnik, Cordelia Schmid, Jean Ponce","year":"2006","venue":"Computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436, 2015.","title":"Deep learning","authors":"Yann Lecun, Yoshua Bengio, Geoffrey Hinton","year":"2015","venue":"Nature","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Lewis et al., 2006] David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, D Grossman, and Jefferson Heard. Building a test collection for complex document information processing. In Proceedings of the 29th annual international ACM SIGIR con- ference on Research and development in information retrieval, pages 665-666. ACM, 2006.","title":"Building a test collection for complex document information processing","authors":"David Lewis, Gady Agam, Shlomo Argamon, Ophir Frieder, D Grossman, Jefferson Heard","year":"2006","venue":"Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Lin et al., 2018] Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object de- tection. IEEE transactions on pattern analysis and machine in- telligence, 2018.","title":"Focal loss for dense object detection","authors":"Tsung-Yi Lin, Priyal Goyal, Ross Girshick, Kaiming He, Piotr Dollár","year":"2018","venue":"IEEE transactions on pattern analysis and machine intelligence","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lowe, 2004] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vi- sion, 60(2):91-110, 2004.","title":"Distinctive image features from scale-invariant keypoints","authors":"David G Lowe","year":"2004","venue":"International journal of computer vision","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mao et al., 2018] Chaojie Mao, Yingming Li, Yaqing Zhang, Zhongfei Zhang, and Xi Li. Multi-channel pyramid person matching network for person re-identification. arXiv preprint arXiv:1803.02558, 2018.","title":"Multi-channel pyramid person matching network for person re-identification","authors":"Mao","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[NIST, 2018] NIST. Nist special database 6. https://www.nist.gov/ srd/nist-special-database-6, 2018. Accessed: 2018-09-30.","title":"NIST. Nist special database 6","authors":null,"year":"2018","venue":null,"doi":null,"url":"https://www.nist.gov/srd/nist-special-database-6,2018","internal_url":null},{"position":21,"raw":"[Pan and Yang, 2010] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-1359, 2010.","title":"A survey on transfer learning","authors":"Jialin Sinno, Qiang Pan, Yang","year":"2010","venue":"IEEE Transactions on knowledge and data engineering","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Patin, 2003] Frédéric Patin. An introduction to digital im- age processing. online]: http://www. programmersheaven. com/articles/patin/ImageProc. pdf, 2003.","title":"An introduction to digital image processing","authors":"Patin ;, Frédéric Patin","year":"2003","venue":null,"doi":null,"url":"http://www.programmersheaven.com/articles/patin/ImageProc.pdf","internal_url":null},{"position":23,"raw":"[Sandler et al., 2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018.","title":"Mobilenetv2: Inverted residuals and linear bottlenecks","authors":"Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen","year":"2018","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Sarkhel and Nandi, 2019] Ritesh Sarkhel and Arnab Nandi. Vi- sual segmentation for information extraction from heterogeneous visually rich documents. In Proceedings of the 2019 Inter- national Conference on Management of Data, pages 247-262. ACM, 2019.","title":"Visual segmentation for information extraction from heterogeneous visually rich documents","authors":"Ritesh Sarkhel, Arnab Nandi","year":"2019","venue":"Proceedings of the 2019 International Conference on Management of Data","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Sarkhel et al., 2016] Ritesh Sarkhel, Nibaran Das, Amit K Saha, and Mita Nasipuri. A multi-objective approach towards cost ef- fective isolated handwritten bangla character and digit recogni- tion. Pattern Recognition, 58:172-189, 2016.","title":"A multi-objective approach towards cost effective isolated handwritten bangla character and digit recognition","authors":"Ritesh Sarkhel, Nibaran Das, Amit K Saha, Mita Nasipuri","year":"2016","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Sarkhel et al., 2017] Ritesh Sarkhel, Nibaran Das, Aritra Das, Ma- hantapas Kundu, and Mita Nasipuri. A multi-scale deep quad tree based feature extraction method for the recognition of isolated handwritten characters of popular indic scripts. Pattern Recogni- tion, 71:78-93, 2017.","title":"A multi-scale deep quad tree based feature extraction method for the recognition of isolated handwritten characters of popular indic scripts","authors":"Ritesh Sarkhel, Nibaran Das, Aritra Das, Mahantapas Kundu, Mita Nasipuri","year":"2017","venue":"Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Smith, 2007] Ray Smith. An overview of the tesseract ocr en- gine. In Document Analysis and Recognition, 2007. ICDAR 2007. Ninth International Conference on, volume 2, pages 629-633. IEEE, 2007.","title":"An overview of the tesseract ocr engine","authors":"Ray Smith","year":"2007","venue":"Document Analysis and Recognition","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Thoma, 2003] GFG Thoma. Ground truth data for document im- age analysis. In Symposium on document image understanding and technology (SDIUT), pages 199-205, 2003.","title":"GFG Thoma. Ground truth data for document image analysis","authors":"Thoma","year":"2003","venue":"Symposium on document image understanding and technology (SDIUT)","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Xu et al., 2018] Yue Xu, Fei Yin, Zhaoxiang Zhang, and Cheng- Lin Liu. Multi-task layout analysis for historical handwritten documents using fully convolutional networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 1057-1063. AAAI Press, 2018.","title":"Multi-task layout analysis for historical handwritten documents using fully convolutional networks","authors":"Yue Xu, Fei Yin, Zhaoxiang Zhang, Cheng-Lin Liu","year":"2018","venue":"Proceedings of the 27th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2018/147"}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}