{"type":"ScholarlyArticle","title":"AgriBERT: Knowledge-Infused Agricultural Language Models for Matching Food and Nutrition","authors":[{"name":"Saed Rezayi","slug":"saed-rezayi","given":"Saed","family":"Rezayi","affiliation":"University of Georgia"},{"name":"Zhengliang Liu","slug":"zhengliang-liu","given":"Zhengliang","family":"Liu","affiliation":"University of Georgia"},{"name":"Zihao Wu","slug":"zihao-wu","given":"Zihao","family":"Wu","affiliation":"University of Georgia"},{"name":"Chandra Dhakal","slug":"chandra-dhakal","given":"Chandra","family":"Dhakal","affiliation":"University of Georgia"},{"name":"Bao Ge","slug":"bao-ge","given":"Bao","family":"Ge","affiliation":"Shanxi Normal University"},{"name":"Chen Zhen","slug":"chen-zhen","given":"Chen","family":"Zhen","affiliation":"University of Georgia"},{"name":"Tianming Liu","slug":"tianming-liu","given":"Tianming","family":"Liu","affiliation":"University of Georgia"},{"name":"Sheng Li","slug":"sheng-li","given":"Sheng","family":"Li","affiliation":"University of Georgia"}],"abstract":"Pretraining domain-specific language models remains an important challenge which limits their applicability in various areas such as agriculture. This paper investigates the effectiveness of leveraging food related text corpora (e.g., food and agricultural literature) in pretraining transformer-based language models. We evaluate our trained language model, called AgriBERT, on the task of semantic matching, i.e., establishing mapping between food descriptions and nutrition data, which is a long-standing challenge in the agricultural domain. In particular, we formulate the task as an answer selection problem, fine-tune the trained language model with the help of an external source of knowledge (e.g., FoodOn ontology), and establish a baseline for this task. The experimental results reveal that our language model substantially outperforms other language models and baselines in the task of matching food description and nutrition.","doi":"10.24963/ijcai.2022/715","url":"https://ijcai.org/proceedings/2022/715","pdf":"https://ijcai.org/proceedings/2022/0715.pdf","markdown":"https://ijcai.org/proceedings/2022/715.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/715","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Research Papers","track":"AI for Good","number":715,"pages":{"first":5150,"last":5156},"keywords":["Natural Language Processing: Named Entities","Natural Language Processing: Applications","Natural Language Processing: Language Models","Multidisciplinary Topics and Applications: Health and Medicine"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38985292","length_sec":639,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38987149","length_sec":639,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Chawla et al., 2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: syn- thetic minority over-sampling technique. Journal of artifi- cial intelligence research, 16, 2002.","title":"Smote: synthetic minority over-sampling technique","authors":"Kevin W Nitesh V Chawla, Lawrence O Bowyer, Philip Hall, Kegelmeyer","year":"2002","venue":"Journal of artificial intelligence research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Deng et al., 2019] Yang Deng, Yuexiang Xie, Yaliang Li, Min Yang, Nan Du, Wei Fan, Kai Lei, and Ying Shen. Multi-task learning with multi-view attention for answer selection and knowledge base question answering. In AAAI, volume 33, 2019.","title":"Multi-task learning with multi-view attention for answer selection and knowledge base question answering","authors":"Yang Deng, Yuexiang Xie, Yaliang Li, Min Yang, Nan Du, Wei Fan, Kai Lei, Ying Shen","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL, 2019.","title":"Bert: Pre-training of deep bidirectional transformers for language understanding","authors":"Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dooley and Griffiths, 2018] Damion M Dooley and Emma J et al. Griffiths. Foodon: a harmonized food ontology to increase global food traceability, quality control and data integration. Science of Food, 2(1), 2018.","title":"Foodon: a harmonized food ontology to increase global food traceability, quality control and data integration","authors":"M Damion, Emma J Dooley","year":"2018","venue":"Science of Food","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Feng et al., 2020] Steven Y Feng, Varun Gangal, Dongyeop Kang, Teruko Mitamura, and Eduard Hovy. Genaug: Data augmentation for finetuning text generators. In DeepIO, 2020.","title":"Genaug: Data augmentation for finetuning text generators","authors":"Varun Steven Y Feng, Dongyeop Gangal, Teruko Kang, Eduard Mitamura, Hovy","year":"2020","venue":"DeepIO","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Feng et al., 2021] Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. A survey of data augmentation approaches for nlp. In ACL-IJCNLP, 2021.","title":"A survey of data augmentation approaches for nlp","authors":"Varun Steven Y Feng, Jason Gangal, Sarath Wei, Soroush Chandar, Teruko Vosoughi, Eduard Mitamura, Hovy","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Grundkiewicz et al., 2019] Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. Neural gram- matical error correction systems with unsupervised pre- training on synthetic data. In ACL BEA, 2019.","title":"Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Kenneth Heafield. Neural grammatical error correction systems with unsupervised pretraining on synthetic data","authors":"Grundkiewicz","year":"2019","venue":"ACL BEA","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Gu et al., 2021] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific lan- guage model pretraining for biomedical natural language processing. ACM Transactions on Computing for Health- care, 3(1), 2021.","title":"Domain-specific language model pretraining for biomedical natural language processing","authors":"Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, Hoifung Poon","year":"2021","venue":"ACM Transactions on Computing for Healthcare","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Jin et al., 2020] Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and en- tailment. In AAAI, volume 34, 2020.","title":"Is bert really robust? a strong baseline for natural language attack on text classification and entailment","authors":"Di Jin, Zhijing Jin, Joey Tianyi Zhou, Peter Szolovits","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Kenter and De Rijke, 2015] Tom Kenter and Maarten De Rijke. Short text similarity with word embeddings. In CIKM, 2015.","title":"Short text similarity with word embeddings","authors":"De Kenter, Tom Rijke, Maarten Kenter, Rijke De","year":"2015","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Laskar et al., 2020] Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque. Contextualized embeddings based transformer encoder for sentence similarity model- ing in answer selection task. In LREC, pages 5505-5514, 2020.","title":"Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task","authors":"Md Tahmid Rahman Laskar, Xiangji Huang, Enamul Hoque","year":"2020","venue":"LREC","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Lee et al., 2020] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioin- formatics, 36(4), 2020.","title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","authors":"Lee","year":"2020","venue":"Bioinformatics","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Liu et al., 2020] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. K-bert: En- abling language representation with knowledge graph. In AAAI, volume 34, 2020.","title":"K-bert: Enabling language representation with knowledge graph","authors":"Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang","year":"2020","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Longpre et al., 2020] Shayne Longpre, Yu Wang, and Chris DuBois. How effective is task-agnostic data augmentation for pretrained transformers? In EMNLP, 2020.","title":"How effective is task-agnostic data augmentation for pretrained transformers?","authors":"Shayne Longpre, Yu Wang, Chris Dubois","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Marcus et al., 1994] Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Fergu- son, Karen Katz, and Britta Schasberger. The penn tree- bank: Annotating predicate argument structure. In Human Language Technology, 1994.","title":"The penn treebank: Annotating predicate argument structure","authors":"Marcus","year":"1994","venue":"Human Language Technology","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[McCann et al., 2017] Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in trans- lation: Contextualized word vectors. In NIPS, 2017.","title":"Learned in translation: Contextualized word vectors","authors":"Bryan Mccann, James Bradbury, Caiming Xiong, Richard Socher","year":"2017","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their composition- ality. NIPS, 26, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Miller, 1995] George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11), 1995.","title":"Wordnet: a lexical database for english","authors":"George A Miller","year":"1995","venue":"Communications of the ACM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Rezayi et al., 2021] Saed Rezayi, Handong Zhao, Sungchul Kim, Ryan Rossi, Nedim Lipka, and Sheng Li. Edge: En- riching knowledge graph embeddings with external text. In NAACL, 2021.","title":"Edge: Enriching knowledge graph embeddings with external text","authors":"Handong Saed Rezayi, Sungchul Zhao, Ryan Kim, Nedim Rossi, Sheng Lipka, Li","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Rücklé et al., 2019] Andreas Rücklé, Nafise Sadat Moosavi, and Iryna Gurevych. Coala: A neural coverage-based approach for long answer selection with small data. In AAAI, volume 33, 2019.","title":"Coala: A neural coverage-based approach for long answer selection with small data","authors":"Andreas Rücklé, Nafise Sadat Moosavi, Iryna Gurevych","year":"2019","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Shi et al., 2021] Haoyue Shi, Karen Livescu, and Kevin Gimpel. Substructure substitution: Structured data aug- mentation for nlp. In ACL-IJCNLP, 2021.","title":"Substructure substitution: Structured data augmentation for nlp","authors":"Shi","year":"2021","venue":"ACL-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wei and Zou, 2019] Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In EMNLP-IJCNLP, 2019.","title":"Eda: Easy data augmentation techniques for boosting performance on text classification tasks","authors":"Zou ; ] Wei, Jason Wei, Kai Zou","year":"2019","venue":"EMNLP-IJCNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Wei et al., 2021] Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, and Shiqi Xu. Few-shot text classi- fication with triplet networks, data augmentation, and cur- riculum learning. In NAACL, 2021.","title":"Few-shot text classification with triplet networks, data augmentation, and curriculum learning","authors":"Jason Wei, Chengyu Huang, Soroush Vosoughi, Yu Cheng, Shiqi Xu","year":"2021","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Wu et al., 2020] Ledell Wu, Fabio Petroni, Martin Josi- foski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In EMNLP, pages 6397-6407, 2020.","title":"Scalable zero-shot entity linking with dense entity retrieval","authors":"Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, Luke Zettlemoyer","year":"2020","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Xia et al., 2019] Mengzhou Xia, Xiang Kong, Antonios Anastasopoulos, and Graham Neubig. Generalized data augmentation for low-resource translation. In ACL, 2019.","title":"Generalized data augmentation for low-resource translation","authors":"Xia","year":"2019","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Yang et al., 2019] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. In NAACL, 2019.","title":"End-to-end open-domain question answering with bertserini","authors":"Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, Jimmy Lin","year":"2019","venue":"NAACL","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yih et al., 2013] Scott Wen-tau Yih, Ming-Wei Chang, Chris Meek, and Andrzej Pastusiak. Question answering using enhanced lexical semantic models. In ACL, 2013.","title":"Question answering using enhanced lexical semantic models","authors":"Wen-Tau Scott, Ming-Wei Yih, Chris Chang, Andrzej Meek, Pastusiak","year":"2013","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Zhao et al., 2018] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL, 2018.","title":"Gender bias in coreference resolution: Evaluation and debiasing methods","authors":"Zhao","year":"2018","venue":"NAACL","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}