{"type":"ScholarlyArticle","title":"Recipe2Vec: Multi-modal Recipe Representation Learning with Graph Neural Networks","authors":[{"name":"Yijun Tian","slug":"yijun-tian","given":"Yijun","family":"Tian","affiliation":"Department of Computer Science , University of Notre Dame , USA"},{"name":"Chuxu Zhang","slug":"chuxu-zhang","given":"Chuxu","family":"Zhang","affiliation":"Department of Computer Science , Brandeis University , USA"},{"name":"Zhichun Guo","slug":"zhichun-guo","given":"Zhichun","family":"Guo","affiliation":"Department of Computer Science , University of Notre Dame , USA"},{"name":"Yihong Ma","slug":"yihong-ma","given":"Yihong","family":"Ma","affiliation":"Department of Computer Science , University of Notre Dame , USA"},{"name":"Ronald Metoyer","slug":"ronald-metoyer","given":"Ronald","family":"Metoyer","affiliation":"Department of Computer Science , University of Notre Dame , USA"},{"name":"Nitesh V. Chawla","slug":"nitesh-v-chawla","given":"Nitesh V.","family":"Chawla","affiliation":"Department of Computer Science , University of Notre Dame , USA"}],"abstract":"Learning effective recipe representations is essential in food studies. Unlike what has been developed for image-based recipe retrieval or learning structural text embeddings, the combined effect of multi-modal information (i.e., recipe images, text, and relation data) receives less attention. In this paper, we formalize the problem of multi-modal recipe representation learning to integrate the visual, textual, and relational information into recipe embeddings. In particular, we first present Large-RG, a new recipe graph data with over half a million nodes, making it the largest recipe graph to date. We then propose Recipe2Vec, a novel graph neural network based recipe embedding model to capture multi-modal information. Additionally, we introduce an adversarial attack strategy to ensure stable learning and improve performance. Finally, we design a joint objective function of node classification and adversarial learning to optimize the model. Extensive experiments demonstrate that Recipe2Vec outperforms state-of-the-art baselines on two classic food study tasks, i.e., cuisine category classification and region prediction. Dataset and codes are available at https://github.com/meettyj/Recipe2Vec.","doi":"10.24963/ijcai.2022/482","url":"https://ijcai.org/proceedings/2022/482","pdf":"https://ijcai.org/proceedings/2022/0482.pdf","markdown":"https://ijcai.org/proceedings/2022/482.md","bibtex":"https://ijcai.org/proceedings/2022/bibtex/482","volume":{"slug":"2022","title":"Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2022","url":"https://ijcai.org/proceedings/2022/","year":2022,"location":"Vienna","dates":"23-29 July 2022","editors":"Luc De Raedt","isbn":"978-1-956792-00-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2022"},"section":"Machine Learning","track":"Main Track","number":482,"pages":{"first":3473,"last":3479},"keywords":["Machine Learning: Multi-modal learning","Machine Learning: Representation learning","Data Mining: Mining Graphs","Data Mining: Mining Heterogenous Data"],"videos":[{"label":"Video #1","url":"https://slideslive.com/38986760","length_sec":74,"provider":"slideslive"},{"label":"Video #2","url":"https://slideslive.com/38984903","length_sec":359,"provider":"slideslive"}],"references":[{"position":1,"raw":"[Carvalho et al., 2018] Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In SIGIR, 2018.","title":"Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings","authors":"Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, Matthieu Cord","year":"2018","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chen et al., 2018] Jing-Jing Chen, Chong-Wah Ngo, Fu-Li Feng, and Tat-Seng Chua. Deep understanding of cooking procedure for cross-modal recipe retrieval. In MM, 2018.","title":"Deep understanding of cooking procedure for cross-modal recipe retrieval","authors":"Jing-Jing Chen, Chong-Wah Ngo, Fu-Li Feng, Tat-Seng Chua","year":"2018","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Dong et al., 2017] Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In KDD, 2017.","title":"metapath2vec: Scalable representation learning for heterogeneous networks","authors":"Dong","year":"2017","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Haussmann et al., 2019] Steven Haussmann, Oshani Seneviratne, Yu Chen, Yarden Ne'eman, James Codella, Ching-Hua Chen, Deborah L. McGuinness, and Mohammed J. Zaki. Foodkg: A semantics-driven knowledge graph for food recommendation. In ISWC, 2019.","title":"Foodkg: A semantics-driven knowledge graph for food recommendation","authors":"Steven Haussmann, Oshani Seneviratne, Yu Chen, Yarden Ne'eman, James Codella, Ching-Hua Chen, Deborah L Mcguinness, Mohammed J Zaki","year":"2019","venue":"ISWC","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He et al., 2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.","title":"Deep residual learning for image recognition","authors":"Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun","year":"2016","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hu et al., 2020] Ziniu Hu, Yuxiao Dong, Kuansan Wang, and Yizhou Sun. Heterogeneous graph transformer. In WWW, 2020.","title":"Heterogeneous graph transformer","authors":"Hu","year":"2020","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Jiang et al., 2020] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. In NeurIPS, 2020.","title":"Robust pre-training by adversarial contrastive learning","authors":"Ziyu Jiang, Tianlong Chen, Ting Chen, Zhangyang Wang","year":"2020","venue":"NeurIPS","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Li and Zaki, 2020] Diya Li and Mohammed J. Zaki. Reciptor: An effective pretrained model for recipe representation learning. In KDD, 2020.","title":"Reciptor: An effective pretrained model for recipe representation learning","authors":"Diya Li, Mohammed J Zaki","year":"2020","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Madry et al., 2018] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.","title":"Towards deep learning models resistant to adversarial attacks","authors":"Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Marin et al., 2019] Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, and Antonio Torralba. Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images. TPAMI, 2019.","title":"Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and food images","authors":"Javier Marin, Aritro Biswas, Ferda Ofli, Nicholas Hynes, Amaia Salvador, Yusuf Aytar, Ingmar Weber, Antonio Torralba","year":"2019","venue":"TPAMI","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Min et al., 2018] Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu, Yong Rui, and Shuqiang Jiang. You are what you eat: Exploring rich recipe information for cross-region food analysis. IEEE Trans. Multimed., 2018.","title":"You are what you eat: Exploring rich recipe information for cross-region food analysis","authors":"Weiqing Min, Bing-Kun Bao, Shuhuan Mei, Yaohui Zhu, Yong Rui, Shuqiang Jiang","year":"2018","venue":"IEEE Trans. Multimed","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Min et al., 2019] Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, and Ramesh Jain. A survey on food computing. arXiv preprint arXiv:1808.07202, 2019.","title":"A survey on food computing","authors":"Weiqing Min, Shuqiang Jiang, Linhu Liu, Yong Rui, Ramesh Jain","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pandey et al., 2017] Paritosh Pandey, Akella Deepthi, Bappaditya Mandal, and Niladri B. Puhan. Foodnet: Recognizing foods using ensemble of deep networks. IEEE Signal Processing Letters, 2017.","title":"Foodnet: Recognizing foods using ensemble of deep networks","authors":"Paritosh Pandey, Akella Deepthi, Bappaditya Mandal, Niladri B Puhan","year":"2017","venue":"IEEE Signal Processing Letters","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Rokicki et al., 2018] M. Rokicki, C. Trattner, and E. Herder. The impact of recipe features, social cues and demographics on estimating the healthiness of online recipes. In ICWSM, 2018.","title":"The impact of recipe features, social cues and demographics on estimating the healthiness of online recipes","authors":"M Rokicki, C Trattner, E Herder","year":"2018","venue":"ICWSM","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Salvador et al., 2021] Amaia Salvador, Erhan Gundogdu, Loris Bazzani, and Michael Donoser. Revamping cross- modal recipe retrieval with hierarchical transformers and self-supervised learning. In CVPR, 2021.","title":"Revamping crossmodal recipe retrieval with hierarchical transformers and self-supervised learning","authors":"Amaia Salvador, Erhan Gundogdu, Loris Bazzani, Michael Donoser","year":"2021","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Schlichtkrull et al., 2018] Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC, 2018.","title":"Modeling relational data with graph convolutional networks","authors":"Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den, Ivan Berg, Max Titov, Welling","year":"2018","venue":"ESWC","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Tian et al., 2021] Yijun Tian, Chuxu Zhang, Ronald Metoyer, and Nitesh V. Chawla. Recipe representation learning with networks. In CIKM, 2021.","title":"Recipe representation learning with networks","authors":"Yijun Tian, Chuxu Zhang, Ronald Metoyer, Nitesh V Chawla","year":"2021","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Tian et al., 2022a] Yijun Tian, Chuxu Zhang, Zhichun Guo, Chao Huang, Ronald Metoyer, and Nitesh V. Chawla. Reciperec: A heterogeneous graph learning model for recipe recommendation. In IJCAI, 2022.","title":"Reciperec: A heterogeneous graph learning model for recipe recommendation","authors":"Tian","year":"2022","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2022/481"},{"position":20,"raw":"[Tian et al., 2022b] Yijun Tian, Chuxu Zhang, Ronald Metoyer, and Nitesh V. Chawla. Recipe recommendation with hierarchical graph attention network. Frontiers in Big Data, 2022. [USDA, 2019] USDA. National nutrient database for standard reference, release 27, 2019.","title":"Recipe recommendation with hierarchical graph attention network","authors":"Tian","year":"2019","venue":"Frontiers in Big Data","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Veličković et al., 2018] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.","title":"Graph attention networks","authors":"Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, Yoshua Bengio","year":"2018","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Wang et al., 2019a] Hao Wang, Doyen Sahoo, Chenghao Liu, Ee-Peng Lim, and Steven C. H. Hoi. Learning cross- modal embeddings with adversarial networks for cooking recipes and food images. In CVPR, 2019.","title":"Learning crossmodal embeddings with adversarial networks for cooking recipes and food images","authors":"Wang","year":"2019","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Wang et al., 2019b] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Peng Cui, Philip S. Yu, and Yanfang Ye. Heterogeneous graph attention network. In WWW, 2019.","title":"Heterogeneous graph attention network","authors":"Wang","year":"2019","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Xie et al., 2020] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In CVPR, 2020.","title":"Adversarial examples improve image recognition","authors":"Xie","year":"2020","venue":"CVPR","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Xu et al., 2019] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In ICLR, 2019.","title":"How powerful are graph neural networks?","authors":"Xu","year":"2019","venue":"ICLR","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhang et al., 2019a] Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V. Chawla. Heterogeneous graph neural network. In KDD, 2019.","title":"Heterogeneous graph neural network","authors":"Zhang","year":"2019","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Zhang et al., 2019b] Chuxu Zhang, Ananthram Swami, and Nitesh V. Chawla. Shne: Representation learning for semantic-associated heterogeneous networks. In WSDM, 2019.","title":"Shne: Representation learning for semantic-associated heterogeneous networks","authors":"Zhang","year":"2019","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}