{"type":"ScholarlyArticle","title":"Hashtag Recommendation for Multimodal Microblog Using Co-Attention Network","authors":[{"name":"Qi Zhang","slug":"qi-zhang","given":"Qi","family":"Zhang","affiliation":"School of Computer Science , Fudan University Shanghai Key Laboratory of Intelligent Information Processing 825 Zhangheng Road , Shanghai , P.R. China"},{"name":"Jiawen Wang","slug":"jiawen-wang","given":"Jiawen","family":"Wang","affiliation":"School of Computer Science , Fudan University Shanghai Key Laboratory of Intelligent Information Processing 825 Zhangheng Road , Shanghai , P.R. China"},{"name":"Haoran Huang","slug":"haoran-huang","given":"Haoran","family":"Huang","affiliation":"School of Computer Science , Fudan University Shanghai Key Laboratory of Intelligent Information Processing 825 Zhangheng Road , Shanghai , P.R. China"},{"name":"Xuanjing Huang","slug":"xuanjing-huang","given":"Xuanjing","family":"Huang","affiliation":"School of Computer Science , Fudan University Shanghai Key Laboratory of Intelligent Information Processing 825 Zhangheng Road , Shanghai , P.R. China"},{"name":"Yeyun Gong","slug":"yeyun-gong","given":"Yeyun","family":"Gong","affiliation":"School of Computer Science , Fudan University Shanghai Key Laboratory of Intelligent Information Processing 825 Zhangheng Road , Shanghai , P.R. China"}],"abstract":"In microblogging services, authors can use hashtags to mark keywords or topics. Many live social media applications (e.g., microblog retrieval, classification) can gain great benefits from these manually labeled tags. However, only a small portion of microblogs contain hashtags inputed by users. Moreover, many microblog posts contain not only textual content but also images. These visual resources also provide valuable information that may not be included in the textual content. So that it can also help to recommend hashtags more accurately. Motivated by the successful use of the attention mechanism, we propose a co-attention network incorporating textual and visual information to recommend hashtags for multimodal tweets. Experimental result on the data collected from Twitter demonstrated that the proposed method can achieve better performance than state-of-the-art methods using textual information only.","doi":"10.24963/ijcai.2017/478","url":"https://ijcai.org/proceedings/2017/478","pdf":"https://ijcai.org/proceedings/2017/0478.pdf","markdown":"https://ijcai.org/proceedings/2017/478.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/478","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning S-Z","track":"Main track","number":478,"pages":{"first":3420,"last":3426},"keywords":["Multidisciplinary Topics and Applications: AI and Social Sciences","Machine Learning: Neural Networks"],"videos":[],"references":[{"position":1,"raw":"[Antol et al., 2015] Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425-2433, 2015.","title":"Vqa: Visual question answering","authors":"Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, Devi Parikh","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Bandyopadhyay et al., 2012] Ayan Bandyopadhyay, Kripa- bandhu Ghosh, Prasenjit Majumder, and Mandar Mitra. Query expansion for microblog retrieval. volume 1, pages 368-380. Inderscience Publishers Ltd, 2012.","title":"Query expansion for microblog retrieval","authors":"Ayan Bandyopadhyay, Kripabandhu Ghosh, Prasenjit Majumder, Mandar Mitra","year":"2012","venue":null,"doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Becker et al., 2010] Hila Becker, Mor Naaman, and Luis Gravano. Learning similarity metrics for event identifica- tion in social media. In Proceedings of WSDM '10, 2010.","title":"Learning similarity metrics for event identification in social media","authors":"Hila Becker, Mor Naaman, Luis Gravano","year":"2010","venue":"Proceedings of WSDM '10","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Bermingham and Smeaton, 2010] Adam Bermingham and Alan F. Smeaton. Classifying sentiment in microblogs: is brevity an advantage? In Proceedings of CIKM'10, 2010.","title":"Classifying sentiment in microblogs: is brevity an advantage?","authors":"Smeaton Bermingham, Adam Bermingham, Alan F Smeaton","year":"2010","venue":"Proceedings of CIKM'10","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Bollen et al., 2011] Johan Bollen, Huina Mao, and Xiaojun Zeng. Twitter mood predicts the stock market. Journal of Computational Science, 2(1):1 -8, 2011.","title":"Twitter mood predicts the stock market","authors":"Johan Bollen, Huina Mao, Xiaojun Zeng","year":"2011","venue":"Journal of Computational Science","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Chen and Zitnick, 2014] Xinlei Chen and C Lawrence Zit- nick. Learning a recurrent visual representation for image caption generation. arXiv preprint arXiv:1411.5654, 2014.","title":"Xinlei Chen and C Lawrence Zitnick. Learning a recurrent visual representation for image caption generation","authors":"Zitnick Chen","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Chen et al., 2008] Hong-Ming Chen, Ming-Hsiu Chang, Ping-Chieh Chang, Ming-Chun Tien, Winston H Hsu, and Ja-Ling Wu. Sheepdog: group and tag recommendation for flickr photos by automatic search-based learning. In Proceedings of the 16th ACM international conference on Multimedia, pages 737-740. ACM, 2008.","title":"Sheepdog: group and tag recommendation for flickr photos by automatic search-based learning","authors":"Chen","year":"2008","venue":"Proceedings of the 16th ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Ding et al., 2013] Zhuoye Ding, Xipeng Qiu, Qi Zhang, and Xuanjing Huang. Learning topical translation model for microblog hashtag suggestion. In IJCAI. Citeseer, 2013.","title":"Learning topical translation model for microblog hashtag suggestion","authors":"Ding","year":"2013","venue":"IJCAI. Citeseer","doi":null,"url":null,"internal_url":"/proceedings/2013/307"},{"position":9,"raw":"[Efron, 2010] Miles Efron. Hashtag retrieval in a microblog- ging environment. In Proceedings of SIGIR '10, 2010.","title":"Hashtag retrieval in a microblogging environment","authors":"Miles Efron, Efron","year":"2010","venue":"Proceedings of SIGIR '10","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Garg and Weber, 2008] Nikhil Garg and Ingmar Weber. Personalized, interactive tag recommendation for flickr. In Proceedings of the 2008 ACM conference on Recom- mender systems, pages 67-74. ACM, 2008.","title":"Personalized, interactive tag recommendation for flickr","authors":"Nikhil Garg, Ingmar Weber","year":"2008","venue":"Proceedings of the 2008 ACM conference on Recommender systems","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Gong and Zhang, 2016] Yuyun Gong and Qi Zhang. Hash- tag recommendation using attention-based convolutional neural network. In IJCAI 2016, Proceedings of the 26rd International Joint Conference on Artificial Intelligence, New York City, USA, 2016.","title":"Hashtag recommendation using attention-based convolutional neural network","authors":"Zhang ; ] Yuyun Gong, Qi Gong, Zhang","year":"2016","venue":"Proceedings of the 26rd International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2016/395"},{"position":12,"raw":"[Guy et al., 2013] Ido Guy, Uri Avraham, David Carmel, Sigalit Ur, Michal Jacovi, and Inbal Ronen. Mining expertise and interests from social media. In Proceedings of WWW '13, 2013.","title":"Mining expertise and interests from social media","authors":"Ido Guy, Uri Avraham, David Carmel, Sigalit Ur, Michal Jacovi, Inbal Ronen","year":"2013","venue":"Proceedings of WWW '13","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Heymann et al., 2008] Paul Heymann, Daniel Ramage, and Hector Garcia-Molina. Social tag prediction. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 531-538. ACM, 2008.","title":"Social tag prediction","authors":"Paul Heymann, Daniel Ramage, Hector Garcia-Molina","year":"2008","venue":"Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kingma and Ba, 2014] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.","title":"Adam: A method for stochastic optimization","authors":"Diederik Kingma, Jimmy Ba","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Krestel et al., 2009] Ralf Krestel, Peter Fankhauser, and Wolfgang Nejdl. Latent dirichlet allocation for tag recommendation. In Proceedings of the third ACM conference on Recommender systems, pages 61-68. ACM, 2009.","title":"Latent dirichlet allocation for tag recommendation","authors":"Ralf Krestel, Peter Fankhauser, Wolfgang Nejdl","year":"2009","venue":"Proceedings of the third ACM conference on Recommender systems","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Kywe et al., 2012] Su Mon Kywe, Tuan-Anh Hoang, Ee- Peng Lim, and Feida Zhu. On recommending hashtags in twitter networks. In International Conference on Social Informatics, pages 337-350. Springer, 2012.","title":"On recommending hashtags in twitter networks","authors":"Kywe","year":"2012","venue":"International Conference on Social Informatics","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Li and Snoek, 2013] Xirong Li and Cees GM Snoek. Clas- sifying tag relevance with relevant positive and negative examples. In Proceedings of the 21st ACM international conference on Multimedia, pages 485-488. ACM, 2013.","title":"Classifying tag relevance with relevant positive and negative examples","authors":"Snoek ; Li, Xirong Li, G M Cees, Snoek","year":"2013","venue":"Proceedings of the 21st ACM international conference on Multimedia","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Lu et al., 2009] Yu-Ta Lu, Shoou-I Yu, Tsung-Chieh Chang, and Jane Yung-jen Hsu. A content-based method to enhance tag recommendation. In IJCAI, volume 9, pages 2064-2069, 2009.","title":"A content-based method to enhance tag recommendation","authors":"Yu-Ta Lu, Shoou-I Yu, Tsung-Chieh Chang, Jane Yung, -Jen Hsu","year":"2009","venue":"IJCAI","doi":null,"url":null,"internal_url":"/proceedings/2009/340"},{"position":19,"raw":"[Ma et al., 2015] Lin Ma, Zhengdong Lu, and Hang Li. Learning to answer questions from image using convolu- tional neural network. arXiv preprint arXiv:1506.00333, 2015.","title":"Learning to answer questions from image using convolutional neural network","authors":"Lin Ma, Zhengdong Lu, Hang Li","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Ohkura et al., 2006] Tsutomu Ohkura, Yoji Kiyota, and Hiroshi Nakagawa. Browsing system for weblog articles based on automated folksonomy. In Proceedings of the WWW 2006 Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics, at WWW, volume 2006, 2006.","title":"Browsing system for weblog articles based on automated folksonomy","authors":"Tsutomu Ohkura, Yoji Kiyota, Hiroshi Nakagawa","year":"2006","venue":"Proceedings of the WWW 2006 Workshop on the Weblogging Ecosystem: Aggregation, Analysis and Dynamics","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Sakaki et al., 2010] Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of WWW '10, 2010.","title":"Earthquake shakes twitter users: real-time event detection by social sensors","authors":"Takeshi Sakaki, Makoto Okazaki, Yutaka Matsuo","year":"2010","venue":"Proceedings of WWW '10","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Sigurbjörnsson and Van Zwol, 2008] Börkur Sigurbjörnsson and Roelof Van Zwol. Flickr tag recommendation based on collective knowledge. In Proceedings of the 17th international conference on World Wide Web, pages 327-336. ACM, 2008.","title":"Börkur Sigurbjörnsson and Roelof Van Zwol. Flickr tag recommendation based on collective knowledge","authors":"Van Sigurbjörnsson, Zwol","year":"2008","venue":"Proceedings of the 17th international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Simonyan and Zisserman, 2014] Karen Simonyan and An- drew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.","title":"Very deep convolutional networks for large-scale image recognition","authors":"Karen Simonyan, Andrew Zisserman","year":"2014","venue":null,"doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey E Hin- ton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Tariq et al., 2013] Amara Tariq, Asim Karim, Fernando Gomez, and Hassan Foroosh. Exploiting topical perceptions over multi-lingual text for hashtag suggestion on twitter. In FLAIRS Conference, 2013.","title":"Exploiting topical perceptions over multi-lingual text for hashtag suggestion on twitter","authors":"Amara Tariq, Asim Karim, Fernando Gomez, Hassan Foroosh","year":"2013","venue":"FLAIRS Conference","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Vinyals et al., 2015] Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156-3164, 2015.","title":"Show and tell: A neural image caption generator","authors":"Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan","year":"2015","venue":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Wang et al., 2011] Xiaolong Wang, Furu Wei, Xiaohua Liu, Ming Zhou, and Ming Zhang. Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. In Proceedings of CIKM '11, 2011.","title":"Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach","authors":"Xiaolong Wang, Furu Wei, Xiaohua Liu, Ming Zhou, Ming Zhang","year":"2011","venue":"Proceedings of CIKM '11","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Yang et al., 2015] Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. arXiv preprint arXiv:1511.02274, 2015.","title":"Stacked attention networks for image question answering","authors":"Yang","year":"2015","venue":null,"doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}