{"type":"ScholarlyArticle","title":"DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis","authors":[{"name":"Sunny Verma","slug":"sunny-verma","given":"Sunny","family":"Verma","affiliation":"Advanced Analytics Institute , School of Computer Science , University of Technology Sydney , Australia; Commonwealth Scientific and Industrial Research Organisation , CSIRO , Data61 , Sydney , Australia"},{"name":"Chen Wang","slug":"chen-wang","given":"Chen","family":"Wang","affiliation":"Commonwealth Scientific and Industrial Research Organisation , CSIRO , Data61 , Sydney , Australia"},{"name":"Liming Zhu","slug":"liming-zhu","given":"Liming","family":"Zhu","affiliation":"Commonwealth Scientific and Industrial Research Organisation , CSIRO , Data61 , Sydney , Australia"},{"name":"Wei Liu","slug":"wei-liu","given":"Wei","family":"Liu","affiliation":"Advanced Analytics Institute , School of Computer Science , University of Technology Sydney , Australia"}],"abstract":"Multimodal sentiment analysis combines information available from visual, textual, and acoustic representations for sentiment prediction. The recent multimodal fusion schemes combine multiple modalities as a tensor and obtain either; the common information by utilizing neural networks, or the unique information by modeling low-rank representation of the tensor.  However, both of these information are essential as they render inter-modal and intra-modal relationships of the data. In this research, we first propose a novel deep architecture to extract the common information from the multi-mode representations. Furthermore, we propose unique networks to obtain the modality-specific information that enhances the generalization performance of our multimodal system. Finally, we integrate these two aspects of information via a fusion layer and propose a novel multimodal data fusion architecture, which we call DeepCU (Deep network with both Common and Unique latent information). The proposed DeepCU consolidates the two networks for joint utilization and discovery of all-important latent information. Comprehensive experiments are conducted to demonstrate the effectiveness of utilizing both common and unique information discovered by DeepCU on multiple real-world datasets. The source code of proposed DeepCU is available at https://github.com/sverma88/DeepCU-IJCAI19.","doi":"10.24963/ijcai.2019/503","url":"https://ijcai.org/proceedings/2019/503","pdf":"https://ijcai.org/proceedings/2019/0503.pdf","markdown":"https://ijcai.org/proceedings/2019/503.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/503","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning M-Z","track":"Main track","number":503,"pages":{"first":3627,"last":3634},"keywords":["Natural Language Processing: Sentiment Analysis and Text Mining","Machine Learning: Deep Learning","Machine Learning: Data Mining","Machine Learning: Feature Selection ; Learning Sparse Models"],"videos":[],"references":[{"position":1,"raw":"[Baltrušaitis et al., 2018] Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.","title":"Multimodal machine learning: A survey and taxonomy","authors":"Tadas Baltrušaitis, Chaitanya Ahuja, Louis-Philippe Morency","year":"2018","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Fukui et al., 2016] Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457-468, 2016.","title":"Multimodal compact bilinear pooling for visual question answering and visual grounding","authors":"Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, Marcus Rohrbach","year":"2016","venue":"Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Guo et al., 2017] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr predic- tion. In Proceedings of the 26th International Joint Con- ference on Artificial Intelligence. AAAI Press, 2017.","title":"Deepfm: a factorization-machine based neural network for ctr prediction","authors":"Guo","year":"2017","venue":"Proceedings of the 26th International Joint Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":"/proceedings/2017/239"},{"position":4,"raw":"[He and Chua, 2017] Xiangnan He and Tat-Seng Chua. Neu- ral factorization machines for sparse predictive analytics. In Proceedings of the 40th International ACM SIGIR con- ference on Research and Development in Information Re- trieval, pages 355-364. ACM, 2017.","title":"Xiangnan He and Tat-Seng Chua. Neural factorization machines for sparse predictive analytics","authors":"Chua He","year":"2017","venue":"Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[He and Sun, 2015] Kaiming He and Jian Sun. Convolu- tional neural networks at constrained time cost. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 5353-5360, 2015.","title":"Convolutional neural networks at constrained time cost","authors":"Sun ; He, Kaiming He, Jian Sun","year":"2015","venue":"Proceedings of the IEEE conference on computer vision and pattern recognition","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.","title":"Long short-term memory","authors":"Schmidhuber Hochreiter, Sepp Hochreiter, Jürgen Schmidhuber","year":"1997","venue":"Neural computation","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Ioffe and Szegedy, 2015] Sergey Ioffe and Christian Szegedy. Batch normalization: accelerating deep net- work training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37. JMLR. org, 2015.","title":"Batch normalization: accelerating deep network training by reducing internal covariate shift","authors":"Szegedy Ioffe, Sergey Ioffe, Christian Szegedy","year":"2015","venue":"Proceedings of the 32nd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim et al., 2016] Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, and Hwanjo Yu. Convolutional ma- trix factorization for document context-aware recommen- dation. In Proceedings of the 10th ACM Conference on Recommender Systems, pages 233-240. ACM, 2016.","title":"Convolutional matrix factorization for document context-aware recommendation","authors":"Donghyun Kim, Chanyoung Park, Jinoh Oh, Sungyoung Lee, Hwanjo Yu","year":"2016","venue":"Proceedings of the 10th ACM Conference on Recommender Systems","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Lahat et al., 2015] Dana Lahat, Tülay Adali, and Christian Jutten. Multimodal data fusion: an overview of methods, challenges, and prospects. Proceedings of the IEEE, 2015.","title":"Multimodal data fusion: an overview of methods, challenges, and prospects","authors":"Dana Lahat, Tülay Adali, Christian Jutten","year":"2015","venue":"Proceedings of the IEEE","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Lin et al., 2015] Tsung-Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear cnn models for fine-grained vi- sual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 1449-1457, 2015.","title":"Bilinear cnn models for fine-grained visual recognition","authors":"Tsung-Yu Lin, Aruni Roychowdhury, Subhransu Maji","year":"2015","venue":"Proceedings of the IEEE International Conference on Computer Vision","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Liu et al., 2013] Wei Liu, Jeffrey Chan, James Bailey, Christopher Leckie, and Kotagiri Ramamohanarao. Min- ing labelled tensors by discovering both their common and discriminative subspaces. In Proceedings of the 2013 SIAM International Conference on Data Mining, 2013.","title":"Mining labelled tensors by discovering both their common and discriminative subspaces","authors":"Wei Liu, Jeffrey Chan, James Bailey, Christopher Leckie, Kotagiri Ramamohanarao","year":"2013","venue":"Proceedings of the 2013 SIAM International Conference on Data Mining","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Liu et al., 2018] Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Efficient low-rank multimodal fusion with modality-specific factors. ACL, 2018.","title":"Efficient low-rank multimodal fusion with modality-specific factors","authors":"Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency","year":"2018","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Park et al., 2014] Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. Computational analysis of persuasiveness in social mul- timedia: A novel dataset and multimodal prediction ap- proach. In Proceedings of the 16th International Confer- ence on Multimodal Interaction. ACM, 2014.","title":"Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach","authors":"Sunghyun Park, Suk Han, Moitreya Shim, Kenji Chatterjee, Louis-Philippe Sagae, Morency","year":"2014","venue":"Proceedings of the 16th International Conference on Multimodal Interaction","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Pérez-Rosas et al., 2013] Verónica Pérez-Rosas, Rada Mi- halcea, and Louis-Philippe Morency. Utterance-level mul- timodal sentiment analysis. In Proceedings of the 51st An- nual Meeting of the Association for Computational Lin- guistics (Volume 1), pages 973-982, 2013.","title":"Utterance-level multimodal sentiment analysis","authors":"Verónica Pérez-Rosas, Rada Mihalcea, Louis-Philippe Morency","year":"2013","venue":"Proceedings of the 51st An-nual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Poria et al., 2015] Soujanya Poria, Erik Cambria, and Alexander Gelbukh. Deep convolutional neural network textual features and multiple kernel learning for utterance- level multimodal sentiment analysis. In Proceedings of the 2015 conference on empirical methods in natural lan- guage processing, pages 2539-2544, 2015.","title":"Deep convolutional neural network textual features and multiple kernel learning for utterancelevel multimodal sentiment analysis","authors":"Poria","year":"2015","venue":"Proceedings of the 2015 conference on empirical methods in natural language processing","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Poria et al., 2016] Soujanya Poria, Erik Cambria, Newton Howard, Guang-Bin Huang, and Amir Hussain. Fusing audio, visual and textual clues for sentiment analysis from multimodal content. Neurocomputing, 174:50-59, 2016.","title":"Fusing audio, visual and textual clues for sentiment analysis from multimodal content","authors":"Poria","year":"2016","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Poria et al., 2017] Soujanya Poria, Erik Cambria, Deva- manyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017.","title":"Context-dependent sentiment analysis in user-generated videos","authors":"Poria","year":"2017","venue":"Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Rendle, 2010s] Steffen Rendle. Factorization machines. In Data Mining (ICDM), 2010 IEEE 10th International Con- ference on, pages 995-1000. IEEE, 2010s.","title":"Factorization machines","authors":"Steffen Rendle, Rendle","year":"2010","venue":"IEEE 10th International Conference on","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Srivastava et al., 2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Re- search, 15(1):1929-1958, 2014.","title":"Dropout: a simple way to prevent neural networks from overfitting","authors":"Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, Ruslan Salakhutdinov","year":"2014","venue":"The Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Verma et al., 2017] Sunny Verma, Wei Liu, Chen Wang, and Liming Zhu. Extracting highly effective features for supervised learning via simultaneous tensor factorization. In AAAI, pages 4995-4996, 2017.","title":"Extracting highly effective features for supervised learning via simultaneous tensor factorization","authors":"Sunny Verma, Wei Liu, Chen Wang, Liming Zhu","year":"2017","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Verma et al., 2018] Sunny Verma, Wei Liu, Chen Wang, and Liming Zhu. Hybrid networks: Improving deep learn- ing networks via integrating two views of images. In Neu- ral Information Processing -25th International Confer- ence, ICONIP , Proceedings, Part I, pages 46-58, 2018.","title":"Hybrid networks: Improving deep learning networks via integrating two views of images","authors":"Sunny Verma, Wei Liu, Chen Wang, Liming Zhu","year":"2018","venue":"Neural Information Processing -25th International Conference, ICONIP , Proceedings, Part I","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zadeh et al., 2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Multimodal sentiment in- tensity analysis in videos: Facial gestures and verbal mes- sages. IEEE Intelligent Systems, 31(6):82-88, 2016.","title":"Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages","authors":"Amir Zadeh, Rowan Zellers, Eli Pincus, Louis-Philippe Morency","year":"2016","venue":"IEEE Intelligent Systems","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zadeh et al., 2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. In Pro- ceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1103-1114, 2017.","title":"Tensor fusion network for multimodal sentiment analysis","authors":"Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, Louis-Philippe Morency","year":"2017","venue":"Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Zadeh et al., 2018a] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis- Philippe Morency. Memory fusion network for multi-view sequential learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, 2018.","title":"Memory fusion network for multi-view sequential learning","authors":"Zadeh","year":"2018","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zadeh et al., 2018b] Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. Multi-attention recurrent network for human communication comprehension. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, 2018.","title":"Multi-attention recurrent network for human communication comprehension","authors":"Zadeh","year":"2018","venue":"Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}