{"type":"ScholarlyArticle","title":"Semantic Visualization for Short Texts with Word Embeddings","authors":[{"name":"Tuan M. V. Le","slug":"tuan-m-v-le","given":"Tuan M. V.","family":"Le","affiliation":"School of Information Systems Singapore Management University"},{"name":"Hady W. Lauw","slug":"hady-w-lauw","given":"Hady W.","family":"Lauw","affiliation":"School of Information Systems Singapore Management University"}],"abstract":"Semantic visualization integrates topic modeling and visualization, such that every document is associated with a topic distribution as well as visualization coordinates on a low-dimensional Euclidean space.  We address the problem of semantic visualization for short texts.  Such documents are increasingly common, including tweets, search snippets, news headlines, or status updates.  Due to their short lengths, it is difficult to model semantics as the word co-occurrences in such a corpus are very sparse.  Our approach is to incorporate auxiliary information, such as word embeddings from a larger corpus, to supplement the lack of co-occurrences.  This requires the development of a novel semantic visualization model that seamlessly integrates visualization coordinates, topic distributions, and word vectors.  We propose a model called GaussianSV, which outperforms pipelined baselines that derive topic models and visualization coordinates as disjoint steps, as well as semantic visualization baselines that do not consider word embeddings.","doi":"10.24963/ijcai.2017/288","url":"https://ijcai.org/proceedings/2017/288","pdf":"https://ijcai.org/proceedings/2017/0288.pdf","markdown":"https://ijcai.org/proceedings/2017/288.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/288","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Machine Learning A-R","track":"Main track","number":288,"pages":{"first":2074,"last":2080},"keywords":["Machine Learning: Data Mining","Machine Learning: Feature Selection/Construction","Machine Learning: Learning Graphical Models"],"videos":[],"references":[{"position":1,"raw":"[Arora et al., 2012] Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models-going beyond svd. In FOCS, pages 1-10. IEEE, 2012.","title":"Learning topic models-going beyond svd","authors":"Sanjeev Arora, Rong Ge, Ankur Moitra","year":"2012","venue":"FOCS","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blei et al., 2003] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. JMLR, 3, 2003.","title":"Latent dirichlet allocation","authors":"D M Blei, A Y Ng, M I Jordan","year":"2003","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Brants and Franz, 2006] Thorsten Brants and Alex Franz. Web 1T 5-gram Version 1. Linguistic Data Consortium, Philadelphia, 2006.","title":"Web 1T 5-gram Version 1. Linguistic Data Consortium","authors":"Franz ; Brants, Thorsten Brants, Alex Franz","year":"2006","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Das et al., 2015] Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian LDA for topic models with word embed- dings. In ACL, 2015.","title":"Gaussian LDA for topic models with word embeddings","authors":"Rajarshi Das, Manzil Zaheer, Chris Dyer","year":"2015","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Deerwester et al., 1990] Scott Deerwester, Susan T Du- mais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. JASIS, 41(6):391, 1990.","title":"Indexing by latent semantic analysis","authors":"Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, Richard Harshman","year":"1990","venue":"JASIS","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Dempster et al., 1977] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical So- ciety, Series B, 39(1):1-38, 1977.","title":"Maximum likelihood from incomplete data via the EM algorithm","authors":"A P Dempster, N M Laird, D B Rubin","year":"1977","venue":"Journal of the Royal Statistical Society, Series B","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[der Maaten and Hinton, 2008] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. JMLR, 9, 2008.","title":"Visualizing data using t-SNE","authors":"L Van Der Maaten, G Hinton","year":"2008","venue":"JMLR","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Greene and Cunningham, 2006] Derek Greene and Pádraig Cunningham. Practical solutions to the problem of diag- onal dominance in kernel document clustering. In ICML, pages 377-384, 2006.","title":"Practical solutions to the problem of diagonal dominance in kernel document clustering","authors":"Cunningham Greene, Derek Greene, Pádraig Cunningham","year":"2006","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hofmann, 1999] T. Hofmann. Probabilistic latent semantic indexing. In SIGIR, 1999.","title":"Probabilistic latent semantic indexing","authors":"T Hofmann","year":"1999","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Hu and Tsujii, 2016] Weihua Hu and Junichi Tsujii. A la- tent concept topic model for robust topic inference using word embeddings. In ACL, page 380, 2016.","title":"A latent concept topic model for robust topic inference using word embeddings","authors":"Tsujii ; ] Weihua Hu, Junichi Hu, Tsujii","year":"2016","venue":"ACL","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Iwata et al., 2007] T. Iwata, K. Saito, N. Ueda, S. Strom- sten, T. L. Griffiths, and J. B. Tenenbaum. Parametric embedding for class visualization. Neural Computation, 19(9), 2007.","title":"Parametric embedding for class visualization","authors":"T Iwata, K Saito, N Ueda, S Stromsten, T L Griffiths, J B Tenenbaum","year":"2007","venue":"Neural Computation","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Iwata et al., 2008] Tomoharu Iwata, Takeshi Yamada, and Naonori Ueda. Probabilistic latent semantic visualization: topic model for visualizing documents. In KDD, pages 363-371, 2008.","title":"Probabilistic latent semantic visualization: topic model for visualizing documents","authors":"Tomoharu Iwata, Takeshi Yamada, Naonori Ueda","year":"2008","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Jin et al., 2011] Ou Jin, Nathan N Liu, Kai Zhao, Yong Yu, and Qiang Yang. Transferring topical knowledge from auxiliary long texts for short text clustering. In CIKM, pages 775-784, 2011.","title":"Transferring topical knowledge from auxiliary long texts for short text clustering","authors":"Ou Jin, Nathan N Liu, Kai Zhao, Yong Yu, Qiang Yang","year":"2011","venue":"CIKM","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Kiros et al., 2015] Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In NIPS, pages 3294-3302, 2015.","title":"Skip-thought vectors","authors":"Ryan Kiros, Yukun Zhu, R Ruslan, Richard Salakhutdinov, Raquel Zemel, Antonio Urtasun, Sanja Torralba, Fidler","year":"2015","venue":"NIPS","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Kruskal, 1964] J. B. Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psy- chometrika, 29(1), 1964.","title":"Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis","authors":"J B","year":"1964","venue":"Psychometrika","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Le and Lauw, 2014a] Tuan M V Le and Hady W Lauw. Manifold learning for jointly modeling topic and visual- ization. In AAAI, 2014.","title":"Manifold learning for jointly modeling topic and visualization","authors":"Le, Lauw Tuan, M V Le, Hady W Lauw","year":"2014","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Le and Lauw, 2014b] Tuan M V Le and Hady W Lauw. Se- mantic visualization for spherical representation. In KDD, pages 1007-1016, 2014.","title":"Semantic visualization for spherical representation","authors":"Le, Lauw ; Tuan, M V Le, Hady W Lauw","year":"2014","venue":"KDD","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Le and Mikolov, 2014] Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, volume 14, pages 1188-1196, 2014.","title":"Distributed representations of sentences and documents","authors":"V Quoc, Tomas Le, Mikolov","year":"2014","venue":"ICML","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Li et al., 2016] Chenliang Li, Haoran Wang, Zhiqian Zhang, Aixin Sun, and Zongyang Ma. Topic modeling for short texts with auxiliary word embeddings. In SIGIR, pages 165-174, 2016.","title":"Topic modeling for short texts with auxiliary word embeddings","authors":"Li","year":"2016","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Liu and Nocedal, 1989] Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale opti- mization. Mathematical Programming, 45:503-528, 1989.","title":"On the limited memory BFGS method for large scale optimization","authors":"Dong C Liu, Jorge Nocedal","year":"1989","venue":"Mathematical Programming","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Metzler et al., 2007] Donald Metzler, Susan Dumais, and Christopher Meek. Similarity measures for short segments of text. In ECIR, pages 16-27, 2007.","title":"Similarity measures for short segments of text","authors":"Donald Metzler, Susan Dumais, Christopher Meek","year":"2007","venue":"ECIR","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Cor- rado, and Jeffrey Dean. Efficient estimation of word rep- resentations in vector space. In Proceedings of Workshop at ICLR, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":"Proceedings of Workshop at ICLR","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Newman et al., 2010] David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In NAACL HLT, pages 100-108, 2010.","title":"Automatic evaluation of topic coherence","authors":"David Newman, Jey, Han Lau, Karl Grieser, Timothy Baldwin","year":"2010","venue":"NAACL HLT","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Nguyen et al., 2015] Dat Quoc Nguyen, Richard Billings- ley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. TACL, 3:299- 313, 2015.","title":"Improving topic models with latent feature word representations","authors":"Nguyen","year":"2015","venue":"TACL","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Nigam et al., 2000] Kamal Nigam, Andrew Kachites Mc- Callum, Sebastian Thrun, and Tom Mitchell. Text classi- fication from labeled and unlabeled documents using em. Machine Learning, 39(2-3):103-134, 2000.","title":"Text classification from labeled and unlabeled documents using em","authors":"Kamal Nigam, Andrew Kachites Mc-Callum, Sebastian Thrun, Tom Mitchell","year":"2000","venue":"Machine Learning","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. EMNLP, 12, 2014.","title":"GloVe: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":27,"raw":"[Phan et al., 2008] Xuan-Hieu Phan, Le-Minh Nguyen, and Susumu Horiguchi. Learning to classify short and sparse text & web with hidden topics from large-scale data col- lections. In WWW, pages 91-100, 2008.","title":"Learning to classify short and sparse text & web with hidden topics from large-scale data collections","authors":"Xuan-Hieu Phan, Le-Minh Nguyen, Susumu Horiguchi","year":"2008","venue":"WWW","doi":null,"url":null,"internal_url":null},{"position":28,"raw":"[Roweis and Saul, 2000] S. T. Roweis and L. K. Saul. Non- linear dimensionality reduction by locally linear embed- ding. Science, 290, 2000.","title":"Nonlinear dimensionality reduction by locally linear embedding","authors":"S T Roweis, L K Saul","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":29,"raw":"[Sriram et al., 2010] Bharath Sriram, Dave Fuhry, Engin Demir, Hakan Ferhatosmanoglu, and Murat Demirbas. Short text classification in twitter to improve information filtering. In SIGIR, pages 841-842, 2010.","title":"Short text classification in twitter to improve information filtering","authors":"Bharath Sriram, Dave Fuhry, Engin Demir, Hakan Ferhatosmanoglu, Murat Demirbas","year":"2010","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":30,"raw":"[Sun, 2012] Aixin Sun. Short text classification using very few words. In SIGIR, pages 1145-1146. ACM, 2012.","title":"Short text classification using very few words","authors":"Aixin Sun, Sun","year":"2012","venue":"SIGIR","doi":null,"url":null,"internal_url":null},{"position":31,"raw":"[Tenenbaum et al., 2000] J. B. Tenenbaum, V. De Silva, and J. C. Langford. A global geometric framework for nonlin- ear dimensionality reduction. Science, 290, 2000.","title":"A global geometric framework for nonlinear dimensionality reduction","authors":"J B Tenenbaum, V De Silva, J C Langford","year":"2000","venue":"Science","doi":null,"url":null,"internal_url":null},{"position":32,"raw":"[Yan et al., 2012] Xiaohui Yan, Jiafeng Guo, Shenghua Liu, Xue-qi Cheng, and Yanfeng Wang. Clustering short text using ncut-weighted non-negative matrix factorization. In CIKM, pages 2259-2262, 2012.","title":"Clustering short text using ncut-weighted non-negative matrix factorization","authors":"Xiaohui Yan, Jiafeng Guo, Shenghua Liu, Xue-Qi Cheng, Yanfeng Wang","year":"2012","venue":"CIKM","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}