{"type":"ScholarlyArticle","title":"A Correlated Topic Model Using Word Embeddings","authors":[{"name":"Guangxu Xun","slug":"guangxu-xun","given":"Guangxu","family":"Xun","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA; {guangxux ,"},{"name":"Yaliang Li","slug":"yaliang-li","given":"Yaliang","family":"Li","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA; {guangxux ,"},{"name":"Wayne Xin Zhao","slug":"wayne-xin-zhao","given":"Wayne Xin","family":"Zhao","affiliation":"School of Information , Renmin University of China , Beijing , China; Beijing Key Laboratory of Big Data Management and Analysis Methods , Beijing , China"},{"name":"Jing Gao","slug":"jing-gao","given":"Jing","family":"Gao","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA; {guangxux ,"},{"name":"Aidong Zhang","slug":"aidong-zhang","given":"Aidong","family":"Zhang","affiliation":"Department of Computer Science and Engineering , SUNY at Buffalo , NY , USA; {guangxux ,"}],"abstract":"Conventional correlated topic models are able to capture correlation structure among latent topics by replacing the Dirichlet prior with the logistic normal distribution. Word embeddings have been proven to be able to capture semantic regularities in language. Therefore, the semantic relatedness and correlations between words can be directly calculated in the word embedding space, for example, via cosine values. In this paper, we propose a novel correlated topic model using word embeddings. The proposed model enables us to exploit the additional word-level correlation information in word embeddings and directly model topic correlation in the continuous word embedding space. In the model, words in documents are replaced with meaningful word embeddings, topics are modeled as multivariate Gaussian distributions over the word embeddings and topic correlations are learned among the continuous Gaussian topics. A Gibbs sampling solution with data augmentation is given to perform inference. We evaluate our model on the 20 Newsgroups dataset and the Reuters-21578 dataset qualitatively and quantitatively. The experimental results show the effectiveness of our proposed model.","doi":"10.24963/ijcai.2017/588","url":"https://ijcai.org/proceedings/2017/588","pdf":"https://ijcai.org/proceedings/2017/0588.pdf","markdown":"https://ijcai.org/proceedings/2017/588.md","bibtex":"https://ijcai.org/proceedings/2017/bibtex/588","volume":{"slug":"2017","title":"Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2017","url":"https://ijcai.org/proceedings/2017/","year":2017,"location":"Melbourne, Australia","dates":"19-25 August 2017","editors":"Carles Sierra","isbn":"978-0-9992411-0-3","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2017"},"section":"Natural Language Processing","track":"Main track","number":588,"pages":{"first":4207,"last":4213},"keywords":["Natural Language Processing: Natural Language Generation","Natural Language Processing: Natural Language Semantics"],"videos":[],"references":[{"position":1,"raw":"[Bengio et al., 2003] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilis- tic language model. journal of machine learning research, 3(Feb):1137-1155, 2003.","title":"A neural probabilistic language model","authors":"Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Jauvin","year":"2003","venue":"journal of machine learning research","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Blei and Lafferty, 2006a] David Blei and John Lafferty. Correlated topic models. Advances in neural information processing systems, 18:147, 2006.","title":"Correlated topic models","authors":"Lafferty Blei, David Blei, John Lafferty","year":"2006","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"[Blei and Lafferty, 2006b] David M Blei and John D Laf- ferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113- 120. ACM, 2006.","title":"Dynamic topic models","authors":"Lafferty ; Blei, M David, John D Blei, Lafferty","year":"2006","venue":"Proceedings of the 23rd international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Blei et al., 2003] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003.","title":"Latent dirichlet allocation","authors":"Andrew Y David M Blei, Michael I Ng, Jordan","year":"2003","venue":"Journal of machine Learning research","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Chen et al., 2013] Jianfei Chen, Jun Zhu, Zi Wang, Xun Zheng, and Bo Zhang. Scalable inference for logistic- normal topic models. In Advances in Neural Information Processing Systems, pages 2445-2453, 2013.","title":"Scalable inference for logisticnormal topic models","authors":"Jianfei Chen, Jun Zhu, Zi Wang, Xun Zheng, Bo Zhang","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Collobert and Weston, 2008] Ronan Collobert and Jason Weston. A unified architecture for natural language pro- cessing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Ma- chine learning, pages 160-167. ACM, 2008.","title":"A unified architecture for natural language processing: Deep neural networks with multitask learning","authors":"Weston ; Collobert, Ronan Collobert, Jason Weston","year":"2008","venue":"Proceedings of the 25th international conference on Machine learning","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Das et al., 2015] Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian lda for topic models with word embed- dings. In Proceedings of the 53nd Annual Meeting of the Association for Computational Linguistics, 2015.","title":"Gaussian lda for topic models with word embeddings","authors":"Rajarshi Das, Manzil Zaheer, Chris Dyer","year":"2015","venue":"Proceedings of the 53nd Annual Meeting of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Griffiths, 2002] Tom Griffiths. Gibbs sampling in the gen- erative model of latent dirichlet allocation. 2002.","title":"Gibbs sampling in the generative model of latent dirichlet allocation","authors":"Tom Griffiths, Griffiths","year":"2002","venue":null,"doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Hofmann, 1999] Thomas Hofmann. Probabilistic latent se- mantic indexing. In Proceedings of the 22nd annual inter- national ACM SIGIR conference on Research and develop- ment in information retrieval, pages 50-57. ACM, 1999.","title":"Probabilistic latent semantic indexing","authors":"Thomas Hofmann","year":"1999","venue":"Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[Holmes et al., 2006] Chris C Holmes, Leonhard Held, et al. Bayesian auxiliary variable models for binary and multi- nomial regression. Bayesian analysis, 1(1):145-168, 2006.","title":"Bayesian auxiliary variable models for binary and multinomial regression","authors":"Chris C Holmes, Leonhard Held","year":"2006","venue":"Bayesian analysis","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Hu et al., 2012] Pengfei Hu, Wenju Liu, Wei Jiang, and Zhanlei Yang. Latent topic model based on gaussian- lda for audio retrieval. In Chinese Conference on Pattern Recognition, pages 556-563. Springer, 2012.","title":"Latent topic model based on gaussianlda for audio retrieval","authors":"Pengfei Hu, Wenju Liu, Wei Jiang, Zhanlei Yang","year":"2012","venue":"Chinese Conference on Pattern Recognition","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"[Huang et al., 2012] Eric H Huang, Richard Socher, Christo- pher D Manning, and Andrew Y Ng. Improving word rep- resentations via global context and multiple word proto- types. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers- Volume 1, pages 873-882. Association for Computational Linguistics, 2012.","title":"Improving word representations via global context and multiple word prototypes","authors":"Eric H Huang, Richard Socher, Christopher D Manning, Andrew Y Ng","year":"2012","venue":"Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers","doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Li et al., 2016] Shaohua Li, Tat-Seng Chua, Jun Zhu, and Chunyan Miao. Generative topic embedding: A contin- uous representation of documents. In Proceedings of The 54th Annual Meeting of the Association for Computational Linguistics (ACL), 2016.","title":"Generative topic embedding: A continuous representation of documents","authors":"Shaohua Li, Tat-Seng Chua, Jun Zhu, Chunyan Miao","year":"2016","venue":"Proceedings of The 54th Annual Meeting of the Association for Computational Linguistics (ACL)","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Li et al., 2017] Shaohua Li, Jun Zhu, and Chunyan Miao. Psdvec: A toolbox for incremental and scalable word em- bedding. Neurocomputing, 237:405-409, 2017.","title":"Psdvec: A toolbox for incremental and scalable word embedding","authors":"Shaohua Li, Jun Zhu, Chunyan Miao","year":"2017","venue":"Neurocomputing","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Liu et al., 2015] Yang Liu, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Topical word embeddings. In AAAI, pages 2418-2424, 2015.","title":"Topical word embeddings","authors":"Yang Liu, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun","year":"2015","venue":"AAAI","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Mei et al., 2008] Qiaozhu Mei, Deng Cai, Duo Zhang, and ChengXiang Zhai. Topic modeling with network regular- ization. In Proceedings of the 17th international confer- ence on World Wide Web, pages 101-110. ACM, 2008.","title":"Topic modeling with network regularization","authors":"Mei","year":"2008","venue":"Proceedings of the 17th international conference on World Wide Web","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Menestrina et al., 2010] David Menestrina, Steven Euijong Whang, and Hector Garcia-Molina. Evaluating entity res- olution results. Proceedings of the VLDB Endowment, 3(1- 2):208-219, 2010.","title":"Evaluating entity resolution results","authors":"David Menestrina, Steven Euijong Whang, Hector Garcia-Molina","year":"2010","venue":"Proceedings of the VLDB Endowment","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Mikolov and Dean, 2013] T Mikolov and J Dean. Dis- tributed representations of words and phrases and their compositionality. Advances in neural information process- ing systems, 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"T Mikolov, Dean","year":"2013","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.","title":"Efficient estimation of word representations in vector space","authors":"Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean","year":"2013","venue":null,"doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Mimno et al., 2011] David Mimno, Hanna M Wallach, Ed- mund Talley, Miriam Leenders, and Andrew McCallum. Optimizing semantic coherence in topic models. In Pro- ceedings of the Conference on Empirical Methods in Natu- ral Language Processing, pages 262-272. Association for Computational Linguistics, 2011.","title":"Optimizing semantic coherence in topic models","authors":"David Mimno, Edmund Hanna M Wallach, Miriam Talley, Andrew Leenders, Mccallum","year":"2011","venue":"Proceedings of the Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Mnih and Hinton, 2009] Andriy Mnih and Geoffrey E Hin- ton. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pages 1081-1088, 2009.","title":"A scalable hierarchical distributed language model","authors":"Andriy Mnih, Geoffrey E Hinton","year":"2009","venue":"Advances in neural information processing systems","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Morin and Bengio, 2005] Frederic Morin and Yoshua Ben- gio. Hierarchical probabilistic neural network language model. In Aistats, volume 5, pages 246-252.","title":"Hierarchical probabilistic neural network language model","authors":"Bengio Morin","year":"2005","venue":"Aistats","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Pennington et al., 2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532-43, 2014.","title":"Glove: Global vectors for word representation","authors":"Jeffrey Pennington, Richard Socher, Christopher D Manning","year":"2014","venue":"EMNLP","doi":null,"url":null,"internal_url":null},{"position":24,"raw":"[Polson et al., 2013] Nicholas G Polson, James G Scott, and Jesse Windle. Bayesian inference for logistic models using pólya-gamma latent variables. Journal of the American statistical Association, 108(504):1339-1349, 2013.","title":"Bayesian inference for logistic models using pólya-gamma latent variables","authors":"James G Nicholas G Polson, Jesse Scott, Windle","year":"2013","venue":"Journal of the American statistical Association","doi":null,"url":null,"internal_url":null},{"position":25,"raw":"[Zhu et al., 2012] Jun Zhu, Amr Ahmed, and Eric P. Xing. Medlda: maximum margin supervised topic models. Jour- nal of Machine Learning Research, 13:2237-2278, 2012.","title":"Medlda: maximum margin supervised topic models","authors":"Jun Zhu, Amr Ahmed, Eric P Xing","year":"2012","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null},{"position":26,"raw":"[Zhu et al., 2014] Jun Zhu, Ning Chen, Hugh Perkins, and Bo Zhang. Gibbs max-margin topic models with data augmentation. Journal of Machine Learning Research, 15(1):1073-1110, 2014.","title":"Gibbs max-margin topic models with data augmentation","authors":"Jun Zhu, Ning Chen, Hugh Perkins, Bo Zhang","year":"2014","venue":"Journal of Machine Learning Research","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}