{"type":"ScholarlyArticle","title":"Learning Topic Models by Neighborhood Aggregation","authors":[{"name":"Ryohei Hisano","slug":"ryohei-hisano","given":"Ryohei","family":"Hisano","affiliation":"Graduate School of Information Science and Technology , The University of Tokyo , Tokyo , Japan"}],"abstract":"Topic models are frequently used in machine learning owing to their high interpretability and modular structure.  However, extending a topic model to include a supervisory signal, to incorporate pre-trained word embedding vectors and to include a nonlinear output function is not an easy task because one has to resort to a highly intricate approximate inference procedure. The present paper shows that topic modeling with pre-trained word embedding vectors can be viewed as implementing a neighborhood aggregation algorithm where messages are passed through a network defined over words. From the network view of topic models, nodes correspond to words in a document and edges correspond to either a relationship describing co-occurring words in a document or a relationship describing the same word in the corpus.  The network view allows us to extend the model to include supervisory signals, incorporate pre-trained word embedding vectors and include a nonlinear output function in a simple manner.  In experiments, we show that our approach outperforms the state-of-the-art supervised Latent Dirichlet Allocation implementation in terms of held-out document classification tasks.","doi":"10.24963/ijcai.2019/347","url":"https://ijcai.org/proceedings/2019/347","pdf":"https://ijcai.org/proceedings/2019/0347.pdf","markdown":"https://ijcai.org/proceedings/2019/347.md","bibtex":"https://ijcai.org/proceedings/2019/bibtex/347","volume":{"slug":"2019","title":"Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence","short_title":"IJCAI 2019","url":"https://ijcai.org/proceedings/2019/","year":2019,"location":"Macao","dates":"10-16 August 2019","editors":"Sarit Kraus","isbn":"978-0-9992411-4-1","publisher":"International Joint Conferences on Artificial Intelligence Organization","doi_prefix":"10.24963/ijcai.2019"},"section":"Machine Learning A-L","track":"Main track","number":347,"pages":{"first":2498,"last":2505},"keywords":["Machine Learning: Classification","Machine Learning: Semi-Supervised Learning"],"videos":[],"references":[{"position":1,"raw":"[Blei et al., 2003] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993-1022, 2003.","title":"Latent dirichlet allocation","authors":"David M Blei, Andrew Y Ng, Michael I Jordan","year":"2003","venue":"J. Mach. Learn. Res","doi":null,"url":null,"internal_url":null},{"position":2,"raw":"[Chang et al., 2009] Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L. Boyd-graber, and David M. Blei. Reading tea leaves: How humans interpret topic models. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 288-296.","title":"Reading tea leaves: How humans interpret topic models","authors":"Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, David M Blei","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":3,"raw":"Curran Associates, Inc., 2009.","title":null,"authors":"Curran Associates, Inc","year":"2009","venue":null,"doi":null,"url":null,"internal_url":null},{"position":4,"raw":"[Dai et al., 2016] Hanjun Dai, Bo Dai, and Le Song. Discriminative embeddings of latent variable models for structured data. In Proceedings of the 33rd International Conference on International Conference on Machine Learning -Volume 48, ICML'16, pages 2702-2711, 2016.","title":"Discriminative embeddings of latent variable models for structured data","authors":"Dai","year":"2016","venue":"Proceedings of the 33rd International Conference on International Conference on Machine Learning","doi":null,"url":null,"internal_url":null},{"position":5,"raw":"[Das et al., 2015] Rajarshi Das, Manzil Zaheer, and Chris Dyer. Gaussian lda for topic models with word embeddings. In ACL (1), pages 795-804. The Association for Computer Linguistics, 2015.","title":"Gaussian lda for topic models with word embeddings","authors":"Rajarshi Das, Manzil Zaheer, Chris Dyer","year":"2015","venue":"ACL (1)","doi":null,"url":null,"internal_url":null},{"position":6,"raw":"[Hamilton et al., 2017] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2017.","title":"Representation learning on graphs: Methods and applications","authors":"William L Hamilton, Rex Ying, Jure Leskovec","year":"2017","venue":"Bulletin of the IEEE Computer Society Technical Committee on Data Engineering","doi":null,"url":null,"internal_url":null},{"position":7,"raw":"[Katharopoulos et al., 2016] Angelos Katharopoulos, Despoina Paschalidou, Christos Diou, and Anastasios Delopoulos. Fast supervised lda for discovering micro-events in large-scale video datasets. In Proceedings of the 2016 ACM on Multimedia Conference, MM '16, pages 332-336, New York, NY, USA, 2016. ACM.","title":"Fast supervised lda for discovering micro-events in large-scale video datasets","authors":"Angelos Katharopoulos, Despoina Paschalidou, Christos Diou, Anastasios Delopoulos","year":"2016","venue":"Proceedings of the 2016 ACM on Multimedia Conference, MM '16","doi":null,"url":null,"internal_url":null},{"position":8,"raw":"[Kim, 2014] Yoon Kim. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1746-1751, 2014.","title":"Convolutional neural networks for sentence classification","authors":"Yoon Kim","year":"2014","venue":"Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014","doi":null,"url":null,"internal_url":null},{"position":9,"raw":"[Kudo et al., 2004] Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. Applying conditional random fields to japanese morphological analysis. In In Proc. of EMNLP, pages 230-237, 2004.","title":"Applying conditional random fields to japanese morphological analysis","authors":"Taku Kudo, Kaoru Yamamoto, Yuji Matsumoto","year":"2004","venue":"Proc. of EMNLP","doi":null,"url":null,"internal_url":null},{"position":10,"raw":"[McAuley et al., 2015] Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15, pages 43-52. ACM, 2015.","title":"Image-based recommendations on styles and substitutes","authors":"Julian Mcauley, Christopher Targett, Qinfeng Shi, Anton Van Den, Hengel","year":"2015","venue":"Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '15","doi":null,"url":null,"internal_url":null},{"position":11,"raw":"[Mcauliffe and Blei, 2008] Jon D. Mcauliffe and David M. Blei. Supervised topic models. In J. C. Platt, D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 121-128.","title":"Supervised topic models","authors":"Jon D Mcauliffe, David M Blei","year":null,"venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":12,"raw":"Curran Associates, Inc., 2008.","title":null,"authors":"Curran Associates, Inc","year":"2008","venue":null,"doi":null,"url":null,"internal_url":null},{"position":13,"raw":"[Mikolov et al., 2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119. 2013.","title":"Distributed representations of words and phrases and their compositionality","authors":"Mikolov","year":"2013","venue":"Advances in Neural Information Processing Systems","doi":null,"url":null,"internal_url":null},{"position":14,"raw":"[Nguyen et al., 2015] Dat Nguyen, Richard Billingsley, Lan Du, and Mark Johnson. Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics, 3, 2015.","title":"Improving topic models with latent feature word representations","authors":"Dat Nguyen, Richard Billingsley, Lan Du, Mark Johnson","year":"2015","venue":"Transactions of the Association for Computational Linguistics","doi":null,"url":null,"internal_url":null},{"position":15,"raw":"[Pal et al., 2006] Chris Pal, Charles Sutton, and Andrew McCallum. Sparse forward-backward using minimum divergence beams for fast training of conditional random fields. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 5, 2006.","title":"Sparse forward-backward using minimum divergence beams for fast training of conditional random fields","authors":"Chris Pal, Charles Sutton, Andrew Mccallum","year":"2006","venue":"IEEE International Conference on Acoustics Speech and Signal Processing Proceedings","doi":null,"url":null,"internal_url":null},{"position":16,"raw":"[Pang and Lee, 2004] Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, 2004.","title":"A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts","authors":"Lee ; Pang, Bo Pang, Lillian Lee","year":"2004","venue":"Proceedings of the ACL","doi":null,"url":null,"internal_url":null},{"position":17,"raw":"[Park et al., 2015] Sungrae Park, Wonsung Lee, and Il-Chul Moon. Supervised dynamic topic models for associative topic extraction with a numerical time series. In Proceedings of the 2015 Workshop on Topic Models, TM '15, pages 49-54, 2015.","title":"Sungrae Park, Wonsung Lee, and Il-Chul Moon. Supervised dynamic topic models for associative topic extraction with a numerical time series","authors":"Park","year":"2015","venue":"Proceedings of the 2015 Workshop on Topic Models, TM '15","doi":null,"url":null,"internal_url":null},{"position":18,"raw":"[Perozzi et al., 2014] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, pages 701-710, 2014.","title":"Deepwalk: Online learning of social representations","authors":"Bryan Perozzi, Rami Al-Rfou, Steven Skiena","year":"2014","venue":"Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14","doi":null,"url":null,"internal_url":null},{"position":19,"raw":"[Ramage et al., 2009] Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D. Manning. Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1 -Volume 1, EMNLP '09, pages 248-256, Stroudsburg, PA, USA, 2009. Association for Computational Linguistics.","title":"Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora","authors":"Daniel Ramage, David Hall, Ramesh Nallapati, Christopher D Manning","year":"2009","venue":"Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":20,"raw":"[Röder et al., 2015] Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM '15, pages 399-408, 2015.","title":"Exploring the space of topic coherence measures","authors":"Michael Röder, Andreas Both, Alexander Hinneburg","year":"2015","venue":"Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM '15","doi":null,"url":null,"internal_url":null},{"position":21,"raw":"[Suzuki et al., 2016] Masatoshi Suzuki, Koji Matsuda, Satoshi Sekine, Naoki Okazaki, and Kentaro Inui. Wikipedia kiji ni taisuru kakucho koyuu hyougen label no tajyuu fuyo [multiple granting of extended named expression labels for wikipedia articles]. In The Association for Natural Language Processing, 2016.","title":"Wikipedia kiji ni taisuru kakucho koyuu hyougen label no tajyuu fuyo [multiple granting of extended named expression labels for wikipedia articles","authors":"Masatoshi Suzuki, Koji Matsuda, Satoshi Sekine, Naoki Okazaki, Kentaro Inui","year":"2016","venue":"The Association for Natural Language Processing","doi":null,"url":null,"internal_url":null},{"position":22,"raw":"[Zeng et al., 2013] Jia Zeng, William K. Cheung, and Jiming Liu. Learning topic models by belief propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(5):1121-1134, May 2013.","title":"Learning topic models by belief propagation","authors":"Jia Zeng, William K Cheung, Jiming Liu","year":"2013","venue":"IEEE Transactions on Pattern Analysis and Machine Intelligence","doi":null,"url":null,"internal_url":null},{"position":23,"raw":"[Zhu et al., 2009] Jun Zhu, Amr Ahmed, and Eric P. Xing. Medlda: Maximum margin supervised topic models for regression and classification. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pages 1257-1264, New York, NY, USA, 2009. ACM.","title":"Medlda: Maximum margin supervised topic models for regression and classification","authors":"Jun Zhu, Amr Ahmed, Eric P Xing","year":"2009","venue":"Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09","doi":null,"url":null,"internal_url":null}],"license_note":"Copyright International Joint Conferences on Artificial Intelligence."}